Showing posts with label Web scraping. Show all posts
Showing posts with label Web scraping. Show all posts

Sunday, 13 July 2014

MIT app inventor - Buttons, Text to Speech, Web and Translate

We just ran this as a workshop at the association of Women in Science conference, so we though we would put it up as an online tutorial too. This will take you through the basics of writing and MIT app inventor App, downloading and installing it to your phone and then following through a few extension challenges beyond those from the standard App inventor tutorials (with a healthy dose of elvish impishness!).

Head over to http://ai2.appinventor.mit.edu/ and login with your google account (or register for one for free) then take a look at the MIT App inventor basics tutorials here: http://goo.gl/sVZMzK.

It's worth noting that you DON'T need an Android phone to test these out as if you have installation privelages to the computers you're working on then you can simply install an emulator. Unfortunately this is only currently an option for OSX and Windows, which also allow direct connecting of an Android phone to a computer with the MIT App inventor editor on by Wifi or USB cable. This awesome little feature instantly clones the app you're creating to your device allowing you to test it in real time. If you're a dedicated linux user like us though, then a nice workaround is to simply use the 'Build' functionality at the top of the page to build your program and then automatically download and install it to your phone using a QR code (however you can end up with a lot of apps installed to your phone this way!).

Our first app: Text to Speech

Aim: An app that speaks some text that the use inputs
        Instructions: Here
Challenge: Extend your app so that it screams when you shake it (check out the .shaking method in the accelerometer sensor)


Our second app: Ball bounce


Aim: An app that allows you to control an on-screen ball
        Instructions: Here
Challenge: Increase the numbers of balls to 5, and allow collisions between balls

Repeater

Aim: An app that recognises voice commands and reads them back to you.

        Instructions: 

Start a new project and set up a basic screen with a button, a label, a speech recognizer and a text-to speech-element. Using the blocks editor, set up these blocks:
Simply, when the button is pushed, the app will try and recognise what has been spoken and will print a copy to the label and then attempt to speak it outloud. Yes, the SpeechRecognizer isn't great at NZ accents. Yes it's a source of much hilarity. Convieniently MIT app inventor have just introduced some new translation functionality in the form of the YandexTranslate design element (found under the 'Media' tab). You can pass this a string of words and it will attempt to translate it into a language of your choice.

Challenge: The 'repeater' app reads back to you in Italian


If you're looking for other languages:

  • en - English
  • es - Spanish
  • de - German
  • no - Norwegian
  • fr - French
  • ru - Russian (however the altered character set will appear in the translation label but not be read out by the text to speech element)

Insulter

Aim: An app that scrapes data from a website and prints it on screen

        Instructions: 

As per usual we're going to start off with a button and a label field, but because we also need data from the internet we will also need the 'Web' element from the Connectivity tab. The set up some blocks:
The first block connects the button with retrieving the html of the website. The second block triggers when that call is complete - and contains and 'if' statement that tests is the website retrieval was successful (responseCode = 200) or not. If it was successful then the returned html is printed to the screen and then spoken by a text to speech element. Initializing the local variable isn't necessary here - but becomes useful in the next step. And yes, you're reading this code right - this will just print the entire html of the website.

So our next step is to 'parse' the website html to allow us to pull out the elements we want. To see the html I'm talking about, head over to http://insult.dream40.org/, right click and select View Page Source. If you look though it you will quickly find two lists of words that the website picks from to get its insults.We can use a 'split' block from the text tab to help us sort through it. I suggest starting by getting your program to grab one insult from the website and display it - then modify that to allow it to grab multiple elements and choose randomly between them. There IS an inbuilt Html Parser method in the Web element, but we were unable to get it working so we just did it by hand!

Challenge: Gets insults from http://insult.dream40.org/ and speaks them.


If you're interested in doing more you can follow our MIT app inventor bluetooth tutorial here: http://goo.gl/QbQhLk

References:

We've borrowed lots of this material from the MIT app inventor team of course! you can find all their wonderful tutorials here: http://appinventor.mit.edu/explore/ai2/tutorials.html and they're by FAR the best way to learn all about App inventor.
We also found this tutorial about getting Yahoo Stock prices really helpful: http://beta.appinventor.mit.edu/learn/tutorials/stockquotes/stockquotes.html
For those that want access to more Android functionality - we suggest Android studio rather than the old Eclipse plugin.

Tuesday, 27 May 2014

Scraping websites with Google spreadsheets

Frequently when working with data collection or visualizations it's useful to get data from an external website that doesn't have an API. There are plenty of tools to help you with this but one of the easiest is certainly google spreadsheets, or you could of course write your own web scraping bot with Python and the Beautiful Soup library (which I will outline in a future tutorial). One thing to note - web scraping is not always legal so make sure to check the website's data use policy before you go scraping it and sharing it with everyone!

To get started go grab a google account and/or login to google documents and create a new google spreadsheet. then go find the URL of the data you want to scrape. To keep things simple I'm going to demonstrate with the highest-grossing films data from Wikipedia. Here's the URL:
 
http://en.wikipedia.org/wiki/List_of_highest-grossing_films#Highest-grossing_films

If you're not familiar with URLs, the # character directs your requests to a specific subset of the page - in this case to the 'highest grossing films' section. If you go to the site you will see a table which is the data we're interested in:



In the top left hand cell of your spreadsheet, use the following command:
 =ImportHtml("http://en.wikipedia.org/wiki/List_of_highest-grossing_films#Highest-grossing_films","Table", 1)

Hit enter and the spreadsheet will slowly populate itself with all the data from the table we're looking at and you can graph it or save it as a .csv or text file for later analysis with another program or script.

If it didn't work there's a few things to check:
  1. Make sure you have enough of the right kind of quotation marks. There should be double quotes around both the URL and the word Table in the above command
  2. Check the URL is correct.
So what precisely is happening? The =ImportHtml() command directs google spreadsheets to the particular page and the next command "Table" tells it too look for an element in the format of a table. You can also use alternative Html elements here like lists ("List"). Finally the value 1 tells it which element to look at. Start with 0 and keep increasing the number by 1 until you find the data you're looking for. 

Have a go changing the number in the above command and see if you can scrape the table from further down the page with films adjusted for inflation.


There's a functional example of the spreadsheet here if you're interested.

It's also worth noting that Google spreadsheet come with several other import tools for importing different types of data including .xml with:
  • ImportHtml()
  • ImportXML()
  • ImportData()
  • ImportFeed()
Finally you can also use the other google spreadsheet commands to sort through the data you get. One of the most useful is the =SPLIT() command that separates the incoming data by a certain character (so the command =SPLIT(A1, " ") would split whatever text is in the cell A1 wherever there was a white space.)  Another useful command is =REGEXEXTRACT(), which searches a sting for a particular pattern of letters and numbers known as a regular expression, and then returns it. Regex commands are a little complicated to get used to but really useful - here's one that searches for the pattern of a year(i.e. 2014) in a string:  
=REGEXEXTRACT(A1,"[0-9]{4}"
)