##Getting Data

###Reading Files

By using stdin and stdout it's easy to create unix like utilities for text processing and pipe them into each other. For example, to count the number of lines in a file that contain numbers:

cat someFile.txt | python egrep.py "[0-9]" | python line_count.py 

###Web Scraping

Be sure you've set up a virtual environment and then just use BeautifulSoup, Requests and html5lib. Please checkout a page's robots.txt and terms before you do something like this.

source_code_of_a_webpage = BeautifulSoup(requests.get(url_of_page).text,'html5lib')

When you're working with json, transform your data to a dictionary and be happy

import json
deserialized = json.loads(serialized_json)

###Twitter API

Get some credentials at https://apps.twitter.com . Please don't check in your consumer_key and secret_key in your repo.

Web-Scraping/twitter.py

twitter.py CONSUMER_KEY SECRET_KEY

##Working with data

The size of the range of a feature should not affect its predictive power so it's usually a good idea to rescale your dataset to have mean 0 and variance 1.

return (data_matrix[i,j] - means[j])/stdevs[j]

You don't want your parser to break if some value is not expected

def try_or_none(f):
    def f_or_none(x):
        try: return f(x)
        except: return f_or_none
    return f_or_none