Data Sources for Cool Data Science Projects: Part 2

Link to Part 1

At The Data Incubator, we run a free eight week data science fellowship to help our Fellows land industry jobs. Our hiring partners love considering Fellows who don’t mind getting their hands dirty with data.  That’s why our Fellows work on cool capstone projects that showcase those skills.  One of the biggest obstacles to successful projects has been getting access to interesting data.  Here are some more cool public data sources you can use for your next project:

Data With a Cause:

1. Environmental Data: Data on household energy usage is available as well as NASA Climate Data.
2. Medical and Biological Data: You can get anything from anonymous medical records, to remote sensor reading for individuals, to data on the Genomes of 1000 individuals.

Miscellaneous:

1. Geo Data: Try looking at these Yelp Datasets for venues near major universities and one for major cities in the Southwest.  The Foursquare API is another good source.  Open Street Map has open data on venues as well.
2. Twitter Data: You can get access to Twitter Data used for sentiment analysis, network Twitter Data, and social Twitter data, on top of their API.
3. Games Data: Datasets for games, including a large dataset of Poker hands, dataset of online Domion Games, and datasets of Chess Games are available. Boardgamegeek.com also has a large database of games, prices, artists, etc.
4. Web Usage Data: Web usage data is a common dataset that companies look at to understand engagement.  Available datasets include anonymous usage data for MSNBC, Amazon purchase history (also anonymized), and Wikipedia traffic.

Metasources: these are great sources for other web pages.

1. Stanford Network Data: http://snap.stanford.edu/index.html
2. Every year, the ACM holds a competition for machine learning called the KDD Cup.  Their data is available online.
3. UCI maintains archives of data for machine learning.
4. US Census Data.
5. Amazon is hosting Public Datasets on s3.
6. Kaggle hosts machine-learning challenges and many of their datasets are publicly available.
7. The cities of Chicago, New York, Washington DC, and SF maintain public data warehouses.
8. Yahoo maintains a lot of data on its web properties which can be obtained by writing them.
9. BigML is a blog that maintains a list of public datasets for the machine learning community.
10. GroupLens Research has collected and made available rating data sets from the MovieLens website.
11. Finally, if there’s a website with data you are interested in, crawl for it!

While building your own project cannot replicate the experience of fellowship at The Data Incubator (our Fellows get amazing access to hiring managers and access to nonpublic data sources) we hope this will get you excited about working in data science.  And when you are ready, you can apply to be a Fellow!

Got any more data sources?  Let us know and we’ll add them to the list!

This post was also published by DataScience101. This, and all things Data Science, can be found here.