Now that you've spent time learning Python, continue to improve your proficiency with the language. You want to spend this day getting familiar with the extensive Python standard library and reinforcing any knowledge gaps you have encountered from the previous day. There is no problem with having to spend more time on the material of the first two days. It's really important you start the program being comfortable programming with Python.

Resources

1. The following are important modules that are part of the Python standard library. It's good to be aware of these before the Fellowship program starts.
   • collections: While you can accomplish much with Python's built-in data structures such as list, tuple, dict, and set, the collections module provides some specialized data structures that make your life more convenient. A few to know about are Counter, namedtuple, and defaultdict.
   • itertools: There are times when you need to come up with all possible permutations or combinations and iterate over them. While you might be able to programmatically create them and store them in a list, that would be memory inefficient, slow, and prone to error. Instead, consider using some of the tools offered by itertools. In general, the module provides a set of fast and memory efficient iterators. A few iterators of note are permutations, combinations, and product.
   • os, glob, and pathlib: You may find yourself needing to do operating system tasks like creating and removing directories. The tools in the os module will help make your code compatible across different operating systems. If you are looking for functions to work with paths, look into os.path. For example, os.path.join is used to join paths intelligently, for example, properly using the correct directory separator for your operating system. The glob module lets you find all file paths satisfying a given pattern. For example, you can find all files in the "data" directory that end in .csv using the pattern data/*.csv. pathlib is an alternative to os.path and glob. Instead of representing paths and files as strings, it provides a Path class, offering an object-oriented way to work with paths. For example, with a Path object, you can have access to methods to check for file existence and iterate over all subdirectories.

Action Items

1. Continue with the Learn python tutorial or this tutorial on the official Python website.
2. Take an assessment of the areas of Python you feel least comfortable with. Create an account on HackerRank, a platform for taking coding challenges. Under the Python section, look for challenges that target the areas you are least comfortable with. HackerRank even has a Python skills assessment test you can take to better understand what areas you need to work on.

Python has a wonderful suite of libraries for doing scientific computing. The two main packages are numpy and scipy.

What is NumPy, and why is it relevant for scientific computing? NumPy is a Python package that provides, among other things, a multi-dimensional array object, the ndarray, which allows for very fast array operations. NumPy is written in the C programming language, allowing the library to leverage the speed of C for fast vectorized operations. Rather than using slow for loops, where possible you should always vectorize your operations using NumPy.

NumPy also forms the basis of, or is used extensively by, other packages such as scipy and scikit-learn, so having a firm understanding of NumPy is useful for using other packages.

scipy is a package that provides many efficient numerical routines such as numeric integration, optimization, linear algebra, and statistics methods. While useful for scientific computing, we won't use scipy much in the Fellowship program.

Resources

1. An excellent tutorial on NumPy.
2. If you like videos, try watching this one and following along with their GitHub code.
3. For those of you familiar with MATLAB, here’s a guide for translating your MATLAB knowledge into NumPy.

Action Items

1. Make certain that you know how to create an array in numpy. There are many different ways this can be done, such as using the array function, zeros and ones for pre-filled in arrays of given sizes, and the arange and linspace functions for sequences.
2. Create some NumPy arrays, manipulate them with basic operations, and try out some of the numpy universal functions.
3. Do the one-star exercises and as many of the two-star exercises as you can from the NumPy 100 exercises page.

pandas is the Python package for data analysis and manipulation. It's a package designed to make working with relational (or tabular) data fast and easy. The package provides many methods for performing data analysis, data manipulation and aggregation, as well as many visualization tools (largely built on top of the matplotlib library). If you are familiar with R, then you'll notice similarities with R's data.frame.

pandas is built on top of the numpy package, hence a good understanding of numpy is valuable for working easily in pandas, and pandas is designed to integrate well with other Python libraries such as scikit-learn and statsmodels. pandas also has great capabilities for working with time series data and includes a large suite of time series-specific methods.

Resources

1. The homepage of pandas is a great place to start, as are the Getting started tutorials. Important things to know include:
   • How to read data from a CSV (comma-separated value) file to create a DataFrame.
   • How to filter data in a DataFrame.
   • How to compute summary statistics for a DataFrame.
   • How to use the groupby method to aggregate data.
   • How to dump results of analysis to a CSV file.
2. There are several nice pandas tutorial videos on YouTube. Here is a playlist of a series we liked. It's broken up by topic so feel free to skip around and focus on areas you need to strengthen.

Action Items

Download the data set about Value of Energy Cost Saving Program for businesses in New York City (under the "Export" option, there is a way to retrieve a CSV file). Answer the following questions.

1. How many different companies are represented in the data set?
2. What is the total number of jobs created for businesses in Queens?
3. How many different unique email domains names are there in the data set?
4. Considering only NTAs with at least 5 listed businesses, what is the average total savings and the total jobs created for each NTA?
5. Save your result for the previous question as a CSV file.

Matplotlib is Python's most popular plotting package. It provides an object-oriented programming API and a procedural API similar to that of MATLAB. While much can be done with Matplotlib, for both nicer looking and statistically focused visualization, consider using Seaborn, which extends and builds on Matplotlib.

With these plotting libraries, don't focus on trying to memorize the entire API. A good starting point is to visit the examples page and look for similar visualizations of what you are trying to build. Altair and Bokeh are two Python packages for creating interactive visualizations. In the Fellowship program, we'll focus on using Altair.

Action Items

1. Go through the Introductory Tutorials on Matplotlib.
2. Using the same data set and results that you were working with in the pandas action items section (Day 4), create a
   • scatter plot of jobs created versus average savings. Use both a standard and a logarithmic scale for the average savings.
   • histogram of the log of the average total savings.
   • line plot of the total jobs created for each month.
3. If you have time, take a look at this short tutorial on Altair.