Accessing Datasets using Python

Introduction

To analyze data with Python, we need to get access to the data and bring them into our Python scripting environment. We've already seen how we can read text files using Python's built-in open function to create a file object and read GIS tables using ArcPy's cursor objects, but Python has several other, more effective means for accessing external data. In this session, we examine a number of helpful Python packages and how they are used to access, fetch, unpack, and manage data in various formats and from various sources.

Lab Prep

To prepare for this lab, we'll need to create a new Conda environment and install a number of packages used in dealing with data. Then we'll clone a GitHub repository containing some sample notebooks and exercises.

1. Create a new Conda environment & adding packages

The following three options are to create your "getting_data" environment created and loaded with the correct packages. These commands are all run at your Python Command prompt.

-Plan A-

• First, we'll try to create the environment from a ".yml" file I created and places in the W:\859_data folder.

  x
  conda create --name getting_data --file W:\859_data\getting_data.yml
  

  This may take a while. If after several minutes it appears to "hang", try right-clicking in the window or hitting enter a few times...

• Activate the new environment

  xxxxxxxxxx
  activate getting_data
  

-Plan B-

If Plan A fails, the fall back is to create the environment from scratch and add the packages ourselves.

• Create a new environment named "getting_data".

  xxxxxxxxxx
  conda create --name getting_data
  

• Activate the new environment:

  xxxxxxxxxx
  activate V:/getting_data
  

• Install packages:

  xxxxxxxxxx
  conda install jupyter requests
  conda install -c conda-forge geopandas
  conda install -c anaconda beautifulsoup4 lxml
  pip install us
  

-Plan C-

If Plan A and B fail, then we'll just use the "getting_data" environment I've created on the W: drive...

More info on these packages can be seen by searching for them, e.g. search for Python us package. Often you'll see links to these packages on the Python Package Index, or "PyPI", web site. The PyPI site for a given package usually includes links to more documentation, the package's GitHub site, and information on how best to install the package.

2. Creating your own copies of the exercise notebooks: 'forking' the repository

The Jupyter notebooks we'll be using for this session are stored in a GitHub repository. However, instead of just downloading static copies of these notebooks to your machine, you'll "fork" the repository, which mean's you'll make a copy of the main repository in your own GitHub account, where you'll have full control over everything.

• Open GitHub in your web browser and sign in.

• Open the class repository containing the exercise files for this session: https://github.com/ENV859/GettingData

• In the upper right corner of the page, you'll see a button to Fork the repository. Click that button.

  → You now have a working copy of that repository in your own GitHub account.

3. Cloning the repository to your local machine

Finally, we'll clone your copy of the "Getting Data" repository to your local machine.

• Open the GitHub Desktop app on your machine.
• Select File > Clone Repository.
• In the dialog box that appears, search for your GettingData repository and clone it to a folder on your machine.
• Open that folder in Windows Explorer and you'll see all the files we'll need to run these exercises.
• Included in these files is are shortcuts to the Jupyter notebook app (depending on which "Plan" above worked). Double-click the appropriate shortcut to ensure it works. If not, create your own shortcut, or contact the instructor or a TA for assistance.

The Exercises:

The specific exercise notebooks are fairly self-explanatory and review an array of methods used to access and download data from the internet. They also touch on a few concepts that we will dig deeper into in upcoming sessions.

Click on the link to fire up Jupyter in your cloned workspace and let's go!

Online viewer of notebooks here: https://nbviewer.jupyter.org/github/ENV859/GettingData/tree/master/

END