Web Services, REST, & APIs
In this session we explore the application of web services to spatial analysis. It’s an exciting and rapidly evolving field: what’s “cutting edge” this year is likely to be old news the next. However, getting in on this technology at an early stage should provide you with a more robust understanding of how it works and thus how you can utilize its power most effectively. At this stage in the development of web services and cloud-based GIS, however, you’ll need to be prepared for bugs and limited documentation. It’s all part of the excitement of being at the forefront of technology.
Here, we begin simply, looking at web services from within web browsers. From there we explore how these services can be controlled via scripts (e.g. Python). Then we examine more advanced interfaces to web services, i.e. via APIs and applications that are integrated with web services (e.g. ArcGIS.com and Desktop ArcGIS).
Learning Objectives:
| Notebook | Learning Objectives |
|---|---|
| Readme.md (Introduction) | • Describe a Google search in terms of a web service • Explain advantages of using web services in GIS • Explain the terms “thin client”, “request”, “response”, “service provider”, “URL”, “REST”, and “API” |
| 1-NWIS-discharge-data-as-API | • Use requests to interactively fetch data• Parse a URL request into service and parameters • Handle server responses as raw text data |
| 2a-Exploring-the-BISON-API 2b-Census-API |
• Navigate API docs to effectively use an API • Construct server requests programmatically • Handle server responses in JSON formats • Convert JSON responses into Pandas dataframes |
| 3-ArcGIS-REST-Service-Demo | • Dissect the ESRI REST endpoint as a service URL • Explore the ESRI REST API documentation • Use the REST endpoint to generate a custom result |
| ↓↓ OPTIONAL NOTEBOOKS BELOW ↓↓ | |
| 4a-Scraping-Data-From-ESRI-MapServers 4b-CharlotteBulkDataScraper 4c-CharlotteDataExplorer |
• Use ESRI REST APIs to programmatically fetch and explore online data • Review how web services work in raw format |
| 5a-Geocoding-With-OSM 5b-Geocoding-With-OSM-in-bulk |
• Use the OpenStreetMap API to geocode an address • Batch geocode several addresses using the OSM API |
Set up:
-
Fork and clone the https://github.com/ENV859/UsingAPIs repository.
This repository includes a series of notebooks examining various aspects of using web services and APIs in a GIS context. These notebooks require the
gisPython environment created earlier in the class. A link to run Jupyter from this environment is included in the workspace.The workspace also describes of the utility of web services more in the README file. Finally, a zipped ArcGIS Pro workspace is provided that includes tools that integrate web services into spatial analysis.
Recording highlights
6.4.1 Intro to Web Services
| Time | Topic |
|---|---|
| 0:15 | Setup: Getting section materials |
| 2:25 | Exploring what a web service is: Google Web Search |
| 6:34 | A more detailed example: “Hacking” the NWIS web interface |
| 11:35 | What is an “API”? Interpreting URLs as API calls… |
6.4.2 Fetching data with web services & APIs
| Time | Topic |
|---|---|
| 0:15 | Demo: Web Services used in an ArcPro tool |
| 4:40 | Invoking an API using the requests package |
| 6:17 | -Decomposing the service URL as a service and parameters |
| 8:05 | -Constructing the service request, and placing the request |
| 9:00 | Handling the response |
| 10:19 | -Handling the response as a text object |
| 14:13 | Another example: The USGS BISON API |
| 14:40 | -Understanding API Documentation |
| 17:44 | Learning by trying: The BISON search service |
| 20:20 | Implementing the API in code |
| 21:13 | Working with JSON formatted responses |
| 27:55 | Using Pandas to work with with JSON responses |
| 30:00 | Using the Census API |
| 30:16 | -What is a service key? What is its purpose? |
| 33:30 | -‘Handling the response.’ |
| 33:50 | -Converting the response to a dataframe |
6.4.3 ArcGIS REST API
| Time | Topic |
|---|---|
| 0:35 | What is an “ESRI REST API endpoint”?? |
| 2:40 | Exploring a map service endpoint |
| 4:50 | Layers and other properties of a map service |
| 6:20 | Service endpoints of a map service |
| 7:30 | Exploring how map service endpoints are used |
| 10:48 | Using the ESRI REST API in Python |
| 16:48 | Looking ahead |