Leveraging ArcGIS Online
In the last session, we contrasted ArcGIS Online with familiar desktop GIS, noting the increasing role that enterprise or cloud-based strategies are playing an increasingly important role in the GIS ecosystem. Here, we dig deeper into that notion and explore the various mechanics of cloud based GIS. We’ll use a data-centric approach, looking at how data stored in ArcGIS online can be accessed not only within ArcGIS Online’s portal, but also within desktop applications like ArcGIS pro, and also more widely through web services.
Topics and Learning Objectives:
Section | Learning Objectives |
---|---|
1. Searching for data using ArcGIS Online | • Deploy strategies for effectively searching for data stored on ArcGIS Online • Navigate the Living Atlas to find authoritative data on various topics |
2. Using AGOL data… | • Locate and use AGOL data with AGOL, within ArcGIS Pro, and via the REST endpoint |
1. Searching for data using ArcGIS Online
https://doc.arcgis.com/en/arcgis-online/reference/search.htm
ArcGIS Online is home to an immense rapidly growing body of data, but those data do you no service if you can’t locate them. So an important skill to foster is how to effectively search for data that folks have shared. Here, we reveal a few key techniques that should improve your data searching skills. In doing so, we’ll also review the different types of data and services that are accessible via ArcGIS Online and how they differ.
♦ The traditional search
The initial search
Say we want to an analysis that calculates distances from wind turbines. Let’s find a dataset we can use.
-
Log in to your AGOL account
-
In the search window, enter “Wind Turbines” and hit enter.
The results page probably shows no results, and that’s because your filter is stuck on searching for datasets shared within your organization (i.e. Duke). There are none.
-
Toggle the
Only Search in Duke University
to the off setting.Now you get results, because we are searching all of ArcGIS Online, including the Living Atlas.
Your initial search is dependent entirely on the key words you enter. Obviously, these should reflect the data you want, but you need to think of ways to tweak your search terms to expand or narrow your search.
Filtering your results
With so many results, we need to narrow this down. Here are some options:
-
Filtering on item type
Note the different types of datasets we have: Web Maps, Feature Layers, Map Images, Tile Layers. This web site lists all the types of items you might find on ArcGIS Online. The item type we want is one we can use in our analsysis, i.e. a Feature Layer, so we’ll filter on that:
- On the left and side, under filters, select Layers, then Feature Layers
-
Filtering on Status
You may also want to filter for records that are tagged authoritative, to remove any guesswork as to whether the data you used are from some teachers 7th grade class or whether it’s been vetted by some official authority.
Sorting your results
You may have already found the data you want, but in case you want to be more picky, you can sort your results
-
Sort on Rating - to see which records have been best reviewed
- Sort on View Count - to see which records have been most popular
- Sort on Data - to see the most recently added files (you can also filter on date)
- Sort on Owner - to see whether a user provides many datasets that fit your general topic
You get the point…
♦The Living Atlas
https://livingatlas.arcgis.com/en/home/
ESRIs’ Living Atlas is a curated repository of great spatial data. It has it’s own browser that is pretty self explanatory. We won’t spend time going into to much detail because it really is quite intuitive. The major take-home is that you do not overlook this resource! Take a moment to get familiar with it, how its organized, the categories of data, and how they are accessed.
♦ More creative searching
The fact that ArcGIS online supports so many apps can be useful for burrowing for data. For example, if you see a dataset that occurs in a Web Map application, it’s possible you can dig into that app and find the feature layers on which you can perform analysis.
Digging into a Web Map
-
Back in AGOL, search for “Solar Siting North Carolina”
-
Filter for Maps/Web Maps
-
In the results should be “North Carolina Solar Siting for Climate Resilience”; Click on the result to open its resource page.
-
On the web map’s resource page, you’ll see a list of layers, the first one being: “NC solar footprints August 2019”; click on that and you’ll be taken to it’s resource page.
You now have found a valuable resource - a feature service - from a Web Map! You can use this layer in your analysis!
Digging into Web Apps
It’s likely that you at some point saw Johns Hopkins COVID-19 dashboard. Maybe you noticed the ESRI attribution somewhere or otherwise discovered this was an ESRI dashboard app.
In this page’s web address, you’ll notice a long unique ID: 52d1342f797c410083adf923f473a28e
. That’s its AGOL item ID, and we can use it to find the resource page for this app and possible the data items associated with it.
- In AGOL, search for
52d1342f797c410083adf923f473a28e
- Open the result’s resource page: No layers to speak up, but it does list the owner’s AGOL name:
adrain_NFCA_USA
- Click on the owner name link and search for his/her items.
The results show other items, some of which may or may not be useful. But now you know how you can trace into related items!
♦ Recap
There are other ways for finding data hosted on AGOL, but we’ll need to get back into coding for that. I’m hopeful, however, that this is enough of a start for you to hone your skills and improve your chances of finding the data you need. Next up: what to do with it when you find it…
2. Using AGOL data…
The resource pages of AGOL resource lists how we can access the data. For example, if you go to the resource page for the NC OneMap Animal Feed Operations Permits page (a feature service), you’ll see the options:
- Open in Map Viewer
- Open in Scene Viewer
- Open in ArcGIS Desktop
- Share
…in ArcGIS Online
The first two option can be easily demonstrated by just trying them out. Click the first and the data is added to an ArcGIS Online Map, and the second opens a Scene (or a 3d map).
…In ArcGIS Pro
Click the third option and your dataset may open in ArcGIS Pro. Sometimes, however, I’ve noticed the link that allows my web browser to open these datasets directly in ArcGIS Pro get’s blocked. If so, there are other ways to open AGOL data within ArcGIS Pro
- First, you can download the link and then drag that into ArcGIS Pro and if its a feature service, it should open right up.
- And if that fails, you have the option of adding data by web link, pasting the web link of your datasets resource page.
- Or you can repeat the search you made to find the dataset in your browser, but in your Portal view in ArcGIS online.
… As a web service
The far less intuitive option (at this point) is accessing data via a “web service”, let alone what a “web service” even is…
-
Navigate toward the bottom of your resource page. There on the left you’ll see a link for the items URL. Open that link by clicking View: https://services2.arcgis.com/kCu40SDxsCGcuUWO/arcgis/rest/services/Animal_Feed_Operation_Permits_(View)/FeatureServer
The format of the page shown may or may not be familiar to you, but it is a pretty cool gateway to the data that can be accessed by a wide variety of other applications. We’ll take a deeper look in the next section when we bust out our Python code, but let’s explore this a bit more right now.
Exploring ESRI’s REST Endpoints
The web service we saw above is called a “REST Endpoint” and comprised a component of ESRI’s REST API. Let’s not get stuck on the jargon right now, but instead concentrate on what’s useful. The steps below may be a bit confusing at first, but just go through the motions (and follow my video recording) for now; it should become more clear later what’s going on.
- First, let’s go back to the REST Endpoint for our animal permit locations (link).
- You’ll see its title indicates this is a Feature Service. Feature Services contain one or more feature layers and/or tables. Look down a bit on the page and you’ll see a list of the layers and/or tables included in the service. Ours includes just one feature layer: “Animal Feed Operation Permits (0)”
- Click on the link to that feature layer. This lists, among other things, the list of fields in this feature layer’s attribute table.
- For now, take note of the last field in the list. It’s name is
ObjectID
and its field type isesriFieldTypeOID
. (This is the feature layers object ID field. It usually appears at the first field, but whatever…)
- For now, take note of the last field in the list. It’s name is
- Now scroll down to the very bottom of the page, and you’ll see what we can do with this service via the API. One of the options is Query. Click that link.
- We have a new window pop up. Again, a lot going on here, but just go through the motions here and it’ll make more sense later.
- In the
Where:
box, enterObjectId < 10
. This is a query to select the first 10 records. - Scroll to the bottom and click the
Query (GET)
button. - After clicking, 10 entries should appear at the bottom of the page.
- In the
- Now we’ll modify our query to show the results in a different format:
- In the same window, just above the
Query (GET)
, change theFormat:
option fromHTML
toGeoJSON
- Click the
Query (GET)
button again. - Now the results appear as a lot of text with many curly braces and colons and quotes - gobbledygook for use, but sweet formatted data for a computer to read. This is a structured JavaScript Object Notation (JSON) formatted file that we’ll be working with soon!
- In the same window, just above the
- Use your browsers back button (or click this link) to get back to the feature layer’s end point.
- At the top of the page, you’ll see the sequence of links used to get to the feature layer.
- Click on the right most link to get back to the feature service serving the feature layer.
- Now click on the services link at the top of that page. (Yes you could have done this from the feature layer’s page too).
- What you see now is a list of all kinds of other services hosted by the owner of our Animal Permit Locations dataset!
Let’s explore another service:
-
In the services page, scroll down to locate Brownfield_Boundary.
→ How many feature layers are in this feature service?
-
Click on the first (and only) feature layer to access its REST endpoint.
-
Find which field is its ObjectID or Feature ID field
-
Query the first 10 records in HTML format
-
Query the first 10 records in GeoJSON format
Recap
So what you just did is mimic a computer (or “client”) request to an ESRI server hosting spatial data. The server posted it’s response to our request returning first in HTML format and then in GeoJSON format. Perhaps this hardly sounds exciting, but it’s how the internet works! And it’s how we tap into an enormous pool of online data and processing capabilities.
In a sense, this is a skeleton key to the cloud, and we are about to explore what new doors we can open with it!