Large custom map_layers with tiles

Something that comes up occasionally is a request to add a custom map_layer to Looker that is very large. Most commonly the questions start when a user realizes that the TopoJSON file for their custom map is hundreds of MB or even GBs and that loading the whole thing into the browser at once isn’t going to work.

Luckily, there are great tools available that are low-cost or even free that make this process reasonably simple. It does require a little bit of patience and some developer tools (like the command line), but if you’ve used Github before, it’s probably well within your skillset.

For the purposes of these instructions, I’m going to assume that you already have a single file that represents your map layer. When you’re uploading the file directly to Looker, that file has to be TopoJSON. But for our purposes, we actually need it to be GeoJSON, KML, or a shapefile. If you have TopoJSON, Mapshaper is a super easy tool for transforming it to GeoJSON.

For this tutorial, I’ll use a GeoJSON file that outlines all 73,057 tracts from the 2010 U.S. Census. This GeoJSON weighs in at just over 1 Gb–definitely not something I want users to load into the browser every time they access it.

But I can take advantage of the same technology that drives Google Maps–tiles–to make the file tractable (pun intended). The basic idea is that if I’m zoomed way in, only a few tracts will be in my viewport, so I only need to load those. Then as I pan, I load the adjacent ones and so on.

If, on the other hand, I’m zoomed way out and can see big regions of the country with thousands of tracts, I don’t actually need all the detail that is in my GeoJSON file. Very rough outlines of the tracts (which weigh far less) will suffice.

Tiling the map at various zoom levels gives me all the raw materials I need to only download the pieces of the map I need right now, at the appropriate level of detail. And Mapbox provides a simple set of tools to do it.

  1. So first, go create a free account at Mapbox. Once you’ve done that, create an Access Token and save it for later. (The default public scopes work fine.)

  2. Then go into Mapbox’s Studio to upload your data. You want to create a new tileset.

  3. Once the tileset is uploaded, it’ll take a bit to process (gotta make all those tiles). After it’s done, click in and you’ll see something like this:

  4. Now comes the trickiest (but not really that tricky) part. We need to calculate the bounding box for each tile region in our shapefile. This is because in order to know which regions to load at any given time, we need to know what the furthest SW and NE points are that encompass the region.

  5. We have a little script that’ll do this for you, which you can download here (11.8 KB). The instructions are in the script, but you’re going to do the following:

  • Install Yarn by opening a command line and typing ( brew install yarn )
  • Run Yarn by typing yarn
  • Generate the extents.json file by typing yarn run extents /my/input/geojson.json extents.json ZCTA5. Except you’re going to replace /my/input/geojson.json with the actual location of your file and ZCTA5 with the ID field in your shapefile that uniquely identifies each region.
  1. Now that you have the extents.json file generated, the last thing you need is to put that somewhere where Looker can access it. Simply uploading to S3 or Google Cloud Storage won’t work, sadly, because of something called “Content-Type Headers.” Luckily, you can upload the file to Github or as a Github gist, and then use this nifty service to fix the headers.

  2. Now we have all the pieces we need to properly define our new map layer in Looker! Go to your model in Looker and as part of the model file, define your new map_layer as follows.

map_layer: tract {
  format: "vector_tile_region"
  url: "https://a.tiles.mapbox.com/v4/[YOUR_MAPBOX_TILESET_ID]/{z}/{x}/{y}.mvt?access_token=[YOUR_MAPBOX_ACCESS_TOKEN]"
  feature_key: "[YOUR_MAPBOX_TILESET_NAME]"
  extents_json_url: "https://rawcdn.githack.com/[YOUR_EXTENTS.JSON_FILE_LOCATION]"
  min_zoom_level: [FROM_MAPBOX]
  max_zoom_level: [FROM_MAPBOX]
  property_key: "[THE_UNIQUE_IDENTIFIER_FOR_REGIONS_IN_YOUR_SHAPEFILE]"
}

To illustrate where you’re going to get all this info, see below:

And:

And that’s it! So, not a 5-second process. But it’s totally worth it :slight_smile:

Explore_Fast_Facts

6 Likes

Thanks for this really awesome write up @Mintz . So I’m having some issues where due to the minimum zoom level being hardcoded/determined by Mapbox, data is not presented at all zoom levels in Looker.

image

Ideally, I’d like to be able to set the zoom level to anything in Looker (and still have data displayed), so I followed Mapbox’s instructions for adjusting zoom extent here: https://docs.mapbox.com/help/troubleshooting/adjust-tileset-zoom-extent/#transform-data-with-tippecanoe

Now what I’m stuck on is this – how do I generate the bounding box for tile regions for an mbtiles filetype?

Good questions, @amm7790! So on the zooms, I think that’s mostly determined by Mapbox’s read of how much detail is in the file. Regardless of what Mapbox sets, though, I think if you set the zoom max/min to what you want when you create the map_layer in Looker, it should work well enough.

The bounding boxes that get transmitted to Mapbox are handled automatically by Looker assuming you have the extents file working right, so you shouldn’t have to worry about those steps with Tippecanoe.

I tried changing the min_zoom_level parameter to 1 in map_layer, but it did not display data in the explore viz until i reached up to zoom level 9, which is the minimum defined zoom level in Mapbox. My read on this is that the min/max zoom layer params in map_layer are only there to prevent a Looker viz in an explore from going into those “restricted” levels and showing a map, but no data?

When you say “Looker assuming you have the extents file working right, so you shouldn’t have to worry about those steps with Tippecanoe”

Do you mean I shouldn’t have to worry about creating another extents.js file for the new mbtiles filetype generated by Tippecanoe? I don’t really have a good grasp on how this new filetype works, but I would assume I’d need to generate something new for this new file.

Hi @Mintz this seems to fit a need I have perfectly! The issue that I am running into is that yarn run extents... returns that ‘Command “extents” not found.’ Is there something that I am missing? Thanks!

Hey Ryan, make sure you’ve installed yarn as outlined above, and then make sure that you’re running the command from the commandline and that you’re in the directory where the the extents.js file that you’ve downloaded is located.

Sounds like you may be in a different directory, which would explain why it can’t find the script.

Sorry, just saw this.

Hopefully you’ve been able to figure it out yourself, but I’ve never run into this exact issue with the zoom levels before. It looks from the documentation like creating the MBTiles with tippecanoe might be the right strategy to manually define the min/max zoom levels.

But as you said, you can’t create an extents file from an MBTiles file. That said, the extents file shouldn’t actually be sensitive to zoom levels, since what it’s doing is defining the far NW and SE coordinates for each region in the map layer. Those aren’t affected by the zoom levels.

So I’d try generating the extents file directly on the GeoJSON file you have, then converting it to an MBTiles file with your preferred zoom limits and uploading that to Mapbox. And see if that works.

1 Like

does this mean i can directly point to the geojson files and display the same too?

Hi,
Not totally sure what you mean by “display the same too”. Can you explain a bit more about what you’re trying to do?