Use pmtiles for OSM data

[geohacker]

Currently we use OSM QA Tiles + Tegola for using OSM data for retraining in PEARL. This vtiles infra has been down since PC took the stack down #15. I think it would be good to eliminate QA Tiles altogether and use something like https://protomaps.com/blog/pmtiles-v3-whats-new/ — we can create small regional extracts sitting on Azure Storage.

cc @developmentseed/pearl

[geohacker]

I started looking into using PMTiles a bit and ran into some snags. I'm now not sure if it would work as an approach because the design for PMTiles is primarily for minimum data storage and rendering. QA Tiles are heavy mbtiles as they try to preserve as much OSM data as possible in a tile.

My workflow so far as been to create PMTiles from QA Tiles. Grab a country extract, download the PMTiles binary, run pmtiles create, upload to S3, and then inspect with pmtiles serve.

Currently there are two pmtiles on S3: Monaco https://pmtiles.s3.amazonaws.com/monaco.pmtiles and Singapore https://pmtiles.s3.amazonaws.com/singapore.pmtiles

When I inspect Singapore https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fpmtiles.s3.amazonaws.com%2Fsingapore.pmtiles I see an error in the console Error: Wrong magic number for PMTiles archive

I'll look at why that's the case a bit today.

cc @srmsoumya @batpad @ingalls

[geohacker]

Also cc @willemarcel

[geohacker]

I made another pmtiles for Seychelles based on QA Tiles https://pmtiles.s3.amazonaws.com/seychelles.pmtiles. The inspector shows 7 tiles but doesn't seem to visualise any. I tried a basic leaflet map with a custom symbolizer which also seems empty. But I do see successful requests to the tileset. https://bl.ocks.org/geohacker/06254ac077a30a7116292f9d47ee1e49

[geohacker]

Disregard the singapore tiles — that was actually an mbtiles I uploaded incorrectly. I'm able to generate pmtiles just fine.

[bdon]

• MBTiles metadata has wrong bounds values so result viewers can't automatically zoom to right location
• protomaps.js has a default levelDiff of 2 so leaflet zoom 14 fetches data tiles on level 12. you can change this to 1, so that leaflet zoom 13 fetches data level 12. at levelDiff=1 a single data tile is 2x2 leaflet tiles = 512x512 screen pixels overall

[geohacker]

Thanks so much @bdon! I'm glad I pinged you — I don't think I'd have figured it out. So this sounds great overall and should work for our usecase. I'll report the metadata problem to HOT since they maintain OSM QA Tiles now.

Few other tips from @bdon:

1. Use maxDataZoom: 12 for overzoom but ideally don't rely on a lot of steps if labelling is involved. I think for us 2-3 steps is all that's needed
2. The interactivity on the protomaps.js isn't super great so we should test how much of that will get in the way of our current userflow.

[geohacker]

I updated the gist pointing to seychelles.pmtiles https://bl.ocks.org/geohacker/06254ac077a30a7116292f9d47ee1e49

[willemarcel]

I did some experiments by executing a request to overpass and exporting it as geojson, then converting to mbtiles with tippecanoe and finally to pmtiles, with the pmtiles binary.

This workflow will be better when tippecanoe supports pmtiles output, so we will have a step less. As Pearl AOIs are quite small, maybe it could be cheaper to use overpass than working with qatiles.

This is the visualization I got of restaurants in London:

@geohacker using your Seychelles pmtiles, I could build a visualization filtering all building polygons and all nodes with the amenity key.

Code: https://github.com/developmentseed/pmtiles-filter/
Live demo app (Seychelles data only): https://luxury-jelly-1c0073.netlify.app/