Serve robots.txt file

[Empty2k12]

I am running a tileserver-gl instance on the internet and have recently noticed that Google has started indexing multiple files from my tileserver. This is not my intention, and it seems like it's not super easy to configure a robots.txt with this project. I would have to run a second Docker container which runs nginx or some other file-server and configure my reverse proxy to serve it when tileserver.my.tld/robots.txt is requested. This is quite cumbersome.

The simplest fix would be to serve a robots.txt when it's available somewhere (configurable) in the file tree, and return a 404 on the robots.txt route if it's not found. This way, you could easily mount a read-only robots.txt into the container.

[acalcutt]

I wonder if anyone considers google indexing their tile server a good thing, should we just put a default robots.txt in "public/resources" which gets served as root of the site right now?

As for a configurable option, i'm not really sure the best way to approach it.

[Empty2k12]

@acalcutt Completely blocking indexing is also fine in my opinion, but I am not sure if there are cases where people would want their tile server to be indexed. But we could just serve a noindex robots.txt and have people override that using their reverse proxy setup should indexing be wanted.

[yetzt]

i always liked googling for intitle:"TileServer GL - Server for vector and raster maps with GL styles" and seeing what maps people make. ;)