data updates

[tyrasd]

from https://github.com/tyrasd/taghistory#todos: implement regular (e.g. daily/weekly) data updates. Possible solutions:

• reprocess the weekly full history dumps (maybe only all keys as a limited subset of tags to limit memory requirements?)
• use Overpass augmented diffs to calculate daily diffs (currently not feasible because of performance issues, see Optimize runtime and memory footprint of augmented diff queries drolbr/Overpass-API#322, possibly fixed by Serve most recent augmented diff from disk cache only drolbr/Overpass-API#342)
• use Overpass augmented diffs by aggregating standard minutely diffs (currently not feasible because of cached augmented diffs differ from on-demand adiffs drolbr/Overpass-API#346)
• download daily planet dumps and calculate tag usage deltas
• maybe someone out there already runs a global, daily (or quicker) updated osm database which could generate daily tag count diffs as a side product of their update process. see also https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment39983
• compare daily results from taginfo

[kocio-pl]

I would be happy if the updates would happen once a week or even month. It is useful tool to see the trends for example when deciding if something is worth rendering in osm-carto.

[tyrasd]

FYI: I've just manually uploaded some updated data (including changes until 2017-01-15).

I'll soon look into setting up more regular updates.

//cc @morray @Klumbumbus @nebulon42

[Klumbumbus]

Great! Thanks.

[morray]

thumbsup

[kocio-pl]

Hi, Martin, any news on regular updates or (at least) manual update after few months?

[tyrasd]

Sorry, I'm currently not working on this. The goal is still to have regular updates eventually, but maybe I can do another manual update soon…

[kocio-pl]

Monthly updates would be highly appreciated and probably enough - be it automatic or even manual. Currently "shop=yes" query shows ~90k in taghistory, while taginfo shows ~126k, so the change is substantial.

[tyrasd]

I just did another manual data update.

Still no real news regarding more regular updates, though. 😒 But let me just repeat my comment/request from https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment39986 here real quick:

if anyone out there already runs a (daily) updated OSM DB which could produce deltas of the counts of tags in their db (for little extra processing cost), please contact me – I'd love to use that data for updating the taghistory service.

[kocio-pl]

Thanks for the update and for sharing where the problem lies - I passed the message on Talk list:

https://lists.openstreetmap.org/pipermail/talk/2017-November/079751.html

[Zverik]

I assume the updates can be done without an OSM DB, but with augmented diffs from Overpass API.

[mmd-osm]

I assume the updates can be done without an OSM DB, but with augmented diffs from Overpass API.

In theory yes, but there's an issue in drolbr/Overpass-API#346 as @tyrasd mentioned before. I added a comment there how to resolve the issue for the time being.

Another option might be to use the cached augmented diffs as per drolbr/Overpass-API#346 (comment)

[kocio-pl]

Hi,

Could we have the 2018 update, please?

What about more regular updates and the approach mentioned in a post above? What is the current state of things?

[chazanov]

@kocio-pl If you really need it, you may do it on your own - the source code is free. You are always demanding - not a good person!

[kocio-pl]

Thanks for updating! It helped me to see another problem with tag changes: gravitystorm/openstreetmap-carto#3158 (comment).

[Marc-marc-marc]

https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment42459
let's see what is possible to have a regular update

[kocio-pl]

Hi,

I'm back again with the kindly question if it's possible to update it? Maybe there's some help you would need for this and regular updates, as @Marc-marc-marc asked? It's valuable resource to me.

[kocio-pl]

This change is very useful, thanks! I'm just curious why now different keys have different last update time (so interpolated lines start in different moments)?

[tyrasd]

I'm just curious why now different keys have different last update time

I did a quick data update using a new approach earlier this year. But this one was limited to only key=* data. See https://twitter.com/tyr_asd/status/1112387792942841857 (sorry, I forgot to keep you updated here)

Btw: this might also be of interest for you:

You might want to give the #ohsome dashboard a try https://ohsome.org/apps/dashboard/ which I've been working on together with my colleagues at @GIScience – it does all what the taghistory page does, but much more! Only downside are the longer query times for global requests.

[tilmanb]

I know about the ohsome dashboard (which is really cool, by the way), but for small things where you only need "the graph" taginfo is still a valuable tool. Plus it now has permalinks ;-). Long story short: could you please update again? Thanks :-)

[tyrasd]

FYI: taginfo now features daily updated tag count statistics for the "most frequent" tags: https://blog.jochentopf.com/2020-11-08-10-years-of-taginfo.html