-
-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Guaranteed Backwards Compatibility #43
Comments
To keep in mind when working on this: Functional Risk Scenarios
Technical Risk Scenarios
|
…to MSVS2019 solution by using a dummy project and prebuild script - added skeleton projects for qiqqa diagnostic and migration work. Related to jimmejardine#43
…erywhere. Part of he move towards achieving jimmejardine#43
Thought: create a minimal v80 derivative which has no changes in the serializable classes and installs in parallel to a regular Qiqqa install - to be used as a fast test base for 'is this stuff still backwards compatible' checks. |
🤔 Are we forever stuck with this collection of protobuf, .NET binary serialization and JSON serialization for the sake of backwards compatibility?! 🤔 When I look through the GetSatisFaction forum at https://getsatisfaction.com/qiqqa/topics/ it looks like both scenarios are true:
Otherwise it's unclear where the activity and usage of Qiqqa is out there. ❓ Anyway; on the tech front I'm not sure if zoning out the backwards-compat crap into another binary is such a swell idea. Heck, right now I don't know what is. Meanwhile riding on the old cruft isn't giving me joy either as there's a few spots that I want to kick the tires of without getting bogged down in the protobuf/serialization funk. To be continued... 🌵 <rant off> |
…ad UX as it takes ages for the sort to complete and the view to update) - this resulted in work done but also the need to nuke the slower dictionary-based PDFDocument setup, which does a lot of casting and conversion back&forth at run-time. As a consequence, serialization will have to be done differently and 'upgrading' old-format records should have been coded into the DESERIALIZER/LOADER anyway, instead of some nasty hack at the (re)write side of things. TODO: complete this work and add in backwards compat load code for issue jimmejardine#43 (jimmejardine#43)
…to MSVS2019 solution by using a dummy project and prebuild script - added skeleton projects for qiqqa diagnostic and migration work. Related to jimmejardine#43
…ad UX as it takes ages for the sort to complete and the view to update) - this resulted in work done but also the need to nuke the slower dictionary-based PDFDocument setup, which does a lot of casting and conversion back&forth at run-time. As a consequence, serialization will have to be done differently and 'upgrading' old-format records should have been coded into the DESERIALIZER/LOADER anyway, instead of some nasty hack at the (re)write side of things. TODO: complete this work and add in backwards compat load code for issue jimmejardine#43 (jimmejardine#43)
…icated library so the mainline codebase doesn't keep cluttered with old stuff, just because we want to be able to load/import old Qiqqa libraries.
…to MSVS2019 solution by using a dummy project and prebuild script - added skeleton projects for qiqqa diagnostic and migration work. Related to #43
Considering
See discussion at #15: as James emphasized backwards compatibility there, it triggered this thought process:
normally, you test for that kind of thing, using stuff like qiqqa test databases and running multiple versions of qiqqa in parallel plus checking if upgrading/downgrading via
setup.exe
actually flies (version checks or otherwise in there might block downgrade paths; currently the Qiqqa installer mentions everything as if it were an upgrade all the time, but that's okay as it allows moving back & forth that way without much trouble.then there's the choice of how far we take this:
Does 'backwards compatibility' mean:
when you upgrade, then downgrade, after the downgrade all your latest activity after the upgrade makes it through? ("full continuation across versions" - tough to accomplish / near impossible), or
when you upgrade, then downgrade, you get 'rolled back' to the time you were upgrading? ("rollback to previous version's SNAPSHOT"?)
when you upgrade (or downgrade, your major data sources are kept intact (using option 1 or 2 above), while the derivative data, e.g. search index caches, are rebuilt on every data format change: if we upgrade Lucene (upgrade LuceneNet #23), or downgrade it as we downgrade Qiqqa, we MUST regenerate the index as the Lucene file format has changed significantly. ("regenerate what you can that's unnoticeable")
or you use a 'weaker' compatibility definition, where 'derivative data' includes BrainStorms, Expedition sessions, etc.: the Qiqqa Configuration and Qiqqa Library Database plus PDF collection (and very probably the OCR text output) are then considered sources which must be kept backwards compatible using option 1 or 2 above, while the rest is marked as 'incompatible' and regenerated upon user action/request: this would mean BrainStorms and such would not always survive past an upgrade or downgrade where the brainstorm file format changes. ("regenerate as much as you can")
Depending on the choice made above (do we go with option 1,2,3 or 4?), there's the technical side of it:
is an extremely tough nut to crack under all circumstances and there's the question of new versions introducing data that old versions don't carry (take add source location/URL to Qiqqa Library record & update source location/URL for an existing PDF #27 which currently is only noticeable to hackers like me until extra work will make it visible via CSL, but how 'faithful' should the backwards compatible move be: kill all the new URI info? You get the point. Some degree of "metadata loss" is always lurking when downgrading.)
snapshotting can be done by
creating a (archived?) backup of the Qiqqa files/libraries and then migrating the files themselves, but that only works out well for upgrades: downgrades need to unpack that archive or they'll handle more-or-less-visibly incompatible data files. This would require more coding.
using (file format) version identifiers in the Qiqqa data file system, such that every version of Qiqqa knows "where to find his own": upgrades then go and look for older versions to migrate upwards when "their own" doesn't exist yet, while downgrades can simply revert to looking at their own data "that was before".
regenerate-what-you-can is the same as option 2, except you exchange extra time for code by regenerating indexes/caches instead of adding cod to migrate them to the new file format/version.
regenerate-even-more is like 3, and thus 2, except you don't code upgrade migration code for some of the Qiqqa derivative data files, e.g. brainstorms, expeditions, ...
Preference
My personal preference would be to go with option 2.ii - snapshotting with version in path (though do we need to snapshot/copy the entire bunch of metadata files as there are (or will be) interrelations, or do we snapshot/copy single files? My pref is the first: one snapshot per version, upgrade what you need upgraded).
Add to that as much option 3 as we can get away with (Lucene search index regeneration, OCR cache regeneration, i.e. re-run OCR task when we change the formats there, or the OCR engine maybe: #35 + #34, autotags, ...)
It's very simple and thus least error prone, which is handy to have for a path less traveled: back-pedaling on a Qiqqa software update/upgrade.
The 'negative' consequences is the user disk slowly filling with Qiqqa library/config snapshots, but that's a 'risk' I am very much okay with: here a 20K+ articles library clocks in at LESS THAN 50MByte of metadata, SQlite3 library main database included.
Of course, it would get worse than that when we change the OCR text cache, but that can very probably be coped with nicely by blowing away the entire OCR cache and regenerating it from scratch.
The (Lucene) Search Index is also quite another matter as that one clocks in at about ❗️700MByte❗️ for that same 20K+ article library, hence we should consider the Lucene 'database' as 'separate' from the rest of the Qiqqa library metadata (.library, .autotags, folder watcher, you name it)
Suggested storage layout
The text was updated successfully, but these errors were encountered: