Model training versioning support in Jaseci/JSORC

[ypkang]

Objective:

• As developer trains and fine-tune models via jac, the jaseci/jsorc infrastructure should automatically manage model versioning and maintain consistent behavior through any pod/server restart.

Solution/Design:

• TLDR -- we will implement a very light-weight MLFlow-esque system in jaseci ai kit and jsorc.
• On the jaseci ai kit module side
  • automatically track version history of trained models (with a default retention range, e.g. last 10 training iterations). With every train action, user will receive an UUID pointing to that version of the model.
  • user can also explicitly save trained version. this will be retaiend forever.
  • All trained models versions are saved in the PV (.jaseci/ directory).
  • And we save a table of all the UUIDs of the versions, pointing to the model path. We will start with this, and expand it to include other meta information such as accuracies, training loss, etc.. So we will need to design a format that is extensible.
  • And we also save the UUID of the latest version, or the last active version.
  • In the SETUP action, we will try to load the latest UUID version first, if it doesn't exist, load the default.
• On the jaseci, jsorc side.
  • For now, we will need to make sure that the auto-reload logic of jaseci actions invoke setup() so in the case of jaseci pod restarts, any previously loaded and trained local module will also load up the latest version pointed to by the latest UUID.
  • In the future, JSORC will handle this.