feature: index migrations

[ra0x3]

• As an index owner I might want to change the data models involved in my index
• I can do this by updating my GraphQL schema and re-deploying assets
• However, this might change the literal tables that I'm using in my index
  • If I want to keep the same tables but use different models I might need to use a migration
• Users should be allowed to include a migrations/ folder that has up and down migrations that can be applied to indices

Additional context

• This might also require a forc index migrate [args] [kwargs] update to the plugin

[ap-atul]

Just putting out some thoughts.

The need to change the schema will occur when I want my indexer to collect more or fewer data. Most cases will want the indexer to start from 0% and start building into the new schema.

Writing migrations will need the dev to know how the indexer stores the data, also there are some nitty bitty things like the object column, data types, and indexed metadata entity table. A lot of stuff to think about. A bit more work on the dev.

When there is a schema mismatch, the indexer can erase the old data and start over with a new schema.

Maybe a feature to extend the schema keeping the old changes and only allowing new entities, which would be more upgradable in a sense.

[ra0x3]

@ap-atul Thanks for the feedback.

I would keep in mind:

• Migrations are just SQL queries that you apply to a table.
• No one needs migration functionality at this time.
• If you don't need migrations, you wouldn't write migrations (it's opt-in).
  • Further, if a dev isn't familiar with migrations, or their index, then they obviously shouldn't be trying to write migrations.
• Migrations would more so targeted towards larger projects where data persistence/consistency is high value.
• I don't think we want to erase any data (unless explicitly told by an index owner to do so)

Maybe a feature to extend the schema keeping the old changes and only allowing new entities, which would be more upgradable in a sense.

• I believe this is existing functionality. If you have Index(A) but upgrade your data model and redeploy with a new namespace/identifier then you have Index(A) and Index(B). You can choose to stop Index(A) if you like (or just let it keep indexing)

[ra0x3]

Update on this:

• This would be a small wrapper around sqlx-cli's migrations
  • https://crates.io/crates/sqlx-cli
• Should support the following things:
  • --database-url URI of database
    • Use default if none is supplied
  • --migrations-dir
    • Same as the flag sqlx-cli uses
  • --up if migrating up
    • Assume --up if neither --up nor --down is passed
  • --down if migrating down
  • Should also allow creating migrations

Again, we're just gonna support those few operations (new migrations, migrate up, migrate down)

Comand can be something like:

• forc index migrate [ARGS] [OPTS]