3️⃣ Spill to disk

[joocer]

Operating on large volumes of data will mean it cannot fit in memory for all operations. Being able to split and merge is likely to be helpful in implementing parallel execution.

Some concerns and initial thoughts:

• how to configure (will need a cache setting and a new data volume setting, this improvement is the cache)
• file format (probably zstd feather or parquet)
• file management (delete your own and date stamp the files and delete older than a day if delete didn't happen)
• new operators will be needed (merge sort, split, merge)
• how the planner will know which algos to use (this will need to be based on predicted volumes, so needs information post binder about volumesbwhich needs the metastore)

Performance will be impacted, but the point of the spill is to prevent crashes, which ultimately will consume more user time.