Clickhouse compatibility: MergeTree engine

[akvlad]

What

Desired functionality.

A client sends a request:

create_table: experiment
fields:
  a: UInt64
  b: String
  c: Float64
engine: Merge
order_by: 
  - a
timestamp:
  field: a
  precision: ms
partition_by:
  - b

Then the client sends a bunch of another requests once in a while:

POST /query?query=INSERT INTO experiment FORMAT JSONEachRow

{ "a": 1, "b": "asdad", "c": 1.2}
{ "a": 2, "b": "asdad", "c": 1.2}
{ "a": 3, "b": "asdad", "c": 1.2}

Desired result.

There is a directory on the server HD /tmp/experiment.

It has a plenty of parquet files.

Once in a while these parquet files get merged into a bigger ones according to the the order by key.

The maximum size of a merged file is 4GB.

Why

We can migrate qryn writer part with little to no changes to DuckDB supported data storage.

In order to do it we should support the simplest clickhouse style insert queries.

Let's start with a simple table creation and JSONEachRow insert function.

In future we can optimize the engine for the time-series data we store.

Tasks

Beta Give feedback

1. Create DuckDB internal schema #29
   +0
   
2. Create a mechanism to write parquet files into a specific hard drive folder and merge them in a while #30
   +0
   
3. Parse and process MergeTree CREATE TABLE expression #31
   +0
4. Parse and process the simplest INSERT INTO ... FORMAT JSONEachRow request #32
   +0

[github-actions]

Thanks for opening an Issue! Please star this repository to motivate developers! ⭐