-
Notifications
You must be signed in to change notification settings - Fork 38
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* feat: add database sqlite3 reader/writer * sqlite3: reset (#43) * none: some env tidy up (#35) 1. Makefile: local may have 'DB2_HOME' already 2. .gitignore: adding ignores for vim and tags Signed-off-by: shane.xb.qian <shane.qian@foxmail.com> * fix: fix split bug when setting is not set (#37) * feat & fix & docs: Add column name to error message when there's an error in reading or writing a column, fix the issue where ignoreOneByOneError does not work when set to false, and add F&Q for the database reader and writer. (#38) * none: some tiny code tidy-up (#34) Signed-off-by: shane.xb.qian <shane.qian@foxmail.com> * docs: Update README.md * feat & docs: improve go1.16 to go 1.20, change package sqlserver to github.com/microsoft/go-mssqldb,and Update all packages to the latest version. * fix: fix some complime error * fix：fix complie error * scripts: do lint for all go version * fix: 修复mysql的问题 * docs: fix go version * feat: in package element add NewBigIntColumnValueFromUint64 and NewDecimalColumnValueFromFloat32, remove Float64 and add Uint64. --------- Signed-off-by: shane.xb.qian <shane.qian@foxmail.com> Co-authored-by: Shane-XB-Qian <shane.qian@live.com> Co-authored-by: 励码万言 <48382427+limawanyan@users.noreply.github.com> * feat: add sqlite3 data read * add Sqlite3 (#52) * none: some env tidy up (#35) 1. Makefile: local may have 'DB2_HOME' already 2. .gitignore: adding ignores for vim and tags Signed-off-by: shane.xb.qian <shane.qian@foxmail.com> * fix: fix split bug when setting is not set (#37) * feat:add sqlite3 --------- Signed-off-by: shane.xb.qian <shane.qian@foxmail.com> Co-authored-by: Shane-XB-Qian <shane.qian@live.com> Co-authored-by: Breeze0806 <53822089+Breeze0806@users.noreply.github.com> * fix： 修复sqlite3测试的问题 * fix: fit go1.20 * fix: makefile --------- Signed-off-by: shane.xb.qian <shane.qian@foxmail.com> Co-authored-by: Shane-XB-Qian <shane.qian@live.com> Co-authored-by: 励码万言 <48382427+limawanyan@users.noreply.github.com>
- Loading branch information
3 people
authored
Sep 28, 2024
1 parent
6c8ae4f
commit a285334
Showing
34 changed files
with
2,358 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| { | ||
| "core" : { | ||
| "container": { | ||
| "job":{ | ||
| "id": 1, | ||
| "sleepInterval":100 | ||
| } | ||
| } | ||
| }, | ||
| "job":{ | ||
| "content":[ | ||
| { | ||
| "reader":{ | ||
| "name": "sqlite3reader", | ||
| "parameter": { | ||
| "column": ["*"], | ||
| "connection": { | ||
| "url": "E:\\Sqlite3\\test.db", | ||
| "table": { | ||
| "db":"main", | ||
| "name":"type_table" | ||
| } | ||
| }, | ||
| "where": "" | ||
| } | ||
| }, | ||
| "writer":{ | ||
| "name": "sqlite3writer", | ||
| "parameter": { | ||
| "writeMode": "insert", | ||
| "column": ["*"], | ||
| "connection": { | ||
| "url": "E:\\Sqlite3\\test.db", | ||
| "table": { | ||
| "db":"main", | ||
| "name":"type_table_copy" | ||
| } | ||
| }, | ||
| "batchTimeout": "1s", | ||
| "batchSize":1000 | ||
| } | ||
| }, | ||
| "transformer":[] | ||
| } | ||
| ] | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| drop table if exists "type_table"; | ||
| create table "type_table" ( | ||
| "t_integer" integer, | ||
| "t_real" real, | ||
| "t_text" text | ||
| ); | ||
|
|
||
| insert into "type_table" values (1, 1.01, 123456); | ||
|
|
||
| drop table if exists "type_table_copy"; | ||
| create table "type_table_copy" ( | ||
| "t_integer" integer, | ||
| "t_real" real, | ||
| "t_text" text | ||
| ); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| { | ||
| "core" : { | ||
| "container": { | ||
| "job":{ | ||
| "id": 1, | ||
| "sleepInterval":100 | ||
| } | ||
| } | ||
| }, | ||
| "job":{ | ||
| "content":[ | ||
| { | ||
| "reader":{ | ||
| "name": "sqlite3reader", | ||
| "parameter": { | ||
| "column": ["*"], | ||
| "connection": { | ||
| "url": "E:\\Sqlite3\\test.db", | ||
| "table": { | ||
| "db":"main", | ||
| "name":"type_table" | ||
| } | ||
| }, | ||
| "where": "" | ||
| } | ||
| }, | ||
| "writer":{ | ||
| "name": "sqlite3writer", | ||
| "parameter": { | ||
| "writeMode": "insert", | ||
| "column": ["*"], | ||
| "connection": { | ||
| "url": "E:\\Sqlite3\\test.db", | ||
| "table": { | ||
| "db":"main", | ||
| "name":"type_table_copy" | ||
| } | ||
| }, | ||
| "batchTimeout": "1s", | ||
| "batchSize":1000 | ||
| } | ||
| }, | ||
| "transformer":[] | ||
| } | ||
| ] | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| # Sqlite3Reader Plugin Documentation | ||
|
|
||
| ## Quick Introduction | ||
|
|
||
| The Sqlite3Reader plugin enables data reading from Sqlite3 databases. Under the hood, Sqlite3Reader connects to remote Sqlite3 databases using `github.com/mattn/go-sqlite3` and executes corresponding SQL statements to query data from the database. | ||
|
|
||
| ## Implementation Principles | ||
|
|
||
| Sqlite3Reader connects to remote Sqlite3 databases using `github.com/mattn/go-sqlite3` and generates SQL queries based on user-provided configuration information. These queries are then sent to the remote Sqlite3 database, and the returned results are assembled into an abstract dataset using go-etl's custom data types. This dataset is then passed to downstream Writer processing. | ||
| Sqlite3Reader implements specific queries by calling go-etl's custom `storage/database` DBWrapper, which is defined in the dbmsreader's query process. DBWrapper encapsulates many interfaces of `database/sql` and abstracts the database dialect, Dialect. For sqlite3, the implementation of Dialect provided by `storage/database/sqlite3` is used. | ||
|
|
||
| ## Functionality Description | ||
|
|
||
| ### Configuration Example | ||
|
|
||
| Configuring a job to synchronize data from a Sqlite3 database to a local system: | ||
|
|
||
| ```json | ||
| { | ||
| "job": { | ||
| "content": [ | ||
| { | ||
| "reader": { | ||
| "name": "sqlite3reader", | ||
| "parameter": { | ||
| "column": [ | ||
| "*" | ||
| ], | ||
| "connection": { | ||
| "url": "E:\\Sqlite3\\test.db", | ||
| "table": { | ||
| "db": "main", | ||
| "name": "type_table" | ||
| } | ||
| }, | ||
| "where": "" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Parameter Explanation | ||
|
|
||
| #### url | ||
|
|
||
| - Description: It is mainly used to configure the path of sqlite3 database files | ||
| - Required: Yes | ||
| - Default: None | ||
|
|
||
| #### table | ||
|
|
||
| Describes the sqlite3 table information. | ||
|
|
||
| ##### name | ||
|
|
||
| - Description: Mainly used to configure the table name of the sqlite3 table. | ||
| - Required: Yes | ||
| - Default: None | ||
|
|
||
| #### column | ||
|
|
||
| - Description: The set of column names that need to be synchronized from the configured table. JSON array syntax is used to describe the column information. Using "*" represents that all columns are used by default, for example, `["*"]`. | ||
|
|
||
| Supports column pruning, which means users can select specific columns for export. | ||
|
|
||
| Supports column reordering, meaning the columns can be exported in an order different from the table schema. | ||
|
|
||
| Supports constant configuration. Users need to follow the sqlite3 syntax format. | ||
|
|
||
| - Required: Yes | ||
| - Default: None | ||
|
|
||
| #### split | ||
|
|
||
| ##### key | ||
|
|
||
| - Description: Mainly used to configure the splitting key for the sqlite3 table. The splitting key must be of type bigInt/string/time, assuming that the data is evenly distributed based on the splitting key. | ||
| - Required: No | ||
| - Default: None | ||
|
|
||
| ##### timeAccuracy | ||
|
|
||
| - Description: Mainly used to configure the time splitting key for the sqlite3 table, mainly to describe the smallest unit of time, such as day (for dates), min (for minutes), s (for seconds), ms (for milliseconds), us (for microseconds), ns (for nanoseconds). | ||
| - Required: No | ||
| - Default: None | ||
|
|
||
| ##### range | ||
|
|
||
| ###### type | ||
| - Description: Mainly used to configure the default value type of the splitting key for the sqlite3 table, with values being bigInt/string/time. Here, it will check the type of the splitting key in the table, so please make sure the type is correct. | ||
| - Required: No | ||
| - Default: None | ||
|
|
||
| ###### left | ||
| - Description: Mainly used to configure the default maximum value of the splitting key for the sqlite3 table. | ||
| - Required: No | ||
| - Default: None | ||
|
|
||
| ###### right | ||
| - Description: Mainly used to configure the default minimum value of the splitting key for the sqlite3 table. | ||
| - Required: No | ||
| - Default: None | ||
|
|
||
| #### where | ||
|
|
||
| - Description: Mainly used to configure the where condition for the select statement. | ||
| - Required: No | ||
| - Default: None | ||
|
|
||
| #### querySql | ||
|
|
||
| - Description: In some business scenarios, the `where` configuration item is not sufficient to describe the filtering conditions, so users can use this configuration item to customize the filtering SQL. When users configure this item, the DataX system will ignore the `table`, `column`, and other configuration items, and directly use the content of this configuration item for data filtering. For example, if you need to perform a join operation on multiple tables before synchronizing the data, you can use `select a,b from table_a join table_b on table_a.id = table_b.id`. | ||
| When the user configures `querySql`, Sqlite3Reader directly ignores the configuration of `table`, `column`, and `where` conditions. The priority of `querySql` is higher than that of `table`, `column`, and `where` options. | ||
| - Required: No | ||
| - Default: None | ||
|
|
||
| #### trimChar | ||
|
|
||
| - Description: Whether to remove leading and trailing spaces for the char type in sqlite3. | ||
| - Required: No | ||
| - Default: false | ||
|
|
||
| ### Type Conversion | ||
|
|
||
| Currently, Sqlite3Reader supports most sqlite3 types, but there are still some individual types that are not supported. Please check your types carefully. | ||
|
|
||
| Below is a list of type conversions that Sqlite3Reader performs for sqlite3 types: | ||
|
|
||
| | go-etl的类型 | sqlite3数据类型 | | ||
| | ------------ |--------------------| | ||
| | string | INTEGER、TEXT、REAL、BLOB | | ||
|
|
||
| ## Performance Report | ||
|
|
||
| To be tested. | ||
|
|
||
| ## Constraints and Limitations | ||
|
|
||
| ### Database Encoding Issues | ||
| Currently, only the utf8 character set is supported. | ||
|
|
||
| ## FAQ |
Oops, something went wrong.