Skip to content

Latest commit

 

History

History
161 lines (139 loc) · 5.16 KB

README.md

File metadata and controls

161 lines (139 loc) · 5.16 KB
Raw

Destination Tester

Pre-requisites

How To Run

  1. Pull the latest docker image from fivetrandocker/fivetran-sdk-tester on Docker Hub.

  2. Run a container using the image with the following command. Make sure to map a local directory for the tool by replacing <local-data-folder> placeholders in the command, and replace <version> with the version of the image you pulled.

docker run --mount type=bind,source=<local-data-folder>,target=/data -a STDIN -a STDOUT -a STDERR -it -e WORKING_DIR=<local-data-folder> -e GRPC_HOSTNAME=host.docker.internal --network=host fivetrandocker/fivetran-sdk-tester:<version>  --tester-type destination --port <port>
  1. To rerun the container from step #2, use the following command:
docker start -i <container-id>

Input Files

Destination tester simulates operations from a source by reading input files from the local data folder. Each input file represents a batch of operations, encoded in JSON format. Data types in common.proto file can be used as column data types.

List of Operations

Table Operations

  • describe_table
  • create_table
  • alter_table

Single Record Operations

  • upsert
  • update
  • delete
  • soft_delete

Bulk Record Operations

  • truncate_before
  • soft_truncate_before

Example input file

Here is an example input file named input_1.json:

{
    "create_table" : {
        "transaction": {
            "columns": {
                "id": "INT",
                "amount" : "DOUBLE",
                "desc": "STRING"
            },
            "primary_key": ["id"]
        },
        "campaign": {
            "columns": {
                "name": "STRING",
                "num": {"type": "DECIMAL", "precision": 6, "scale": 3}
            },
            "primary_key": []
        }
    },
    "alter_table" : {
        "transaction": {
            "columns": {
                "id": "INT",
                "amount" : "FLOAT",
                "desc": "STRING"
            },
            "primary_key": ["id"]
        }
    },
    "describe_table" : [
        "transaction"
    ],
    "ops" : [
        {
            "upsert": {
                "transaction": [
                    {"id":1, "amount": 100.45, "desc": null},
                    {"id":2, "amount": 150.33, "desc": "two"}
                ],
                "campaign": [
                    {"_fivetran_id": "abc-123-xyz", "name": "Christmas", "num": 100.23},
                    {"_fivetran_id": "vbn-543-hjk", "name": "New Year", "num": 200.56}
                ]
            }
        },
        {
            "truncate_before": [
                "campaign"
            ]
        },
        {
            "update": {
                "transaction": [
                    {"id":1, "amount": 200}
                ]
            }
        },
        {
            "soft_truncate_before": [
                "transaction"
            ]
        },
        {
            "upsert": {
                "transaction": [
                    {"id":10, "amount": 100, "desc": "thee"},
                    {"id":20, "amount": 50, "desc": "mone"}
                ],
                "campaign": [
                    {"_fivetran_id": "dfg-890-lkj", "name": "Christmas 2", "num": 400.32}
                ]
            }
        },
        {
            "delete": {
                "transaction": [
                    {"id":3}
                ],
                "campaign": [
                    {"_fivetran_id": "abc-123-xyz"}
                ]
            }
        },
        {
            "soft_delete": {
                "transaction": [
                    {"id":4}
                ],
                "campaign": [
                    {"_fivetran_id": "dfg-890-lkj"}
                ]
            }
        }
    ]
}

CLI Arguments

The tester supports the following optional CLI arguments to alter its default behavior. You can append these options to the end of the docker run command provided in step 2 of How To Run section above.

--port

This option defines the port the tester should run on.

--plain-text

This option disables encryption and compression of batch files for debugging purposes.

--input-file

The tester by default reads all input files from local data folder and executes them in the alphabetical order they appear. You can specify a single input file to be read and executed using this option. Providing just the filename is sufficient.

--schema-name

The tester by default creates a schema named tester. This option allows the tester to run with a custom schema name by specifying --schema-name <custom_schema_name> where <custom_schema_name> is your custom schema name.

--disable-operation-delay

The tester by default adds a delay to operation for real-time simulation. Specifying this argument disables the delay.