Skip to content

redis-field-engineering/redis-connect-dist

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Redis Connect

Redis Connect 🚀 is a distributed platform that enables real-time event streaming, transformation, and propagation of changed-data events from heterogeneous data platforms to Azure Cache for Redis, Redis Cloud, and Redis Enterprise.

✅ No Code ✅ Seamless Integration ✅ Multi-Tenancy ✅ Linear-Scalability ✅ High-Availability ✅ Support

End-to-End demos

Redis Connect DB2 demo Redis Connect MySQL demo Redis Connect Oracle demo Redis Connect Postgres demo Redis Connect SQL Server demo
Redis Connect MongoDB demo Redis Connect Gemfire/Apache Geode demo Redis Connect Splunk HEC demo Redis Connect Files demo Redis Connect Vertica demo

Table of Contents

Background

Cloud-Native Architecture
Redis Connect has a cloud-native and shared-nothing architecture which allows any node to operate stand-alone or as a cluster member. Its platform-agnostic and lightweight design requires minimal infrastructure and avoids complex dependencies on 3rd-party platforms. All you need is Redis Enterprise.
Redis Connect Architecture
 
Redis Insight Multi-Tenancy | Partitioning | Linear Scalability
Redis Connect manages multi-tenant replication pipelines. A pipeline from source to sink is known as a job. Jobs with different source types can be collocated without becoming noisy neighbors. Jobs can be partitioned for linear scalability across one or more cluster nodes.
 
High-Availability | Recovery
Redis Connect jobs update their checkpoint upon each committed changed-data event within a transactional scope. In the event of a node failure or network split, in-flight jobs will fail over to another node and seamlessly begin replication from the last committed checkpoint. Data is not lost, and order is preserved. Redis Connect works in container orchestration environments such as Kubernetes and OpenShift.
Redis Connect Cluster
 
Redis Connect Custom Transformation Custom Transformations
Redis Connect jobs support user-defined business logic. You can create custom workflows that include user-defined stages for proprietary business rules, custom transformations, de-tokenization, and more. You can also extend the supported list of target sinks.
 
REST API | CLI | Swagger UI
Redis Connect is entirely data-driven and relies on Redis Enterprise as its metadata store. You can configure, start, stop, migrate, and restart jobs using the built-in REST API and interactive CLI. Redis Connect also exposes a Swagger UI to simplify endpoint discovery and operational experience.
Redis Connect Swagger UI
 
Redis Enterprise ACL Enterprise-Grade Security
Redis Connect jobs are stateless, so changed-data events are always in-transit. Redis Connect benefits from Redis Enterprise’s security, including RBAC, TLS, and more. Credentials, secrets, and trust-store passwords are never stored in Redis Connect; these secrets can be dynamically rotated with minimal disruption to the replication pipeline. Vault integration is supported.

Requirements

Minimum production hardware requirements

  • 1 GB of RAM
  • 4 CPU cores
  • 20 GB of disk space
  • 1 Gbps network

Runtime requirements

  • JRE 11+ (JRE 17+ version 0.10.7 onwards) e.g. Azul OpenJDK

Quick Start

You can run Redis Connect as a container or by downloading the code and running in your environment of choice.

Docker

You can run and deploy Redis Connect using the Redis Connect Docker image.

Download

Download the latest release for Linux or Windows and unarchive redis-connect-<version>.<build>.[tar.gz|zip] archive

The following subdirectories will be extracted under /redis-connect:

  • bin – Startup scripts
  • lib – Dependencies
  • config – Credentials property files, jobmanager.properties, and job-config (JSON) examples
  • extlib – Custom/external dependencies (e.g., custom stages, source-database drivers, etc.)

Launch Redis Connect

Redis Connect includes scripts for launching a single instance. You can run the scripts as follows:

On Linux

redis-connect/bin$ ./redisconnect.sh    
-------------------------------
Redis Connect startup script.
*******************************
Please ensure that the value of REDISCONNECT_JOB_MANAGER_CONFIG_PATH points to the correct jobmanager.properties in /home/viragtripathi/qa/vm/redis-connect/bin/redisconnect.conf before executing any of the options below
Check the value of redis.connection.url and credentials.dir.path in jobmanager.properties e.g.
redis.connection.url=redis://redis-19836.c9.us-east-1-2.ec2.cloud.redislabs.com:19836
credentials.dir.path=/var/secrets/redis
*******************************
Usage: [-h|cli|start]
options:
-h: Print this help message and exit.
cli: init Redis Connect CLI
start: init Redis Connect Instance (Cluster Member)
-------------------------------

On Windows

redis-connect\bin> redisconnect.bat
Prerequisite Configuration ❗
Update credentials.dir.path and redis.connection.url in /config/jobmanager.properties
Example - jobmanager.properties

Starting an instance

Linux:

redis-connect/bin$ ./redisconnect.sh start

Windows:

redis-connect\bin> redisconnect.bat start

Redis Connect Start Log

Swagger UI

Redis Connect Swagger UI is available on port 8282 by default. If you're running locally, you can point your browser to http://localhost:8282/swagger-ui/index.html


For quick start, use 'cdc_job' as jobName

Create Job Configuration - /connect/api/vi/job/config/{jobName}
For quick start, use a sample job configuration: DB2, Files, Gemfire, MongoDB, MySQL, Oracle, PostgreSQL, SQL Server and VERTICA

Prerequisite Configuration ❗
Update credentialsDirectoryPath, databaseURL, database.dbname (database.names for SQL Server), database.hostname, database.port and columns within sample job configuration for source and target, where applicable.

Start Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
For quick start, use 'stream' as jobType

Confirm Job Claim - /connect/api/vi/jobs/claim/{jobStatus}
For quick start, use 'all' as jobStatus

Once you've configured a job, try inserting some records into the source database. Then confirm that they have arrived in Redis.

Copyright

Redis Connect is developed by Redis, Inc. Copyright (C) 2023 Redis, Inc.