Doc shared CDC source transactions and refactor example section #1754

emile-00 · 2024-01-11T01:55:35Z

Info

Description
For both MySQL and PG CDC:
- Add note on transaction for shared CDC source
- Refactor examples section to include create single CDC table and create multiple tables from shared source
Notes
- [ Include any supplementary context or references here. ]
Related code PR
- feat(cdc): support transaction for shared cdc source risingwave#14375
Related doc issue

Resolves Document: feat(cdc): support transaction for shared cdc source #1732, Bug: refactor the MySQL and Postgres native CDC page #1726

For reviewers

Preview
- [ Paste the preview link to the updated page(s) here. Edit this item after the preview site is ready. To find the updated pages, scroll down to locate and open the Amplify preview link and select the dev version of the documentation. ]
Key points
- [ Parts that may need revision or extra consideration. ]

Before merging

I have checked the doc site preview, and the updated parts look good.
I have acquired the approval from the owner (and optionally the reviewers) of the code PR and at least one tech writer (CharlieSYH, emile-00, & hengm3467).

aws-amplify-us-east-1 · 2024-01-11T02:08:04Z

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-1754.d2fbku9n2b6wde.amplifyapp.com

ShanlanLi

LGTM

ShanlanLi · 2024-01-11T03:15:57Z

docs/guides/ingest-from-mysql-cdc.md

@@ -8,7 +8,7 @@ slug: /ingest-from-mysql-cdc
  <link rel="canonical" href="https://docs.risingwave.com/docs/current/ingest-from-mysql-cdc/" />
 </head>

-Change Data Capture (CDC) refers to the process of identifying and capturing data changes in a database, then delivering the changes to a downstream service in real time.
+Change Data Capture (CDC) refers to the process of identifying and capturing data changes in a database and then delivering the changes to a downstream service in real time.


I meet similar conditions like this. I guess it's Grammarly's suggestion to remove the comma and use "and". However, I personally think the comma is better. Because it makes the sentence shorter and easier to read. If use "and", the sentence is too long and a bit hard to understand, or it costs me more time to understand.

Maybe we can use both the comma and the conjunction when meeting this? like:
Change Data Capture (CDC) refers to the process of identifying and capturing data changes in a database**, and** then delivering the changes to a downstream service in real time.

This way, we meet both the requirements of grammar standards and reader-friendability.

StrikeW · 2024-01-11T04:09:30Z

Do we have a rendered link of the PR? This link https://pr-1754.d2fbku9n2b6wde.amplifyapp.com/ doesn't contain the modification of this PR.

StrikeW · 2024-01-11T04:18:11Z

Btw I found a bug not related to this PR: we should not claim transaction support for the Citus CDC connector. The Citus connector is not full fledged like mysql and postgres and it is in limited support.

emile-00 · 2024-01-12T00:23:42Z

Do we have a rendered link of the PR? This link https://pr-1754.d2fbku9n2b6wde.amplifyapp.com/ doesn't contain the modification of this PR.

Here they are: MySQL, PG

The preview page is on v1.5 and the updates are on v1.6 of the docs.

emile-00 · 2024-01-12T00:25:31Z

Btw I found a bug not related to this PR: we should not claim transaction support for the Citus CDC connector. The Citus connector is not full fledged like mysql and postgres and it is in limited support.

Got it! I can remove the option in this PR since it's a minor change.

StrikeW · 2024-01-12T07:08:19Z

docs/guides/ingest-from-mysql-cdc.md

@@ -155,7 +155,7 @@ To ensure all data changes are captured, you must create a table and specify pri
 ### Syntax

 ```sql
-CREATE TABLE [ IF NOT EXISTS ] source_name (
+CREATE {TABLE | SOURCE} [ IF NOT EXISTS ] source_name (


The syntax for CREATE SOURCE is not correct. We cannot specify table schema for the cdc source.
Btw I think we should remove [ FORMAT DEBEZIUM ENCODE JSON ], it is not compatible with CREATE SOURCE. User don't need to aware the format when they use native cdc connector.

StrikeW · 2024-01-12T07:12:47Z

docs/guides/ingest-from-mysql-cdc.md

+  username = 'root',
+  password = '123456',
+  database.name = 'mydb',
+  server.id = 5888


server.id in shared source is required, we may make it optional in future.

StrikeW · 2024-01-12T07:13:58Z

docs/guides/ingest-from-postgres-cdc.md

@@ -8,7 +8,7 @@
  <link rel="canonical" href="https://docs.risingwave.com/docs/current/ingest-from-postgres-cdc/" />
 </head>

-Change Data Capture (CDC) refers to the process of identifying and capturing data changes in a database, then delivering the changes to a downstream service in real time.
+Change Data Capture (CDC) refers to the process of identifying and capturing data changes in a database, and then delivering the changes to a downstream service in real time.

 RisingWave supports ingesting CDC data from PostgreSQL. Versions 10, 11, 12, 13, and 14 of PostgreSQL are supported.


We also support version 15 of PG

StrikeW · 2024-01-12T07:15:08Z

docs/guides/ingest-from-postgres-cdc.md


 ## Create a table using the native CDC connector

-To ensure all data changes are captured, you must create a table and specify primary keys. See the [`CREATE TABLE`](/sql/commands/sql-create-table.md) command for more details. The data format must be Debezium JSON.
+To ensure all data changes are captured, you must create a table or source and specify primary keys. See the [`CREATE TABLE`](/sql/commands/sql-create-table.md) command for more details. The data format must be Debezium JSON.


We don't need to remind user for the data format, it should be hide from users.

StrikeW · 2024-01-12T07:17:04Z

docs/guides/ingest-from-postgres-cdc.md


 ### Syntax

 ```sql
- CREATE TABLE [ IF NOT EXISTS ] source_name (
+ CREATE {TABLE | SOURCE} [ IF NOT EXISTS ] source_name (


This syntax is not correct. We cannot specify table schema for the cdc Source

StrikeW · 2024-01-12T07:19:00Z

docs/guides/ingest-from-postgres-cdc.md


 :::note
-RisingWave implements CDC via PostgresQL replication. Inspect the current progress via the [`pg_replication_slots`](https://www.postgresql.org/docs/14/view-pg-replication-slots.html) view. Remove inactive replication slots via [`pg_drop_replication_slot()`](https://www.postgresql.org/docs/current/functions-admin.html#:~:text=pg_drop_replication_slot).
+RisingWave implements CDC via PostgreSQL replication. Inspect the current progress via the [`pg_replication_slots`](https://www.postgresql.org/docs/14/view-pg-replication-slots.html) view. Remove inactive replication slots via [`pg_drop_replication_slot()`](https://www.postgresql.org/docs/current/functions-admin.html#:~:text=pg_drop_replication_slot).


Add a note that RW won't drop replication slot, user need to drop useless slots.

StrikeW · 2024-01-12T07:20:11Z

docs/guides/ingest-from-mysql-cdc.md

@@ -181,27 +181,54 @@ All the fields listed below are required.
 |database.name| Name of the database. Note that RisingWave cannot read data from a built-in MySQL database, such as `mysql`, `sys`, etc.|
 |table.name| Name of the table that you want to ingest data from. |
 |server.id| Optional. A numeric ID of the database client. It must be unique across all database processes that are running in the MySQL cluster. If not specified, RisingWave will generate a random ID.|
-|transactional| Optional. Specify whether you want to enable transactions for the CDC table that you are about to create. For details, see [Transaction within a CDC table](/concepts/transactions.md#transactions-within-a-cdc-table).|
+|transactional| Optional. Specify whether you want to enable transactions for the CDC table that you are about to create. This feature is also supported for shared CDC sources for multi-table transactions. For details, see [Transaction within a CDC table](/concepts/transactions.md#transactions-within-a-cdc-table).|

 ### Data format

 Data is in Debezium JSON or Debezium AVRO format. [Debezium](https://debezium.io) is a log-based CDC tool that can capture row changes from various database management systems such as PostgreSQL, MySQL, and SQL Server and generate events with consistent structures in real time. The MySQL CDC connector in RisingWave supports JSON or AVRO as the serialization format for Debezium data. The data format does not need to be specified when creating a table with `mysql-cdc` as the source.


We don't support AVRO at all. This should be mistake.

StrikeW

LGTM, thank you!

Doc shared CDC source transactions and refactor example section

3719a64

emile-00 requested review from StrikeW, CharlieSYH, hengm3467, ShanlanLi and WanYixian January 11, 2024 01:55

ShanlanLi approved these changes Jan 11, 2024

View reviewed changes

emile-00 added 2 commits January 11, 2024 16:26

Remove transaction from Citus

cfc9c86

Syntax fix

b724c1a

CharlieSYH approved these changes Jan 12, 2024

View reviewed changes

StrikeW reviewed Jan 12, 2024

View reviewed changes

Fixes for MySQL and PG CDC

28d4ce6

StrikeW approved these changes Jan 13, 2024

View reviewed changes

emile-00 and others added 2 commits January 16, 2024 11:28

Merge branch 'main' into transaction-shared-cdc

b2b5235

Update versioned docs

f3400a1

emile-00 merged commit e931ef1 into main Jan 16, 2024
3 checks passed

emile-00 deleted the transaction-shared-cdc branch January 16, 2024 20:23

This was referenced Jan 16, 2024

Bug: refactor the MySQL and Postgres native CDC page #1726

Closed

Document that postgres-cdc won't drop replication slots on upstream #1747

Closed

xiangjinwu mentioned this pull request Aug 12, 2024

fix(source): PostgreSQL/MySQL CDC source vs table relationship #2449

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Doc shared CDC source transactions and refactor example section #1754

Doc shared CDC source transactions and refactor example section #1754

emile-00 commented Jan 11, 2024

aws-amplify-us-east-1 bot commented Jan 11, 2024

ShanlanLi left a comment

ShanlanLi Jan 11, 2024

StrikeW commented Jan 11, 2024 •

edited

Loading

StrikeW commented Jan 11, 2024

emile-00 commented Jan 12, 2024

emile-00 commented Jan 12, 2024

StrikeW Jan 12, 2024 •

edited

Loading

StrikeW Jan 12, 2024

StrikeW Jan 12, 2024

StrikeW Jan 12, 2024

StrikeW Jan 12, 2024 •

edited

Loading

StrikeW Jan 12, 2024

StrikeW Jan 12, 2024

StrikeW left a comment •

edited

Loading

Doc shared CDC source transactions and refactor example section #1754

Doc shared CDC source transactions and refactor example section #1754

Conversation

emile-00 commented Jan 11, 2024

Info

For reviewers

Before merging

aws-amplify-us-east-1 bot commented Jan 11, 2024

ShanlanLi left a comment

Choose a reason for hiding this comment

ShanlanLi Jan 11, 2024

Choose a reason for hiding this comment

StrikeW commented Jan 11, 2024 • edited Loading

StrikeW commented Jan 11, 2024

emile-00 commented Jan 12, 2024

emile-00 commented Jan 12, 2024

StrikeW Jan 12, 2024 • edited Loading

Choose a reason for hiding this comment

StrikeW Jan 12, 2024

Choose a reason for hiding this comment

StrikeW Jan 12, 2024

Choose a reason for hiding this comment

StrikeW Jan 12, 2024

Choose a reason for hiding this comment

StrikeW Jan 12, 2024 • edited Loading

Choose a reason for hiding this comment

StrikeW Jan 12, 2024

Choose a reason for hiding this comment

StrikeW Jan 12, 2024

Choose a reason for hiding this comment

StrikeW left a comment • edited Loading

Choose a reason for hiding this comment

StrikeW commented Jan 11, 2024 •

edited

Loading

StrikeW Jan 12, 2024 •

edited

Loading

StrikeW Jan 12, 2024 •

edited

Loading

StrikeW left a comment •

edited

Loading