Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

canal-server1.1.4遇到很大数据量的DML语句时,出现重复订阅 #2767

Closed
2 tasks done
VincentLiSH opened this issue May 26, 2020 · 1 comment
Closed
2 tasks done

Comments

@VincentLiSH
Copy link

VincentLiSH commented May 26, 2020

  • I have searched the issues of this repository and believe that this is not a duplicate.
  • I have checked the FAQ of this repository and believe that this is not a duplicate.

environment

  • canal version 1.1.4
  • mysql version 8
  • zookeeper latest

Issue Description

我在数据库执行create table xxx_etl_init as select * from xxx 这样的dml语句,对xxx表存量数据做初始化etl,
对于数据量几千的表没什么问题,
但是当遇到数据量9万以上的表时,canal-server会出现以下日志——最初22:35的两条是正常的 客户端订阅轮训后产生的,但是接下来23:13:47以及之后不断出现的日志 则会导致重复订阅 :

2020-05-25 22:35:55.005 [New I/O server worker #1-1] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to qcdbv2..*
2020-05-25 22:35:55.006 [New I/O server worker #1-1] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^qcdbv2..$
2020-05-25 23:13:47.771 [New I/O server worker #1-2] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to qcdbv2..

2020-05-25 23:13:47.772 [New I/O server worker #1-2] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^qcdbv2..$
2020-05-25 23:13:52.883 [New I/O server worker #1-3] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to qcdbv2..

2020-05-25 23:13:52.883 [New I/O server worker #1-3] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^qcdbv2..$
2020-05-25 23:41:35.519 [New I/O server worker #1-4] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to qcdbv2..

2020-05-25 23:41:35.519 [New I/O server worker #1-4] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^qcdbv2..*$

Steps to reproduce

canal.instance.memory.buffer.size =1024
canal.instance.memory.buffer.memunit = 1048576
canal.instance.memory.batch.mode = MEMSIZE

Expected behaviour

Actual behaviour

If there is an exception, please attach the exception trace:

Just put your stack trace here!
@agapple
Copy link
Member

agapple commented Aug 21, 2020

这个日志是你客户端在发生重新订阅,检查一下客户端是否有异常?

@agapple agapple closed this as completed Aug 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants