Hello, does the sql source plugin now support bulk data sending?

[junf-liu]

Please describe your problem in detail, if necessary, you can upload the log file through the attachment:
Hello, does the sql source plugin now support bulk data sending?

[Yisaer]

Could you provide more detail about supporting bulk data querying in sql source.

[junf-liu]

您能否提供有关在 sql 源中支持批量数据查询的更多详细信息。

Hello, the demand is this, to the rough processing data in the database for statistical analysis, need to get multiple data; I know you will recommend me to use the window, but it is not easy to use, I want to get all the data at the same time, thank you very much!

[ngjaying]

Do you mean SQL source ingest an array instead of multiple tuples each time? This is not supported now. We can consider it for v2 release.

[junf-liu]

Do you mean SQL source ingest an array instead of multiple tuples each time? This is not supported now. We can consider it for v2 release.

Thank you very much, I am trying to modify the SQL source plugin to get the full amount of data, do you think this is a good idea?

[ngjaying]

Sure. This is going to be supported in v2, maybe we can discuss the implementation like in weekly comunity meeting.

[junf-liu]

Sure. This is going to be supported in v2, maybe we can discuss the implementation like in weekly comunity meeting.

with pleasure

[junf-liu]

@ngjaying
Hello, yesterday my plug-in was successfully built and can be used normally; I have two questions for you,

1. If the TemplateSql attribute is not filled in the mysql configuration, the sqlSource node will fail to be saved (the same problem occurs when you try to use the official sqlSource plug-in), but you do not open the editing permission on the flow interface, is this not considered comprehensively?
   1 error log
   GetQueryGenerator fail with error: read properties map[delimiter: format:json internalSqlQueryCfg:map[dateTimeFormat:map[control:text default: hint:map[en_US:dateTimeFormat zh_CN:dateTimeFormat] label:map[en_US:DateTime Format zh_CN:时间格式] name:dateTimeFormat optional:true type:string] indexField:map[control:text default: hint:map[en_US:index field zh_CN:索引字段名] label:map[en_US:Index Field zh_CN:索引字段名] name:indexField optional:true type:string] indexFieldType:map[control:select default: hint:map[en_US:Column type for the indexField zh_CN:索引字段的列类型] label:map[en_US:Index Field Type zh_CN:索引字段格式] name:indexFieldType optional:true type:string values:[DATETIME INT]] indexValue:map[control:text default: hint:map[en_US:index init value zh_CN:索引字段初始值] label:map[en_US:Index init value zh_CN:索引字段初始值] name:indexValue optional:true type:string] limit:map[control:text default:10 hint:map[en_US:query result limit zh_CN:查询结果条数限制] label:map[en_US:Limit zh_CN:查询条数限制] name:limit optional:true type:int] table:map[control:text default: hint:map[en_US:table name to query zh_CN:指定查询的数据库表名] label:map[en_US:Table Name zh_CN:表名] name:table optional:true type:string]] interval:1000 key: lookup:map[cache:map[control:radio default:true hint:map[en_US:Whether to enable cache zh_CN:是否启用缓存] label:map[en_US:Enable lookup cache zh_CN:开启查询缓存] name:cache optional:true type:bool] cacheMissingKey:map[control:radio default:true hint:map[en_US:Whether to cache nil value for a key zh_CN:是否对空值进行缓存] label:map[en_US:Cache missing key zh_CN:缓存未命中的 Key] name:cacheMissingKey optional:true type:bool] cacheTtl:map[control:text default:600 hint:map[en_US:The time to live of the cache in seconds zh_CN:缓存的生存时间（s)] label:map[en_US:Cache TTL zh_CN:缓存时间] name:cacheTtl optional:true type:int]] name:251mysql templateSqlQueryCfg:map[TemplateSql:map[control:textarea default: hint:map[en_US:SQL statement template zh_CN:SQL 语句模版] label:map[en_US:Query template zh_CN:查询语句模版] name:TemplateSql optional:false type:string] dateTimeFormat:map[control:text default: hint:map[en_US:Data time format for the index field zh_CN:索引字段的时间格式] label:map[en_US:DateTime Format zh_CN:时间格式] name:dateTimeFormat optional:true type:string] indexField:map[control:text default: hint:map[en_US:Which column for the table act as index to record the offset zh_CN:表的哪一列作为索引来记录偏移量] label:map[en_US:Index Field zh_CN:索引字段名] name:indexField optional:true type:string] indexFieldType:map[control:select default: hint:map[en_US:Column type for the indexField zh_CN:索引字段的列类型] label:map[en_US:Index Field Type zh_CN:索引字段格式] name:indexFieldType optional:true type:string values:[DATETIME INT]] indexValue:map[control:text default: hint:map[en_US:Initial index value zh_CN:初始索引值] label:map[en_US:Index init value zh_CN:索引字段初始值] name:indexValue optional:true type:string]] url:mysql://root:12345@192.168.1.251:3306/cloud2022?parseTime=true] fail with error: 9 error(s) decoding: * 'internalSqlQueryCfg.dateTimeFormat' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:text default: hint:map[en_US:dateTimeFormat zh_CN:dateTimeFormat] label:map[en_US:DateTime Format zh_CN:时间格式] name:dateTimeFormat optional:true type:string]' * 'internalSqlQueryCfg.indexField' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:text default: hint:map[en_US:index field zh_CN:索引字段名] label:map[en_US:Index Field zh_CN:索引字段名] name:indexField optional:true type:string]' * 'internalSqlQueryCfg.indexFieldType' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:select default: hint:map[en_US:Column type for the indexField zh_CN:索引字段的列类型] label:map[en_US:Index Field Type zh_CN:索引字段格式] name:indexFieldType optional:true type:string values:[DATETIME INT]]' * 'internalSqlQueryCfg.limit' expected type 'int', got unconvertible type 'map[string]interface {}', value: 'map[control:text default:10 hint:map[en_US:query result limit zh_CN:查询结果条数限制] label:map[en_US:Limit zh_CN:查询条数限制] name:limit optional:true type:int]' * 'internalSqlQueryCfg.table' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:text default: hint:map[en_US:table name to query zh_CN:指定查询的数据库表名] label:map[en_US:Table Name zh_CN:表名] name:table optional:true type:string]' * 'templateSqlQueryCfg.dateTimeFormat' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:text default: hint:map[en_US:Data time format for the index field zh_CN:索引字段的时间格式] label:map[en_US:DateTime Format zh_CN:时间格式] name:dateTimeFormat optional:true type:string]' * 'templateSqlQueryCfg.indexField' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:text default: hint:map[en_US:Which column for the table act as index to record the offset zh_CN:表的哪一列作为索引来记录偏移量] label:map[en_US:Index Field zh_CN:索引字段名] name:indexField optional:true type:string]' * 'templateSqlQueryCfg.indexFieldType' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:select default: hint:map[en_US:Column type for the indexField zh_CN:索引字段的列类型] label:map[en_US:Index Field Type zh_CN:索引字段格式] name:indexFieldType optional:true type:string values:[DATETIME INT]]' * 'templateSqlQueryCfg.templateSql' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[control:textarea default: hint:map[en_US:SQL statement template zh_CN:SQL 语句模版] label:map[en_US:Query template zh_CN:查询语句模版] name:TemplateSql optional:false type:string]'.
   1 image

2. Can I use the script node after the sqlSource node? Description Failed to save the rule.
   2 image

[junf-liu]

1 image

2 image

[ngjaying]

@junf-liu issue 1 is a manager problem, try to use REST API. issue 2: must build ekuiper with script tag.

[junf-liu]

@junf-liu issue 1 is a manager problem, try to use REST API. issue 2: must build ekuiper with script tag.

Hello, the problem has been solved according to the way you said; For example, if the value of the TemplateSql attribute is select * from A and the value of the dataSource attribute of the sqlSource node is B, the rule queries data according to the value of the TemplateSql attribute, but the dataSouces attribute does not take effect. Is it my improper operation? (Based on official plugin operation)

[ngjaying]

Manager is not supported now. Use REST API.

[junf-liu]

Manager is not supported now. Use REST API.

OK, thank you