feat(expr): add jsonb table functions #9977

wangrunji0408 · 2023-05-24T04:08:18Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Based on #9949, this PR adds the following jsonb table functions:

jsonb_array_elements ( jsonb ) → setof jsonb
jsonb_array_elements_text ( jsonb ) → setof text
jsonb_each ( jsonb ) → setof record ( key text, value jsonb )
jsonb_each_text ( jsonb ) → setof record ( key text, value text )
jsonb_object_keys ( jsonb ) → setof text

Checklist For Contributors

I have written necessary rustdoc comments (will update docs after doc(expr): add docs for #[function] macro #10010)
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

SQL commands, functions, and operators

Release note

Add jsonb table functions:

jsonb_array_elements ( jsonb ) → setof jsonb
jsonb_array_elements_text ( jsonb ) → setof text
jsonb_each ( jsonb ) → setof record ( key text, value jsonb )
jsonb_each_text ( jsonb ) → setof record ( key text, value text )
jsonb_object_keys ( jsonb ) → setof text

https://www.postgresql.org/docs/15/functions-json.html

Signed-off-by: Runji Wang <wangrunji0408@163.com>

wangrunji0408 · 2023-05-26T03:55:16Z

e2e_test/batch/types/jsonb.slt.part

+query T
+select jsonb_each('{"a":"foo", "b":"bar"}'::jsonb);
+----
+(a,"""foo""")
+(b,"""bar""")


Found a problem in string formatting in struct. cc @xiangjinwu

[Diff] (-expected|+actual) - (a,"""foo""") - (b,"""bar""") + (a,"foo") + (b,"bar")

The quote " is escaped to \" in array, but "" in record.

postgres=# select row(1, '"str"'); row --------------- (1,"""str""") postgres=# select array['"str"']; array ------------- {"\"str\""}

Is there any documents for this behavior? How should we handle it?

Yes it is buggy right now: #4769

Yes it is buggy right now: #4967

Seems not related. Wrong issue number?

My bad. Updated to #4769

Thanks! Then I'd like to change the test result to fit the current format and leave this issue for the future.

Yes, that works. Alternatively, we can write the tests without relying on the display of struct:

select (st).key, (st).value from (select jsonb_each('{"a":"foo", "b":"bar"}'::jsonb)) as t(st); ---- a "foo" b "bar"

codecov · 2023-05-26T08:38:12Z

Codecov Report

Merging #9977 (46860fb) into main (71729ac) will decrease coverage by 0.03%.
The diff coverage is 52.88%.

@@            Coverage Diff             @@
##             main    #9977      +/-   ##
==========================================
- Coverage   70.93%   70.91%   -0.03%     
==========================================
  Files        1233     1234       +1     
  Lines      210884   210990     +106     
==========================================
+ Hits       149600   149625      +25     
- Misses      61284    61365      +81

Flag	Coverage Δ
rust	`70.91% <52.88%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/common/src/array/arrow.rs	`65.84% <0.00%> (-0.55%)`	⬇️
src/common/src/array/data_chunk.rs	`87.83% <0.00%> (+0.12%)`	⬆️
src/common/src/types/jsonb.rs	`43.22% <0.00%> (-5.82%)`	⬇️
src/common/src/types/mod.rs	`63.23% <0.00%> (-0.95%)`	⬇️
src/common/src/types/struct_type.rs	`62.00% <0.00%> (-38.00%)`	⬇️
src/expr/src/table_function/mod.rs	`72.72% <ø> (ø)`
src/expr/src/table_function/unnest.rs	`57.14% <0.00%> (ø)`
src/expr/src/table_function/jsonb.rs	`22.72% <22.72%> (ø)`
src/expr/src/table_function/generate_series.rs	`92.00% <50.00%> (-0.08%)`	⬇️
src/frontend/src/expr/table_function.rs	`40.96% <71.42%> (-8.10%)`	⬇️
... and 8 more

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

BugenZhao

Generally LGTM!

BugenZhao · 2023-05-26T11:13:23Z

src/common/src/types/mod.rs

+/// To make sure there is `as_scalar_ref` for all scalar ref types.
+pub trait SelfAsScalarRef {
+    fn as_scalar_ref(&self) -> Self;
+}


Why we need this?

The purpose is to allow both owned type and reference type as function return value. (e.g. -> &str and -> Box<str>). So we want both types have a method .as_scalar_ref() -> &str so that we can call it to convert owned type into reference and keep reference type unchanged in the macro. Currently this method exists in the Scalar trait. At first I added this method to the ScalarRef trait, but it leads to conflicts for those types that share both traits (e.g. primitive types: i32, i64...). Therefore I created this trait to add as_scalar_ref for types which impl ScalarRef but not impl Scalar. Maybe a better solution is to extract an AsScalarRef trait. But it's too hard for me to do type exercise. 🤪

BugenZhao · 2023-05-26T11:16:04Z

src/common/src/types/struct_type.rs

+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        if s == "record" {


What format is the str in? Is it SQL representation?

It is almost auto-generated by Copilot from the above Display code. 😄

Signed-off-by: Runji Wang <wangrunji0408@163.com> Co-authored-by: xxchan <xxchan22f@gmail.com>

github-actions bot added type/feature user-facing-changes Contains changes that are visible to users labels May 24, 2023

wangrunji0408 mentioned this pull request May 24, 2023

Tracking: jsonb operations #7714

Closed

23 tasks

Base automatically changed from wrj/set-returning-function-macro to main May 25, 2023 11:49

wangrunji0408 added 7 commits May 26, 2023 11:01

add 3 jsonb table functions

ce991bb

Signed-off-by: Runji Wang <wangrunji0408@163.com>

use prost struct as TableFunctionType

f69674e

Signed-off-by: Runji Wang <wangrunji0408@163.com>

add e2e tests

071020a

Signed-off-by: Runji Wang <wangrunji0408@163.com>

add jsonb_each[_text]

fd878dd

Signed-off-by: Runji Wang <wangrunji0408@163.com>

flatten columns when returning struct type

88f8b64

Signed-off-by: Runji Wang <wangrunji0408@163.com>

add e2e tests for jsonb_each

8b575be

Signed-off-by: Runji Wang <wangrunji0408@163.com>

optimize struct output

a9b8f3c

Signed-off-by: Runji Wang <wangrunji0408@163.com>

wangrunji0408 force-pushed the wrj/json-table-functions branch from ec25dd6 to a9b8f3c Compare May 26, 2023 03:02

wangrunji0408 commented May 26, 2023

View reviewed changes

wangrunji0408 requested review from xiangjinwu, BugenZhao and xxchan May 26, 2023 03:55

wangrunji0408 marked this pull request as ready for review May 26, 2023 03:55

wangrunji0408 added 2 commits May 26, 2023 15:47

fix jsonb e2e tests

5790df6

use datatype at runtime

bcd3783

BugenZhao approved these changes May 26, 2023

View reviewed changes

wangrunji0408 added 2 commits June 1, 2023 13:07

Merge remote-tracking branch 'origin/main' into wrj/json-table-functions

096d358

update docs

6cbf58e

wangrunji0408 enabled auto-merge June 1, 2023 05:48

Merge branch 'main' into wrj/json-table-functions

8abb5ec

kwannoel mentioned this pull request Jun 1, 2023

fix(sqlsmith): ignore neg exponential #10121

Merged

7 tasks

Merge branch 'main' into wrj/json-table-functions

46860fb

wangrunji0408 added this pull request to the merge queue Jun 1, 2023

Merged via the queue into main with commit 1ac7732 Jun 1, 2023

wangrunji0408 deleted the wrj/json-table-functions branch June 1, 2023 12:02

kwannoel pushed a commit that referenced this pull request Jun 6, 2023

feat(expr): add jsonb table functions (#9977)

d93dcfa

Signed-off-by: Runji Wang <wangrunji0408@163.com> Co-authored-by: xxchan <xxchan22f@gmail.com>

hengm3467 mentioned this pull request Jun 9, 2023

Document jsonb table functions risingwavelabs/risingwave-docs#962

Closed

hengm3467 added the 📖✓ Covered or will be covered in the user docs. label Jun 9, 2023

neverchanje mentioned this pull request Aug 9, 2023

jsonb_array_elements / jsonb_array_elements_text #11570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(expr): add jsonb table functions #9977

feat(expr): add jsonb table functions #9977

wangrunji0408 commented May 24, 2023 •

edited

Loading

wangrunji0408 May 26, 2023

xiangjinwu May 26, 2023 •

edited

Loading

wangrunji0408 May 26, 2023

xiangjinwu May 26, 2023

wangrunji0408 May 26, 2023

xiangjinwu May 26, 2023

codecov bot commented May 26, 2023 •

edited

Loading

BugenZhao left a comment

BugenZhao May 26, 2023

wangrunji0408 May 29, 2023

BugenZhao May 29, 2023

BugenZhao May 26, 2023

wangrunji0408 May 29, 2023

		fn from_str(s: &str) -> Result<Self, Self::Err> {
		if s == "record" {

feat(expr): add jsonb table functions #9977

feat(expr): add jsonb table functions #9977

Conversation

wangrunji0408 commented May 24, 2023 • edited Loading

What's changed and what's your intention?

Checklist For Contributors

Checklist For Reviewers

Documentation

Types of user-facing changes

Release note

Choose a reason for hiding this comment

xiangjinwu May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 26, 2023 • edited Loading

Codecov Report

BugenZhao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangrunji0408 commented May 24, 2023 •

edited

Loading

xiangjinwu May 26, 2023 •

edited

Loading

codecov bot commented May 26, 2023 •

edited

Loading