Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement](doris-future) Support REGR_AVGX_AVGY_COUNT aggregation functions #39278

Closed
wants to merge 5 commits into from
Closed

[Enhancement](doris-future) Support REGR_AVGX_AVGY_COUNT aggregation functions #39278

wants to merge 5 commits into from

Conversation

therehello
Copy link

@therehello therehello commented Aug 13, 2024 •
edited
Loading

Proposed changes

Issue Number: close #38974

mysql> select regr_avgx(y, x) from t;
+--------------------+
| regr_avgx(y, x)    |
+--------------------+
| 10.872839868571429 |
+--------------------+
1 row in set (0.044 sec)
mysql> select regr_avgy(y, x) from t;
+--------------------+
| regr_avgy(y, x)    |
+--------------------+
| 60.417901200833334 |
+--------------------+
1 row in set (0.046 sec)
mysql> select * from t;
+------+------+------+
| id   | x    | y    |
+------+------+------+
|    3 |   13 |    3 |
|    5 |    4 |    7 |
|    1 |    1 |   18 |
|    2 |    6 | NULL |
|    4 | NULL |    6 |
+------+------+------+
5 rows in set (0.034 sec)

mysql> select regr_count(x, y) from t;
+------------------+
| regr_count(x, y) |
+------------------+
|                3 |
+------------------+
1 row in set (0.026 sec)

doc：apache/doris-website#1042

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 13, 2024
@therehello therehello force-pushed the regr_avgx_and_regr_avgy_and_regr_count branch from c8ddae3 to 333f2e8 Compare August 22, 2024 11:54
github-actions[bot]
github-actions bot reviewed Aug 22, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/aggregate_functions/aggregate_function_regr_count.h Show resolved Hide resolved
be/src/vec/aggregate_functions/aggregate_function_regr_count.h Show resolved Hide resolved
be/src/vec/aggregate_functions/aggregate_function_regr_count.h Show resolved Hide resolved
be/src/vec/aggregate_functions/aggregate_function_regr_count.h Show resolved Hide resolved
github-actions[bot]
github-actions bot reviewed Aug 22, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/aggregate_functions/aggregate_function_regr_count.h Outdated Show resolved Hide resolved
be/src/vec/aggregate_functions/aggregate_function_regr_count.h Outdated Show resolved Hide resolved
be/src/vec/aggregate_functions/aggregate_function_regr_count.h Outdated Show resolved Hide resolved
@therehello therehello force-pushed the regr_avgx_and_regr_avgy_and_regr_count branch from 7eacc91 to cf6b603 Compare August 22, 2024 13:03
@therehello therehello marked this pull request as ready for review August 22, 2024 14:04
@therehello therehello mentioned this pull request Aug 22, 2024
Open
@therehello
Copy link
Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37820 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf6b6031c3a0c196f41ffe9fd9f9d8d3fdbcc42d, data reload: false

------ Round 1 ----------------------------------
q1	17665	4446	4320	4320
q2	2019	182	178	178
q3	11682	972	1105	972
q4	10517	740	721	721
q5	7785	2836	2831	2831
q6	226	138	136	136
q7	980	623	601	601
q8	9350	2050	2061	2050
q9	7230	6521	6580	6521
q10	6984	2176	2167	2167
q11	459	244	244	244
q12	397	223	230	223
q13	17763	3005	3045	3005
q14	275	244	238	238
q15	521	495	493	493
q16	488	397	394	394
q17	985	661	727	661
q18	7315	7019	6762	6762
q19	1384	998	1059	998
q20	660	331	324	324
q21	3928	3075	2977	2977
q22	1091	1004	1015	1004
Total cold run time: 109704 ms
Total hot run time: 37820 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4331	4262	4262
q2	378	293	275	275
q3	2860	2641	2672	2641
q4	1936	1632	1664	1632
q5	5574	5715	5690	5690
q6	224	130	142	130
q7	2261	1821	1836	1821
q8	3296	3371	3428	3371
q9	8902	8819	8834	8819
q10	3581	3425	3358	3358
q11	618	501	509	501
q12	859	644	681	644
q13	14404	3162	3235	3162
q14	324	310	297	297
q15	542	488	497	488
q16	511	457	465	457
q17	1823	1532	1556	1532
q18	8028	7878	7718	7718
q19	1713	1597	1568	1568
q20	2195	1914	1866	1866
q21	5653	5498	5375	5375
q22	1115	1018	996	996
Total cold run time: 71145 ms
Total hot run time: 56603 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191622 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf6b6031c3a0c196f41ffe9fd9f9d8d3fdbcc42d, data reload: false

query1	1255	894	878	878
query2	6294	1909	1810	1810
query3	10609	3920	3761	3761
query4	59966	23839	23187	23187
query5	5376	494	488	488
query6	421	158	160	158
query7	5772	296	301	296
query8	308	212	213	212
query9	8942	2501	2483	2483
query10	490	289	274	274
query11	18310	15157	15122	15122
query12	170	114	107	107
query13	1566	372	371	371
query14	11043	7296	7326	7296
query15	220	176	164	164
query16	7600	445	496	445
query17	1148	578	597	578
query18	2060	334	290	290
query19	302	144	148	144
query20	121	108	113	108
query21	203	102	99	99
query22	4458	4604	4416	4416
query23	34226	33890	33476	33476
query24	5965	2836	2885	2836
query25	521	377	375	375
query26	687	155	157	155
query27	1802	284	276	276
query28	3654	2042	2027	2027
query29	690	390	410	390
query30	230	161	152	152
query31	954	773	748	748
query32	77	57	57	57
query33	437	292	277	277
query34	868	461	474	461
query35	832	746	698	698
query36	1081	937	940	937
query37	142	89	83	83
query38	3949	3893	4026	3893
query39	1463	1369	1398	1369
query40	201	118	121	118
query41	48	47	46	46
query42	126	98	100	98
query43	513	480	478	478
query44	1097	747	753	747
query45	201	175	170	170
query46	1082	758	732	732
query47	1912	1773	1788	1773
query48	367	288	294	288
query49	779	433	449	433
query50	808	427	407	407
query51	7209	7199	7130	7130
query52	100	91	95	91
query53	263	193	180	180
query54	571	465	466	465
query55	82	78	75	75
query56	290	267	279	267
query57	1199	1078	1078	1078
query58	228	228	238	228
query59	3012	2693	2710	2693
query60	309	283	284	283
query61	126	136	142	136
query62	761	625	723	625
query63	205	177	180	177
query64	4293	2263	1730	1730
query65	3259	3163	3169	3163
query66	699	338	347	338
query67	15452	15263	15152	15152
query68	3172	558	576	558
query69	398	270	276	270
query70	1152	1111	1121	1111
query71	373	276	269	269
query72	6148	2335	2071	2071
query73	743	321	321	321
query74	9240	8804	8858	8804
query75	3379	2657	2673	2657
query76	1542	1035	902	902
query77	517	318	323	318
query78	9567	9329	9161	9161
query79	1027	541	524	524
query80	683	488	502	488
query81	451	234	227	227
query82	235	189	131	131
query83	171	147	148	147
query84	259	76	75	75
query85	672	292	290	290
query86	310	302	274	274
query87	4495	4211	4315	4211
query88	3111	2290	2284	2284
query89	382	290	287	287
query90	1970	207	199	199
query91	125	96	107	96
query92	62	48	53	48
query93	1057	530	530	530
query94	742	289	300	289
query95	380	263	309	263
query96	586	269	264	264
query97	3188	3088	3108	3088
query98	214	207	195	195
query99	1503	1283	1272	1272
Total cold run time: 307258 ms
Total hot run time: 191622 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf6b6031c3a0c196f41ffe9fd9f9d8d3fdbcc42d, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.10	0.07
query5	0.50	0.50	0.50
query6	1.14	0.73	0.73
query7	0.01	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.47	0.49
query10	0.55	0.53	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.63	0.59	0.59
query14	0.76	0.80	0.79
query15	0.88	0.83	0.83
query16	0.36	0.37	0.37
query17	1.08	1.04	1.06
query18	0.21	0.20	0.20
query19	1.84	1.73	1.83
query20	0.01	0.01	0.01
query21	15.39	0.66	0.65
query22	4.42	7.12	2.04
query23	18.23	1.39	1.26
query24	2.11	0.22	0.24
query25	0.16	0.08	0.08
query26	0.28	0.18	0.18
query27	0.07	0.08	0.08
query28	13.19	1.03	1.01
query29	12.63	3.41	3.36
query30	0.25	0.06	0.06
query31	2.86	0.41	0.39
query32	3.25	0.49	0.49
query33	2.98	2.99	2.99
query34	17.04	4.40	4.39
query35	4.49	4.42	4.47
query36	0.65	0.49	0.51
query37	0.20	0.16	0.16
query38	0.16	0.15	0.16
query39	0.04	0.03	0.05
query40	0.16	0.13	0.13
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.65 s
Total hot run time: 30.99 s

@therehello
Copy link
Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38302 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed7841c8102cc08c8cbb4c4aab80b2036642e36e, data reload: false

------ Round 1 ----------------------------------
q1	17628	4624	4351	4351
q2	2023	189	177	177
q3	11779	989	1119	989
q4	10506	830	756	756
q5	7766	2870	2852	2852
q6	226	137	137	137
q7	994	611	603	603
q8	9343	2133	2100	2100
q9	7506	6587	6605	6587
q10	7003	2265	2205	2205
q11	451	248	244	244
q12	397	221	223	221
q13	17778	3011	3039	3011
q14	295	236	231	231
q15	506	479	486	479
q16	494	407	387	387
q17	1009	681	658	658
q18	7431	6964	6877	6877
q19	1383	1084	1129	1084
q20	705	339	346	339
q21	4064	2993	3075	2993
q22	1117	1021	1033	1021
Total cold run time: 110404 ms
Total hot run time: 38302 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4434	4396	4317	4317
q2	388	275	273	273
q3	2878	2615	2631	2615
q4	1927	1718	1692	1692
q5	5697	5693	5716	5693
q6	234	138	136	136
q7	2244	1807	1786	1786
q8	3360	3449	3470	3449
q9	8881	8859	8834	8834
q10	3640	3441	3370	3370
q11	636	496	496	496
q12	831	622	662	622
q13	15022	3183	3197	3183
q14	344	312	286	286
q15	526	495	493	493
q16	497	459	445	445
q17	1860	1554	1570	1554
q18	8234	7812	7932	7812
q19	1744	1602	1678	1602
q20	2168	1949	1893	1893
q21	5899	5413	5466	5413
q22	1120	1049	1039	1039
Total cold run time: 72564 ms
Total hot run time: 57003 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191900 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed7841c8102cc08c8cbb4c4aab80b2036642e36e, data reload: false

query1	1227	885	866	866
query2	6283	1904	1873	1873
query3	10612	4081	4030	4030
query4	60025	24204	23283	23283
query5	5419	509	480	480
query6	403	158	167	158
query7	5773	294	304	294
query8	291	214	213	213
query9	8975	2490	2494	2490
query10	482	281	267	267
query11	17992	15025	15248	15025
query12	147	101	104	101
query13	1587	392	388	388
query14	11364	7585	6997	6997
query15	241	176	177	176
query16	7078	486	523	486
query17	1098	599	600	599
query18	1225	318	310	310
query19	293	152	151	151
query20	122	111	113	111
query21	214	107	105	105
query22	4919	4500	4816	4500
query23	34444	33272	33345	33272
query24	5987	2890	2915	2890
query25	543	405	400	400
query26	688	159	159	159
query27	1769	285	283	283
query28	3852	2055	2039	2039
query29	663	430	421	421
query30	246	159	146	146
query31	924	770	757	757
query32	89	56	57	56
query33	450	302	292	292
query34	874	475	495	475
query35	822	733	731	731
query36	1065	916	958	916
query37	143	89	82	82
query38	4014	3778	3849	3778
query39	1445	1411	1390	1390
query40	206	116	119	116
query41	48	54	46	46
query42	117	99	101	99
query43	513	450	473	450
query44	1096	748	745	745
query45	194	170	173	170
query46	1109	757	745	745
query47	1911	1822	1810	1810
query48	376	300	304	300
query49	784	444	450	444
query50	821	423	436	423
query51	7219	7031	7078	7031
query52	101	89	86	86
query53	252	185	181	181
query54	579	473	464	464
query55	79	78	75	75
query56	296	273	266	266
query57	1179	1066	1101	1066
query58	227	235	244	235
query59	2994	2819	2719	2719
query60	286	265	263	263
query61	100	96	100	96
query62	774	646	644	644
query63	220	183	184	183
query64	3260	1746	1784	1746
query65	3184	3171	3134	3134
query66	657	332	343	332
query67	15568	15310	15397	15310
query68	4434	556	558	556
query69	422	280	269	269
query70	1089	1127	1103	1103
query71	424	276	274	274
query72	6395	2301	2072	2072
query73	777	326	331	326
query74	9433	8754	8877	8754
query75	3349	2667	2733	2667
query76	1776	1033	983	983
query77	563	319	335	319
query78	9727	9303	9064	9064
query79	1081	539	532	532
query80	798	508	511	508
query81	553	237	226	226
query82	257	135	144	135
query83	173	150	151	150
query84	309	83	73	73
query85	831	276	278	276
query86	340	297	301	297
query87	4376	4336	4265	4265
query88	3127	2342	2351	2342
query89	398	293	283	283
query90	2062	198	193	193
query91	124	98	98	98
query92	60	51	50	50
query93	1361	538	547	538
query94	769	329	299	299
query95	353	264	259	259
query96	593	275	270	270
query97	3234	3081	3096	3081
query98	217	212	199	199
query99	1536	1283	1270	1270
Total cold run time: 308879 ms
Total hot run time: 191900 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed7841c8102cc08c8cbb4c4aab80b2036642e36e, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.03	0.04
query3	0.22	0.04	0.05
query4	1.68	0.08	0.07
query5	0.50	0.50	0.50
query6	1.14	0.74	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.54	0.49	0.48
query10	0.55	0.55	0.54
query11	0.17	0.12	0.11
query12	0.16	0.13	0.12
query13	0.60	0.59	0.59
query14	0.77	0.80	0.77
query15	0.85	0.82	0.81
query16	0.38	0.37	0.41
query17	1.07	1.07	1.02
query18	0.22	0.21	0.22
query19	1.97	1.86	1.86
query20	0.01	0.01	0.01
query21	15.42	0.65	0.64
query22	3.68	7.03	2.54
query23	18.23	1.41	1.31
query24	2.04	0.22	0.23
query25	0.15	0.09	0.08
query26	0.27	0.19	0.18
query27	0.08	0.08	0.07
query28	13.34	1.04	1.01
query29	12.58	3.37	3.29
query30	0.25	0.06	0.05
query31	2.89	0.40	0.39
query32	3.26	0.48	0.48
query33	2.96	3.02	3.00
query34	17.28	4.34	4.40
query35	4.46	4.45	4.42
query36	0.66	0.47	0.50
query37	0.19	0.17	0.16
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.32 s
Total hot run time: 31.42 s

morrySnow
morrySnow requested changes Aug 28, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/RegrAvgXToAvg.java
/**
* RegrAvgXToAvg
*/
public class RegrAvgXToAvg implements ExpressionPatternRuleFactory {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regr_avgx and regr_avgy is not same with avg, we could not rewrite them to avg. refer to https://docs.snowflake.com/en/sql-reference/functions/regr_avgx

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@therehello therehello closed this Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@morrySnow morrySnow morrySnow requested changes

Assignees
No one assigned
Labels
doing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement](doris-future) Support "REGR_" aggregation functions (PART I)
3 participants
@therehello @doris-robot @morrySnow