Skip to content

[fix](be) Report stream load size limits in MiB - #66224

Open
0AyanamiRei wants to merge 2 commits into
apache:masterfrom
0AyanamiRei:fix/stream-load-json-size-unit
Open

[fix](be) Report stream load size limits in MiB#66224
0AyanamiRei wants to merge 2 commits into
apache:masterfrom
0AyanamiRei:fix/stream-load-json-size-unit

Conversation

@0AyanamiRei

@0AyanamiRei 0AyanamiRei commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Stream Load body-size limit errors display raw byte counts next to BE configuration items whose names and values are expressed in MB, making the message misleading. Report both the actual body size and configured limit in MiB for JSON Stream Load, CSV Stream Load, and HTTP Stream. The limit calculation and rejection behavior are unchanged.

Release note

Report Stream Load body-size limit errors consistently in MiB.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    ./run-be-ut.sh --run --filter=StreamLoadTest.* -j48 (4 tests passed)

  • Behavior changed:

    • No.
    • Yes. JSON and CSV Stream Load size-limit errors now display both values in MiB.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@0AyanamiRei
[fix](be) Clarify JSON stream load size limit units
fcd414a 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: JSON Stream Load body-size errors displayed both byte counts without units next to a configuration named streaming_load_json_max_mb, making the configured limit misleading. Keep the existing threshold and byte comparison unchanged, label both exact byte values, and show the snapshotted configured MB value. Add a BE unit test using the reported request size.

### Release note

Clarify units in the JSON Stream Load body-size limit error message.

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run --filter=StreamLoadTest.* -j48 (2 tests passed)
- Behavior changed: Yes. The JSON Stream Load size-limit error now identifies byte values and the configured MB value.
- Does this need documentation: No
@0AyanamiRei
[fix](be) Report stream load size limits in MiB
09fa875 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Stream Load size-limit errors mixed byte counts with configuration names expressed in MB, which made both JSON and CSV failures misleading. Report the actual request size and configured limit consistently in MiB for JSON Stream Load, CSV Stream Load, and HTTP Stream without changing any thresholds or rejection behavior. Add BE unit coverage for all three paths.

### Release note

Report Stream Load body-size limit errors consistently in MiB.

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run --filter=StreamLoadTest.* -j48 (4 tests passed)
- Behavior changed: Yes. JSON and CSV Stream Load size-limit errors now display both values in MiB.
- Does this need documentation: No
@hello-stephen

hello-stephen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei

0AyanamiRei commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28941 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 09fa8751ae439d0d4d2a2f5ffc73ca52ec6c1a0d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18122	3970	3933	3933
q2	2426	341	203	203
q3	10817	1359	812	812
q4	4964	459	338	338
q5	8591	847	563	563
q6	205	168	136	136
q7	794	796	613	613
q8	11975	1506	1555	1506
q9	7112	4326	4362	4326
q10	7563	1718	1467	1467
q11	809	345	317	317
q12	941	573	463	463
q13	19193	3396	2709	2709
q14	287	265	240	240
q15	q16	814	780	697	697
q17	1327	1016	1017	1016
q18	7553	5967	5415	5415
q19	1686	1236	1021	1021
q20	784	697	602	602
q21	5577	2528	2264	2264
q22	413	349	300	300
Total cold run time: 111953 ms
Total hot run time: 28941 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4404	4271	4285	4271
q2	299	320	211	211
q3	4579	4910	4399	4399
q4	2015	2123	1349	1349
q5	4666	4385	4567	4385
q6	247	179	127	127
q7	1827	1698	1483	1483
q8	2255	2062	1990	1990
q9	7578	7760	7572	7572
q10	4642	4579	4174	4174
q11	558	401	390	390
q12	767	750	522	522
q13	3288	3514	2944	2944
q14	283	292	260	260
q15	q16	666	697	614	614
q17	1265	1258	1245	1245
q18	7681	7017	6937	6937
q19	1112	1101	1074	1074
q20	2210	2244	2024	2024
q21	5418	4803	4666	4666
q22	535	502	411	411
Total cold run time: 56295 ms
Total hot run time: 51048 ms

@hello-stephen

hello-stephen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 176946 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 09fa8751ae439d0d4d2a2f5ffc73ca52ec6c1a0d, data reload: false

query5	4394	630	489	489
query6	510	242	200	200
query7	4974	610	331	331
query8	450	194	171	171
query9	8744	4106	4080	4080
query10	510	362	309	309
query11	5883	2312	2145	2145
query12	157	105	105	105
query13	1281	572	427	427
query14	6796	5181	4859	4859
query14_1	4234	4224	4196	4196
query15	227	204	185	185
query16	4079	485	483	483
query17	1254	745	562	562
query18	2605	490	374	374
query19	298	180	144	144
query20	112	102	101	101
query21	924	163	133	133
query22	14748	14382	13619	13619
query23	17280	16471	16012	16012
query23_1	16137	16213	16266	16213
query24	7542	1768	1275	1275
query24_1	1287	1273	1309	1273
query25	565	453	388	388
query26	1416	359	216	216
query27	2257	566	393	393
query28	4385	2017	2007	2007
query29	925	620	494	494
query30	481	258	220	220
query31	1116	1091	988	988
query32	126	60	59	59
query33	558	311	260	260
query34	1239	1128	647	647
query35	758	767	664	664
query36	1016	1020	888	888
query37	142	99	85	85
query38	1904	1693	1646	1646
query39	894	864	870	864
query39_1	847	843	836	836
query40	363	157	144	144
query41	68	65	66	65
query42	97	91	91	91
query43	325	323	279	279
query44	1398	779	763	763
query45	189	180	172	172
query46	1010	1228	765	765
query47	2141	2092	2007	2007
query48	412	395	294	294
query49	663	413	308	308
query50	1032	408	327	327
query51	10748	10714	10429	10429
query52	86	87	76	76
query53	265	271	206	206
query54	325	241	229	229
query55	74	71	67	67
query56	300	295	287	287
query57	1672	1343	1173	1173
query58	293	268	253	253
query59	1549	1640	1439	1439
query60	322	270	250	250
query61	151	146	142	142
query62	538	495	431	431
query63	243	199	193	193
query64	1761	984	839	839
query65	4736	4601	4621	4601
query66	1723	509	378	378
query67	29413	29293	29080	29080
query68	3252	1516	1044	1044
query69	653	297	265	265
query70	906	820	794	794
query71	381	327	331	327
query72	3517	2704	2438	2438
query73	842	779	452	452
query74	5078	4960	4729	4729
query75	2542	2496	2123	2123
query76	1884	1153	796	796
query77	348	364	268	268
query78	11969	11985	11338	11338
query79	2094	1156	735	735
query80	1281	555	478	478
query81	720	331	277	277
query82	645	155	120	120
query83	371	322	302	302
query84	472	163	132	132
query85	1148	643	513	513
query86	458	231	223	223
query87	1821	1819	1746	1746
query88	3748	2790	2797	2790
query89	455	362	321	321
query90	1945	197	191	191
query91	201	192	182	182
query92	57	62	55	55
query93	1877	1514	949	949
query94	770	361	307	307
query95	777	493	571	493
query96	1051	884	365	365
query97	2638	2598	2509	2509
query98	243	205	203	203
query99	1153	1102	972	972
Total cold run time: 272654 ms
Total hot run time: 176946 ms

@hello-stephen

hello-stephen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 09fa8751ae439d0d4d2a2f5ffc73ca52ec6c1a0d, data reload: false

query1	0.01	0.00	0.01
query2	0.45	0.05	0.05
query3	0.70	0.14	0.13
query4	2.34	0.13	0.13
query5	0.23	0.22	0.23
query6	1.82	1.12	1.08
query7	0.03	0.01	0.00
query8	0.06	0.04	0.04
query9	0.66	0.34	0.31
query10	0.55	0.58	0.56
query11	0.36	0.13	0.15
query12	0.24	0.14	0.14
query13	0.48	0.47	0.48
query14	1.03	1.01	1.01
query15	0.62	0.58	0.59
query16	0.32	0.33	0.30
query17	1.11	1.13	1.09
query18	0.23	0.21	0.22
query19	2.33	1.94	2.16
query20	0.02	0.01	0.01
query21	16.96	0.19	0.13
query22	4.79	0.05	0.06
query23	16.88	0.30	0.13
query24	15.26	0.45	0.37
query25	0.15	0.07	0.04
query26	0.76	0.20	0.14
query27	0.04	0.05	0.03
query28	3.51	0.93	0.58
query29	12.85	4.16	3.29
query30	0.28	0.16	0.15
query31	2.88	0.59	0.32
query32	3.43	0.59	0.49
query33	3.20	3.19	3.22
query34	15.64	4.27	3.53
query35	3.51	3.53	3.54
query36	0.61	0.44	0.43
query37	0.19	0.07	0.06
query38	0.06	0.04	0.04
query39	0.09	0.03	0.03
query40	0.21	0.16	0.15
query41	0.27	0.03	0.03
query42	0.15	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 115.35 s
Total hot run time: 25.16 s

@hello-stephen

hello-stephen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.78% (31991/42216)
Line Coverage 60.38% (356364/590175)
Region Coverage 56.91% (298654/524769)
Branch Coverage 58.31% (134441/230571)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@0AyanamiRei @hello-stephen