[feature](function) Support Spark/Hive-compatible stack table function

[HappenLee]

What problem does this PR solve?

Issue Number: close #66687

Related PR: apache/doris-website#4059

Problem Summary: Add Spark/Hive-compatible stack(num_rows, ...) support with row-major output, NULL padding, column type validation, and recursive constant folding.

Release note

Support Spark/Hive-compatible stack table-generating function semantics.

Check List (For Author)

• Test: FE StackTest 6/6; BE TableFunctionTest.vstack; targeted regression generation and rerun; FE build/checkstyle; BE format/clang-tidy
• Behavior changed: Yes (adds stack table function compatibility)
• Does this need documentation: Yes (apache/doris-website#4059)

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[HappenLee]

Review requested: implementation and the reported FE/BE/regression validation are complete. Please review the stack table-function compatibility changes.

[HappenLee]

/review

[github-actions]

Request changes: three blocking issues remain in stack's analysis and execution paths.

Review completion: converged after three full FE, BE, and risk-focused rounds. Every final-round reviewer returned NO_NEW_VALUABLE_FINDINGS, and every candidate was independently adjudicated against the live inline-thread set.

• Goal and test plan: The implementation covers the intended row-major layout, NULL padding, scalar/struct outputs, and same-column type checks, but the test plan misses the three blocking boundaries called out inline.
• Focusedness: The change is focused on adding stack; no unrelated production behavior was found.
• Concurrency: Table-function state is operator-local, and no new shared mutable state or race was found.
• Lifecycle: Normal row/null-map/cursor handling and close/reset behavior are coherent; the inline BE issue is the retained block-wide constant expansion.
• Configuration: No configuration or session-variable contract is added.
• Compatibility: FE registration, BE fake-function registration, and the table-function factory are symmetric. No storage, protocol, or persisted-state format changes were found.
• Parallel and special paths: Scalar versus struct output, typed/all-NULL values, multiple child rows, batch splits, and Cartesian products with other table functions were traced; no additional defect survived.
• Conditions and error handling: Alias cardinality and row-count legality/evaluation do not consistently produce stable analysis errors; see the two FE comments.
• Tests: Existing FE/BE/regression tests cover ordinary results and several invalid inputs, but not short/long aliases, context-dependent or unsupported deterministic constant row counts, or multi-row literal amplification.
• Expected outputs: The checked-in ordered results match the intended row-major and padding semantics for the covered cases.
• Observability: No new logging or metrics are required, but invalid SQL should surface AnalysisException rather than internal exceptions.
• Transactions and persistence: Not applicable; no transactional or persisted-state path is changed.
• Data writes: Not applicable; the function only generates query rows.
• FE/BE handoff: The function name and argument-0 constness metadata line up, and ordinary accepted inputs reach BE with coherent physical types; the remaining FE evaluator failures are inline.
• Performance: Constant value children are expanded by input-block cardinality and retained; see the BE comment.
• Other issues: No additional correctness, security-boundary, compatibility, or observability issue survived the final sweep.
• User focus: review_focus.txt contained no additional focus; the complete PR was reviewed.

Validation performed: static full-diff and upstream/downstream control-flow review only, as required by this review runner; I did not run local builds or tests. At review time, CheckStyle and Clang Formatter were passing, while the macOS BE unit-test job was still pending.

[github-actions]

[P1] Validate the dynamic stack alias count before expansion

stack can derive a different number of struct fields for each call, but the generic binder later iterates every field and indexes the supplied alias list without checking its size. For example, lateral view stack(2, 1, 2, 3, 4, 5) s as a, b derives three fields and reaches aliases.get(2), producing an internal IndexOutOfBoundsException; supplying four aliases silently drops the fourth. Please validate the alias count against the derived stack schema (while preserving the intended one-alias whole-result form if required) and return an AnalysisException; add short/long cardinality tests.

[HappenLee]

Fixed in 2351b0f. Multi-column Stack now always enters STRUCT expansion and exact alias-count validation, including the previously bypassed one-alias case. A 3-column stack with 1, 2, or 4 aliases returns AnalysisException; exactly 3 aliases still analyzes successfully. Single-column Stack and the existing one-alias behavior of other struct-returning generators are unchanged. Added a direct Nereids analyzer unit test plus the one-alias regression case.

[github-actions]

[P1] Keep constant stack arguments compact per input block

convert_to_full_column_if_const() expands each literal to block->rows() physical values and _value_columns retains all of them until the child block is exhausted. Literal-heavy stack calls therefore multiply live memory/CPU by input_rows * constant_arguments * value_size (for example, 100 constant 1 KiB strings over 4096 rows is about 400 MiB of duplicated source payload). Preserve/unpack the const column and read row 0 for constants versus _row_idx for ordinary columns; add a multi-row constant-value case so the common constant form does not amplify per block.

[HappenLee]

/review

[HappenLee]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/31705761691

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[HappenLee]

run buildall

[github-actions]

Request changes: two distinct P1 correctness gaps remain in stack analysis. A logical one-column result is confused with the multi-column STRUCT carrier when the value itself is STRUCT, and compatible ordinary literal types are rejected before the custom signature can widen them.

Review status: the full 14-file change and its parser/analyzer/translator/operator handoffs were reviewed through three cyclic rounds. Round 3 returned no new valuable findings, every candidate has been accepted, duplicate-fenced, or dismissed with concrete reachability evidence, and the review converged. No additional user focus was provided, so the full PR was reviewed without extra narrowing.

Critical checkpoint conclusions:

1. Goal and proof: the PR wires a Spark/Hive-style stack table-generating function through FE and BE and includes FE unit, BE unit, and regression coverage. The ordinary scalar, NULL-padding, const, and true multi-column cases are implemented coherently, but the two inline cases show that the compatibility goal is not yet complete.
2. Scope: the implementation is focused on function registration, dynamic schema analysis, translation, execution, and tests. No unrelated production behavior was found in the changed files.
3. Concurrency: no new threads, shared mutable state, locks, or lock ordering are introduced. Generator instances and retained columns are operator/query local.
4. Lifecycle/static initialization: FE registration and BE factory registration follow existing patterns; retained evaluated columns are scoped to table-function processing and released on close/block transitions. No cross-translation-unit static-initialization dependency or lifecycle leak was found.
5. Configuration: no configuration item or dynamic-reload behavior is added.
6. Compatibility: no storage/persistence format changes are introduced. FE lookup, legacy translation, BE fake-function lookup, and runtime factory lookup are present; an older BE fails as unsupported rather than decoding mismatched state. The remaining compatible-value rejection is called out inline.
7. Parallel paths: parser return-many handling, Nereids binding, legacy expression translation, BE fake preparation, and table-function creation were traced. No missing parallel registration path was found.
8. Conditions and errors: positive constant row-count, output-width, alias-count, and NULL-padding checks were inspected. Existing live threads already cover the residual null-context/physical-const concern; the two new, distinct condition defects are inline.
9. Test coverage: coverage is substantial across FE, BE, and regression layers, including negative cases, but it lacks width-one STRUCT values and compatible mixed-width integer/VARCHAR literals. Both missing regressions are requested inline.
10. Test results: no tests or builds were run in this bundle-only review environment. The committed test code and expected outputs were inspected; no incorrect expected row was found.
11. Observability: this is a synchronous expression-analysis/execution feature with existing analysis errors and operator status propagation; no new logs or metrics are required.
12. Transactions/persistence: not applicable; no transaction state, EditLog, replay, or persisted metadata changes.
13. Data writes/atomicity: not applicable; the change produces query rows and does not mutate stored data.
14. FE-BE variables: no new session/config variable or scattered thrift variable is introduced. Function metadata and dynamic return type use the existing expression serialization path.
15. Performance: const input materialization is compact at the current head. The suspected row-wise Cartesian repeat cost was dismissed because the mandatory Stack alias Project prevents Stack from occupying the repeated generator position in a production plan; no reachable blocking performance issue remains.
16. Other: FE/BE width and row-major indexing agree for ordinary arities, trailing padding and nullable/nested values are aligned, and no additional correctness, memory-safety, security-boundary, or style blocker was substantiated.

[hello-stephen]

BE UT Coverage Report

Increment line coverage 91.49% (86/94) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	60.36% (26508/43914)
Line Coverage	44.81% (270619/603860)
Region Coverage	40.58% (215980/532184)
Branch Coverage	42.02% (99454/236702)

[HappenLee]

/review

[github-actions]

Request changes: no new inline thread is warranted, but the existing Stack row-count folding blocker remains unresolved on the current head.

Existing blocker: #66734 (comment). The new isConstant() guard does not fully make the null-context schema fold safe. KEY db.key parses as a deterministic EncryptKeyRef over two string literals, so it passes isConstant(); FoldConstantRuleOnFE.evaluate(..., null) then dispatches to visitEncryptKeyRef(), which dereferences context.cascadesContext and raises an internal NPE instead of Stack's documented AnalysisException. CAST(KEY db.key AS INT) reaches the same visitor, so a top-level type check alone is insufficient. This is the same changed call and mechanism already covered by the unresolved thread, so I did not open a duplicate inline comment. Please make the schema-time fold context-safe (or pass the real rewrite context) and add a negative analyzer regression for this case.

Review status: complete and converged after two full FE/test, BE/cross-layer, and risk-focused rounds over all 14 changed files. Every Round 2 reviewer returned NO_NEW_VALUABLE_FINDINGS; the residual reproducer above was independently validated and deduplicated against the live thread set. No other distinct issue survived the final sweep.

Critical checkpoint conclusions:

1. Goal and proof: the PR implements row-major Spark/Hive-style stack, NULL padding, dynamic output width, and scalar/STRUCT results across FE and BE, with FE unit, BE unit, and regression coverage. Ordinary covered behavior is coherent, but the unresolved analysis-error boundary above prevents approval.
2. Scope and reuse: the production change is focused on registration, schema analysis, translation, execution, and tests, and follows existing table-function and expression-folding infrastructure. No unrelated behavior was found.
3. Concurrency: no new threads, shared mutable state, locks, or lock-ordering concerns are introduced. Table-function execution state is local to the operator instance.
4. Lifecycle and memory safety: VStack retains owning ColumnPtrs only for the active child block, releases them through process_close(), reads compact constants from physical row 0, and appends nullable scalar/complex values with consistent payload/null-map structure. No cross-translation-unit initialization dependency, ownership leak, or distinct memory-accounting issue was found.
5. Configuration: no configuration item or dynamic-reload behavior is added.
6. Compatibility and FE-BE contract: FE and BE agree on the lowercase symbol, INT constant row count, row-major width, casts, dynamic return type, and nullable fields. An older BE lacking the new symbol fails explicitly; no storage, persistence, or wire-format change is introduced. The resolved exact-type discussion is consistent with Spark's exact non-NULL type check and NullType-only coercion.
7. Parallel and special paths: mixed-case names, short/long aliases, sequential/merged generators, width-one genuine STRUCT values, width-many synthetic STRUCT carriers, all-NULL and nested complex values, const/nonconst sources, cross products, repeated output, batch splits, and final-row padding were traced. No distinct issue survived.
8. Conditions and error handling: alias count, positive/range row counts, output width, NULL padding, and BE invariants were inspected. The existing row-count thread remains the one blocking error-path defect because invalid SQL can still surface an internal NPE.
9. Tests and expected results: coverage is substantial and checked-in ordered results match the row-major/padding inputs. The missing KEY negative analyzer case belongs to the existing unresolved blocker. No local build or test was run because this review runner explicitly forbids it.
10. Observability: existing synchronous analysis errors and operator status propagation are sufficient; no new logs or metrics are required.
11. Transactions, persistence, and writes: not applicable; the change neither persists metadata nor mutates stored data.
12. FE-BE variables: no new configuration/session variable or scattered thrift field is introduced; existing function-expression metadata carries the required contract on all traced paths.
13. Performance: the current head keeps constant arguments compact per input block, and no other reachable blocking CPU, allocation, or redundant-copy issue was found.
14. Other risks: no additional correctness, lifecycle, security-boundary, compatibility, performance, style, or observability issue was substantiated.

User focus: review_focus.txt contained no additional focus, so the complete PR was reviewed without extra narrowing.

Validation performed: static full-diff and upstream/downstream control-flow review only. At submission preparation time, CheckStyle, Clang Formatter, dependency review, license, and secret checks were passing; the macOS BE unit-test job was still in progress.

[HappenLee]

/review

[github-actions]

Request changes: one P1 analysis-error boundary remains in Stack's new context-free row-count folding. A deterministic password(KEY db.key) expression reaches the new evaluator, but visitPassword() assumes its child is already a string literal and leaks an internal IllegalArgumentException instead of Stack's documented AnalysisException; see the inline comment.

Review status: complete and converged after two full FE, BE/tests, and risk-focused rounds over all 15 changed paths. Every round-2 reviewer returned NO_NEW_VALUABLE_FINDINGS; the sole candidate was independently verified, checked against the refreshed live thread set, and all other risks were dismissed with concrete evidence or duplicate-fenced.

Critical checkpoint conclusions:

1. Goal and proof: the PR implements Spark/Hive-style row-major stack, NULL padding, dynamic output width, and scalar/STRUCT results through FE and BE. The ordinary paths are coherent, but the inline internal-exception case prevents approval.
2. Scope and reuse: the changes remain focused on function registration, schema analysis, translation, execution, and tests, using existing generator and expression-folding infrastructure. No unrelated production behavior was found.
3. Concurrency: no threads, shared mutable state, locks, or lock-ordering concerns are introduced. Table-function state and retained columns are operator-local.
4. Lifecycle and memory safety: child columns are owned for the active input block and released on close; const columns remain compact, nullable/STRUCT insertion is aligned, and batch/cartesian roll-reset paths were traced without another issue.
5. Configuration: no configuration item, session variable, or dynamic-reload contract is added.
6. Compatibility and FE-BE contract: FE registration, expression serialization, BE fake-function registration, and table-function lookup agree on the symbol, constant first argument, dynamic return shape, and nullable fields. Older components reject the unknown function explicitly; no storage or persisted-state format changes are introduced. Spark's exact non-NULL per-column type rule and NullType-only coercion were also checked, so the existing widening discussion is not repeated.
7. Parallel and special paths: mixed-case names, alias cardinality, width-one genuine STRUCT values, width-many synthetic STRUCT carriers, all-NULL and nested values, const/nonconst sources, multiple input rows, Cartesian products, batch splits, and final-row padding were inspected. No distinct issue survived.
8. Conditions and error handling: positive/range row counts, foldability, output width, alias count, NULL padding, and BE invariants were reviewed. The inline nested password(KEY ...) path is the remaining condition that surfaces an internal exception.
9. Tests and expected results: FE unit, BE unit, and ordered regression coverage exercise primary positive/negative, row/column, constant, NULL, STRUCT, and alias behavior; the checked-in expected rows match those inputs. The missing nested Password/EncryptKeyRef negative regression is requested inline.
10. Observability: existing synchronous analysis errors and operator status propagation are sufficient; no new log or metric is needed, but invalid SQL must not leak an unchecked internal exception.
11. Transactions, persistence, and writes: not applicable; the change does not persist metadata, change transaction state, or mutate stored data.
12. Performance: the current implementation keeps literal inputs compact per block and bounds output production by the requested batch size; no additional blocking CPU, allocation, or retained-memory issue was substantiated.
13. Other risks: no further correctness, lifecycle, compatibility, security-boundary, performance, style, or observability blocker was found.

User focus: review_focus.txt contained no additional focus, so the entire PR was reviewed without extra narrowing.

Validation performed: static authoritative-diff and upstream/downstream control-flow review only, as required by this review runner; no local build or test was run.

corner case

[HappenLee]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 28946 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1fbb2a5f19ff8c290d203671e3432f63d6cfc8b4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17689	3993	3895	3895
q2	2358	318	201	201
q3	10233	1429	814	814
q4	4697	476	345	345
q5	7765	836	554	554
q6	184	174	139	139
q7	749	811	589	589
q8	9953	1605	1629	1605
q9	6718	4057	4077	4057
q10	7217	1652	1356	1356
q11	504	353	319	319
q12	800	573	458	458
q13	18156	3336	2747	2747
q14	280	264	239	239
q15	q16	727	730	660	660
q17	1051	1041	937	937
q18	6678	5606	5521	5521
q19	1176	1252	1096	1096
q20	812	678	581	581
q21	5822	2854	2529	2529
q22	424	367	304	304
Total cold run time: 103993 ms
Total hot run time: 28946 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4711	4689	4571	4571
q2	302	322	206	206
q3	4864	5256	4461	4461
q4	2257	2337	1430	1430
q5	4531	4465	4463	4463
q6	229	185	139	139
q7	1870	1696	1547	1547
q8	2356	2019	2000	2000
q9	7220	7169	7002	7002
q10	4226	4213	3816	3816
q11	511	377	355	355
q12	730	712	499	499
q13	3010	3299	2773	2773
q14	271	277	244	244
q15	q16	677	685	603	603
q17	1258	1207	1196	1196
q18	12094	11006	11759	11006
q19	1092	1085	1062	1062
q20	2197	2194	1938	1938
q21	5238	4512	4579	4512
q22	510	450	406	406
Total cold run time: 60154 ms
Total hot run time: 54229 ms

[hello-stephen]

TPC-DS: Total hot run time: 158439 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1fbb2a5f19ff8c290d203671e3432f63d6cfc8b4, data reload: false

query5	4326	559	439	439
query6	458	215	220	215
query7	4845	562	316	316
query8	332	170	155	155
query9	8807	4086	4069	4069
query10	451	369	286	286
query11	5864	2272	1985	1985
query12	156	96	96	96
query13	1256	607	428	428
query14	6106	4263	4006	4006
query14_1	3805	3784	3803	3784
query15	198	191	178	178
query16	1012	457	439	439
query17	932	710	540	540
query18	2444	449	337	337
query19	236	181	141	141
query20	101	98	100	98
query21	241	159	136	136
query22	13006	13034	12790	12790
query23	15776	14918	14632	14632
query23_1	14592	14676	14637	14637
query24	7571	1705	1248	1248
query24_1	1244	1231	1234	1231
query25	554	458	384	384
query26	1330	354	220	220
query27	2619	597	397	397
query28	4655	2012	2013	2012
query29	1073	632	492	492
query30	356	270	227	227
query31	1177	1115	1041	1041
query32	117	69	62	62
query33	520	325	250	250
query34	1230	1199	648	648
query35	742	769	641	641
query36	785	768	686	686
query37	162	117	99	99
query38	1840	1768	1691	1691
query39	836	817	797	797
query39_1	809	778	797	778
query40	257	173	156	156
query41	72	76	70	70
query42	95	93	97	93
query43	317	330	278	278
query44	1446	772	760	760
query45	196	180	171	171
query46	1031	1183	735	735
query47	1535	1525	1481	1481
query48	413	395	312	312
query49	599	443	307	307
query50	1157	428	340	340
query51	10706	10743	10691	10691
query52	90	90	75	75
query53	267	270	205	205
query54	310	261	232	232
query55	78	76	68	68
query56	303	311	322	311
query57	1022	1011	944	944
query58	291	267	264	264
query59	1510	1573	1352	1352
query60	347	282	266	266
query61	178	171	172	171
query62	406	326	274	274
query63	228	198	215	198
query64	3001	1038	846	846
query65	3862	3801	3820	3801
query66	1842	478	360	360
query67	20092	20048	20021	20021
query68	3279	1559	1001	1001
query69	400	299	266	266
query70	866	780	752	752
query71	367	330	317	317
query72	2954	2646	2340	2340
query73	809	763	434	434
query74	4613	4483	4290	4290
query75	2359	2314	1988	1988
query76	2345	1158	764	764
query77	334	359	274	274
query78	11103	11185	10564	10564
query79	1397	1081	756	756
query80	1245	544	468	468
query81	514	331	283	283
query82	640	177	141	141
query83	370	320	289	289
query84	330	159	131	131
query85	951	645	519	519
query86	394	233	226	226
query87	1987	1973	1837	1837
query88	3756	2816	2772	2772
query89	384	308	281	281
query90	1915	206	202	202
query91	197	187	167	167
query92	68	62	60	60
query93	1690	1522	985	985
query94	716	352	293	293
query95	806	582	495	495
query96	1099	838	367	367
query97	2514	2454	2313	2313
query98	196	189	180	180
query99	757	721	617	617
Total cold run time: 246014 ms
Total hot run time: 158439 ms

[hello-stephen]

ClickBench: Total hot run time: 23.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1fbb2a5f19ff8c290d203671e3432f63d6cfc8b4, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.05	0.04
query3	0.25	0.13	0.13
query4	1.60	0.14	0.14
query5	0.24	0.23	0.22
query6	1.16	0.88	0.82
query7	0.03	0.01	0.00
query8	0.05	0.03	0.04
query9	0.37	0.31	0.32
query10	0.56	0.55	0.58
query11	0.19	0.14	0.13
query12	0.18	0.14	0.13
query13	0.46	0.47	0.48
query14	1.04	0.99	1.00
query15	0.60	0.61	0.59
query16	0.34	0.32	0.31
query17	1.09	1.04	1.09
query18	0.20	0.20	0.20
query19	2.00	1.95	1.97
query20	0.02	0.02	0.01
query21	15.43	0.19	0.15
query22	4.98	0.05	0.05
query23	16.16	0.30	0.12
query24	2.93	0.41	0.35
query25	0.12	0.05	0.04
query26	0.73	0.19	0.14
query27	0.06	0.04	0.03
query28	3.42	0.71	0.34
query29	12.47	3.97	3.19
query30	0.27	0.16	0.15
query31	2.77	0.55	0.32
query32	3.22	0.58	0.49
query33	3.16	3.20	3.32
query34	15.55	3.98	3.26
query35	3.25	3.21	3.22
query36	0.56	0.44	0.43
query37	0.09	0.06	0.07
query38	0.05	0.03	0.03
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.07 s
Total hot run time: 23.89 s

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 98.94% (93/94) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.12% (32588/42809)
Line Coverage	61.10% (366435/599734)
Region Coverage	57.81% (308596/533831)
Branch Coverage	59.16% (140015/236686)