fix: drop a deep BinaryExpr chain iteratively to avoid a stack overflow#23198
fix: drop a deep BinaryExpr chain iteratively to avoid a stack overflow#23198mdashti wants to merge 3 commits into
BinaryExpr chain iteratively to avoid a stack overflow#23198Conversation
100b14c to
f7ed788
Compare
test_stack_overflow against a deep-recursion stack overflowtest_stack_overflow cases on a large explicit stack
f7ed788 to
a46cc36
Compare
test_stack_overflow cases on a large explicit stacka46cc36 to
ff9369c
Compare
test_stack_overflow cases on a large stack
adriangb
left a comment
There was a problem hiding this comment.
i'm curious if this is a test-only concern or also a real production issue. should we implement an iterative Drop?
| std::thread::Builder::new() | ||
| .stack_size(64 * 1024 * 1024) |
There was a problem hiding this comment.
doesn't this defeat the purpose of the test? can we keep the smaller stack size for the planning (which is what we are trying to ensure is iterative) but move the value into a large stack thread to run drop()?
There was a problem hiding this comment.
Thanks. You're right. It's a real production issue, so I went with the iterative Drop. The BinaryExpr children hold a BoxedExpr whose Drop tears a deep chain down off the call stack (39e458c5c), so the original test_stack_overflow cases pass with no stack trick. impl Drop for Expr wasn't viable (E0509 from the by-value destructuring in map_children), so the newtype wraps just the recursive edges. Repurposed the PR for it.
5a710be to
39e458c
Compare
test_stack_overflow cases on a large stackBinaryExpr chain iteratively to avoid a stack overflow
| std::thread::Builder::new() | ||
| .stack_size(64 * 1024 * 1024) |
There was a problem hiding this comment.
Thanks. You're right. It's a real production issue, so I went with the iterative Drop. The BinaryExpr children hold a BoxedExpr whose Drop tears a deep chain down off the call stack (39e458c5c), so the original test_stack_overflow cases pass with no stack trick. impl Drop for Expr wasn't viable (E0509 from the by-value destructuring in map_children), so the newtype wraps just the recursive edges. Repurposed the PR for it.
A long `OR`/`AND` chain builds an `Expr` as deep as the chain, so its recursive `Drop` overflows the stack. The `BinaryExpr` children hold a smart pointer whose `Drop` tears the chain down iteratively.
39e458c to
b347ae3
Compare
The derived `Debug` printed the wrapper and changed `BinaryExpr`'s debug output, breaking the `Expr` doctest. Forward to the inner `Expr` so the representation is unchanged.
The unparser raises `recursive`'s process-global minimum stack size for a deep unparse. In the lib test binary it shared that global with hundreds of concurrent `expr_to_sql` tests whose `StackGuard`s reset it on drop, so a reset on another thread could drop the red zone mid-unparse and overflow on amd64. Its own test binary keeps anything else from touching the red zone while it runs.
003a983 to
e16fd43
Compare
|
The deep-unparse regression test now runs in its own test binary, so the suite is stable. The underlying cross-thread race on |
adriangb
left a comment
There was a problem hiding this comment.
This seems right to me, I could not figure out any way to make this simpler or less churn. @xudong963 since you work on the recursion guards originally I wonder if you can spot any alternative to making this change?
Which issue does this close?
None filed.
Rationale for this change
This PR makes a deep
BinaryExprchain drop iteratively instead of recursively.A left-associative chain like
a OR b OR ...parses with shallow recursion (the left side accumulates in a loop), sorecursion_limitnever bounds its length.sql_expr_to_logical_exprbuilds it iteratively (the stack machine from #1444), so planning is fine. But the resultingExpris as deep as the chain, and its recursiveDropthrough the boxed children overflows the stack. The tree-walk passes are#[recursive]-protected, so theDropis the one recursion the protection can't reach.impl Drop for Exprisn't viable: it forbids moving fields out ofExpr, whichmap_childrenand the rewriters do by value (E0509).What changes are included in this PR?
Wrap the recursive
BinaryExpr.left/rightedges in aBoxedExprnewtype whoseDropdetaches each node's children into a heap work-list before the node drops.ExprandBinaryExprkeep noDrop, so the by-value destructuring elsewhere is untouched, andBinaryExpr::newkeeps itsBox<Expr>signature. The unbounded-depth edges are only these chains; other nesting is capped byrecursion_limit.The rest of the diff is mechanical: a few consumers move
BinaryExproperands by value, so each*leftbecomes*left.into_inner().Are these changes tested?
Yes.
deep_binary_expr_chain_drops_without_overflowingbuilds a 200000-deepORchain and drops it on a normal stack. The existingtest_stack_overflowcases cover the SQL planning path, and this fix is what lets them pass without a large-stack workaround.Are there any user-facing changes?
No.