Skip to content

Commit 48940be

Browse files
yaooqinndongjoon-hyun
authored andcommitted
[SPARK-55877][UI] Side-by-side Initial vs Final plan comparison for AQE queries
### What changes were proposed in this pull request? Add a side-by-side comparison view for AQE plan evolution on the SQL execution detail page. When AQE rewrites a query plan, the Plan Details section now shows a **Unified | Split** toggle: - **Unified** (default): full plan text as before - **Split**: two columns showing Initial Plan (left) vs Final Plan (right) For non-AQE queries (where initial == final), the raw text is shown directly without the toggle. ### Why are the changes needed? Understanding how AQE rewrites a plan is important for query performance analysis. Currently the plan text mixes both versions in a single block, making it hard to compare. The split view makes differences immediately visible (e.g., added ShuffleQueryStage, AQEShuffleRead coalescing, statistics). ### Does this PR introduce _any_ user-facing change? Yes — Plan Details section on the SQL execution page now shows a Unified/Split toggle when AQE has rewritten the plan. ### How was this patch tested? Compilation verified. Manual testing with AQE-enabled queries. ### Was this patch authored or co-authored using generative AI tooling? Yes, co-authored with GitHub Copilot. Closes #54806 from yaooqinn/SPARK-55877. Authored-by: Kent Yao <kentyao@microsoft.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 7eef6f7 commit 48940be

1 file changed

Lines changed: 75 additions & 5 deletions

File tree

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala

Lines changed: 75 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -193,17 +193,87 @@ class ExecutionPage(parent: SQLTab) extends WebUIPage("execution") with Logging
193193
"%s/jobs/job/?id=%s".format(UIUtils.prependBaseUri(request, parent.basePath), jobId)
194194

195195
private def physicalPlanDescription(physicalPlanDescription: String): Seq[Node] = {
196+
val (initialPlan, finalPlan) = extractInitialAndFinalPlans(physicalPlanDescription)
197+
val hasDiff = initialPlan.nonEmpty && finalPlan.nonEmpty
198+
199+
// scalastyle:off line.size.limit
196200
<div>
197-
<span data-action="clickPhysicalPlanDetails">
201+
<span class="collapse-table" data-bs-toggle="collapse"
202+
data-bs-target="#physical-plan-details"
203+
aria-expanded="false" aria-controls="physical-plan-details"
204+
data-collapse-name="collapse-plan-details">
198205
<h4>
199-
<span id="physical-plan-details-arrow" class="arrow-closed"></span>
206+
<span class="collapse-table-arrow arrow-closed"></span>
200207
<a>Plan Details</a>
201208
</h4>
202209
</span>
210+
<div class="collapsible-table collapse" id="physical-plan-details">
211+
{if (hasDiff) {
212+
<div>
213+
<ul class="nav nav-pills nav-pills-sm mb-2" role="tablist">
214+
<li class="nav-item" role="presentation">
215+
<button class="nav-link active btn-sm" data-bs-toggle="pill"
216+
data-bs-target="#plan-unified-tab" type="button" role="tab">Unified</button>
217+
</li>
218+
<li class="nav-item" role="presentation">
219+
<button class="nav-link btn-sm" data-bs-toggle="pill"
220+
data-bs-target="#plan-split-tab" type="button" role="tab">Split</button>
221+
</li>
222+
</ul>
223+
<div class="tab-content">
224+
<div class="tab-pane fade show active" id="plan-unified-tab" role="tabpanel">
225+
<pre>{physicalPlanDescription}</pre>
226+
</div>
227+
<div class="tab-pane fade" id="plan-split-tab" role="tabpanel">
228+
<div class="row">
229+
<div class="col-6">
230+
<h6 class="fw-bold text-muted">Initial Plan</h6>
231+
<pre class="border rounded p-2" style="font-size: 0.8rem; max-height: 600px; overflow: auto;">{initialPlan}</pre>
232+
</div>
233+
<div class="col-6">
234+
<h6 class="fw-bold text-muted">Final Plan</h6>
235+
<pre class="border rounded p-2" style="font-size: 0.8rem; max-height: 600px; overflow: auto;">{finalPlan}</pre>
236+
</div>
237+
</div>
238+
</div>
239+
</div>
240+
</div>
241+
} else {
242+
<pre>{physicalPlanDescription}</pre>
243+
}}
244+
</div>
203245
</div>
204-
<div id="physical-plan-details" style="display: none;">
205-
<pre>{physicalPlanDescription}</pre>
206-
</div>
246+
// scalastyle:on line.size.limit
247+
}
248+
249+
/**
250+
* Extract Initial Plan and Final Plan tree sections from the physicalPlanDescription.
251+
* Returns (initialPlan, finalPlan). If the plan doesn't contain AQE sections, returns
252+
* empty strings.
253+
*/
254+
private def extractInitialAndFinalPlans(
255+
description: String): (String, String) = {
256+
val lines = description.split("\n")
257+
var initialLines = Seq.empty[String]
258+
var finalLines = Seq.empty[String]
259+
var section = "" // "", "final", "initial"
260+
261+
for (line <- lines) {
262+
val trimmed = line.trim
263+
if (trimmed.contains("== Final Plan ==")) {
264+
section = "final"
265+
} else if (trimmed.contains("== Initial Plan ==")) {
266+
section = "initial"
267+
} else if (section.nonEmpty && trimmed.startsWith("(") && trimmed.contains(")") &&
268+
!trimmed.startsWith("(+") && !trimmed.startsWith("(-")) {
269+
section = ""
270+
} else if (section == "final") {
271+
finalLines :+= line
272+
} else if (section == "initial") {
273+
initialLines :+= line
274+
}
275+
}
276+
(initialLines.mkString("\n").trim, finalLines.mkString("\n").trim)
207277
}
208278

209279
private def jobsTable(

0 commit comments

Comments
 (0)