[SPARK-58013][PS] Fix loc setitem dropping writes for reordered columns on pandas 3#57504
Open
Vivek1106-04 wants to merge 2 commits into
Open
[SPARK-58013][PS] Fix loc setitem dropping writes for reordered columns on pandas 3#57504Vivek1106-04 wants to merge 2 commits into
Vivek1106-04 wants to merge 2 commits into
Conversation
…ns on pandas 3 The pandas 3 CoW branch in `LocIndexerLike.__setitem__` returned early (no-op) when a scalar was assigned to a list of columns not in the frame's internal order. pandas 3 actually applies such writes, so the guard silently dropped valid assignments. Remove it; both column orders now use the normal path. The change is behind `pandas >= 3.0.0`, so pandas 2 is unaffected.
Vivek1106-04
force-pushed
the
SPARK-58013-pandas-tests
branch
from
July 24, 2026 16:25
f46e9a2 to
0753c3a
Compare
uros-b
reviewed
Jul 25, 2026
Member
There was a problem hiding this comment.
After the deletion, selected_column_labels is unpacked but never used anywhere else in the method (its only other references were inside the removed block; missing_keys handling uses missing_keys/data_spark_columns, not this name). Please revert the unpack to _, data_spark_columns, _, _, _ so the now-misleading name does not imply it is live.
uros-b
reviewed
Jul 25, 2026
uros-b
left a comment
uros-b
left a comment
Member
There was a problem hiding this comment.
Thank you @Vivek1106-04 for the cleanup, I left one suggestion, otherwise looks good
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Removes an early-return in
LocIndexerLike.__setitem__(python/pyspark/pandas/indexing.py) that, under pandas 3, turned a scalar.locassignment into a no-op when the target columns were passed in a different order than the frame's internal column order.Why are the changes needed?
The branch (added in SPARK-55296) assumed pandas 3 does not apply such writes. It does, e.g.
df.loc[["viper", "sidewinder"], ["shield", "max_speed"]] = 10sets both columns. The guard silently dropped valid writes, failingtest_frame_loc_setitem(op1) on pandas 3. For a scalar assignment the column order is irrelevant, and the CoW view-decoupling the branch was meant to protect is already handled by the normal assignment path.Does this PR introduce any user-facing change?
Yes. With pandas 3,
DataFrame.loc[rows, cols] = scalarnow correctly applies whencolsis a list in non-internal order; previously it was a silent no-op. No change on pandas 2 (the removed code was behind apandas >= 3.0.0check).How was this patch tested?
Ran
pyspark.pandas.tests.indexes.test_indexing_loc(classic and Connect parity) on pandas 3.0.5. The previously failingtest_frame_loc_setitempasses, and the fullindexessuite is green with no regressions.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code