Skip to content

Commit 5aeb251

Browse files
committed
[SPARK-58615][ML] Avoid FPGrowthModel temporary column conflicts
### What changes were proposed in this pull request? This follow-up to #57806 gives the transient association-rules column in `FPGrowthModel.transform` a generated name instead of the fixed name `rules`. It also adds regression coverage for an input dataset that already contains a `rules` column. ### Why are the changes needed? The fixed temporary column can conflict with an input column of the same name after the join, causing ambiguous-column analysis failures or dropping the user's column. ### Does this PR introduce _any_ user-facing change? Yes. `FPGrowthModel.transform` now supports and preserves an input column named `rules`. ### How was this patch tested? - `build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 mllib/Test/compile` - Added `FPGrowthSuite` coverage for an input `rules` column. The focused suite has not been run locally yet. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) Closes #57816 from zhengruifeng/ml_fpgrowth_temp_rules_column. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
1 parent eb318ba commit 5aeb251

1 file changed

Lines changed: 4 additions & 3 deletions

File tree

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,7 @@ class FPGrowthModel private[ml] (
280280
override def transform(dataset: Dataset[_]): DataFrame = {
281281
transformSchema(dataset.schema, logging = true)
282282
val dt = dataset.schema($(itemsCol)).dataType
283+
val rulesCol = Identifiable.randomUID("rules")
283284
// For each rule, examine the input items and summarize the consequents.
284285
val predictFunc = (items: Seq[Any], rules: Seq[Row]) => {
285286
if (items != null) {
@@ -293,9 +294,9 @@ class FPGrowthModel private[ml] (
293294
val predictUDF = SparkUserDefinedFunction(predictFunc, dt, Nil)
294295
dataset.join(
295296
associationRules.select("antecedent", "consequent")
296-
.agg(collect_set(struct("antecedent", "consequent")).as("rules")))
297-
.withColumn($(predictionCol), predictUDF(col($(itemsCol)), col("rules")))
298-
.drop("rules")
297+
.agg(collect_set(struct("antecedent", "consequent")).as(rulesCol)))
298+
.withColumn($(predictionCol), predictUDF(col($(itemsCol)), col(rulesCol)))
299+
.drop(rulesCol)
299300
}
300301

301302
@Since("2.2.0")

0 commit comments

Comments
 (0)