Feature request: make it easier for users to evaluate if sail already supports required PySpark functions

[keen85]

I'd love to use sail for faster local development of PySpark code - and on the long run maybe even execute production workloads on sail.
However, currently sail does not have 100% feature parity with PySpark.

So, I need to check if at the PySpark functions that we use are already supported by sail - and I need to repeat this after every sail release.

What I do is to search our code repo for import pyspark.sql.function occurrences and identify the function names that we use.
Then I need to cross reference this with the sail documentation: https://docs.lakesail.com/sail/main/guide/functions/scalar.html

I think it would be very helpful if sail came with a "compatibility checker" that automates this and highlights gaps.

[shehabgamin]

@keen85 Agreed, this would be super useful. I'll brainstorm a couple of ways to create a script for this.

Lemme know if you have any ideas as well!

[keen85]

@shehabgamin I already have a Python script for analyzing which PySpark functions are used in *.py and *.ipynb files for a given root directory and counts how many times these occur.

I could create a PR to show what I got so far.

But I'm unsure how to handle the "look up if supported" part. I could use the doc *.md files like this one.
Are these generated or maintained manually?

[shehabgamin]

@shehabgamin I already have a Python script for analyzing which PySpark functions are used in *.py and *.ipynb files for a given root directory and counts how many times these occur.

I could create a PR to show what I got so far.

But I'm unsure how to handle the "look up if supported" part. I could use the doc *.md files like this one.
Are these generated or maintained manually?

@keen85 Yeah, that would be a great first start!

The docs are manually maintained for now, we haven't come up with a good story around how to generate them yet.

@linhr may have some additional thoughts here, I know he's been thinking about docs generation.

[linhr]

@linhr may have some additional thoughts here, I know he's been thinking about docs generation.

We'll later consolidate the function docs with test cases so passing tests will indicate supported functions. I'm still figuring out the details of the idea.

[keen85]

@shehabgamin, I created a PR containing my rudimentary scanner for PySpark function calls: #1075

[keen85]

@shehabgamin & @linhr, I used the new functionality to analyze my orgs current (JVM) PySpark project, and found several functions that are already supported by Sail but that are missing in Sail's documentation currently.

I offer to put some more effort into updating the JSONs containing the support status. I already started doing so #1127, but I realized that it is quite time-consuming so I wanted to check back with you guys if this is considered useful.

[linhr]

This would be super helpful! Thanks for following up on this!