Make scancode.api.get_licenses more widely useful by reducing dependencies

[gotmax23]

Short Description

In https://fedora.gitlab.io/sigs/go/go-vendor-tools/, we use the scancode.api.get_licenses() Python API to detect license texts contained within license files and then produce an SPDX expression (relevant code). The scancode license detector is very powerful due to its large corpus of license texts, but it has a huge dependency footprint which has prevented us from making it the default license scanner in go-vendor-tools (currently, askalono is the default). Many of these dependencies are to support package scanning features that we don't use. I wonder what it would look like to extract the license scanning–related APIs into a smaller subpackage (or maybe to make more dependencies optional Python extras) without disrupting scancode-toolkit as a whole. Do you think that'd be feasible?

Possible Labels

• new feature
• improve-license-detection
• installation and packaging
• core and api

Select Category

• Enhancement
• Add License/Copyright
• Scan Feature
• Packaging
• Documentation
• Expand Support
• Other

Describe the Update

A mechanism to use the scancode/licensedcode file license detection API (scancode.api.get_licenses()) without pulling in so many dependencies.

How This Feature will help you/your organization

See earlier comment about Go Vendor Tools' license scanning functionality.

Possible Solution/Implementation Details

See above. Reudction of dependencies in the main scancode-toolkit package (e.g., by using extras) or extracting the license scanning API are possible solutions.

Example/Links if Any

Can you help with this Feature

[AyanSinhaMahapatra]

@gotmax23 thank you for your input, this is certainly possible.

See https://github.com/aboutcode-org/scancode-toolkit/blob/develop/pyproject-licensedcode-index.toml where we are using github.com/aboutcode-org/flot to only release the scancode license index as a seperate wheel, see #5055. This will soon also be the case for the commoncode module, with #5055

The plan was also to release packagedcode and licensedcode modules seperately, but there are some work that needs to be done:

• decoupling code and import to be able to release these modules
• not releasing the plugins specific code, as this is relative to a lot of other parts in scancode (scan post-processing)
• adding a pyproject.toml with flot specific metadata to enable releaseing a wheel
• add tests to make sure the module works seperately
• the api functions need to be moved back to the modules possibly
• update and add release scripts and add release tests

We would still have some dependencies for the licensedcode/packagedcode modules but only what is required and not all of them as before.

[gotmax23]

Thanks! I'm happy to see there's work ongoing here :).

For what it's worth, I left some feedback about the adoption of flot in #4847 (review) but am otherwise on board with the minimization efforts.