FEAT: Dataset Loading Changes

[ValbuenaVC]

Description

Features:

• Addition of filters argument to get_all_dataset_names, which rejects datasets that don't meet filter criteria. filters has type SeedDatasetFilters.
• SeedDatasetProviders have two options for storing static metadata (dynamic metadata, like derived attributes at runtime, has been scoped out of this PR):
  • If they are remote datasets, they are stored directly as named class attributes (e.g. harm_categories), and use types like SeedDatasetFoobar.
  • If they are local datasets, they are stored in the *.prompt file as tags, and extracted from it.
• In all cases, SeedDatasetMetadata acts as a unified schema and ground truth for logic related to parsing metadata, and we expect only a few class attributes to count as metadata.
• Datasets that don't have any metadata fields are excluded from filtering logic, with the sole exception that if the filter asks for tags = {"all"}, all filtering logic is bypassed.

Notes for Follow-Up PRs:

• Populating dataset metadata is a follow-up item that's out of scope here.
• Derived attributes (e.g. exact size after downloading a dataset) are also out of scope.

Tests and Documentation

• Addition of test_seed_dataset_metadata.py under unit tests.
• Addition of two integration tests under test_seed_dataset_provider_integration.py in tests.integration.datasets to account for local and remote metadata population.

[hannahwestra25]

lgtm thanks for addressing all my comments 🙂