HyperTools 1.0: architecture refactor + bug hunt

.github/workflows/test.yml

@@ -2,7 +2,7 @@ name: Tests
 
 on:
   push:
-    branches: [ master, dev, dev-2.0 ]
+    branches: [ master, dev, dev-1.0, dev-1.0-refactor ]
   pull_request:
     branches: [ master, dev ]
 
@@ -13,6 +13,10 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest, macos-latest]
+        # datawrangler 0.5.0+ supports pandas 3.0 (dw#30/#31) and py3.10-3.13. The pandas<3
+        # ceiling is lifted, so the resolver naturally picks pandas 3.x on py3.11-3.13 and
+        # pandas 2.x on py3.10 (pandas 3.0 requires py>=3.11) -- the matrix exercises both.
+        # A dedicated pinned-pandas-3 gate runs below on ubuntu/py3.12.
         python-version: ['3.10', '3.11', '3.12', '3.13']
 
     steps:
@@ -45,7 +49,11 @@ jobs:
       if: matrix.os == 'ubuntu-latest'
       run: |
         sudo apt-get update
-        sudo apt-get install -y ffmpeg
+        # fonts-noto-cjk (GH #205): ubuntu-latest ships no CJK-covering font
+        # at all, so tests/test_multibyte.py's `requires_covering_font`
+        # skipif would silently skip every CJK-dependent test on this OS.
+        sudo apt-get install -y ffmpeg fonts-noto-cjk
+        fc-cache -f
 
     - name: Install system dependencies (macOS)
       if: matrix.os == 'macos-latest'
@@ -59,20 +67,75 @@ jobs:
       continue-on-error: true
 
     - name: Install package with dev dependencies
+      # GH #130 (hyp.io.lsl_stream, tests/test_lsl_streaming.py): pylsl is
+      # in the [dev] extra (pyproject.toml) with NO separate native liblsl
+      # install step on any of the three platforms below -- investigated
+      # during Task 13's implementation, pylsl>=1.16's wheels BUNDLE the
+      # native liblsl library for every platform this matrix runs on:
+      #   - ubuntu-latest (x86_64): manylinux_2_35_x86_64 wheel -> ships
+      #     pylsl/lib/liblsl.so (ubuntu-latest's glibc satisfies 2.35+)
+      #   - macos-latest (arm64/x86_64): macosx_11_0_universal2 wheel ->
+      #     ships pylsl/lib/liblsl.dylib; verified working end-to-end
+      #     locally (real in-process StreamOutlet -> StreamInlet round trip)
+      #   - windows-latest (x86_64): win_amd64 wheel -> ships
+      #     pylsl/lib/lsl.dll
+      # so all three CI platforms are "provisioned" and none is excluded;
+      # tests/test_lsl_streaming.py::test_ci_has_pylsl fails the build (not
+      # skip) if pylsl is ever unimportable here, e.g. if a future pylsl
+      # release drops the bundled binary for one of these wheel tags.
       run: |
         python -m pip install --upgrade pip
-        pip install -e ".[dev]"
+        # [torch] (also pulled in transitively by [dev]) is installed
+        # explicitly too so a future change to the dev extra can't
+        # silently drop torch coverage of the six autoencoder reducers
+        # (GH #162, tests/test_autoencoders.py). torch>=2.0 ships CPU
+        # wheels for all three CI platforms/Python versions here.
+        pip install -e ".[dev,torch]"
+
+    - name: Rebuild matplotlib font cache (Ubuntu only)
+      # GH #205: matplotlib scans installed fonts once and caches the
+      # result (~/.cache/matplotlib/fontlist-*.json). On ubuntu-latest
+      # runners that cache is built (by whatever process first imports
+      # matplotlib.font_manager -- here, `pip install`'s dependency
+      # resolution/build step, or a previous cached run) BEFORE
+      # fonts-noto-cjk is installed above, so `find_covering_font`
+      # (hypertools/plot/fonts.py) would scan a stale font list missing
+      # every Noto Sans CJK file and skip every CJK-dependent test in
+      # tests/test_multibyte.py. Force one rebuild here (expensive -- do
+      # it once per job, not from inside fonts.py on every call).
+      if: matrix.os == 'ubuntu-latest'
+      run: |
+        python -c "import matplotlib.font_manager as fm; fm._load_fontmanager(try_read_cache=False)"
 
     - name: Run pytest
       env:
         MPLBACKEND: Agg
+        # Authenticate the fivethirtyeight loader's GitHub API listing calls
+        # (tests/test_load_538_kaggle.py). The default GITHUB_TOKEN raises the
+        # REST API quota from 60 to 5000 requests/hour, so the ~12 concurrent
+        # matrix jobs sharing a runner IP pool don't exhaust the shared
+        # unauthenticated limit (403 rate-limit failures). See
+        # hypertools/io/sources.py::_github_api_headers.
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       run: |
         pytest -v --tb=short
 
+    - name: Acceptance gate — pandas 3.0 (Ubuntu Python 3.12 only)
+      # Guarantees a pandas-3 run regardless of resolver choices (dw#30/#31 fix).
+      if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
+      env:
+        MPLBACKEND: Agg
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      run: |
+        pip install "pandas>=3.0"
+        python -c "import pandas as pd; assert pd.__version__.split('.')[0] == '3', pd.__version__; print('pandas', pd.__version__)"
+        pytest -q
+
     - name: Run pytest with coverage (Ubuntu Python 3.12 only)
       if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
       env:
         MPLBACKEND: Agg
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       run: |
         pytest --cov=hypertools --cov-report=xml --cov-report=term-missing
 

CLAUDE.md

@@ -4,17 +4,17 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-HyperTools is a Python library for visualizing and manipulating high-dimensional data. It provides a unified interface for dimensionality reduction, data alignment, clustering, and visualization, built on top of matplotlib, scikit-learn, and seaborn.
+HyperTools is a Python library for visualizing and manipulating high-dimensional data. It provides a unified interface for dimensionality reduction, data alignment, clustering, and visualization, built on top of matplotlib, plotly (interactive backend), scikit-learn, seaborn, and pydata-wrangler (core data-wrangling).
 
 ## Key Commands
 
 ### Testing
-- `pytest` - Run all tests from the hypertools/ directory
+- `pytest` - Run all tests (run from the repo root; `pyproject.toml` sets `testpaths = ["tests"]`)
 - `pytest tests/test_<module>.py` - Run tests for a specific module
 - `pytest tests/test_<module>.py::test_<function>` - Run a specific test function
 
 ### Development Setup
-- `pip install -e ".[dev]"` - Install in development mode with test dependencies (all deps declared in pyproject.toml)
+- `pip install -e ".[dev]"` - Install in development mode with test/dev dependencies. `[dev]` does NOT pull in every optional extra: `text` (pydata-wrangler[hf], HF text embeddings) and `predict-hf` (chronos-forecasting) are left out, and their code paths are `importorskip`-guarded in tests.
 - `pip install -r docs/doc_requirements.txt` - Install documentation dependencies
 
 ### Documentation
@@ -26,36 +26,61 @@ HyperTools is a Python library for visualizing and manipulating high-dimensional
 ### Core Components
 
 **DataGeometry Class** (`hypertools/datageometry.py`)
-- Central data container that holds raw data, transformed data, and transformation parameters
-- Stores matplotlib figure/axes handles and animation objects
-- Contains normalization, reduction, and alignment model parameters
+- INTERNAL, unpickle-only legacy shell (its own docstring: "not part of the public API"); not used by any current API function
+- Kept solely so `hypertools.load()` can unpickle hosted example-dataset geo files (created by hypertools < 1.0) and extract their raw data via `get_data()`
+- In 1.0, `plot()` returns a matplotlib `Figure` and `load()` returns raw data -- no `DataGeometry` is ever constructed or returned to users
 
 **Main API Functions** (`hypertools/__init__.py`)
 - `plot()` - Primary visualization function
-- `analyze()` - Data analysis and dimensionality reduction
+- `analyze()` - Classic manip -> normalize -> reduce -> align -> cluster pipeline dispatcher
 - `reduce()` - Dimensionality reduction utilities
 - `align()` - Data alignment across datasets
 - `normalize()` - Data normalization
 - `describe()` - Data description and summary
 - `cluster()` - Clustering functionality
+- `manip()` - Manipulator dispatcher (Normalize/ZScore/Smooth/Resample, etc.)
+- `predict()` - Timeseries forecasting (ARIMA, Kalman, GP, autoregression, Chronos, etc.)
+- `impute()` - Missing-data imputation (PPCA, Kalman, sklearn imputers)
 - `load()` - Data loading utilities
+- `save()` - Save data/results to disk
+- `apply_model()` - Apply an arbitrary sklearn-API model to data
+- `Pipeline` - Class for chaining/reusing manip/reduce/align/cluster stages
+- `set_interactive_backend()` - Switch the plotting backend (matplotlib/plotly)
+- `io` submodule - I/O helpers, including `io.lsl_stream()` for Lab Streaming Layer input
 
 **Tools Module** (`hypertools/tools/`)
-- `align.py` - Hyperalignment and Procrustes alignment
-- `reduce.py` - Dimensionality reduction (PCA, t-SNE, UMAP, etc.)
-- `normalize.py` - Data normalization methods
-- `cluster.py` - K-means and other clustering algorithms
+- `align.py` - Classic array/mode `align()` API; thin compatibility wrapper over `hypertools/align/`
+- `analyze.py` - `analyze()` pipeline dispatcher
+- `normalize.py` - `normalize()` (z-scoring) with cross-module stage kwargs
 - `format_data.py` - Data preprocessing and formatting
 - `text2mat.py` - Text-to-matrix conversion
 - `df2mat.py` - DataFrame-to-matrix conversion
-- `load.py` - Data loading from various sources
+- `gensim_models.py` - sklearn-API wrappers around gensim topic/embedding models
 - `missing_inds.py` - Missing data handling
-- `procrustes.py` - Procrustes analysis
+
+The dev-1.0 refactor moved several tools into their own top-level subpackages (each a folder with a main module plus helpers):
+- `hypertools/reduce/` - Dimensionality reduction (`reduce.py`, `describe.py`, `autoencoders.py`, `common.py`)
+- `hypertools/cluster/` - Clustering (`cluster.py`, `common.py`)
+- `hypertools/align/` - Alignment (`align.py`, `hyperalign.py`, `procrustes.py`, `srm.py`, `null.py`, `common.py`)
+- `hypertools/manip/` - Manipulators (`manip.py`, `normalize.py`, `zscore.py`, `smooth.py`, `resample.py`, `common.py`)
+- `hypertools/io/` - Loading/saving/streaming (`load.py`, `save.py`, `sources.py`, `streaming.py`, `lsl.py`)
+- `hypertools/predict/` - Forecasting models (`predict.py`, `arima.py`, `autoreg.py`, `gp.py`, `kalman.py`, `laplace.py`, `chronos.py`, `common.py`)
+- `hypertools/impute/` - Imputation models (`impute.py`, `ppca.py`, `kalman.py`, `sklearn_imputers.py`, `common.py`)
+- `hypertools/core/` - Shared config/exceptions and `apply_model()`/`Pipeline` (`configurator.py`, `exceptions.py`, `model.py`, `pipeline.py`, `shared.py`)
 
 **Plot Module** (`hypertools/plot/`)
 - `plot.py` - Main plotting interface and logic
-- `backend.py` - matplotlib backend configuration
-- `draw.py` - Low-level drawing functions
+- `backend.py` - Backend selection (`set_interactive_backend()`)
+- `matplotlib_backend.py` - Low-level matplotlib drawing (`draw.py` is now a 3-line compatibility shim over this)
+- `plotly_backend.py` - Low-level plotly drawing (interactive backend)
+- `interactive.py` - Interactive-mode helpers
+- `animate.py` - Animation support
+- `colors.py` - Color handling
+- `density.py` / `surface.py` - Density and iso-surface rendering
+- `morph.py` / `trails.py` - Morph and trail effects
+- `meshutil.py` - Mesh utilities
+- `multiindex.py` - Pandas MultiIndex handling
+- `fonts.py` - Font handling
 
 **External Dependencies** (`hypertools/_externals/`)
 - `ppca.py` - Probabilistic Principal Component Analysis
@@ -82,10 +107,10 @@ HyperTools is a Python library for visualizing and manipulating high-dimensional
 
 - The package follows a functional programming style with separate modules for each operation
 - All major functions are designed to work with multiple input formats
-- The DataGeometry class serves as the central data container and state manager
+- `plot()` returns a matplotlib `Figure`; `load()` returns raw data -- there is no central state-container object in 1.0 (`DataGeometry` is an internal unpickle-only legacy shell; see above)
 - Tests are located in `tests/` directory and follow pytest conventions
 - Documentation is built with Sphinx and uses example galleries
-- The codebase maintains compatibility with Python 3.9+
+- The codebase requires Python 3.10+ (`requires-python = ">=3.10"`)
 
 ## Testing Strategy