[WIP] Gaia charts#362
Open
mrosseel wants to merge 378 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace SD image build/mount/extract cycle with direct closure assembly. This eliminates ext4 filesystem overhead, reducing tarball size to fit in Pi 4's 924 MB tmpfs. Also factors out configTxt and adds migration-boot-firmware package to deduplicate boot partition assembly across profiles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nix store paths are read-only and root-owned; rm -rf without sudo fails on the copied closure paths. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge upstream changes including: - IMU integrator selection (classic/quaternion) - Eq-mount support under IMU dead-reckoning - Chinese locale and font support - Harris Globular Cluster catalog - T9 input support - PAM-based web auth (replacing su) - Stellarium+ Mobile support - Various bug fixes NixOS adjustments: - Add quaternion (numpy-quaternion) to python-packages.nix - Add quaternion to mypy ignore list in pyproject.toml - Add get_initialized_solved_dict() lost during merge resolution - Fix formatting in integrator.py - Keep requirements.txt and version.txt deleted (managed by nix) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SHA256 is now fetched at runtime from the .sha256 sidecar file in the GitHub release, eliminating the need for CI to update the migration branch after each build. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: network-online.target was unreliable because NetworkManager-wait-online was disabled, so pifinder-first-boot ran before internet was available. - Add curl-based connectivity check with 5-minute retry loop - Add Restart=on-failure with 15s delay - Re-enable NetworkManager-wait-online (with 30s timeout) - Add sudo permissions for systemctl/journalctl (remote recovery) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Starts boot-splash in animation mode while downloading the full system closure, so the user sees activity instead of a static screen. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
solution() can return None when solve_state() is truthy, causing TypeError in base.py screen_update. Also fetch first-boot target from GitHub pifinder-build.json with baked-in fallback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merges upstream PiFinder changes including: - Bottle → Flask/Jinja webserver migration (.tpl → .html) - Harris and Lynga catalog loaders + data + DB update - Selenium-based webserver test suite NixOS-specific decisions: - Keep deletion of requirements.txt, requirements_dev.txt, noxfile.py, version.txt (managed via flake.nix / pifinder-build.json) - Keep Python 3.13 target (was reset to py39 by upstream) - Keep stub sys_utils_fake (uses sys_utils_base) - Replace bottle/cheroot with flask/flask-babel/waitress in nixos/pkgs/python-packages.nix; add selenium to devPackages - Reimplement Bottle-era network status message in Jinja syntax
Includes nixos/pkgs/pifinder-src.nix fix: drop fetchurl for hip_main.dat, which upstream now ships as a committed file in astro_data/.
- nixos/RELEASE.md: document version flow + release/dev pipelines - software.py: MIN_NIXOS_VERSION 2.5.0 → 3.0.0 - python-packages.nix: add pyerfa (used by calc_utils since upstream brickbots#423, silently dropped during upstream merge because requirements.txt is not mirrored into the Nix env) - python-packages.nix: include hardwarePackages in devEnv so nix develop matches the runtime import surface - python-packages.nix: select simplejpeg wheel by host arch (was hard-pinned to aarch64; failed to import on x86_64 dev shells) - flake.nix: apply libcamera -Dpycamera=enabled overlay to the x86_64 devShell and export PYTHONPATH so picamera2 finds the python bindings Verified: nix develop --command python -c 'import …' on x86_64 succeeds for all 34 imports (erfa, picamera2, libcamera, PyHotKey, pynput, hardware packages, etc.). RPi.GPIO still raises its own "only on a Raspberry Pi" RuntimeError at import time — expected, matches upstream pip behavior on non-Pi hardware. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. gpsd-add-uart: rename /dev/ttyAMA3 → /dev/ttyAMA1 (6 sites). The uart3 overlay surfaces as ttyAMA1, matching hardware.nix's udev rule and the Debian image's gpsd.conf. 2. /etc/default/gpsd: drop the custom USBAUTO+GPSD_SOCKET pair, write upstream pi_config_files/gpsd.conf's three lines verbatim. DEVICES now opens the on-board UART at startup. gpsd-add-uart kept as the boot-time socket-activation kick; can retire after on-Pi confirmation. 3. pifinder-upgrade: replace fragile `nix build --dry-run | grep` progress with `nix --log-format internal-json build … --max-jobs 0` parsed by gawk, counting type=100 (actCopyPath) start/stop events. Stable across Nix ≥ 2.4. Validated against a real cache.nixos.org substitute (5/5). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build.yml was defaulting VERSION=2.5.0 on push triggers (and the workflow_dispatch default also read 2.5.0), so this branch's auto-build was publishing v2.5.0-migration tarballs while the migration branch's downloader (software.py _MIGRATION_VERSION_INFO and the brickbots/PiFinder release branch's migration_gate.json) points at v3.0.0-migration. Bump both the workflow_dispatch default and the push-trigger fallback to 3.0.0 so a normal push to nixos publishes the artifact at the URL the migration branch actually downloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Lint & Test workflow was using DeterminateSystems/magic-nix-cache-action,
which is backed by GitHub's Actions Cache and gets HTTP-418 rate-limited
under sustained traffic — exactly the failure mode that just broke
type-check ("--install-types failed: substituter disabled, rate limit
exceeded"). The Nix substituter is then disabled mid-run and dependent
commands like mypy --install-types fall over.
Replace it with cachix/cachix-action@v17 pointed at the pifinder cache
(read-only, no auth token needed). Same backing as build.yml, so dev-shell
substitutes hit the same store paths the system closure was built against.
cache.nixos.org remains the default fallback.
Also bump actions/checkout@v4 → @v6 in this file to align with the Node 24
migration in build.yml/release.yml.
This is a stop-gap. The real fix is standing up Attic with an S3 backend
so both build.yml and lint.yml can retire cachix.org and MNC together —
tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rong) Previous commit on this branch swapped DeterminateSystems/magic-nix-cache-action for cachix/cachix-action@v17 thinking the MNC HTTP-418 rate-limit was the root cause of the failed lint/type-check. That swap made things worse: the pifinder Cachix only contains the NixOS *system closure*, not the *dev shell* (cedar-detect-server's Rust crate builds). With MNC removed, the dev shell had to rebuild from source, which fetched crate tarballs from a crates.io mirror and hit 403s. MNC was carrying real weight by caching locally-built derivations between runs. Restoring it. The original MNC rate-limit was a transient flake — re-runs work around it. Real fix is standing up Attic with S3-backed storage so both build.yml and lint.yml can retire MNC and cachix.org together. The checkout@v4 → @v6 bump from the swap commit is preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-frame PedestalEstimator supplied a pedestal_override that shadowed the profile/calibration bias_offset. Now that the black level is a measured per-model constant (238 ADU, shipped in the profile and refinable per-unit via the CAL wizard), the estimator only adds risk: under live steady auto-exposure it never gets the >=2x exposure span it needs, and the one time it does condition (a transient ramp) it can fit a low intercept and drive SQM too bright. Its slope is sky-dominated so it can't recover dark current either. Remove the override from the solver; SQM now always uses the profile/calibration bias_offset.
The raw-green photometry path skipped every cycle on the imx462/imx290: their cropped green frame (~490px) tripped a hardcoded 512px floor, so SQM never updated despite good plate-solves. Lower the floor so only missing or malformed frames are rejected; photometry runs at the green frame's own scale. The menubar also rendered the 20.15 default as if it were a reading. Only show a value once one has actually been measured (last_update set). (ported from the nixos branch) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
A monotonic exposure ramp lets temporal sky drift (dawn twilight, passing haze) alias into an apparent exposure dependence when a sweep is analysed — both 2026-07-15 sweeps show it (fake slopes to -0.44 mag/dex from a ~4% sky-rate creep). Random order decorrelates time from exposure; file names carry the exposure so analysis is unaffected. Also record the tested-and-rejected faint-star mzero cut in ADR 0020. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
The solver dispatches align-queue commands by type (isinstance(command, ReloadSqmCalibration)), and align.py sends typed objects. The SQM calibration wizard instead enqueued a raw list ['reload_sqm_calibration'], which matched nothing and fell through to 'Unknown solver command' -- so the wizard's reload never fired and new calibration only took effect on a full restart. Send the typed command.
A monotonic exposure ramp lets temporal sky drift (dawn twilight, passing haze) alias into an apparent exposure dependence when a sweep is analysed — both 2026-07-15 sweeps show it (fake slopes to -0.44 mag/dex from a ~4% sky-rate creep). Random order decorrelates time from exposure; file names carry the exposure so analysis is unaffected. Also: record the tested-and-rejected faint-star mzero cut in ADR 0020, and drop a stray patch backup file (solver.py.orig) from the tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
Sweeps carried no record of the constants they were reduced with. Save the calibration-aware bias_offset, band offset and colour coefficient (and pass camera_type from the sweep capture path, which never did). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
Sweeps carried no record of the constants they were reduced with. Save the calibration-aware bias_offset, band offset and colour coefficient (and pass camera_type from the sweep capture path, which never did). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
…nstant The offset is (sensor passband, fixed) x (sky spectrum, environmental). Calibrated under LP-dominated suburban sky where it measures constant to ~0.05 mag; expect a different, wandering value under airglow-dominated dark skies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
…nstant The offset is (sensor passband, fixed) x (sky spectrum, environmental). Calibrated under LP-dominated suburban sky where it measures constant to ~0.05 mag; expect a different, wandering value under airglow-dominated dark skies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
The instrument is stable within a night; its only weak number is the absolute scale, which depends on the night sky spectrum through the per-sensor band offset (calibrated for one sky regime). The anchor measures exactly that unknown: aim the reference meter at the camera field, enter its reading on the new ANCHOR marking-menu entry of the SQM screen, and the session is shifted by delta = reference - current. - UISQMAnchor: meter-value entry (sweep-wizard input pattern), shows the live PiFinder value, 0 clears the anchor; delta lives in shared state only (resets on restart - tonight's spectrum is tonight's) - solver applies the delta to sqm_final and sqm_altitude_corrected and reports it in details; SQM screen shows "anch +/-x.xx" when active - every anchor event is appended to ~/PiFinder_data/sqm_anchors.jsonl (reference, measured, deltas, camera, location) - accumulating the per-site band-offset calibration dataset as a side effect of use - marking menu reworked: ANCHOR (everyday action), CAL (noise constants), SWEEP (diagnostic; label was CORRECT which now belongs to the anchor) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
The instrument is stable within a night; its only weak number is the absolute scale, which depends on the night sky spectrum through the per-sensor band offset (calibrated for one sky regime). The anchor measures exactly that unknown: aim the reference meter at the camera field, enter its reading on the new ANCHOR marking-menu entry of the SQM screen, and the session is shifted by delta = reference - current. - UISQMAnchor: meter-value entry (sweep-wizard input pattern), shows the live PiFinder value, 0 clears the anchor; delta lives in shared state only (resets on restart) - solver applies the delta to sqm_final and sqm_altitude_corrected and reports it in details; SQM screen shows "anch +/-x.xx" when active - anchor events append to ~/PiFinder_data/sqm_anchors.jsonl, building the per-site band-offset calibration dataset - marking menu reworked: ANCHOR (everyday), CAL, SWEEP (was CORRECT) - drop a stray patch reject file (camera_interface.py.rej) from the tree Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
The archived imx296 sweep (100 frames, ref 17.8-17.9) replaces three placeholders with measurements: - bias_offset 32 -> 60: the on-sky exposure-ramp intercept is 60.3, matching the Sony-standard black level (240 @ 12-bit = 60 @ 10-bit, same convention as the imx462 at 238). The old 32 was a mis-measurement and biased the background subtraction. - color_coefficient 0 -> +0.21 (460 stars, mzero vs B-V). Small because the Pregius mono QE falls through the NIR, unlike the STARVIS colour sensors. - sqm_band_offset 0 -> -0.46, PROVISIONAL: one moonlit sweep, one hand reference; the mono band reads darker than the meter. The session anchor absorbs the uncertainty until more nights exist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
bias_offset 32 -> 60 (Sony-standard black level, confirmed by on-sky ramp intercept 60.3); color_coefficient 0 -> +0.21 (460 stars); sqm_band_offset 0 -> -0.46 PROVISIONAL (one moonlit sweep, one hand reference; the anchor absorbs the uncertainty). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
Brings the SQM redesign branch (PR #28) into nixos as a true merge so the histories converge instead of running on parallel ported commits: raw-green photometry, measured black levels, colour terms, wing correction, robust mzero, band offsets (imx462/hq/imx296), session anchor, sweep randomization and metadata. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
SQM photometry is raw-only; the 8-bit processed image is for solving and display. The PNG path was unreachable in production and measured strictly worse (0.6-2.4 mag within-sweep swings, errors to 1.7 mag, frames failing photometry outright) at identical CPU cost. - solver: create_sqm_calculator always builds the raw calculator; update_sqm loses image_processed/use_raw_green and the processed fallback branch - camera_profiles: drop the four _processed profiles and the sqm_use_raw_green flag - CAL wizard: captures, analyzes, verifies and saves the RAW calibration only; verification runs the raw pipeline on the raw sky frames (green extraction + centroid rescale + 70% saturation cut) - SQM screen CAL indicator keys off the raw calibration file - sweep metadata records the raw camera type - SNR auto-exposure uses the controller defaults (its from_camera_profile existed only to consume _processed profiles; 8-bit defaults 15/30/100 are equivalent) - noise-floor tests run on a synthetic registered profile instead of imx296_processed Sweep regression after the deletion is bit-identical (18.275/std 0.070 on the 2026-07-11 reference sweep). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
… deletion) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo
mrosseel
force-pushed
the
deepchart
branch
3 times, most recently
from
fc5b854 to
6337356
Compare
stdenv's minimal bash lacked programmable completion, breaking interactive bash launched inside the direnv devshell (shopt progcomp / complete errors, raw PS1 escapes).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a gaia achart view to all objects, down to mag 17 but with an auto limiting magnitude taking into account SQM and equipment. This auto limit is overridable.
TODO: