Bug: URGENT all court report downloads are broken

[compwron]

Reported by stakeholder

Nancy King, Office Manager at Voices for Children Montgomery, reports via email:

Volunteers, Case Supervisors, and myself are getting the following error when trying to generate a report on the tracking form. I have tried Microsoft Edge and Google Chrome.

Note: the actual error text/screenshot did not come through in the email we received, so the specific failure is not yet known. We have asked for a follow-up with the exact message.

Impact

• Affects all roles at this org (volunteers, case supervisors, and org staff), not one user.
• Reproduces in both Edge and Chrome, so it is unlikely to be browser-specific.

Info still needed

• Screenshot or exact text of the error
• Which page/button — "tracking form" is stakeholder language, not a term in the app. Most likely candidates:
  • Court Reports page (/case_court_reports) → "Generate Report" (produces the .docx)
  • Reports / Export Data page (/reports) → one of the CSV exports
• The case number (or date range) being used, so we can reproduce with their data
• Approximate timestamp of a failure, to line it up with production logs

Triage pointers

If it is the court report .docx generation, the relevant path is
CaseCourtReportsController#generate (app/controllers/case_court_reports_controller.rb),
which renders JSON and surfaces e.to_s to the user for any unhandled error, plus a
"Template is not found" message on Zip::Error. Voices for Children Montgomery uses a
custom court report template (see closed #1826 / #2265 / #3698), so a template/placeholder
mismatch is worth ruling out — app/documents/templates/montgomery_report_template.docx.

If it is a CSV export, the relevant services are under app/services/*_export_csv_service.rb.

Related history (all closed, for context)

• Give Montgomery their own logo and format for court report #1826 / Use new Montgomery court report templates instead of old one #2265 / Fix Montgomery court report letterhead to include their logo #3698 — Montgomery-specific court report template and letterhead
• Generate Report is broken #1039 — "Generate Report is broken"
• [bug] exception when downloading report with options #1253 — exception when downloading report with options
• Contact_type Group filter causes an unhandled error when generating reports #1556 — contact_type group filter caused an unhandled error when generating reports

[compwron]

Root cause found: Ruby 4.0's stripped cgi stdlib + Active Storage Azure

The stakeholder error text is undefined method 'parse' for class CGI. This is not a Montgomery template problem — it is every Active Storage read/write in production.

Mechanism

Ruby 4.0 ships a cut-down cgi in the stdlib: only cgi.rb, cgi/util.rb, cgi/escape.rb. There is no cgi/core.rb, so CGI.parse does not exist.

$ ruby -v
ruby 4.0.3 (2026-04-21 revision 85ddef263a)
$ ruby --disable-gems -e 'require "cgi"; puts CGI.respond_to?(:parse)'
false

Production uses Azure for Active Storage (config.active_storage.service = :microsoft, config/environments/production.rb:49), and azure-storage-common calls CGI.parse in two places:

call site	what it signs	user-visible break
azure/storage/common/core/auth/shared_access_signature_generator.rb:382	SAS download URLs	active_storage/blobs/redirect#show → the Bugsnag events above (org logos, downloading the .docx)
azure/core/auth/shared_key.rb:117	upload request signature	casa_case.court_reports.attach(...) in CaseCourtReportsController#save_report → "Generate report" fails

That covers both halves of the report — generating it and downloading it — plus every other attachment in the app (logos, profile pictures, banners). The Bugsnag stack line CGI::parse(uri.query || "")... is an exact match for the SAS generator.

The reason the user saw raw Ruby internals instead of a friendly message: CaseCourtReportsController#generate has a blanket rescue => e that renders e.to_s straight into the flash.

When it broke — cgi in Gemfile.lock

date	commit	cgi in lock
2025-11-24	c9f2595	0.5.0 added transitively
2025-12-15	9913c6d	→ 0.5.1
2026-03-25	37e716e	Ruby 3.3.10 → 4.0.1 (still fine — the gem was in the bundle)
2026-05-27	90679bb (#6955 "Upgrade Rails from 7.2 to 8.0")	cgi removed from the lock entirely
2026-07-30	c5a7455 (#7088 "Fix org button")	re-added as gem "cgi", "~> 0.4.2"

The Rails 8 upgrade is the regression. Rails 8 / rack 3.2 no longer pull cgi in transitively, so it dropped out of the lock, and require "cgi" started falling through to Ruby 4.0's stripped stdlib copy. Worth noting the gem is fine at any version — cgi-0.5.1 still ships lib/cgi/core.rb with parse. It is only the stdlib fallback that lacks it. That timeline lines up with Jocie's 2026-07-06 email.

Current state

The fix is already on main (the gem "cgi", "~> 0.4.2" pin from #7088), and it works — verified locally that the real Azure signer runs green under the bundle:

$ bundle exec ruby -e 'require "azure/storage/blob"; require "uri"
  k = Azure::Storage::Common::Core::Auth::SharedKey.new("acct", "a2V5")
  puts CGI.method(:parse).source_location.first
  puts k.send(:canonicalized_resource, URI("https://acct.blob.core.windows.net/c/f.docx?comp=block&blockid=abc"))'
.../gems/cgi-0.4.2/lib/cgi/core.rb
/acct/c/f.docx
blockid:abc
comp:block

But production is still broken right now. The Bugsnag URL from the report still 500s:

$ curl -s -o /dev/null -w "%{http_code}" "https://casavolunteertracking.org/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MTI0OCwicHVyIjoiYmxvYl9pZCJ9fQ==--e36a834ec8498e117f9bddd855a53668825891b3/CASA%20LOGO.jpg"
500

So main has the fix but prod has not picked it up. Given #7086 "Fix deploy." and the "retrigger deploy for patch notes" commit, the deploy pipeline itself is worth checking.

Recommended next steps

1. Deploy main to production and re-check that curl. This is the whole outage for this org (and every other org on Azure storage).
2. Consider relaxing the pin to gem "cgi", "~> 0.5.1" instead of ~> 0.4.2. 0.5.1 still ships the full library including CGI.parse, is current, and avoids pinning back to an older C extension built against Ruby 4.0 (a possible deploy-time build failure). 0.4.2 does work — this is a robustness preference, not a correctness fix.
3. Add a regression guard. Nothing in CI catches this: the test env uses Disk storage, so the Azure code path is never exercised. A cheap guard is a spec asserting CGI.respond_to?(:parse) with a comment pointing here; a better one is a smoke spec around the Azure SAS signer.
4. azure-storage-blob 2.0.3 / azure-storage-common 2.0.4 are effectively unmaintained (last release 2021), and the pin comment already says "Remove when azure-storage-blob is dropped." Migrating Active Storage to S3 or GCS would remove this class of breakage.
5. Separately, CaseCourtReportsController#generate's rescue => e → generate_error(e.to_s) is why the report to us was an inscrutable Ruby message. It should report to Bugsnag and show users a generic failure message.

Answering the open questions in the issue description

• Which page/button — the Court Reports page (/case_court_reports) → "Generate Report", not the CSV exports. The CSV path does not touch Active Storage.
• Montgomery template — ruled out. The failure is org-independent; Montgomery just reported it first.
• Affects all roles — consistent: the break is below authorization, in blob signing.

[compwron]

Confirmed: prod is running July 14 code, which predates the fix

Last production deploy was 2026-07-14, i.e. b7b87cd6 "retrigger deploy for patch notes". The gem "cgi", "~> 0.4.2" fix landed 2026-07-30 in c5a7455. So prod has the Rails 8 upgrade (the regression) but not the pin — exactly matching the 500s still happening tonight.

That also explains the reporting timeline: Jocie's email was 2026-07-06, before the July 14 deploy, so the break came in with an earlier deploy of the Rails 8 change.

The catch: main is 14 commits ahead, and most of it is the design rebuild

eb742e8c 2026-08-01  Casadesign part3 (#7091)
c5a74558 2026-07-30  Fix org button (#7088)        <-- contains the cgi pin
464b78f8 2026-07-30  Part2 of design rebuild - DO NOT MERGE YET (#7087)
3f9b4693 2026-07-29  5563 - Replace helper spec stubs with real tests (#7084)
4ea31d49 2026-07-28  Fix deploy. (#7086)
ad9c1498 2026-07-28  Casadesign (#7051)
... 8 more (dependabot, reimbursement badge, spec work)
b7b87cd6 2026-07-14  retrigger deploy for patch notes   <-- currently in production

Deploying main fixes the outage but simultaneously ships three PRs of Tailwind UI rebuild, one of which is literally titled "DO NOT MERGE YET". That is a lot of surface area to introduce while resolving a P1.

Two options

A. Hotfix branch off the deployed commit (recommended). Branch from b7b87cd6, add the two lines, deploy. The entire change is:

 Gemfile
+# Ruby 4.0 ships a cgi stdlib with only escape/unescape; azure-storage-common needs the
+# full library's CGI.parse to sign blob URLs. Remove when azure-storage-blob is dropped.
+gem "cgi", "~> 0.4.2"

 Gemfile.lock
+    cgi (0.4.2)        # GEM section
+  cgi (~> 0.4.2)       # DEPENDENCIES section

Restores every attachment in the app with essentially zero regression risk, and decouples the outage from the design rollout.

B. Deploy main. Fixes it too, no cherry-picking, but the design rebuild ships at the same time and any fallout gets tangled with this incident.

Either way the verification step is the same — that blob URL should stop returning 500:

curl -s -o /dev/null -w "%{http_code}" "https://casavolunteertracking.org/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MTI0OCwicHVyIjoiYmxvYl9pZCJ9fQ==--e36a834ec8498e117f9bddd855a53668825891b3/CASA%20LOGO.jpg"

[compwron]

Opened #7095 for the two hardening follow-ups:

1. Pin cgi forward to ~> 0.5.1 (resolves 0.5.2) rather than back to ~> 0.4.2. The gem still ships the full library at 0.5.x, so this avoids compiling an older C extension against Ruby 4.0 at build time.
2. spec/lib/azure_storage_signing_spec.rb — exercises both real Azure signing call sites so a missing cgi gem fails CI instead of production. Confirmed the guard reproduces the exact reported error when CGI.parse is removed: NoMethodError: undefined method 'parse' for class CGI.

#7095 does not resolve this issue. The functional fix has been on main since #7088 (2026-07-30); production is still on 2026-07-14 code and still 500ing. This issue stays open until something ships to prod and that blob URL stops returning 500.

[compwron]

Hotfix branch ready to deploy: hotfix-7093-cgi-pin (34aa6f2).

Branched off b7b87cd6 — the commit currently in production — and carries nothing but the pin. Full diff against prod is 6 lines:

 Gemfile
+# Ruby 4.0 ships a cgi stdlib with only escape/unescape; azure-storage-common needs the
+# full library's CGI.parse to sign blob URLs. Rails 8 no longer pulls cgi in transitively,
+# so without this the stripped stdlib wins and every Active Storage read/write 500s. See #7093.
+gem "cgi", "~> 0.5.1"

 Gemfile.lock
+    cgi (0.5.2)      # GEM
+  cgi (~> 0.5.1)     # DEPENDENCIES

The lockfile was hand-edited rather than re-resolved, so nothing else moved — a plain bundle lock on this old lock also wanted to drop the ruby platform and the generic nokogiri entries, which is unrelated churn for a hotfix. Verified the result is deploy-valid under the same frozen mode Heroku uses:

$ BUNDLE_FROZEN=true bundle install
Bundle complete! 96 Gemfile dependencies, 247 gems now installed.

And both Azure signing paths work on the branch:

cgi 0.5.2 — .../gems/cgi-0.5.2/lib/cgi/core.rb
upload sig: casaaccount:W6OsZRzCVtc5...
download url signed: true

After deploying, this should stop returning 500:

curl -s -o /dev/null -w "%{http_code}" "https://casavolunteertracking.org/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MTI0OCwicHVyIjoiYmxvYl9pZCJ9fQ==--e36a834ec8498e117f9bddd855a53668825891b3/CASA%20LOGO.jpg"

#7095 still carries the same pin plus the CI regression guard for main.