Skip to content

Add big-request guardrails and honest resource diagnostics - #9669

Open
jhonabreul wants to merge 7 commits into
QuantConnect:masterfrom
jhonabreul:feature-big-request-guardrails
Open

Add big-request guardrails and honest resource diagnostics#9669
jhonabreul wants to merge 7 commits into
QuantConnect:masterfrom
jhonabreul:feature-big-request-guardrails

Conversation

@jhonabreul

@jhonabreul jhonabreul commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds four independent warn-don't-fail diagnostics for big-request failure modes that today surface as opaque or misleading fatal errors. Runs that would have succeeded are unaffected; all thresholds are config overridable.

  • Honest OOM diagnostic in ZipDataCacheProvider.Fetch: allocation failures wrapped by Ionic were logged as Corrupt zip file/entry and swallowed. An OutOfMemoryException anywhere in the chain is now rethrown as a memory diagnostic. Truly corrupt zips keep the old behavior.
  • Large history request warning (in the QCAlgorithm.History funnel, so all overloads): warns once over an estimated data-cell threshold (default 5M), and warns once when 30 consecutive large calls have overlapping time windows (the re-fetch-a-long-lookback-every-day pattern).
  • Early slow time step warning: AlgorithmTimeLimitManager logs a warning once per time step over a threshold (default 1 minute) instead of staying silent until the isolator kill, and TimeMonitor now names the running scheduled event when it crosses the one-minute mark.
  • Large option universe warning: warns once when option universe selections exceed a contract threshold (default 500), aggregated across all option universes; suggests narrowing the filter or using OptionChain() + AddOptionContract.

Related Issue

N/A

Motivation and Context

These failure modes cost hours: a wide option universe dying as "corrupt zip" (actually OOM), billion-cell history re-fetches stalling backtests, and 10-minute isolator kills that never name the slow handler. Upfront, named warnings make them self-correcting.

Requires Documentation Change

No. The new config keys only tune warnings and have safe defaults.

How Has This Been Tested?

  • New unit tests per diagnostic, each proven red on pre-change code: OOM rethrow + corrupt-zip regression guard, history size/overlap warnings, time-step warning (once per step, re-arms on new step), named scheduled event log, option universe warning (under/over threshold, once per algorithm).
  • Touched fixtures all green: zip cache, time-limit manager, isolator, universe selection, real-time handler, plus the full AlgorithmHistoryTests fixture.
  • BasicTemplateOptionsAlgorithm (C# + Python) statistics unchanged.
  • End-to-end Launcher runs verified each warning fires with the expected text and the runs complete normally.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which improves implementation)
  • Performance (non-breaking change which improves performance. Please add associated performance test and results)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Non-functional change (xml comments/documentation/etc)

Checklist:

  • My code follows the code style of this project.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

Moves the large/overlapping history request warning state and logic out
of the QCAlgorithm partial into a private LargeHistoryRequestDiagnostics
nested class with a single entry point, keeping the algorithm class
surface small. No behavior change.
Keep the new user-facing warnings and code comments short and direct so
they are easy to act on. No behavior change.
@jhonabreul
jhonabreul marked this pull request as ready for review August 12, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant