Run a reduced results analysis periodically during backtests

[jhonabreul]

Description

Runs a reduced suite of the backtest results analyses periodically while the backtest is still running, so error conditions (order response errors, margin calls, stale fills, non-positive equity, high margin usage) surface to the user early instead of only after the backtest completes.

The analyzer:

• A single ResultsAnalyzer serves both modes through named factories: CreateForFinalAnalysis runs the full set once against the completed result; CreateForInRunAnalysis builds the long-lived in-run instance.
• The analyses are stateless and created once per instance. The equity/benchmark curves, which need a benchmark history request, are only built for the final analysis.
• Each analysis declares whether it can run in-run (BaseResultsAnalysis.RunsInRun, default true) and whether it reads current state instead of the order event and log streams (IsStateBased). The in-run set is the final set filtered by RunsInRun, so it cannot drift from it.

The in-run flow:

• BacktestingResultHandler invokes the analyzer on the periodic intermediate result updates, with a 1s time budget since it runs on the result handler thread, handing it the intermediate BacktestResult it already builds for storage.
• That result carries only the last 100 orders and order events, so each cycle's cost is effectively constant. The analyzer dedupes the newest-first order event window against a watermark (the last analyzed order id / event id pair); events evicted from the window between runs are missed until the final analysis re-scans the complete streams.
• The analyzer supplies itself what the result doesn't carry: the handler passes in the full log list and the analyzer slices off the lines produced since the previous run, and it samples the engine's speed counters (handed to it at construction) on each run.
• Stream-based findings accumulate across runs (first sample kept, counts totaled); state-based findings are replaced on every run.
• Statistics are withheld until warm-up ends and the first equity sample exists: before that they are all-zero defaults that would flag a false non-positive equity finding.
• Findings are sent to the browser in their own result packet.
• A finding is returned a capped number of times (3) and then muted, so recurring findings are not re-sent forever, polluting the context of LLMs consuming them. Only findings actually returned count toward the cap: one that a truncated run never got to report is not silenced before it is seen.
• When the time limit truncates a run, the log position and event watermark still advance: analyses that didn't run miss that delta until the final analysis. Accepted behavior (stress runs finish in a fraction of the budget), documented in the code.

New AlgorithmSpeedAnalysis, tracking the algorithm's speed so the user can decide to stop a slow backtest early:

• The analyzer samples the engine counters (data points, history data points, processed/total days, elapsed) into an AlgorithmSpeedTracker that computes cumulative and recent-window rates, calendar-days-per-second, projected remaining time, and history-request share.
• Sub-findings: SlowExecution (recent pace below the 40k data points/s benchmark), LongProjectedRuntime (over an hour left at the recent pace, or stalled calendar progress), ThroughputDegradation (recent pace below half the early-run baseline), HistoryRequestLoad (most processed data points served by history requests).
• Guardrails: a one-minute minimum sampled span, and every condition must hold for two consecutive windows, so warm-up noise or a single slow cycle doesn't flag (or clear) a finding. The counters are not sampled during warm-up so its pace doesn't skew the metrics; the in-run analyses themselves do run during warm-up, so conditions like orders submitted while warming up surface immediately.
• Also runs on the final analysis: SendFinalResult hands the final instance the in-run tracker, completed with one last sample (CompleteSpeedTracking). A completed backtest omits the remaining-time projection and self-suppresses the long-runtime finding; an aborted run still reflects its final pace.
• Replaces the previous ExecutionSpeedAnalysis: when the tracked metrics cannot measure the speed, the engine's completion log line is parsed for the whole-run average as a fallback, with the same threshold (SlowDataPointsPerSecond) and minimum-runtime rule. The line only exists once the backtest ends, so the fallback only fires on the final analysis. The finding is now named AlgorithmSpeedAnalysis / SlowExecution instead of ExecutionSpeedAnalysis.

New SingleTimeLoopTimeoutRuntimeErrorAnalysis (final only, fatal weight 100):

• Detects all the Isolator timeout terminations — a single time loop over its per-loop limit, the run outliving the maximum allowed runtime, and "Operation was canceled" when code keeps running after a stop request — since they share the same root cause and advice.
• Reads the runtime error from the result state, falling back to the "Runtime Error:" log line, and recommends how to avoid the timeouts.

Related Issue

N/A

Motivation and Context

Users currently only get the results analysis findings when a backtest finishes. For long backtests, surfacing failures like margin calls or persistent order errors while the run is in progress lets the user stop it early instead of waiting for completion.

Requires Documentation Change

N/A

How Has This Been Tested?

• Ran a backtest through the Launcher with a packet-capturing messaging handler and an algorithm deliberately triggering order response errors: the findings arrived in their own result packets during the run (including one produced during warm-up, with no false non-positive-equity finding), and the final result carried them plus the final-only findings (AlgorithmSpeedAnalysis / SlowExecution, StatisticalSignificanceOfDailyReturnsAnalysis).

• Stress-tested the in-run analyzer with an instrumented build (local-only timing instrumentation, not part of this PR) — worst observed cycle: 365 ms against the 1 s budget. These numbers predate the switch to analyzing the truncated intermediate result: deltas were unbounded then, while cycles are now capped at the last 100 orders and order events, so they are a conservative upper bound on the current design's cost.

  Stress test setup and per-cycle numbers

  • Baseline run — 4 minute-resolution symbols, ~30 orders per time step: a cycle processing a delta of 31,489 order events + 764 log lines against 14,818 accumulated orders took 85 ms.

  • Heavy run — 133 minute-resolution symbols over a month, ~75 orders per time step (~30k orders/day) with fills, cancellations, rejections, and a log line per bar:

    Cycle	Order-event delta	Accumulated orders	Elapsed
    1	65,035	29,589	133 ms
    2	194,935	118,282	252 ms
    3	194,086	206,588	297 ms
    4	194,262	294,978	311 ms
    5	124,329	351,544	365 ms
    6	64,135	380,803	289 ms

• Reproduced each of the timeout runtime errors with temporary local algorithms and verified the final result contains the corresponding finding with the error message as its sample.

• Unit tests (75, all passing):

  • ResultsAnalyzerInRunTests — the in-run mode, driven through intermediate results carrying truncated order event windows plus the accumulated logs: watermark dedup (each event analyzed exactly once, in order, advancing on truncated runs), evicted events being missed, accumulation vs state-based replacement, the reported-occurrences cap, weight ranking and the findings cap, the snapshot contents, statistics withholding, speed sample tracking and the final hand-off, and the in-run subset.
  • ResultsAnalyzerTests — the mode-independent core: analysis set overriding and reuse, weight-ordered execution, the time-limit and max-failures early exits, skipped equity curves, the pinned final-only and state-based declarations, and the in-run entry points throwing on a final-analysis instance.
  • PortfolioValueIsNotPositiveAnalysisTests — withheld statistics, the non-positive ending equity flag, no finding on positive equity.
  • AlgorithmSpeedTrackerTests — rate math, history data point share, remaining-time estimation, sample validation.
  • AlgorithmSpeedAnalysisTests — each sub-finding's threshold, the warm-up grace span, the two-window hysteresis, unwired counters, the completed-run projection, and the completion-log fallback (fires without tracked metrics, ignores fast/short/absent lines, superseded by conclusive metrics).
  • SingleTimeLoopTimeoutRuntimeErrorAnalysisTests — each timeout message shape, state and log-fallback detection, no false positives, language-aware solution formatting.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which improves implementation)
• Performance (non-breaking change which improves performance. Please add associated performance test and results)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Non-functional change (xml comments/documentation/etc)

Checklist:

• My code follows the code style of this project.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>