fix(webapp): bound logs search memory and fix pagination at scale (#4012)

## Summary

The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.

## Fix

Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:

- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.

Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.

Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.

## ClickHouse 26.2

The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Author: ericallam

.github/workflows/unit-tests-internal.yml

@@ -95,7 +95,7 @@ jobs:
           }
           echo "Pre-pulling Docker images with authenticated session..."
           pull postgres:14
-          pull clickhouse/clickhouse-server:25.4-alpine
+          pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
           pull redis:7.2
           pull testcontainers/ryuk:0.14.0
           pull electricsql/electric:1.2.4

.github/workflows/unit-tests-packages.yml

@@ -95,7 +95,7 @@ jobs:
           }
           echo "Pre-pulling Docker images with authenticated session..."
           pull postgres:14
-          pull clickhouse/clickhouse-server:25.4-alpine
+          pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
           pull redis:7.2
           pull testcontainers/ryuk:0.14.0
           pull electricsql/electric:1.2.4

.github/workflows/unit-tests-webapp.yml

@@ -95,7 +95,7 @@ jobs:
           }
           echo "Pre-pulling Docker images with authenticated session..."
           pull postgres:14
-          pull clickhouse/clickhouse-server:25.4-alpine
+          pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
           pull redis:7.2
           pull testcontainers/ryuk:0.14.0
           pull electricsql/electric:1.2.4

.server-changes/logs-search-memory-and-pagination.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: fix
+---
+
+Keep logs search within bounded ClickHouse memory when browsing long time ranges, and fix pagination that could skip or duplicate entries sharing a timestamp.

apps/webapp/app/env.server.ts

@@ -1642,6 +1642,34 @@ const EnvironmentSchema = z
     CLICKHOUSE_LOGS_LIST_MAX_THREADS: z.coerce.number().int().default(2),
     CLICKHOUSE_LOGS_LIST_MAX_ROWS_TO_READ: z.coerce.number().int().default(10_000_000),
     CLICKHOUSE_LOGS_LIST_MAX_EXECUTION_TIME: z.coerce.number().int().default(120),
+    // Bound read-in-order memory on object-storage reads: each part opens a per-column read
+    // stream, and the default ~1 MiB+ S3 buffers dominate peak memory. These two byte sizes
+    // cap the per-stream buffers and exist on every supported ClickHouse, so they are always on.
+    CLICKHOUSE_LOGS_LIST_PREFETCH_BUFFER_SIZE: z.coerce.number().int().nonnegative().default(262_144),
+    CLICKHOUSE_LOGS_LIST_MAX_READ_BUFFER_SIZE: z.coerce.number().int().nonnegative().default(262_144),
+    // The decisive lever on Cloud SharedMergeTree, but it only exists on newer ClickHouse and
+    // is a no-op on local-disk MergeTree, so it is opt-in: unset means it is never sent (safe on
+    // any self-hosted version). Set to 0 on object-storage deployments to get the memory win.
+    CLICKHOUSE_LOGS_LIST_FILESYSTEM_CACHE_PREFER_BIGGER_BUFFER_SIZE: z.coerce
+      .number()
+      .int()
+      .nonnegative()
+      .optional(),
+
+    // Logs list pagination tuning (page sizing + recent-first probe windows).
+    LOGS_LIST_DEFAULT_PAGE_SIZE: z.coerce.number().int().positive().default(50),
+    LOGS_LIST_MAX_PAGE_SIZE: z.coerce.number().int().positive().default(100),
+    // Days back from the page ceiling to probe before widening to the full requested window,
+    // comma-separated. Empty disables narrowing (a single full-window query).
+    LOGS_LIST_RECENT_FIRST_PROBE_DAYS: z
+      .string()
+      .default("1,7")
+      .transform((s) =>
+        s
+          .split(",")
+          .map((v) => Number(v.trim()))
+          .filter((n) => Number.isFinite(n) && n > 0)
+      ),
 
     // Query feature flag
     QUERY_FEATURE_ENABLED: z.string().default("1"),