fix(hosting): disable ClickHouse system-log telemetry tables and apply profile settings via users.d

[Leafgard]

fixes #4343

Problem

The self-hosting ClickHouse ships with every system log table enabled and unbounded (no TTL by default). On the recommended webapp machine (3+ vCPU / 6+ GB), the wide telemetry tables (metric_log ~1200 columns, text_log, asynchronous_metric_log) grow until their background merges no longer fit under the memory cap (max_server_memory_usage_to_ram_ratio 0.9 ≈ 5.2 GiB on a 6 GB box — ClickHouse measured a default metric_log merge peaking around 6 GB in ClickHouse/ClickHouse#89811). Failed non-replicated merges have no retry backoff, so ClickHouse retries them forever: in our production this burned ~270% CPU at ~350k MEMORY_LIMIT_EXCEEDED/day, each failure logging a stack trace into text_log and feeding the loop — and eventually the webapp's own inserts into trigger_dev.task_runs_v2/metrics_v1 started failing with the same error, so runs went missing from the dashboard. 14 days after a fresh data dir, 5.96 GiB of the 6.0 GiB of MergeTree data on the box was ClickHouse telemetry; actual Trigger data was ~40 MiB.

The dev stack fixed exactly this in #3565 (docker/config/clickhouse-disable-system-logs.xml); it was never ported to hosting/docker. The ClickHouse low-RAM guide prescribes disabling these tables on <16 GB machines: https://clickhouse.com/docs/operations/tips

Separately, the <profiles> block in hosting/docker/clickhouse/override.xml is silently ignored: profile settings only apply from the users config tree (users.d), never from config.d — so the advertised low-memory settings (max_block_size 8192, etc.) have never applied.

Changes

• hosting/docker/clickhouse/override.xml: disable the same system log tables as the dev stack, extended with the newer ones (latency_log, query_metric_log, opentelemetry_span_log, query_views_log). Keep query_log and error_log, bounded with a config-level <ttl> (7/30 days) — config-level TTL survives table recreation, unlike ALTER … MODIFY TTL. The ineffective <profiles> block is removed.
• New hosting/docker/clickhouse/users-override.xml, mounted at /etc/clickhouse-server/users.d/override.xml: carries those profile settings so they actually apply, plus explicit zeros for the memory/query profilers (their samples were the main trace_log firehose).
• hosting/docker/webapp/docker-compose.yml: add the users.d mount.

Validation

Deployed on our production for 7 days (see #4343 for the full soak log): ClickHouse CPU 275% → ~2%, MEMORY_LIMIT_EXCEEDED from ~350k/day to zero for 7 straight days, webapp insert failures from ~40/day to zero — including a 79-database backup fleet run (~470 task runs) — and 6 GiB of disk reclaimed.

Note for existing deployments: disabling a log table stops new writes but doesn't delete existing data. To reclaim disk: DROP TABLE system.<name> SYNC for each disabled table (plus any *_log_<N> leftovers from config-change renames). Happy to add that to the self-hosting docs if useful.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 75f4f59

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Walkthrough

ClickHouse now disables fifteen system telemetry log tables and retains query_log for 7 days and error_log for 30 days. Low-memory settings move from override.xml to users-override.xml, which Docker Compose mounts into ClickHouse’s users.d directory. A hosting changelog entry documents these changes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes disable affected telemetry, retain bounded query and error logs, and move profile settings into users.d as required by #4343.
Out of Scope Changes check	✅ Passed	All configuration, Compose, and hosting-documentation changes directly support the ClickHouse telemetry and users.d objectives in #4343.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check	✅ Passed	The title clearly summarizes the main ClickHouse telemetry and profile configuration changes.
Description check	✅ Passed	The description explains the problem, changes, issue reference, and validation results, but it omits the template checklist and screenshots section.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b241327b-92c1-4cb0-9f5b-f4f07d2ef9a1

📥 Commits

Reviewing files that changed from the base of the PR and between 90e8bd5 and 14fde64.

📒 Files selected for processing (4)

• .server-changes/hosting-clickhouse-system-logs.md
• hosting/docker/clickhouse/override.xml
• hosting/docker/clickhouse/users-override.xml
• hosting/docker/webapp/docker-compose.yml

📜 Review details

🧰 Additional context used

🧠 Learnings (2)

📚 Learning: 2026-05-14T14:54:39.095Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3545
File: .server-changes/agent-view-sessions.md:10-10
Timestamp: 2026-05-14T14:54:39.095Z
Learning: In the `trigger.dev` repository, do not flag inconsistent dot vs slash notation in route/path strings inside `.server-changes/*.md` files. These markdown files are consumed verbatim into the changelog, so the mixed notation (e.g., `resources.orgs.../runs.$runParam/...`) is intentional and should be preserved as-is.

Applied to files:

• .server-changes/hosting-clickhouse-system-logs.md

📚 Learning: 2026-07-26T13:14:02.968Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 4378
File: .server-changes/realtime-run-reads-from-primary.md:0-0
Timestamp: 2026-07-26T13:14:02.968Z
Learning: For files in the .server-changes directory, the body text is published verbatim as dashboard-facing user release notes. Write entries in terms of user-visible behavior (what users can do/see), and avoid implementation-oriented details such as environment-variable names, internal mechanisms, or configuration knobs. If you need to include operational/configuration specifics, put those details in the PR description instead of the .server-changes entry.

Applied to files:

• .server-changes/hosting-clickhouse-system-logs.md

🔇 Additional comments (5)

hosting/docker/clickhouse/override.xml (2)

32-32: 🗄️ Data Integrity & Integration

Verify latency_log against the target ClickHouse image.

The supplied context does not include the ClickHouse image tag. The upstream ClickHouse configuration I checked does not define latency_log. (github.com) Confirm that the image used by hosting/docker/webapp/docker-compose.yml supports this key. Also verify the full disable list against system.tables after startup.

Source: MCP tools

---

13-20: LGTM!

Also applies to: 37-46, 48-50

hosting/docker/clickhouse/users-override.xml (2)

11-18: 🚀 Performance & Scalability

Disable the server-wide memory profiler, or verify it is already disabled.

This file disables the per-query profiler settings only. The ClickHouse 26.2 stock configuration sets the server-level total_memory_profiler_step to 4194304; this setting is not controlled by the users.d profile. (raw.githubusercontent.com)

If hosting/docker/clickhouse/override.xml near Lines 48-50 does not set total_memory_profiler_step to 0, the server-wide profiler can continue collecting discarded trace samples. Add that setting to hosting/docker/clickhouse/override.xml, or verify the effective merged value.

---

1-10: LGTM!

Also applies to: 19-21

hosting/docker/webapp/docker-compose.yml (1)

180-180: LGTM!