Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ class GroupHistory(Model):

## Tombstones

Tracking the absence of a record is complicated. It is far simpler to track the presence of another object. When a cross region resource (like a user) is deleted we create and propagate a marker of what once was aka a *Tombstone*. The presence of a tombstone is used to cleanup data that was previously related now deceased record.
Tracking the absence of a record is complicated. It is far simpler to track the presence of another object. When a cross region resource (like a user) is deleted we create and propagate a marker of what once was aka a _Tombstone_. The presence of a tombstone is used to cleanup data that was previously related now deceased record.

Tombstones have a few properties:

Expand Down Expand Up @@ -60,7 +60,7 @@ sequenceDiagram
In step 5 and 6 of the above diagram we reconcile the tombstone changes with the rest of the data in the region. Tombstones needs to be reconciled for each relation that the removed record had. For example, removing a user will:

- Delete the user’s saved searches, dashboards, and more using the `CASCADE` action.
- Null out the user’s in alert rules, releases and more using the `SET NULL` action.
- Null out the user’s in alerts, releases and more using the `SET NULL` action.

### Tombstone Reconciliation

Expand Down
36 changes: 18 additions & 18 deletions develop-docs/integrations/slack/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -74,24 +74,24 @@ Add the following scopes to **Bot Scopes**:
Features which are not currently available for self-hosted. You can freely omit them.
</Alert>

| Scope | Reason |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `assistant:write` | Enables accessing the app as a personal Slack Agent. |
| `app_mentions:read` | Receive `app_mention` events when the bot is mentioned (e.g. @Sentry) in a message. |
| `channels:read` | List and validate public channels when configuring alert-rule destinations, and resolve channel names to IDs. |
| `channels:history` | (Optional) Allows Sentry to read messages from public channels. Sentry uses this to gather thread context for conversations with Seer Agent. |
| `chat:write` | Send and update messages, such as alerts, notifications, and responses. |
| `chat:write.customize` | Allow sending messages with a custom bot name and icon. |
| `chat:write.public` | Post notifications to public channels the bot hasn't been invited to. |
| `commands` | Power the `/sentry` slash command: `link`/`unlink` (identity), `link team`/`unlink team` (channel-to-team mapping), and `help`. |
| `groups:read` | Same as `channels:read` but for private channels. |
| `groups:history` | (Optional) Same as `channels:history` but for private channels. |
| `im:history` | Receive `message.im` events for the bot's help responses in DMs, and fetch DM thread context for Seer Agent. |
| `im:read` | Receive direct-message events and resolve DM channel metadata, enabling slash commands and messages sent directly to the bot. |
| `links:read` | Receive `link_shared` events when a user pastes a Sentry URL in Slack so the integration can generate a rich preview. |
| `links:write` | Attach rich unfurl previews (issues, explore queries, dashboard charts, metric alerts) to Sentry URLs shared in Slack. |
| `team:read` | Fetch the workspace name and icon during installation for display in the Sentry integration settings page. |
| `users:read` | Look up workspace users to validate alert-rule recipients, search by username/display name, and automatically link Sentry accounts to Slack identities by email after installation. |
| Scope | Reason |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `assistant:write` | Enables accessing the app as a personal Slack Agent. |
| `app_mentions:read` | Receive `app_mention` events when the bot is mentioned (e.g. @Sentry) in a message. |
| `channels:read` | List and validate public channels when configuring alert destinations, and resolve channel names to IDs. |
| `channels:history` | (Optional) Allows Sentry to read messages from public channels. Sentry uses this to gather thread context for conversations with Seer Agent. |
| `chat:write` | Send and update messages, such as alerts, notifications, and responses. |
| `chat:write.customize` | Allow sending messages with a custom bot name and icon. |
| `chat:write.public` | Post notifications to public channels the bot hasn't been invited to. |
| `commands` | Power the `/sentry` slash command: `link`/`unlink` (identity), `link team`/`unlink team` (channel-to-team mapping), and `help`. |
| `groups:read` | Same as `channels:read` but for private channels. |
| `groups:history` | (Optional) Same as `channels:history` but for private channels. |
| `im:history` | Receive `message.im` events for the bot's help responses in DMs, and fetch DM thread context for Seer Agent. |
| `im:read` | Receive direct-message events and resolve DM channel metadata, enabling slash commands and messages sent directly to the bot. |
| `links:read` | Receive `link_shared` events when a user pastes a Sentry URL in Slack so the integration can generate a rich preview. |
| `links:write` | Attach rich unfurl previews (issues, explore queries, dashboard charts, alerts) to Sentry URLs shared in Slack. |
| `team:read` | Fetch the workspace name and icon during installation for display in the Sentry integration settings page. |
| `users:read` | Look up workspace users to validate alert recipients, search by username/display name, and automatically link Sentry accounts to Slack identities by email after installation. |

Add the following scopes to **User Scopes**:

Expand Down
5 changes: 2 additions & 3 deletions develop-docs/self-hosted/backup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Specifically, the following data is generally included in a partial JSON backup:
- Organizations, their members, and their settings.
- Projects, their members, and their settings.
- Teams, their members, and their settings.
- Alert rules, incidents, and monitors.
- Alerts, incidents, and monitors.

### Backup

Expand Down Expand Up @@ -66,7 +66,7 @@ which you will then need to manually remove.

If you'd like more granular control over what gets backed up, and are using a `self-hosted` release
greater than or equal to 23.11.1, you may use the `./sentry-admin.sh export` command instead. In
particular, this command allows you export only a certain ***scope*** of data. There are four such
particular, this command allows you export only a certain **_scope_** of data. There are four such
scopes:

- `User`: Exports data associated with Sentry users only, like their emails, permissions, and login
Expand Down Expand Up @@ -101,7 +101,6 @@ Alternatively, if you are using Google Cloud Platform's [Key Management
Service](https://cloud.google.com/security/products/security-key-management?hl=en) to manage keys on
your behalf, you can use that service directly from the CLI instead:


```shell
# Authenticate with Google Cloud (requires the `gcloud` CLI to be installed):
gcloud auth login
Expand Down
38 changes: 19 additions & 19 deletions develop-docs/self-hosted/troubleshooting/snuba.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,40 +8,40 @@ Snuba is the service that handles Sentry's search and analytics. It's separated

## What Snuba subscription consumers are responsible for

Snuba subscriptions implement Sentry's alert rules, meaning periodic queries that run on schedule and emit results back to Sentry. There are two roles per dataset (events, transactions, metrics, generic-metrics, eap-items):
Snuba subscriptions implement Sentry's alerts, meaning periodic queries that run on schedule and emit results back to Sentry. There are two roles per dataset (events, transactions, metrics, generic-metrics, eap-items):

- **Subscription scheduler** — decides *when* to run each subscribed query. It does **not** consume event data. Instead, it tails a small "clock" topic (the commit log) and emits one Kafka message per scheduled query.
- **Subscription scheduler** — decides _when_ to run each subscribed query. It does **not** consume event data. Instead, it tails a small "clock" topic (the commit log) and emits one Kafka message per scheduled query.
- **Subscription executor** — picks up those scheduled queries, runs them against ClickHouse, and produces the answers to a results topic that Sentry consumes.

There is also a combined `subscriptions-scheduler-executor` binary that fuses both stages in one process (this is what self-hosted typically runs for `events` / `transactions` / `metrics`).

### Which topics they consume from — and why this matters

| Role | Reads from | Writes to | Where defined |
|---|---|---|---|
| Role | Reads from | Writes to | Where defined |
| --------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------ |
| Scheduler | `snuba-commit-log` (events) / `snuba-transactions-commit-log` / `snuba-metrics-commit-log` / `snuba-generic-metrics-*-commit-log` | `scheduled-subscriptions-<entity>` | `commit_log_topic` + `subscription_scheduled_topic` in each storage YAML |
| Executor | `scheduled-subscriptions-<entity>` | `<entity>-subscription-results` | `subscription_scheduled_topic` + `subscription_result_topic` |
| Executor | `scheduled-subscriptions-<entity>` | `<entity>-subscription-results` | `subscription_scheduled_topic` + `subscription_result_topic` |

Critically: the scheduler does not consume from `events` / `transactions` / `snuba-metrics`. It consumes the *commit log* of those topics. The commit log is written by the main ingest consumer once per commit (i.e. periodically batched), so it has dramatically lower throughput than the data topic itself.
Critically: the scheduler does not consume from `events` / `transactions` / `snuba-metrics`. It consumes the _commit log_ of those topics. The commit log is written by the main ingest consumer once per commit (i.e. periodically batched), so it has dramatically lower throughput than the data topic itself.

The scheduler reads the `orig_message_ts` header from each commit-log message and uses that as a clock to decide which subscriptions are due. See `subscriptions_scheduler.py` for the design — tick consumer → tick buffer → commit-strategy step → query producer.

### Why offsets may appear static or end offsets missing

This is **expected** in most healthy self-hosted deployments. Reasons, in order of likelihood:

1. **Low traffic on the source topic.** The scheduler reads the *commit log*,
1. **Low traffic on the source topic.** The scheduler reads the _commit log_,
which only gets a record when the upstream events/transactions consumer
flushes a batch (typically every few seconds, and only if there's data).
On a quiet self-hosted instance, that's a tiny trickle.
2. **No active subscriptions.** Sentry alert rules are what create
2. **No active subscriptions.** Sentry alerts are what create
subscription rows; if no alerts are configured for a dataset, the
scheduler emits nothing to `scheduled-subscriptions-*`, so the executor's
input topic stays empty and its committed offset never moves.
End offset = last committed offset = no apparent change.
3. **GLOBAL watermark mode buffers ticks.** For most entities (transactions,
metrics, eap-items, generic-metrics) `subscription_scheduler_mode: global`.
The scheduler waits until *every* partition of the source topic has
The scheduler waits until _every_ partition of the source topic has
advanced past a timestamp before scheduling — so on a single-partition
self-hosted topic it advances the moment a commit-log message arrives, but
on a multi-partition setup with one stalled partition it can stall
Expand All @@ -50,7 +50,7 @@ This is **expected** in most healthy self-hosted deployments. Reasons, in order
end offset for a topic with zero recent produces; this is a UI artifact,
not a consumer problem. `docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --describe --all-groups` will show
the real `LOG-END-OFFSET`.
5. **At-least-once commit policy.** The scheduler only commits the *earliest*
5. **At-least-once commit policy.** The scheduler only commits the _earliest_
commit-log offset whose tick has been fully scheduled and produced. It
deliberately holds the committed offset back relative to the read
position; small static lag is normal.
Expand All @@ -63,7 +63,7 @@ This is **expected** in most healthy self-hosted deployments. Reasons, in order
range. Zero movement is fine **if** the source dataset isn't currently
ingesting.
- **Executor:** committed offset on `scheduled-subscriptions-<entity>`
advances every time an alert fires. With *no* configured Sentry alerts for
advances every time an alert fires. With _no_ configured Sentry alerts for
that dataset, expect the offset to never move — that's not a bug, there's
literally no work.
- **Result topic:** `<entity>-subscription-results` should receive one
Expand All @@ -82,12 +82,12 @@ Quick sanity checks from the host:

### Differences vs. the events / regular Snuba consumers

| | Ingest consumer (e.g. errors, transactions) | Subscription scheduler | Subscription executor |
|---|---|---|---|
| Source topic | High-volume data topic (`events`, `transactions`, `snuba-items`…) | Low-volume **commit log** of that data topic | Internal `scheduled-subscriptions-*` topic |
| Driven by | Event throughput from Sentry/Relay | Commit cadence of the ingest consumer + configured alert rules | Existence of configured alert rules |
| What it writes | ClickHouse INSERTs + commit-log entry | Scheduled-query Kafka messages | Query results to Kafka + ClickHouse SELECTs |
| Expected lag pattern | Continuous offset advance under normal load | Periodic, batched advance — long flat stretches between commits are normal | Static if no alerts are configured; otherwise advances per alert tick |
| If offset is static | Likely a problem (ingestion stalled) | Probably fine (no upstream commits or no alerts) | Probably fine (no Sentry alert rules for this dataset) |
| | Ingest consumer (e.g. errors, transactions) | Subscription scheduler | Subscription executor |
| -------------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| Source topic | High-volume data topic (`events`, `transactions`, `snuba-items`…) | Low-volume **commit log** of that data topic | Internal `scheduled-subscriptions-*` topic |
| Driven by | Event throughput from Sentry/Relay | Commit cadence of the ingest consumer + configured alerts | Existence of configured alerts |
| What it writes | ClickHouse INSERTs + commit-log entry | Scheduled-query Kafka messages | Query results to Kafka + ClickHouse SELECTs |
| Expected lag pattern | Continuous offset advance under normal load | Periodic, batched advance — long flat stretches between commits are normal | Static if no alerts are configured; otherwise advances per alert tick |
| If offset is static | Likely a problem (ingestion stalled) | Probably fine (no upstream commits or no alerts) | Probably fine (no Sentry alerts for this dataset) |

The headline point: **a static offset on a subscription consumer is only a problem if the corresponding feature in Sentry is broken.** Verify by checking whether (a) ingestion is working and (b) any alert rules exist for the dataset — not by staring at offsets in isolation.
The headline point: **a static offset on a subscription consumer is only a problem if the corresponding feature in Sentry is broken.** Verify by checking whether (a) ingestion is working and (b) any alerts exist for the dataset — not by staring at offsets in isolation.
2 changes: 1 addition & 1 deletion docs/account/user-settings/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Clicking "Sign out of all devices" will end your sessions with any device logged

In **Notifications**, you can configure your personal settings for the following Sentry notifications:

- Alerts (issue alerts only)
- Alerts
- Deploy
- Workflow
- Weekly Reports
Expand Down
Loading
Loading