> For the complete documentation index, see [llms.txt](https://docs.spike.sh/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.spike.sh/integrations-guideline/integrate-spike-with-monte-carlo.md). # Integrate Spike with Monte Carlo Send Monte Carlo data observability alerts to Spike so your data on-call rotation is paged by phone, SMS, Slack or Teams, and incidents resolve themselves when the alert is fixed. [Monte Carlo](https://www.montecarlodata.com) watches your warehouse for freshness, volume and schema anomalies and for custom rule breaches. Its Webhook notification channel posts a JSON payload when an alert opens and again every time somebody works on that alert, so Spike can page the data on-call rotation on the first anomaly and close the incident when the alert is marked fixed in Monte Carlo. Nothing is installed anywhere. You create an audience in Monte Carlo, point its Webhook channel at a Spike integration URL, and the two stay in sync from there. ## What Spike does with each webhook | Webhook | What happens in Spike | | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `type: open_anomaly_incident` | Opens an incident for that alert and pages the escalation policy | | `type: open_custom_rule_anomaly_incident` | Opens an incident for the rule breach and pages the escalation policy | | Alert status set to `fixed`, `expected`, `no_action_needed` or `false_positive` | Auto-resolves the open incident. Dropped when nothing is open | | Alert acknowledged, or status set to `investigating`, `work_in_progress` or `no_status` | Added as an event on the open incident. Escalation continues | | Owner changed, alert marked or unmarked as an incident, external ticket attached | Added as an event on the open incident. They never page anyone | Spike identifies the incident by `payload.incident_id`, the same id Monte Carlo puts in the alert URL. Every update for that alert lands on the one incident, so an alert that is acknowledged, reassigned, escalated to SEV-1 and finally resolved pages your team once. {% hint style="info" %} Resolving the incident in Spike does not change the alert in Monte Carlo, and an update that arrives once the incident is closed has nothing left to attach to and is dropped. Set the alert status in Monte Carlo and let it resolve the incident in Spike to keep the two sides in step. {% endhint %} ## Incident titles Titles come from the alert itself, so they stay readable when Spike reads them out on a phone call: * Anomaly: `Has 1,204 rows vs. 58,930 expected on analytics:prod.finance.daily_ledger`, built from `event_list[0].event_details` and `event_list[0].table_name` * Custom rule breach: `Rule broken: Refund amounts must never be negative on snowflake:prod.payments.refunds`, built from the rule description. A rule that carries no table reads `Rule broken: Daily signups below 40% of 7-day average` * An alert that carries nothing to build a title from: `Monte Carlo incident `, or `Monte Carlo alert with no details` when even the id is missing The `Rule broken:` prefix is there so a custom rule reads as an alert rather than as a reminder of the rule. Anomaly titles never get it, because the event details already read as a problem. A title longer than 200 characters keeps its `on {table}` part whole and shortens only the subject, at a word boundary, so the table a responder needs is never the part that gets cut: ``` Distribution shift detected in field shipping_country: share of rows with value US dropped from 61.4% to 12.9% while value unknown rose from 0.2% to 48.7%... on snowflake:prod.logistics.shipments ``` The alert URL, the group id, the table and the rule comparisons stay in the payload and show up on the incident page. Use a [Title Remapper](/alerts/title-remapper.md) if you would rather see the warehouse or the domain in the title: ```handlebars {{payload.event_list.0.table_name}} — {{payload.event_list.0.event_details}} ``` ## Severity Set severity on a Monte Carlo incident with [alert rules](/alerts/alert-rules.md). The same rules route the incident to another escalation policy or suppress it entirely, which is how you send freshness alerts on a staging dataset somewhere quieter than the pager. {% hint style="warning" %} The SEV you declare in Monte Carlo does not reach the Spike incident. Monte Carlo sends `declared_alert_severity` only on a later "marked as incident" update, never on the alert that opens the incident, and Spike sets severity when an incident is created or reopened rather than on an update that joins one that is already open. {% endhint %} Spike does read the value when it ever arrives on an opening payload, and maps it onto its three severities: | `declared_alert_severity` | Severity in Spike | | ------------------------- | ----------------- | | `SEV-1` | SEV1 | | `SEV-2` | SEV2 | | `SEV-3` | SEV3 | | `SEV-4` | SEV3 | | `null` or absent | Left unset | Monte Carlo has four severities and Spike has three, which is why `SEV-4` lands on SEV3. Read more about [priority and severity](/incidents/priority-and-severity.md). ## Step 1 — Create the integration in Spike In Spike, go to **Integrations → Add integration → Monte Carlo**, attach it to a service and an escalation policy, and copy the webhook URL. It looks like `https://hooks.spike.sh//push-events`. {% content-ref url="/pages/-Ln3rb4wLUFeI-OAPcmq" %} [Create integration and service on our dashboard](/integrations-guideline/create-integration-and-service-on-dashboard.md) {% endcontent-ref %} ## Step 2 — Create the audience in Monte Carlo {% tabs %} {% tab title="Setup on Monte Carlo" %} 1. **Open notification settings:** Go to [**Settings → Notifications**](https://getmontecarlo.com/settings/notifications) in Monte Carlo. 2. **Create the audience:** Click **Create audience** and select **Webhook** from the **Channel** drop down. 3. **Point it at Spike:** Paste the webhook URL from Step 1 into the **URL** field. Leave **Secret** blank, see the note below. 4. **Choose what the audience covers:** Pick the alert types, domains, tables or monitors this audience should notify on. An audience scoped to your production domain keeps development noise out of the pager, and you can create a second audience for a second Spike integration when different teams own different domains. 5. **Save and test:** Save the audience, then use **Send test notification** if the channel offers it. Otherwise wait for the next real alert, or trigger a custom rule against a test table. The incident shows up in Spike within a few seconds. {% endtab %} {% endtabs %} {% hint style="warning" %} Monte Carlo sends webhooks from a fixed set of public IP addresses listed on your **Account information** page. If egress to `hooks.spike.sh` runs through a proxy you control, nothing needs allowlisting on the Spike side. {% endhint %} ## Which events to send The Webhook channel has no per-event switch. Once an audience notifies on an alert, Monte Carlo sends the alert creation and every later update for it: acknowledgement, status change, owner change, external ticket attached, marked or unmarked as an incident, and resolution. Spike pages on the creation, resolves on the resolving statuses, and records the rest on the incident, so there is nothing to turn off. The audience filters are therefore the only knob that matters. Scope the audience to the alerts a human should be woken up for. ## The webhook secret Monte Carlo can sign the body with a secret and send the result in an `x-mcd-signature` header. Spike does not verify that header today, so leave **Secret** blank. Your webhook URL carries a token that only your integration has, which is the same model every other Spike integration uses. If the URL leaks, archive the integration and create a new one. {% content-ref url="/pages/-MRcwUUCSJO-4crz5xXn" %} [Archive an integration](/integrations-guideline/archive-an-integration.md) {% endcontent-ref %} ## Payload reference Monte Carlo sends `application/json` over POST, version `v0.1`. An anomaly alert opens with: ```json { "account": { "id": "9a59c6f1-203a-4c72-8cd7-132db7f21b92" }, "action": "incident", "type": "open_anomaly_incident", "version": "v0.1", "payload": { "incident_id": "f83867b6-8f25-428b-807c-08518f63384c", "group_id": "zeus:ares", "url": "https://getmontecarlo.com/incidents/f83867b6-8f25-428b-807c-08518f63384c", "event_list": [ { "event_details": "Has 42 rows vs. 101 expected", "table_name": "artemis" } ] } } ``` A custom rule breach carries `type: open_custom_rule_anomaly_incident` and a `rule` object instead of the table level details: ```json { "account": { "id": "9a59c6f1-203a-4c72-8cd7-132db7f21b92" }, "action": "incident", "type": "open_custom_rule_anomaly_incident", "version": "v0.1", "payload": { "incident_id": "f83867b6-8f25-428b-807c-08518f63384c", "url": "https://getmontecarlo.com/incidents/f83867b6-8f25-428b-807c-08518f63384c", "event_list": [ { "data": { "metric_display_name": "custom_metric_1234", "rule_comparisons": [ { "metric": "custom_metric_1234", "operator": ">", "threshold": "49.0" } ], "hits": [{ "measurement_timestamp": "Jan 15, 06:10PM", "value": "50.0" }] } } ], "rule": { "uuid": "0c8e06aa-a44b-4f2d-8a37-5ed3f394431d", "description": "Orders row count must stay above 49", "rule_type": "custom_sql", "comparisons": [ { "metric": "custom_metric_1234", "operator": "GT", "threshold": 49.0 } ] } } } ``` An update repeats `incident_id` and adds the fields that changed: | Field | Values | Sent when | | ------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------- | | `alert_feedback` | `investigating` | The alert is acknowledged, or its status is set to investigating | | `alert_feedback` | `work_in_progress`, `no_status` | The status is updated and the alert stays open | | `alert_feedback` | `fixed`, `expected`, `no_action_needed`, `false_positive` | The alert is resolved. Spike resolves the incident | | `declared_alert_severity` | `SEV-1`, `SEV-2`, `SEV-3`, `SEV-4` | The alert is marked as an incident | | `declared_alert_severity` | `null` | The alert is unmarked as an incident | | `owner` | An email address | The alert owner changes | {% hint style="info" %} Monte Carlo documents these update fields in a table rather than as a sample payload. Spike reads them whether they arrive at the top level or inside `payload`, so an update resolves the right incident either way. An update is recorded on the incident, including the SEV declared on it, but it does not change the incident's severity. {% endhint %} ## Troubleshooting
No incidents show up in Spike Check that the audience is enabled and that its filters actually match an alert that fired. An audience scoped to a domain or a set of tables only notifies on alerts for those assets, and a Monte Carlo alert that nobody is notified about never reaches the Webhook channel at all. Confirm the URL in the channel is the full `https://hooks.spike.sh//push-events`, with no trailing characters.
Incidents open but never resolve Spike resolves on `fixed`, `expected`, `no_action_needed` and `false_positive`. Setting an alert to `investigating` or `work_in_progress` keeps the incident open on purpose, so the escalation stays alive while somebody digs into the anomaly. If your team resolves alerts in the Monte Carlo UI but incidents stay open in Spike, check that the same audience is still attached to the alert, since the resolution follows the same channel as the creation.
One alert opened several incidents Spike groups on `payload.incident_id`. Two incidents for what looks like the same problem usually means Monte Carlo opened two alerts, for example a freshness alert and a volume alert on the same table, each carrying its own id. That is one incident per alert by design. Repeats of the same alert are [grouped](/incidents/grouping-incidents.md) under the incident already open.
A title reads as an id, or as "Monte Carlo alert with no details" Both are fallbacks for an alert that carried nothing to build a title from. A custom rule with no description is the usual cause, so give the rule a description in Monte Carlo, or write a [Title Remapper](/alerts/title-remapper.md) that builds the title from `payload.rule.comparisons`.
Severity never gets set That is expected today. The SEV is declared on a later update, and Spike does not change the severity of an incident that is already open, so the declared SEV stays in the incident's activity rather than on the incident itself. Set severity from [alert rules](/alerts/alert-rules.md) instead, which also gets it onto the incident from the first page.