Ctrlk
DashboardWebsiteRoadmap
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Integrate Spike with Icinga 2

Send Icinga 2 host and service notifications to Spike so on-call is paged by phone, SMS, Slack or Teams, and incidents resolve and acknowledge themselves.

Icinga 2 has no webhook output. It runs a notification command instead, so Spike ships a small bash script that turns an Icinga notification into an incident. A hard CRITICAL or DOWN pages your on-call rotation, the recovery resolves the incident, and an acknowledgement made in Icinga Web stops the escalation in Spike.

The script works both with the stock env style of notification commands and with --flag value arguments, which is what you need when you manage configuration through Icinga Director.

Running Icinga 1, Nagios, Naemon or Shinken? Those read Nagios object configuration, so use the Nagios guide and the Nagios tile instead.

What Spike does with each notification

Notification type
What happens in Spike

PROBLEM

Opens an incident for that host and service, or adds an event to the one already open

RECOVERY

Auto-resolves the open incident. Dropped when nothing is open

ACKNOWLEDGEMENT

Acknowledges the open incident and records who acknowledged it and their comment. Dropped when nothing is open

CUSTOM, FLAPPINGSTART, FLAPPINGEND, DOWNTIMESTART, DOWNTIMEEND, DOWNTIMEREMOVED

Added as an event to the open incident. Dropped when nothing is open. They never page anyone

There is one incident per host and service pair, and one incident per host for host checks. Repeat notifications from interval, a WARNING that turns into a CRITICAL, and a flapping check all land on that one incident, so a noisy check never pages the team twice.

Incident titles stay the same for every repeat, which is what makes them readable when Spike reads them out on a phone call:

  • Service problem: Disk / on db-01 is CRITICAL

  • Host problem: Host db-01 is DOWN

Check output, attempt counts and host groups are on the incident page rather than in the title.

Severity comes from the state Icinga reports:

State
Severity

CRITICAL, DOWN

SEV1

WARNING

SEV2

UNKNOWN, OK, UP

SEV3

Severity is set when the incident is created and does not move on repeats. Alert rules can override it, route the incident elsewhere, or suppress it entirely.

Prerequisites

  • Root or sudo access on the Icinga 2 master

  • bash and curl on that host. The script needs nothing else, no jq and no Python

  • An Icinga 2 integration in Spike and its webhook URL

Step 1 — Create the integration in Spike

In Spike, go to Integrations → Add integration → Icinga 2, attach it to a service and an escalation policy, and copy the webhook URL. It looks like https://hooks.spike.sh/<your-token>/push-events.

Create integration and service on our dashboard

Step 2 — Install the notification script

Save the script below as /etc/icinga2/scripts/spike-notify.sh on the master that sends notifications, and make it executable by the icinga user.

spike-notify.sh

Step 3 — Add the notification commands, user and rules

Create /etc/icinga2/conf.d/spike.conf with the objects below and paste your webhook URL into vars.spike_webhook_url on the spike user.

A few things to adjust for your setup:

  • assign where decides what gets sent to Spike. The rules above cover every host with an address and every service. Narrow them the usual way, for example assign where host.vars.notify_spike == true or assign where "production" in host.groups.

  • interval = 30m re-notifies every 30 minutes while a problem is open. Those repeats are added to the incident already open in Spike, they do not page again. Set interval = 0 if you only want the first notification.

  • Keeping the URL in vars.spike_webhook_url on the user rather than in the command means it is not repeated in every log line, and both commands pick it up through $user.vars.spike_webhook_url$.

Step 4 — Validate and reload

Then send a test incident. It opens an incident called Spike test on <hostname> is CRITICAL and resolves it two seconds later:

A resolved Spike test incident on your service means Icinga can reach Spike and the integration is wired to the right escalation policy.

Icinga Director

Director cannot set env on a command, so define the same two commands with arguments instead. The script accepts every value as a --flag, and flags always win over the environment.

In Director, go to Commands → Add and create a Notification Plugin Command pointing at /etc/icinga2/scripts/spike-notify.sh, then add one argument row per flag. This is the configuration Director generates:

The host command is the same with "--kind" = "host", the --host-* flags, "--host-state-type" = "$host.state_type$", "--host-output" = "$host.output$" and "--host-groups" = "$host.groups$", and no --service-* flags.

Then create the user and the notifications in Director as usual: a User named spike carrying the custom variable spike_webhook_url, and two Notifications using the commands above with the types and states from Step 3.

Things worth knowing

  • Icinga 2 hosts are only UP or DOWN. There is no UNREACHABLE host state, so a host incident from Icinga 2 is always SEV1.

  • Icinga 2 has no problem id. Spike identifies an incident by host and service name, so nothing depends on one. The same identity is used for Nagios, which is why both tools behave identically in Spike.

  • Acknowledging in Icinga Web acknowledges in Spike. The acknowledgement carries the author and the comment, and the incident activity reads Acknowledged in Icinga 2 by jane: looking into it. Escalation stops there.

  • Notifications are not retried by Icinga. The script retries once on its own and gives up after roughly 23 seconds, well inside Icinga's 60 second command timeout. It then exits non-zero, which Icinga logs as a failed notification in /var/log/icinga2/icinga2.log.

  • Output with quotes, backslashes or newlines is safe. The script JSON-escapes every value it sends, so plugin output never breaks the payload.

Payload reference

The script posts a flat JSON object. Every value is a string.

There is no top-level message key in this payload on purpose. Spike builds the incident title from the host and service fields, which keeps the title identical across a problem, its repeats and its acknowledgement.

Troubleshooting

Nothing arrives in Spike

Run the script by hand as the icinga user with SPIKE_DEBUG=1 to see the payload and the curl error:

If curl reports a connection or timeout error, the Icinga master cannot reach hooks.spike.sh on port 443. If the script says it has no webhook URL, $user.vars.spike_webhook_url$ did not resolve, which usually means the notification is pointing at a different user object.

Icinga logs a failed notification

Check /var/log/icinga2/icinga2.log for the script's stderr, and confirm the file is executable and readable by the icinga user. On SELinux systems the script also needs a context Icinga is allowed to execute.

Incidents are created but never resolve

Recovery has to be in the types list of both the notification rule and the user object, otherwise Icinga never sends it. A recovery that arrives when nothing is open in Spike is dropped, which is what should happen if you resolved the incident by hand first.

Host notifications arrive with the wrong host name

HOSTNAME is set by bash itself, so the env block has to set it explicitly, as in Step 3. If you wrote your own command definition and left HOSTNAME out, every notification will carry the Icinga master's hostname instead of the monitored host's.

PreviousIntegrate Spike with Hyperping
NextIntegrate Spike with Instana

Last updated

Was this helpful?