Notifications
AME Notifications alert users or external systems about new events or event changes in Alert Manager Enterprise (AME).
Notification Schemes specify the criteria for triggering notifications and the target channels they are sent to.
The Notifications UI is shown below:
Managing Notifications
See Role Overview for permissions required to manage notifications.
Use these buttons to manage notifications:
| Button | Function |
|---|---|
 | Add Notification Scheme |
 | Clone Notification Scheme |
 | Save Notification Scheme |
 | Edit Custom Tag |
 | Add Flow |
 | Add Notification Target |
 | Add Template Target |
Filter by notification name using the search field or by Notification Scheme using the dropdown.
Only admins or users with the ame.admin role can access this page and its features.
Setting Up a Notification Scheme
To create a notification scheme for all AME-managed events:
- Click
Add Notification Schemeat the bottom of the list to open a modal:
- Assign a unique name to the notification scheme.
- Click
Add Flow. - Give the flow a unique name.
- Select a trigger.
- Add a trigger condition (requires subscription).
- Configure a notification target and notification template for the flow:
Triggers
Available notification triggers include:
- Event created
- Event updated
- Event assigned
- Event appended
- Event commented
- Event deleted
- Bulk update
- Bulk delete
- Bulk comment
- Rule matched
- Event violated SLA
- Event fulfilled SLA
- SLA Violation Imminent
Trigger Conditions
Trigger conditions function similarly to rules:
For these triggers, event changes can serve as notification conditions:
- Event updated
- Rule matched
- Bulk update
Available fields for trigger conditions on source:
| Field | Type | Description | Examples |
|---|---|---|---|
| changes | String | Fields that changed | status_name, assignee |
| values | String | Values that changed | new, admin |
| keyword | String | Keyword set in a rule | rule_number_one |
Example:
Let's assume we want to notify only on status updates for in_progress and resolved statuses. In pseudo code the condition looks
like this:
(changed contains “ame.status_name” ) AND (values contains “resolved” OR values contains “in_progress")
Note that we use contains as changes and values are lists. Using the UI it looks like this:
The available fields can be shown by clicking on the Trigger Condition Scheme button.
Limitations for Community Edition
Following Trigger Conditions are available for the Community Edition:
| Trigger | Conditions |
|---|---|
| Event created | none |
| Event assigned | none |
| Event updated | ame.status_name |
| Event appended | ame.status_name |
| BulkUpdate | ame.status_name |
Update and delete a notification scheme
To update a notification scheme, revise the information and press the Save button. To delete a notification scheme,
press the Delete button button.
Notification Targets
The Notification Target tab is used to configure where notifications are sent to.
Currently, AME Supports the following targets:
- Slack
- Webhook
- MS Teams
- Alert Action
The Mail target is already preconfigured. Additional targets can be added by pressing the Add Notification Target button.
A modal will be opened.
Depending on the target, different configuration options are available. For some targets, a Squash Updates selector is available.
Enabling this option will combine multiple notifications into one notification.
Mail
Mail sends an email to the recipients, as well as the cc and bcc entries. You can choose a user name from the dropdown or enter
a valid email address by hand.
The keyword assignee will translate into the assignee's email if it is available to Splunk.
Slack
Slack sends a Slack message to Slack. The channel supports Slack App notifications. Enter the URL and if messages should be squashed.
Webhook
Webhook will send a POST request to a specified URL. Additionally, Headers can be configured.
To create an Incoming Webhook, follow this guide from Slack: Sending messages using incoming webhooks
Teams
Teams will send an MS Teams message to an MS Teams URL with a specified channel. Teams' messages can be squashed.
To create an Incoming Webhook, follow this guide from Microsoft: Create an Incoming Webhook - Teams
Alert Action
Alert Action will allow the usage of existing Alert Actions. The results of an AME event will be passed to the selected Alert
Action command.
The Alert Action Channel supports parametrization. Static values can be used for parameters, or results can be referenced
using the $<result.field>$ syntax.
See more examples below.
Update and delete a notification targets
To update a notification target, revise the information and press the Save button. To delete a notification target, press
the Delete button.
Examples
Automatically open a Jira Task with Atlassian Jira Issue Alerts
The goal in this example is to create a Jira Task when the status of an AME event is automatically changed.
Step-by-step Configuration
The Atlassian JIRA Issue Alerts add-on for Splunk is used to send events to Jira. The add-on should be appropriately installed and configured.
An alert action can be run manually if all parameters are correctly provided to the sendalerts command. Typically, all params are
listed in the app directory in README/alert_actions.conf.spec. For this, the following parameters are available:
[jira_issue_alert]
param.project = <string> Project. It's a required parameter.
param.summary = <string> Summary. It's a required parameter.
param.issue_type = <string> Issue Type. It's a required parameter. Its default value is Task.
param.priority = <string> Priority. It's a required parameter.
param.description = <string> Description. It's a required parameter.
param.assignee = <string> Assignee.
param.labels = <string> Labels.
param.components = <string> Component/s.
param.additional_field_1_name = <string> Additional field 1 - name.
param.additional_field_1_value = <string> Additional field 1 - value.
param.append_alert_result = <list> Append alert result. Its default value is 1.
The sendalert command takes the alert_action name as the first parameter. All further parameters are found in the spec file.
Note that search results can be accessed with the $result.<field>$ syntax:
| makeresults
| eval summary="foo", description="bar"
| sendalert jira_issue_alert param.priority="Medium" param.project="SCRUMTEST" param.summary="$result.summary$" param.append_alert_result="2" param.description="$result.description"
If the search runs successfully, a notification scheme can be created, e.g., using internal AME fields for parametrization.
| Parameter | Value |
|---|---|
project | SOC |
priority | medium |
summary | $result.ame.event_title$ |
append_alert_result | 1 |
description | EventId=$result.ame._key$ |
Automatically send a Splunk Mobile Alert
The goal in this example is to automatically send a Splunk Mobile Alert when changing the status of an AME event.
Step-by-step Configuration
Splunk Secure Gateway must be appropriately installed and configured.
An alert action can be run manually if all parameters are correctly provided to the sendalerts command. For this alert following parameters are available:
[ssg_mobile_alert]
param.alert_message = <string>
param.alert_recipients = <string>
param.alert_severity = <int> # 0=info, 1=Low, 2=Medium, 3=High, 4=Critical
param.alert_subject = <string>
param.alert.dashboard_toggle = <bool>
# (Optional) Following settings are used if dashboard_toggle is set to true
param.alert.alert_dashboard_id = <url> # The url of the dashboard
param.token_name = <string> # (Optional) Dashboard token
param.result_fieldname = <string> # (Optional ) Dashboard result fieldname
# (Optional) The following settings are used for alert calls to action
parameters.alert_call_to_action_label = <string>
param.alert_call_to_action_url = <url>
With this information, a search command can be created to validate all params. The sendalert command takes the alert_action nameas
the first parameter. All further parameters are found in the spec file. Note that search results can be accessed with the
$result.<field>$ syntax:
| makeresults
| eval summary="foo", description="foobarbaz"
| sendalert ssg_mobile_alert param.alert_message="foo" param.alert_recipients="admin" param.alert_subject="bar" param.dashboard_toggle="0" param.alert_severity="4"
If the search succeeds, configure a notification scheme using internal AME fields for parameterization:
| Parameter | Value |
|---|---|
alert_subject | "AME Alert - Status $result.ame.status$" |
alert_message | $result.ame.event_title$ |
alert_recipients | admin |
alert_severity | 0 |
dashboard_toggle | 1 |
alert_dashboard_id | https://127.0.0.1:8089/servicesNS/nobody/alert_manager_enterprise/data/ui/views/splunk_mobile_event_summary |
result_fieldname | ame._key |
token_name | event_id_token |
If an assignee is set, use $result.ame.assignee$ to dynamically set alert_recipients.
When triggered, the notification appears in the Splunk Mobile Event Summary:
Notification Templates
Notification Templates define the format of AME notifications:
AME includes a Template Library for all triggers and targets. Access it by clicking the Arrow Down button next to Add Notification Target:
To use a library template:
- Name the template.
- Select the target type.
- Specify the flow type (trigger).
To create custom templates (requires subscription), click Add Notification Target to open a modal:
- Name the template.
- Select the target type.
- Specify the flow type (trigger).
- Enter the template in text and, if available, structured format.
Templates use Jinja templating. Refer to Jinja Docs for syntax. Examine existing templates for examples of looping through updates.
Available substitutions (fields marked optional may be unavailable):
All Contexts
| Field | Description |
|---|---|
actor | User invoking the action |
ame_host | Hostname |
ame_link | Link to instance/app |
Event Update Context
| Field | Description |
|---|---|
actor | User invoking the action |
ame._index | Event index in the database |
ame._key | Unique event identifier |
ame.assignee | User assigned to the event |
ame.count | Event count |
ame.event_title | Event title |
ame.event_ttl | Event time-to-live |
ame.first_seen | First event timestamp |
ame.impact | Event impact |
ame.most_recent | Most recent event timestamp |
ame.notifications | Event notifications |
ame.notable_fields | Notable fields |
ame.originQuery.app | Origin query app context (optional) |
ame.originQuery.description | Origin query description (optional) |
ame.originQuery.query_earliest | Origin query earliest time (optional) |
ame.originQuery.query_latest | Origin query latest time (optional) |
ame.originQuery.query_string | Origin query string (optional) |
ame.originQuery.query_view | Origin query view (optional) |
ame.priority | Event priority |
ame.priority_name | Priority name |
ame.resolution | Event resolution (optional) |
ame.resolution_name | Resolution name (optional) |
ame.search_name | Search name |
ame.status | Event status |
ame.status_name | Status name |
ame.tags | Event tags |
ame.template | Event template (optional) |
ame.template_name | Template name (optional) |
ame.tenant_uid | Tenant identifier |
ame.ttl_target | TTL target |
ame.urgency | Event urgency |
ame_host | Hostname |
ame_link | Link to instance/app |
comment | Action comment (if applicable) |
link_to_event | Deep-link to event |
updates | List of updated fields |
Event Updates-Item
| Field | Description |
|---|---|
attribute | Updated field |
new_value | New value |
old_value | Previous value |
Event Assigned Context
| Field | Description |
|---|---|
actor | User invoking the action |
ame._index | Event index in the database |
ame._key | Unique event identifier |
ame.assignee | User assigned to the event |
ame.count | Event count |
ame.event_title | Event title |
ame.event_ttl | Event time-to-live |
ame.first_seen | First event timestamp |
ame.impact | Event impact |
ame.most_recent | Most recent event timestamp |
ame.notifications | Event notifications |
ame.notable_fields | Notable fields |
ame.originQuery.app | Origin query app context (optional) |
ame.originQuery.description | Origin query description (optional) |
ame.originQuery.query_earliest | Origin query earliest time (optional) |
ame.originQuery.query_latest | Origin query latest time (optional) |
ame.originQuery.query_string | Origin query string (optional) |
ame.originQuery.query_view | Origin query view (optional) |
ame.priority | Event priority |
ame.priority_name | Priority name |
ame.resolution | Event resolution (optional) |
ame.resolution_name | Resolution name (optional) |
ame.search_name | Search name |
ame.status | Event status |
ame.status_name | Status name |
ame.tags | Event tags |
ame.template | Event template (optional) |
ame.template_name | Template name (optional) |
ame.tenant_uid | Tenant identifier |
ame.ttl_target | TTL target |
ame.urgency | Event urgency |
ame_host | Hostname |
ame_link | Link to instance/app |
assignee | New assigned user |
link_to_event | Deep-link to event |
Event Appended Context
| Field | Description |
|---|---|
actor | User invoking the action |
ame._index | Event index in the database |
ame._key | Unique event identifier |
ame.assignee | Assigned user |
ame.count | Event count |
ame.event_title | Event title |
ame.event_ttl | Event time-to-live |
ame.first_seen | First event timestamp |
ame.impact | Event impact |
ame.most_recent | Most recent event timestamp |
ame.notifications | Event notifications |
ame.notable_fields | Notable fields |
ame.originQuery.app | Origin query app context (optional) |
ame.originQuery.description | Origin query description (optional) |
ame.originQuery.query_earliest | Origin query earliest time (optional) |
ame.originQuery.query_latest | Origin query latest time (optional) |
ame.originQuery.query_string | Origin query string (optional) |
ame.originQuery.query_view | Origin query view (optional) |
ame.priority | Event priority |
ame.priority_name | Priority name |
ame.resolution | Event resolution (optional) |
ame.resolution_name | Resolution name (optional) |
ame.search_name | Search name |
ame.status | Event status |
ame.status_name | Status name |
ame.tags | Event tags |
ame.template | Event template (optional) |
ame.template_name | Template name (optional) |
ame.tenant_uid | Tenant identifier |
ame.ttl_target | TTL target |
ame.urgency | Event urgency |
ame_host | Hostname |
ame_link | Link to instance/app |
count | New count |
link_to_event | Deep-link to event |