Overview

AME Notifications alert users or external systems about new events or event changes in Alert Manager Enterprise (AME).

Notification Schemes specify the criteria for triggering notifications and the target channels they are sent to.

The Notifications UI is shown below:

Managing Notifications

info

See Role Overview for permissions required to manage notifications.

Use these buttons to manage notifications:

Button	Function
	Add Notification Scheme
	Clone Notification Scheme
	Save Notification Scheme
	Edit Custom Tag
	Add Flow
	Add Notification Target
	Add Template Target

Filter by notification name using the search field or by Notification Scheme using the dropdown.

info

Only admins or users with the ame.admin role can access this page and its features.

Setting Up a Notification Scheme

To create a notification scheme for all AME-managed events:

1. Click Add Notification Scheme at the bottom of the list to open a modal:

2. Assign a unique name to the notification scheme.
3. Click Add Flow.
4. Give the flow a unique name.
5. Select a trigger.
6. Add a trigger condition (requires subscription).
7. Configure a notification target and notification template for the flow:

Triggers

Available notification triggers include:

• Event created
• Event updated
• Event assigned
• Event appended
• Event commented
• Event deleted
• Rule matched
• Event violated SLA
• Event fulfilled SLA
• SLA Violation Imminent
• Bulk update
• Bulk delete
• Bulk comment
• Direct Search Data Trigger
• Risk Events Change

Notification Alert Action

Notification Alert Actions can be used to trigger a notification through a search, without creating an Event. Use the Direct Search Data Trigger.

Trigger Conditions

Trigger conditions function similarly to rules:

For these triggers, event changes can serve as notification conditions:

• Event updated
• Rule matched
• Bulk update

Available fields for trigger conditions on source:

Field	Type	Description	Examples
changes	String	Fields that changed	status_name, assignee
values	String	Values that changed	new, admin
keyword	String	Keyword set in a rule	rule_number_one

Example:

Let's assume we want to notify only on status updates for in_progress and resolved statuses. In pseudo code the condition looks like this:

(changed contains “ame.status_name” ) AND (values contains “resolved” OR values contains “in_progress")

Note that we use contains as changes and values are lists. Using the UI it looks like this:

The available fields can be shown by clicking on the Trigger Condition Scheme button.

Limitations for Community Edition

Following Trigger Conditions are available for the Community Edition:

Trigger	Conditions
Event created	none
Event assigned	none
Event updated	ame.status_name
Event appended	ame.status_name
BulkUpdate	ame.status_name

Update and delete a notification scheme

To update a notification scheme, revise the information and press the Save button. To delete a notification scheme, press the Delete button button.

Notification Targets

The Notification Target tab is used to configure where notifications are sent to.

Currently, AME Supports the following targets:

• Mail
• Slack
• Webhook
• MS Teams
• Alert Action

The Mail target is already preconfigured. Additional targets can be added by pressing the Add Notification Target button.

A modal will be opened.

Depending on the target, different configuration options are available. For some targets, a Squash Updates selector is available. Enabling this option will combine multiple notifications into one notification.

Mail

Mail sends an email to the recipients, as well as the cc and bcc entries. You can choose a user name from the dropdown or enter a valid email address by hand.

info

The keyword assignee will translate into the assignee's email if it is available to Splunk.

Slack

Slack sends a Slack message to Slack. The channel supports Slack App notifications. Enter the URL and if messages should be squashed.

Webhook

Webhook will send a POST request to a specified URL. Additionally, Headers can be configured.

info

To create an Incoming Webhook, follow this guide from Slack: Sending messages using incoming webhooks

Teams

Teams will send an MS Teams message to an MS Teams URL with a specified channel. Teams' messages can be squashed.

info

To create an Incoming Webhook, follow this guide from Microsoft: Create an Incoming Webhook - Teams

Alert Action

Alert Action will allow the usage of existing Alert Actions. The results of an AME event will be passed to the selected Alert Action command.

The Alert Action Channel supports parametrization. Static values can be used for parameters, or results can be referenced using the $<result.field>$ syntax.

See more examples below.

Update and delete a notification targets

To update a notification target, revise the information and press the Save button. To delete a notification target, press the Delete button.

Examples

Automatically open a Jira Task with Atlassian Jira Issue Alerts

The goal in this example is to create a Jira Task when the status of an AME event is automatically changed.

Step-by-step Configuration

Prerequisites

The Atlassian JIRA Issue Alerts add-on for Splunk is used to send events to Jira. The add-on should be appropriately installed and configured.

An alert action can be run manually if all parameters are correctly provided to the sendalerts command. Typically, all params are listed in the app directory in README/alert_actions.conf.spec. For this, the following parameters are available:

[jira_issue_alert]
param.project = <string> Project. It's a required parameter.
param.summary = <string> Summary. It's a required parameter.
param.issue_type = <string> Issue Type. It's a required parameter. Its default value is Task.
param.priority = <string> Priority. It's a required parameter.
param.description = <string> Description. It's a required parameter.
param.assignee = <string> Assignee.
param.labels = <string> Labels.
param.components = <string> Component/s.
param.additional_field_1_name = <string> Additional field  1 - name.
param.additional_field_1_value = <string> Additional field  1 - value.
param.append_alert_result = <list> Append alert result.  Its default value is 1.

The sendalert command takes the alert_action name as the first parameter. All further parameters are found in the spec file. Note that search results can be accessed with the $result.<field>$ syntax:

| makeresults 
| eval summary="foo", description="bar" 
| sendalert jira_issue_alert param.priority="Medium" param.project="SCRUMTEST" param.summary="$result.summary$" param.append_alert_result="2" param.description="$result.description"

If the search runs successfully, a notification scheme can be created, e.g., using internal AME fields for parametrization.

Parameter	Value
project	SOC
priority	medium
summary	$result.ame.event_title$
append_alert_result	1
description	EventId=$result.ame._key$

Automatically send a Splunk Mobile Alert

The goal in this example is to automatically send a Splunk Mobile Alert when changing the status of an AME event.

Step-by-step Configuration

Prerequisites

Splunk Secure Gateway must be appropriately installed and configured.

An alert action can be run manually if all parameters are correctly provided to the sendalerts command. For this alert following parameters are available:

[ssg_mobile_alert]
param.alert_message = <string>
param.alert_recipients = <string>
param.alert_severity = <int> # 0=info, 1=Low, 2=Medium, 3=High, 4=Critical
param.alert_subject = <string>
param.alert.dashboard_toggle = <bool>

# (Optional) Following settings are used if dashboard_toggle is set to true
param.alert.alert_dashboard_id = <url> # The url of the dashboard
param.token_name = <string> # (Optional) Dashboard token
param.result_fieldname = <string> # (Optional ) Dashboard result fieldname

# (Optional) The following settings are used for alert calls to action
parameters.alert_call_to_action_label = <string>
param.alert_call_to_action_url = <url>

With this information, a search command can be created to validate all params. The sendalert command takes the alert_action nameas the first parameter. All further parameters are found in the spec file. Note that search results can be accessed with the $result.<field>$ syntax:

| makeresults 
| eval summary="foo", description="foobarbaz" 
| sendalert ssg_mobile_alert param.alert_message="foo" param.alert_recipients="admin" param.alert_subject="bar" param.dashboard_toggle="0" param.alert_severity="4"

If the search succeeds, configure a notification scheme using internal AME fields for parameterization:

Parameter	Value
alert_subject	"AME Alert - Status $result.ame.status$"
alert_message	$result.ame.event_title$
alert_recipients	admin
alert_severity	0
dashboard_toggle	1
alert_dashboard_id	https://127.0.0.1:8089/servicesNS/nobody/alert_manager_enterprise/data/ui/views/splunk_mobile_event_summary
result_fieldname	ame._key
token_name	event_id_token

info

If an assignee is set, use $result.ame.assignee$ to dynamically set alert_recipients.

When triggered, the notification appears in the Splunk Mobile Event Summary:

Notification Templates

Notification Templates define the format of AME notifications:

AME includes a Template Library for all triggers and targets. Access it by clicking the Arrow Down button next to Add Notification Target:

To use a library template:

1. Name the template.
2. Select the target type.
3. Specify the flow type (trigger).

To create custom templates (requires subscription), click Add Notification Target to open a modal:

1. Name the template.
2. Select the target type.
3. Specify the flow type (trigger).
4. Enter the template in text and, if available, structured format.

Templates use Jinja templating. Refer to Jinja Docs for syntax. Examine existing templates for examples of looping through updates.

Available substitutions (fields marked optional may be unavailable):