Notifications
AME Notifications are helpful when users or other systems need to be informed about new events or changes for events.
Notification Schemes define when a notification should be triggered by which criteria and to which targets (channels) they are sent.
The following image shows the Notifications UI:
Managing Notifications
See Role Overview for capabilities required to manage notifications.
Use the following buttons to manage notifications:
| Button | Function |
|---|---|
 | Add Notification Scheme |
 | Clone Notification Scheme |
 | Save Notification Scheme |
 | Edit custom tag |
 | Add Flow |
 | Add Notification Target |
 | Add Template Target |
The search field can be used to filter by notification name. The dropdown allows filtering by a Notification Scheme.
Only an admin or a user with the role ame.admin can see this page
and use its features.
Set up a notification scheme
To create a notifications scheme that will apply to all events managed by AME:
Click the
Create Notification Schemebutton at the bottom of the list. A Modal will open:
Give the notification a unique name.
Click on
Add flow.Give the flow a unique name.
Select a trigger
Add a trigger condition (subscription required)
Add a configuration to add a notification target and notification template to the flow.
Triggers
The following notification triggers are available:
- Event created
- Event updated
- Event assigned
- Event appended
- Event commented
- Event deleted
- Bulk update
- Bulk delete
Trigger Conditions
Trigger conditions work similarly as in rules.
Additionally, for the following triggers, changes to the event can be used for notification conditions:
- Event updated
- Rule matched
- Bulk update
The following fields are available for Trigger Conditions on source.
| Field | Type | Description | Examples |
|---|---|---|---|
| changes | String | The fields that have changed | status_name, assignee |
| values | String | The values that have changed | new, admin |
| keyword | String | The keyword set in a rule | rule_number_one |
Example:
Let's assume we want to notify only on status updates for in_progress and resolved statuses. In pseudo code the condition looks
like this:
(changed contains “ame.status_name” ) AND (values contains “resolved” OR values contains “in_progress")
Note that we use contains as changes and values are lists. Using the UI it looks like this:
The available fields can be shown by clicking on the Trigger Condition Scheme button.
Limitations for Community Edition
Following Trigger Conditions are available for the Community Edition:
| Trigger | Conditions |
|---|---|
| Event created | none |
| Event assigned | none |
| Event updated | ame.status_name |
| Event appended | ame.status_name |
| BulkUpdate | ame.status_name |
Update and delete a notification scheme
To update a notification scheme, revise the information and press the Save button. To delete a notification scheme,
press the Delete button button.
Notification Targets
The Notification Target tab is used to configure where notifications are sent to.
Currently, AME Supports the following targets:
- Slack
- Webhook
- MS Teams
- Alert Action
The Mail target is already preconfigured. Additional targets can be added by pressing the Add Notification Target button.
A modal will be opened.
Depending on the target, different configuration options are available. For some targets, a Squash Updates selector is available.
Enabling this option will combine multiple notifications into one notification.
Mail
Mail sends an email to the recipients, as well as the cc and bcc entries. You can choose a user name from the dropdown or enter
a valid email address by hand.
The keyword assignee will translate into the assignee's email if it is available to Splunk.
Slack
Slack sends a Slack message to Slack. The channel supports Slack App notifications. Enter the URL and if messages should be squashed.
Webhook
Webhook will send a POST request to a specified URL. Additionally, Headers can be configured.
To create an Incoming Webhook, follow this guide from Slack: Sending messages using incoming webhooks
Teams
Teams will send an MS Teams message to an MS Teams URL with a specified channel. Teams' messages can be squashed.
To create an Incoming Webhook, follow this guide from Microsoft: Create an Incoming Webhook - Teams
Alert Action
Alert Action will allow the usage of existing Alert Actions. The results of an AME event will be passed to the selected Alert
Action command.
The Alert Action Channel supports parametrization. Static values can be used for parameters, or results can be referenced
using the $<result.field>$ syntax.
See more examples below.
Update and delete a notification targets
To update a notification target, revise the information and press the Save button. To delete a notification target, press
the Delete button.
Examples
Automatically open a Jira Task with Atlassian Jira Issue Alerts
The goal in this example is to create a Jira Task when the status of an AME event is automatically changed.
Step-by-step Configuration
The Atlassian JIRA Issue Alerts add-on for Splunk is used to send events to Jira. The add-on should be appropriately installed and configured.
An alert action can be run manually if all parameters are correctly provided to the sendalerts command. Typically, all params are
listed in the app directory in README/alert_actions.conf.spec. For this, the following parameters are available:
[jira_issue_alert]
param.project = <string> Project. It's a required parameter.
param.summary = <string> Summary. It's a required parameter.
param.issue_type = <string> Issue Type. It's a required parameter. Its default value is Task.
param.priority = <string> Priority. It's a required parameter.
param.description = <string> Description. It's a required parameter.
param.assignee = <string> Assignee.
param.labels = <string> Labels.
param.components = <string> Component/s.
param.additional_field_1_name = <string> Additional field 1 - name.
param.additional_field_1_value = <string> Additional field 1 - value.
param.append_alert_result = <list> Append alert result. Its default value is 1.
The sendalert command takes the alert_action name as the first parameter. All further parameters are found in the spec file.
Note that search results can be accessed with the $result.<field>$ syntax:
| makeresults
| eval summary="foo", description="bar"
| sendalert jira_issue_alert param.priority="Medium" param.project="SCRUMTEST" param.summary="$result.summary$" param.append_alert_result="2" param.description="$result.description"
If the search runs successfully, a notification scheme can be created, e.g., using internal AME fields for parametrization.
| Parameter | Value |
|---|---|
project | SOC |
priority | medium |
summary | $result.ame.event_title$ |
append_alert_result | 1 |
description | EventId=$result.ame._key$ |
Automatically send a Splunk Mobile Alert
The goal in this example is to automatically send a Splunk Mobile Alert when changing the status of an AME event.
Step-by-step Configuration
Splunk Secure Gateway must be appropriately installed and configured.
An alert action can be run manually if all parameters are correctly provided to the sendalerts command. For this alert following parameters are available:
[ssg_mobile_alert]
param.alert_message = <string>
param.alert_recipients = <string>
param.alert_severity = <int> # 0=info, 1=Low, 2=Medium, 3=High, 4=Critical
param.alert_subject = <string>
param.alert.dashboard_toggle = <bool>
# (Optional) Following settings are used if dashboard_toggle is set to true
param.alert.alert_dashboard_id = <url> # The url of the dashboard
param.token_name = <string> # (Optional) Dashboard token
param.result_fieldname = <string> # (Optional ) Dashboard result fieldname
# (Optional) The following settings are used for alert calls to action
parameters.alert_call_to_action_label = <string>
param.alert_call_to_action_url = <url>
With this information, a search command can be created to validate all params. The sendalert command takes the alert_action nameas
the first parameter. All further parameters are found in the spec file. Note that search results can be accessed with the
$result.<field>$ syntax:
| makeresults
| eval summary="foo", description="foobarbaz"
| sendalert ssg_mobile_alert param.alert_message="foo" param.alert_recipients="admin" param.alert_subject="bar" param.dashboard_toggle="0" param.alert_severity="4"
If the search runs successfully, a notification scheme can be created, e.g., using internal AME fields for parametrization.
| Parameter | Value |
|---|---|
alert_subject | "AME Alert - Status $result.ame.status$" |
alert_message | $result.ame.event_title$ |
alert_recipients | admin |
alert_severity | 0 |
dashboard_toggle | 1 |
alert_dashboard_id | https://127.0.0.1:8089/servicesNS/nobody/alert_manager_enterprise/data/ui/views/splunk_mobile_event_summary |
result_fieldname | ame._key |
token_name | event_id_token |
If an assignee is set you can set the alert_recipients dynamically with $result.ame.assignee$
When the notification is triggered, the Splunk Mobile Event Summary will be shown:
Notification Templates
Notification Templates specify the format of notifications. AME provides.
AME comes with a Template Library for all triggers as well as all targets.
Using the arrow down button on the right of the Add Notification Target, the template library modal will be opened:
- Give the template a name
- Select the target type for the template
- Add a flow type (trigger) for which the template should be used.
It is also possible to create your own templates (subscription required).
Click on Add Notification Target and a modal will open.
- Give the template a name
- Select the target type for the template
- Add a flow type (trigger) for which the template should be used.
- Add the template in text and (where available) structured format.
The template uses Jinja templating. See Jijna Docs for syntax.
See the existing templates on how to loop through updates.
The following substitutions are available:
All Contexts
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
Event Update Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
comment | Comment on the action that caused the change (if applicable) |
updates | List of fields that were updated |
Event Updates-Item
| Field | Description |
|---|---|
attribute | Field that was updated |
old_value | Old value |
new_value | New value |
Event Assigned Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
assignee | New assigned user |
Event Appended Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
count | New count |
Event Commented Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
comments | List of new comments |
Event Comment-Item
| Field | Description |
|---|---|
text | Comment |
actor | User |
timestamp_formatted | Timestamp of the comment |
Rule Matched Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
keyword | Defined keyword |
message | Defined message |
rule_name | Rule that was matched |
Event Deleted Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
Event Created Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
search_name | Name of the search that created the event |
fields | Alert - fields |
To access event fields, use following Syntax: {{ alert_data.<field_name> or 'n/a' }}
Bulk Update Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
event_count | Number of events that were affected |
updates | List of events that were updated |
Bulk Updates-Item
| Field | Description |
|---|---|
attribute | Field that was updated |
new_value | New value |
Bulk Delete Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
event_count | Number of events that were affected |
Additional Single-Event Notification Contexts
| Field | Description |
|---|---|
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
comment | Comment on the action that caused the change (if applicable) |
SLA Fulfilled Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
keyword | Defined keyword |
message | Defined message |
sla_name | Name of the SLA |
sla_start_time | Start time of the SLA |
sla_window_end | Window end time of the SLA |
sla_fulfilled_time_end | Fufilled time of the SLA |
SLA Violated Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
keyword | Defined keyword |
message | Defined message |
sla_name | Name of the SLA |
sla_start_time | Start time of the SLA |
sla_window_end | Windows end time of the SLA |
sla_violation_time | Violation time of the SLA |
SLA Violation Imminent Context
| Field | Description |
|---|---|
actor | User that invoked the action |
ame_host | Hostname |
ame_link | Link to instance / app |
ame._key | Unique identifier for the event |
ame._index | Index of the event in the database |
ame.event_title | Title of the event |
ame.tenant_uid | Tenant identifier |
ame.impact | Impact of the event |
ame.urgency | Urgency of the event |
ame.priority | Priority of the event |
ame.assignee | User assigned to the event |
ame.search_name | Search name of the event |
ame.count | Number of events |
ame.status | Status of the event |
ame.notifications | Notifications for the event |
ame.resolution | Resolution of the event (optional) |
ame.tags | Tags for the event |
ame.notable_fields | Notable fields for the event |
ame.first_seen | Timestamp of the first event |
ame.most_recent | Timestamp of the most recent event |
ame.event_ttl | Time-to-live for the event |
ame.ttl_target | Target for the time-to-live |
ame.template | Template for the event (optional) |
ame.priority_name | Name of the priority |
ame.status_name | Name of the status |
ame.template_name | Name of the template (optional) |
ame.resolution_name | Name of the resolution (optional) |
link_to_event | Deep-Link to the event |
sla_name | Name of the SLA |
sla_start_time | Start time of the SLA |
sla_window_end | Windows end time of the SLA |
sla_violation_in_real_time | Violation real time of the SLA |
sla_violation_in_service_time | Violation service time of the SLA |