Skip to main content
Version: Next

Working with Events

This section explains how to manage events in Alert Manager Enterprise (AME).

Assigning an Event

To assign an event to a user:

  1. Click the assignee button on the right side of the summary panel.
  2. Select a user from the dropdown.

If a user’s Full Name is configured, it appears in the dropdown and as a tooltip in the Assignee column.

note

Only users with the tenant’s power user role appear as options.

Updating the Notification Scheme

To update an event’s notification scheme:

  1. Click the notifications button on the right side of the summary panel.
  2. Choose a scheme to apply when the event’s status changes.
note

Only schemes accessible to the current power user for the event’s tenant are displayed.

Updating the Status

To change an event’s status:

  1. Click the status button on the right side of the summary panel.
  2. Select the new status.
info

Status changes may trigger notifications if defined in the notification scheme.

  1. Add a comment if the update requires explanation.
note

Comments are indexed as ame-audit-record entries and quoted in notifications.

Updating the Urgency

To modify an event’s urgency:

  1. Click the urgency button on the right side of the summary panel.
  2. Select the desired urgency level.
note

Urgency updates generate an ame-audit-record and may trigger notifications.

To re-run an event’s original search:

  1. Click the search button on the right side of the summary panel.
note

Results may be inaccessible if the current user lacks permissions for the original index.

Bulk Updating Events

Select or deselect multiple events with these buttons:

ButtonFunction
Edit/Delete
Reset Selection

To update multiple events simultaneously:

  1. Check the boxes next to the events, then click Edit/Delete to open the menu:
  1. Choose an action (edit or comment) from the menu.
  2. Review the number of affected events by tenant, adjust attributes or add a comment, then confirm:
danger

Only users belonging to the event’s tenant can be assigned; assignments to non-tenant users are skipped.

Editing Event Tags

To edit an event’s tags:

  1. Select Edit Tags from the Actions menu:
  1. Select or deselect tags and optionally add a comment:
  1. Verify a tag by clicking it or remove it using the bin icon in the tag description:

Adding a Comment

To comment on an event:

  1. Expand the event and open the Comments tab:
  1. Use these buttons to edit, preview, or save the comment:
ButtonFunction
Edit Raw Text
Preview Formatted Text
Save Comment

Comments support Markdown formatting. Preview them with the Preview Formatted Text button:

Press Enter for a new line, or use Ctrl+Enter or Save Comment to save. Comments appear once indexed:

danger

Indexed comments cannot be edited.

Invoking a Workflow Action

To run a Splunk Workflow Action on an event result:

  1. Expand the event and locate the action under the Notable Fields or Data tab.
  2. Click the Run Workflow Action button:
ButtonFunction
Run Workflow Action
  • Notable Fields tab example:
  • Data tab example:
  1. Select the desired action from the dropdown:

Workflow Actions are available if all required fields are present.

info

AME ignores the Show action in setting, applying all matching Workflow Actions.

note

On Splunk Cloud, only HTTPS-based Workflow Actions are permitted; HTTP actions are filtered out.

Examples

Example: Webhook Workflow Action

Following is an example of a Workflow action that triggers the built-in Splunk Webhook Action manually.

The search uses makeresults and eval to pass the tenant's name ame.tenant_name and a result-field action to the send alert command. Depending on the alert action, various parameters can be set. In this case, the destination web server URL is specified.

Full search command:

| makeresults 1  
| eval action=$action$, ame_tenant="$ame.tenant_name$"
| sendalert webhook param.url="https://webserver:10000

When running the workflow action from the AME dropdown, the web server will receive a request as follows:

POST / HTTP/1.1
Accept-Encoding: identity
Content-Length: 226
Host: localhost:10000
Content-Type: application/json
User-Agent: Splunk/D1F702EF-402A-4659-A582-C88B3D7380EC
Connection: close

{"sid": "1679403559.11097", "search_name":" ", "app": "search", "owner": "admin", "results_link": "https://<mysplunkserver>:8000/app/search/@go?sid=1679403559.11097", "result": {"_time": "1679403559", "ame_tenant": "Default Tenant"}

Filtering Workflow Actions

Workflow Actions are shown in AME when all fields in the Apply only to the following fields settings that match. The list of Workflow Actions can become very long, and not all actions are helpful for an event.

It is possible to filter Workflow Actions further down using the name of the saved search, or by template.

The following attributes can be set in workflow_actions.conf:

[<workflowaction>]
ame_templates = [template_name|template_uid](,[template_name|template_uid])+
ame_savedsearches = savedsearch_name(,savedsearch_name)+

Examples

Example: Show workflowaction1 only to the `mytest search1` and `mytest search2`
[workflowaction1]
ame_savedsearches = "mytest search1", "mytest search2"
Example: Hide `workflowaction2` for all events
[workflowaction2]
ame_template = ThisTemplateDoesNotExist

Manually Creating an Event

Sometimes, you may need to add an event not generated by a Splunk Alert. Use these buttons in the top-left corner of the Event Summary:

ButtonFunction
Create Event
Test Query

Click Create Event to open a modal:

Enter these fields (mandatory fields marked with *):

  • Title* - Event title
  • Tenant* - Destination tenant
  • Impact* - Event impact
  • Urgency* - Event urgency
  • Assignee* - Event assignee
  • Status* - Initial status
  • Notifications* - Notification scheme
  • Tags - Comma-separated tags
  • Notable Fields - Comma-separated fields
  • Earliest - Earliest search time
  • Latest - Latest search time
  • Query - Search query for re-runs
  • Fields - Key-value field list

Test the query by clicking Test Query (opens in a new tab), then click Submit to create the event:

Managing SLAs

Events can have SLAs assigned automatically or manually. View them in the SLA tab:

Add SLAs with this button:

ButtonFunction
Activate SLA
info

Requires an AME subscription.