Observables

Observables in Alert Manager Enterprise (AME) is a powerful feature that collects, manages, and enriches identity and asset data using Splunk Alert Actions. This data is stored in tenant-specific KV Store collections, enabling dynamic enrichment of AME events and Splunk search results.

info

An AME subscription is required to manage more than 100 assets and 100 identities. The free version supports up to 100 of each.

What Are Observables?

Observables allow AME to gather and organize critical data—like user identities (e.g., usernames, roles) and asset details (e.g., device IPs, hostnames)—through a Splunk Alert Action. This data is stored in KV Store collections, separated by tenant, and can be used to add context to security and IT operations, making investigations and event management more effective.

Key Capabilities

• Data Collection: Runs scheduled activities to continueously import and update your asset and identity data into AME, through configurable Alert Actions.
• Tenant-Specific Storage: Stores data in KV Store collections, isolated per tenant for secure management.
• Event Enrichment: Adds context to AME events, like user roles or asset ownership, for better insights.
• Search Enhancement: Integrates with Splunk lookups to enrich your event data for improved accuracy and results.
• Customizable Fields: AME observables support customisable fields, allowing you to bring in any observable metadata that makes sense for your needs

Common Use Cases

• Event Investigation: Enrich AME events with details like "Who owns this device?" or "What’s this user’s role?"
• Event Enrichment: Enhance event handling with added details, such as asset criticality or user department.
• Search Optimization: Improve Splunk search precision by linking observable data to lookup tables.
• Contextual Management: Bring in data from your CMDB or asset management system that makes sense for your environment. For example: upcoming changes or outage windows. Correlate these against your events for automatic management/closure.
• Observable History: Track the presence of an observable accross multiple events, allowing analysts to better investigations and root cause analysis.

Getting Started with Observables

Set up Observables by creating a Splunk search and Alert Action to ingest data, then refine, group, and manage it in AME. Follow these steps:

1. Create a Splunk Search to Collect Observable Data

Write a Splunk search to gather asset or identity data, including required and optional fields for ingestion into AME Observables. Trigger this search with the Ingest Observables Alert Action.

Required and Optional Fields

Field	Required?	Description
uid	Yes	Unique identifier (e.g., server01, user123) to merge data from multiple searches
observable_criticality	No	Criticality level (e.g., high, low) to influence handling in AME; defaults to medium if unset
observable_state	No	State of the observable: existing (default) or removed (deletes it, bypassing age-out)
Other Fields	No	Custom data fields (e.g., ip, name, fqdn); only use letters, numbers, and underscores

index=assetdata sourcetype=cmdb
| eval uid=coalesce(hostname, fqdn, ip)
| table uid hostname ip fqdn

Set this search to trigger the Ingest Observables Alert Action, targeting a tenant’s KV Store collection.

2. Configure the Ingest Observables Alert Action

In Splunk, add the Ingest Observables Alert Action to your search and configure it:

Field	Description
Tenant-UID	Choose the tenant (e.g., Default Tenant) to store data
Observable-Type	Select Asset or Identity to categorize the data
Confidence	Set a value (0–100) to prioritize this source; higher values overwrite existing data (see below)

Confidence Level Explained

Confidence levels allow you to manage conflicts and merge attributes on observables when you have assets or identities appearing in multiple sources. It is not uncommon for an environment to have; CMDB data, LDAP data as well as inventory from vulnerability scanners in Splunk. Using confidence levels you can merge the attributes returned by these lists for an observable, merging the attributes accordingly.

• Range: 0–100 (e.g., 90 for high-confidence sources).
• Behavior: Higher confidence overwrites existing fields. If confidence matches, the new value wins if the source name alphabetically follows the existing one (e.g., SourceB beats SourceA).

Example

• Results for one asset from three different sources:

Search Name	UID	Host	Country	OS	Memory	CPU Cores	IP	Confidence
Network Scanner	server01	server01	USA	(empty)	(empty)	(empty)	192.168.1.10	90
CMDB	server01	server01	(empty)	Windows 11	16GB	8	192.168.1.11	70
Hardware Monitor	server01	server01	Canada	Windows 10	32GB	4	(empty)	50

• Merged Results for Asset uid = "server01":

Field	Value	Confidence	Source	Origin Type
uid	server01	90	Network Scanner	search
host	server01	90	Network Scanner	search
country	USA	90	Network Scanner	search
os	Windows 11	70	CMDB	search
memory	16GB	70	CMDB	search
cpu_cores	8	70	CMDB	search
ip	192.168.1.10	90	Network Scanner	search
observable_priority	high	90	Network Scanner	search
observable_state	existing	90	Network Scanner	search

note

Avoid identical confidence values across sources to prevent unexpected overwrites.

3. Refine Observables

Use the Refinements tab in AME’s Observables menu to tweak ingested data with rules. Refinement rules allow you to transform your data post ingestion, using an easy to use visual interface.

Example Use-Case:

Set a normalised platform for your operating systems. Your analysts are interested on whether a computing resource is running some version of Windows, or some flavour of Unix, such as AIX, HPUX, OpenBSD etc. For your purposuses you want to categorise these into either: Windows on Unix

note

The refinement functionality is intended to transform and contextualise your observables post collection. For pre-ingestion normalisation, refer to the search section above, where you can use the Splunk command pallette, including eval to perform transformations on your data

How to Add a Refinement Rule

1. Go to Observables > Refinements.
2. Click Add Asset Refinement Rule or Add Identity Refinement Rule.
3. Fill in:

Field	Description	Example
Name	Unique rule name.	Standardize OS
Tenant	Target tenant.	default
Description	Optional purpose note.	Set OS Version
Scope	Asset or Identity.	Asset
Confidence	Priority (higher evaluated first).	95
Condition	Logical filter (e.g., os matches Windows Versions)	{{os = "Windows NT 10.0" OR os = "Windows NT 10 Build 22621"}}
Field Set	New field value to apply.	os = Windows 11

Example

Standardize OS values for assets where os is Windows NT 10.0 or Windows NT 10 Build 22621 to Windows 11.

• Input values:

uid	host	os	ip	confidence	source	origin_type
host1	host1	Windows NT 10 Build 22621	192.168.1.20	85	OS Scanner	search
host2	host2	Windows NT 10.0	192.168.1.21	60	Legacy System	search

• Refined Results for Hosts:

uid	host	os	ip	source	origin_type
host1	host1	Windows 11	192.168.1.20	Standardize OS	refinement
host2	host2	Windows 11	192.168.1.21	Standardize OS	refinement

4. Click Recalculate to apply rules instantly.

4. Group Observables

Organize observables into groups via the Groups tab to categorize and enrich data based on common attributes. Observable groups allows you to create additional context for your observables using Names and Descriptions.

Example Use-Case:

• Add all assets that exist in a specific region to a group. Eg: Add all computing resources in Germany to the DE Asset Group
• Set groups for different PCI network zones: For example PCI customers can add groups for Cardholder Data Environment (CDE) zones, DMZ, Corporate and Wireless zones

How to Create a Group

1. Go to Observables > Groups.
2. Click Add Asset Group or Add Identity Group.
3. Define:
   • Name: e.g., DE Assets.
   • Condition: e.g., country = "DE"
4. Define a "catch-all" rule for all others
5. Use Reorder to set priority

6. UsePreview to check the resulting grouping. Note that only the first 1000 assets are shown.

7. Recalculate for immediate updates.

Groups update daily or on recalculation, appearing as Aggregated by observable-group in the UI. For example, grouping assets by country (e.g., USA, Germany) helps organize data for regional analysis.

Example: Group five hosts across three countries by the country field.

• Asset Data for Hosts:

uid	host	country
host1	host1	DE
host2	host2	DE
host3	host3	CH
host4	host4	AT
host5	host5	empty

• Output after group assignment:

observable_group	uid	host	country
DE Assets	host1	host1	DE
DE Assets	host2	host2	DE
CH Assets	host3	host3	CH
AT Assets	host4	host4	AT
Missed Assets	host5	host5	empty

5. Create Reporting Groups

For reporting purposes Observable Groups can be further grouped for reporting purposes.

Reporting groups can be simple groups, but also supports nesting groups to cover more complex reporting requirements.

To create a new Observable Reporting Group click the `Add Asset Observable Reporting Group button. A modal will open:

Select Observables Group and/or Observable Reporting Groups to add them to the new group.

6. Configure Observable Settings

To prevent stale and decommissioned assets from taking up space in your collections you can setup aging rules that will age-out observables if not seen for a period of time. Manage retention in the Configuration tab

Setting	Default	Description
Days Before Age-Out	90	Data expires after 90 days unless re-detected

Adjust this to balance storage and relevance.

7. Explore and Manage Observables

AME Provides a rich graphical interface to manage your observables. You can obtain a quick view of any observable by adding a filter to sort and graph by. Use-cases include, quickly obtaining a view your identities based on a field in either the data or meta data, eg: What percentage of assets are without an owner.

In the Observables menu, use the Overview tab to:

• Filter: Use Observable Type Filter or Add filter.
• Visualize: Add charts with Add chart (toggle with Hide Charts).
• View Details: Click the icon on the right of each row to access in-depth info.

Observables Table Layout

Field	Description
uid	Unique ID (e.g., server01)
first_seen	First detection time
last_seen	Latest sighting
criticality	Severity (e.g., low, high)
observable_group	Assigned group (e.g., US Assets)
risk	Risk score
custom fields	Data fields

8. Use Lookups

The observable data can be used with standard search tooling by means of lookups. AME presents your observable collections as lookups, that can be used with inputlookup and lookup commands.

The naming scheme for obserable lookups is ame_<tenant>_observable_<observable_type>

Following lookup definitions will be created automatically for the default tenant:

• Assets: ame_default_observable_assets
• Identities: ame_default_observable_identities

Supported meta fields for both observable types are as follows:

• _key
• tenant_uid
• type
• uid
• first_seen
• last_seen
• observable_group_name
• criticality
• risk

Data fields can be accessed directly.

Example search

<base search> | lookup ame_default_observable_assets ip as src_ip OUTPUT hostname as host fqdn as host_fqdn observable_group_name as group

info

Lookup definitions will be created and updated automatically when an asset or identity collection is created or modified. This ensures, that all fields are usable in lookups. Updates run daily.

Automatic Lookups

For automatic lookups use the configuration tab to configure input field mappings. The UI will ensure, that the correct configuration is created.

note

For automatic lookups, replication will be enabled by AME. See Splunk Docs for more details

Detailed Observable Information

When you click the icon on the right of a row in the Observables table, you’ll access a detailed view with comprehensive information about the observable. This view includes several sections:

Observable Details, Risk change and Event participation

Following detail information is available for an observable:

• UID: The unique identifier for the observable.
• First Seen: The timestamp of the first detection.
• Last Seen: The timestamp of the most recent sighting.
• Criticality: The severity level of the observable.
• Risk: A numeric risk score associated with the observable.

Following visual trends are availabe:

• Risk Change: A chart showing changes in the risk score over time, helping you track risk trends.
• Event Participation: A chart displaying the observable’s involvement in events over time, aiding in identifying patterns or anomalies.

Following field details are available:

• Field: The name of the data field (e.g., country, ip).
• Value: The value associated with the field.
• Confidence: A numerical confidence level (0–100) indicating the reliability of the data.
• Origin: The source of the data (e.g., search or refinement).
• Origin Type: The type of origin, such as search or refinement.
• First Created: The timestamp when the field was first added.
• Last Updated: The timestamp of the most recent update.
• Action: An option to delete the field (e.g., a button or icon).

info

Fields can be deleted, but may re-appear when new data is ingested

Risk Details

info

For a complete explanation about risk scoring in AME, see Add risk scores

• Occurrence: The timestamp when the risk event occurred.
• Matched Value: The value triggering the risk.
• Risk Change: The change in risk score.
• Related Search: The source or search generating the risk data.

Use the drilldown button to find the contributing AME event.

Events Details

• First Occurrence: The timestamp of the first event occurrence.
• Last Occurrence: The timestamp of the most recent occurrence.
• Total Occurrences: The number of times the event has occurred.
• Risk Change: The change in risk score triggered by the event.
• Event Risk: The risk level of the event.
• Event Title: A description of the event.
• Event Status: The current status of the event.

Use the drilldown button to find the contributing AME event.

Next Steps

• Configure templates to map asset and identity information to alert results (see Templates)
• Use Observables to manage Risks (see Risk Scoring)