SPLK-1004 Exploring Alerts

Exploring Alerts Detailed Explanation

1. What are Alerts?

An alert in Splunk is a feature that monitors your data automatically and notifies you when something important happens. Think of alerts as watchdogs that run your saved searches and let you know if a specific condition is true.

Example situations where alerts are useful:

• A server stops sending logs (indicating it's down).

• A spike in 404 errors on a website.

• A user logs in from two distant locations in a short time (possible compromise).

Alerts help you act quickly, often without needing to manually monitor dashboards or run searches yourself.

2. Types of Alerts

Splunk provides two main types of alerts:

a) Scheduled Alerts

• These run at specific time intervals, such as every 5 minutes, every hour, or daily.

• They check historical data over a defined time window.

• Best for routine monitoring, like detecting daily patterns or thresholds.

Example:

Run every hour to check if failed logins > 50 in the last 60 minutes.

b) Real-Time Alerts

• These run continuously and trigger as soon as a condition is met.

• Used when you need instant notification (e.g., security breach, system crash).

• Real-time alerts can be resource-intensive, so use with care.

Example:

Trigger immediately when CPU usage exceeds 95%.

3. Alert Conditions

An alert is triggered when a search result meets specific criteria.

There are two common condition types:

a) Based on Number of Results

Trigger if the number of events returned by the search is greater than 0.

Example:

index=auth_logs action="failed"

If any results are returned, the alert is triggered.

b) Based on Field Value

Trigger if an aggregated value crosses a threshold.

Example:

index=web_logs
| stats avg(response_time) as avg_time
| where avg_time > 1000

This triggers if the average response time exceeds 1000ms.

4. Alert Actions

Once an alert is triggered, it can perform one or more actions. These actions notify users or systems that something has happened.

Common Actions:

• Send Email
  Sends an email to one or more recipients with the alert results.

• Run Script
  Executes a custom script on the server (e.g., restart a service, block an IP).

• Webhook Notification
  Sends data to an external system using a HTTP POST request (often JSON payload).

• Log to Index
  Writes the alert result into a specific Splunk index for later analysis or reporting.

• Output to Lookup
  Stores the result into a lookup table for comparison in future searches.

• Slack / Microsoft Teams
  Integrates with collaboration platforms via webhook to send notifications to a channel.

Example alert action:

If 5 or more failed logins in 5 minutes, send email + log to "security_alerts" index.

5. Throttling and Suppression

Without control, alerts might trigger too often, especially during a burst of activity. This can flood your email or logs.

To avoid that, Splunk provides throttling, which is a way to suppress repeated alerts for a defined period.

Throttling

• Temporarily blocks the same alert from triggering again too soon.

• You can define it per field value (e.g., per user, per host).

Example:

Suppress alerts for the same user for 30 minutes.

This means: once an alert is triggered for user123, no new alert for that user will fire for 30 minutes.

6. Alert Management

Alerts are created and managed in the Splunk UI under:

Settings > Searches, Reports, and Alerts

From there, you can:

• Set permissions (private or shared with a team).

• Define scheduling frequency (how often the alert runs).

• Configure alert actions (email, webhook, etc.).

• Edit alert conditions at any time.

Alerts are stored as saved searches, and you can find/edit them through the "Searches, Reports, and Alerts" page or use | rest searches to inspect configurations.

Summary Table: Alerts Overview

Feature	Description
What is it?	A saved search with a trigger condition and one or more actions
Types	Scheduled, Real-Time
Trigger Conditions	Result count > 0 or field-based threshold
Actions	Email, script, webhook, log to index, Slack/Teams
Throttling	Prevents repeat alerts for same value in a time window
Management Location	Splunk UI → Searches, Reports, and Alerts

Exploring Alerts (Additional Content)

1. Three Types of Trigger Conditions

Splunk alerts can be configured to trigger based on different types of search result conditions. While two types are most commonly known, a third option is often included in SPLK-1004 exam scenarios.

a) Number of Results Greater Than 0

This is the default trigger condition. It simply checks whether the search returns any results.

Example:

index=security sourcetype=syslog action="blocked"

If one or more results are returned, the alert fires.

b) Field-Based Threshold with stats or where

This involves using stats, timechart, or where commands to calculate values and compare them to thresholds.

Example:

... | stats avg(response_time) as avg_resp
| where avg_resp > 3000

Only when the calculated average response time exceeds the threshold will the alert be triggered.

c) Custom Condition Expressions

This is a third, often misunderstood trigger type where you define an expression directly in the alert configuration panel.

Splunk allows you to specify a custom condition such as:

• Number of results > 10

• Custom field value = “error”

• Specific logic combining multiple fields or values

Example:

From the UI trigger condition dropdown, select:

Trigger alert if: Custom condition is met
Custom condition: count > 10

In this case, the search might be:

... | stats count

The alert will only trigger if the count exceeds 10.

Exam Tip: This setting is configured directly in the “Trigger Condition” section of the alert editor and can often be misunderstood as a WHERE clause, but it’s a post-search evaluation of the result.

2. Scheduled Alert – Time Range vs. Alert Window

This is a frequent exam trap. Candidates often confuse these two.

a) Time Range

This defines the period of historical data the search will cover when the alert runs.

Example:
If set to last 15 minutes, the alert will search data from now - 15m to now.

b) Alert Window (Overlapping Searches)

When the alert frequency (cron schedule) is shorter than the search window, Splunk introduces an alert window overlap.

Example Scenario:

• Scheduled to run every 5 minutes

• Search time range is last 10 minutes

This creates a 5-minute overlap between runs.

Why does this matter?

• Helps prevent missed alerts in case of data delays or short-lived spikes

• Especially important for critical monitoring like security breaches or system failures

Exam Focus: You may see questions asking whether an alert is likely to miss a condition if the time window and alert frequency are misaligned.

3. Alert Action – Permissions and System Requirements

Many alert actions require proper permissions or configurations to execute correctly. These often appear as troubleshooting scenarios in the exam.

a) Run a Script

• Requires the script to be predefined and located in $SPLUNK_HOME/bin/scripts/

• The script must exist on the Splunk server and be executable

• Only available in on-premises deployments

b) Send Email

• Requires a configured SMTP server

• User must have the sendemail capability (controlled via role settings)

• Alert results can be included as CSV or inline content

If these conditions are not met, the alert will fail silently or log an error, which can be tested in the UI but is also relevant in scenario-based exam questions.

c) Write to Lookup (outputlookup)

• Requires the user to have the output_file capability

• Without this, the outputlookup command will fail — even if the search runs successfully

This is a common permission-related failure in multi-user environments.

Summary: Key Additions to Remember

Area	Key Point
Trigger Condition – Custom	Use count > x or field=value in the alert editor
Scheduled Alert Logic	Time range defines what is searched; alert window defines overlap
Script Action Requirements	Must predefine and install scripts on Splunk server
Email Alert Dependencies	Needs SMTP configuration and correct role capability
outputlookup Restrictions	output_file capability required to write to lookup files