SPLK-3001 Tuning Correlation Searches

Tuning Correlation Searches Detailed Explanation

1. What Are Correlation Searches?

Correlation searches are scheduled SPL queries used in Splunk ES to detect suspicious activity across multiple data sources. When these searches find matching patterns, they usually trigger a Notable Event, which appears in the Incident Review Dashboard.

They form the core detection logic in Splunk ES. However, if not properly tuned, they can:

• Generate too many alerts (many of them false positives)

• Consume excessive system resources

• Overwhelm analysts and reduce effectiveness

That’s why tuning correlation searches is both critical and ongoing.

2. Why Is Tuning Crucial?

a. Reduce Alert Fatigue

• Overly broad or aggressive searches may result in hundreds or thousands of daily alerts.

• This makes it hard for analysts to identify truly critical events.

b. Improve System Performance

• Complex searches running too frequently can slow down Splunk, especially if they use large data models or expensive operations like joins and subsearches.

• Tuning helps reduce resource load while maintaining alert quality.

3. Tuning Techniques

There are several ways to fine-tune correlation searches for better efficiency and precision.

a. Modify Search Criteria

Adjust the actual SPL to make the search more targeted.

• Narrow time ranges:

  • Avoid searching across unnecessarily large time windows.

  • Example: Change from last 24 hours to last 30 minutes if the alert is meant to be real-time.

• Add specific filters:

  • Include conditions for known threat sources, severity, or asset criticality.

  • Example: Only look for failed logins on production systems or high-privilege accounts.

This helps filter out irrelevant noise.

b. Adjust Scheduling

Correlation searches are run on a scheduled basis using cron expressions.

• Best practice: Avoid running too many high-cost searches at the same time.

• Spread out searches across different hours or minutes.

• Consider how frequently a rule needs to run. For example:

  • Run critical rules every 5 minutes.

  • Run low-risk or broad-spectrum searches hourly or daily.

Additionally:

• Search Window: Defines the time range the search looks at each time it runs.

• Throttle Settings: Prevent repetitive notable events within a set time frame.

Example:

If a user fails to log in 10 times in 10 minutes, only alert once — not 10 times.

This reduces duplication and keeps dashboards clean.

c. Use Suppression Rules

• Suppression rules temporarily hide expected or low-priority alerts.

• Common use case: suppress alerts during known vulnerability scans or penetration tests.

These rules help:

• Avoid distracting analysts with known, safe behaviors.

• Reduce data noise during events like compliance testing.

Suppression can be:

• Time-bound

• Based on specific fields (e.g., source IP or hostname)

4. Risk Scoring Optimization

Instead of triggering a notable event for every suspicious event, consider using Risk-Based Alerting (RBA) techniques.

a. Assign Risk Scores

• Configure correlation searches to assign numeric risk scores to a user or asset.

• Do not immediately generate a notable event.

b. Aggregate Multiple Events

• When several low-risk events occur for the same user or host over time, the total risk score may reach a threshold.

• This then triggers a single, more meaningful notable event.

This reduces false positives while maintaining visibility.

c. Customize Risk Thresholds

• Different types of users or assets should have different thresholds.

  • Example: 10 failed logins on a test server may not be alarming.

  • But 3 failed logins on a CFO’s laptop might be serious.

By fine-tuning thresholds based on:

• Asset role

• Business function

• Risk category

You ensure that alerts are both context-aware and actionable.

5. Summary of Tuning Correlation Searches

Area	Description
Modify Search Criteria	Use filters and tighter time ranges to make searches more relevant
Adjust Scheduling	Spread out queries, set realistic intervals, and apply throttle settings
Suppression Rules	Suppress alerts from known benign activity to reduce noise
Risk Scoring Optimization	Assign risk scores, aggregate events, and trigger alerts only when necessary
Customize Risk Thresholds	Apply thresholds based on asset or user importance

Tuning Correlation Searches (Additional Content)

Throttle Settings – Configuration Example

Throttle settings are used to suppress repetitive Notable Events triggered by the same entity within a short time window, helping to reduce noise and avoid alert fatigue.

Without throttling, correlation searches may generate dozens—or even hundreds—of duplicate alerts for repeated behavior within a short time frame.

Practical Example: Suppress Repeated Alerts by User

Imagine you have a correlation search that triggers a Notable Event for failed login attempts.

If a single user fails to log in 10 times within 30 minutes, you don’t want 10 separate alerts—you want just one.

How to configure this in the Splunk ES UI:

• Throttle Field: user

• Throttle Period: 30m

This configuration tells Splunk:

“If this same user triggers this search again within 30 minutes, don’t alert again.”

This setting is applied in the Correlation Search Editor under:

Configure > Content Management > [Select Search] > Edit > Notable Event Actions

Visual Summary:

Throttle Field	Throttle Period	Outcome
user	30m	Suppresses duplicate Notable Events from same user for 30 mins

This mechanism is especially useful for:

• Repeated login failures

• Port scans from the same IP

• Data exfiltration attempts from the same host

Why It Matters:

• Reduces alert volume and prevents dashboard clutter

• Helps analysts focus on unique and high-value events

• Ensures better incident triage efficiency

Summary of Enhancement

Topic	Enhancement Description
Throttle Settings	Introduces real UI-aligned example (user field throttled over 30m)
Learning Benefit	Makes the abstract concept tangible and memorable for learners/admins