SPLK-3002 Correlation and Multi KPI Searches

Correlation and Multi KPI Searches Detailed Explanation

1. Purpose of Correlation and Multi-KPI Searches

What Is Correlation in ITSI?

Correlation in ITSI means combining multiple KPIs or data sources to identify complex problems or patterns of failure.

Why is this useful?

Because in real-life IT environments, issues rarely show up in one place:

• A single spike in CPU might be harmless

• But a spike in CPU + memory + error rate might indicate a serious issue

By correlating KPIs, you can:

• Spot root causes faster

• Reduce false positives

• Trigger smarter, more accurate alerts

2. What Are Multi-KPI Searches?

A Multi-KPI Search is a custom query that:

• Looks at several KPIs at once

• Applies logic or math to their values

• Triggers an action if the combined condition is met

Example:

“If CPU usage is over 90% and Memory usage is over 85% → raise an alert.”

This type of condition can’t be detected by a single KPI—it requires combining metrics.

3. Correlation Search Features

Correlation searches in ITSI are built using SPL (Search Processing Language)—the same language used in Splunk.

Here’s what they can do:

a. Use Logical or Mathematical Conditions

• Combine KPIs using AND, OR, >, <, etc.

• Example:

  | search CPU > 90 AND Memory > 85
  

b. Compare Across Services or Entities

You can include KPIs from:

• Multiple services (e.g., API + Database)

• Multiple hosts (e.g., Server A + Server B)

This helps you see the full system behavior, not just part of it.

c. Apply Time-Based Logic

You can build conditions that check:

• If one KPI spikes before or after another

• If multiple KPIs show issues within the same time window

Example:

“If API error rate increases within 10 minutes of a DB query spike.”

d. Tagging and Thresholding

You can:

• Add tags (e.g., "critical", "performance", "database")

• Define severity levels based on how many KPIs were affected

• Use the results to trigger Notable Events

4. Use Cases for Correlation and Multi-KPI Searches

a. Detecting Cascading Failures

• A problem in one component causes issues in others

• Example:

  • DB query time goes up → App error rate rises → Website slows down

A correlation search helps tie these events together into one meaningful alert.

b. Identifying Behavioral Patterns

• Example:

  • A memory leak causes high memory every day at the same time

  • Correlation detects this pattern across multiple servers

This helps you see trends you might otherwise miss.

c. Enhancing Alert Accuracy

• If you only alert on a single KPI, you may get false positives (normal fluctuations)

• If you alert only when multiple KPIs fail together, alerts become more meaningful

This improves your signal-to-noise ratio.

5. Best Practices

a. Start with Well-Defined KPIs

Before you build a correlation search, make sure your KPIs:

• Are accurate

• Have proper thresholds

• Are actively collecting reliable data

Garbage in = garbage out.

b. Validate Search Performance

Multi-KPI correlation searches can be resource-intensive.

Tips:

• Limit time ranges

• Avoid overly complex logic

• Use summary indexing when needed

This keeps your searches fast and scalable.

c. Test with Historical Incidents

Before going live:

• Run your search against known past outages

• See if it would have detected them

• Tune the logic as needed

This helps you build trust in your correlation rules.

Summary: What to Remember About Correlation and Multi-KPI Searches

• Correlation searches combine multiple KPIs to detect complex issues that can’t be found using a single metric.

• They use logical, mathematical, and time-based conditions to generate smarter alerts.

• Use them to detect cascading failures, behavioral patterns, and enhance alert accuracy.

• Always validate KPIs, test search logic, and optimize for performance.

Correlation and Multi KPI Searches (Additional Content)

1. Clarifying Scope: Correlation Search ≠ Only Multi-KPI

While many correlation searches are built on multiple KPIs, it’s important to understand that correlation searches are not limited to KPI data.

They can also be based on:

• Log patterns

• Metric anomalies

• Event triggers

• External system alerts

This makes correlation searches a general-purpose detection mechanism for any type of Splunk data, not just KPI summaries.

Clarification to include:

Correlation searches in ITSI can include non-KPI-based logic too—such as log patterns or scheduled events—making them a powerful tool beyond KPI analysis alone.

2. SPL Example: Simplified vs Realistic Query

In exam material or early training, correlation logic is often shown in shorthand form like:

“CPU > 90 AND Memory > 85”

However, a real correlation search in ITSI typically uses metrics from the itsi_summary index, often written in full SPL:

| mstats avg(cpu) as cpu_avg avg(mem_used) as mem_avg 
  WHERE index=itsi_summary 
  BY host 
  span=1m
| where cpu_avg > 90 AND mem_avg > 85

Side-by-side illustration:

Abstract Logic	Actual SPL Syntax (ITSI Context)
CPU > 90 AND Memory > 85	`

This helps students translate conceptual logic into executable SPL—a crucial skill for both the exam and practice.

3. How Correlation Searches Trigger Actions

Once you’ve written and enabled a correlation search, you must configure what happens when it matches.

Typical options include:

• Generating Notable Events in Episode Review

• Triggering automated workflows (e.g., through adaptive response actions)

• Initiating external actions, such as:

  • Creating a ServiceNow ticket

  • Sending alerts to Slack, PagerDuty, or email

  • Running a custom script

Suggested exam-ready summary line:

Once a correlation search is created and enabled, it can be set to trigger Notable Events, initiate workflows, or create custom alert actions such as ticket creation in ServiceNow or Slack notifications.

4. Common Misunderstanding: Correlation Rule vs. Multi-KPI Trend View

Some users confuse two very different features in ITSI:

Feature	Purpose
Correlation Search	Alerting logic that generates Notable Events based on defined rules
Multi-KPI Panel (Trends)	Monitoring panel for side-by-side KPI tracking (no alerts generated)

A correlation rule actively detects patterns and triggers alerts. A Multi-KPI view (like in Service Analyzer or a dashboard) is purely observational, allowing analysts to manually compare multiple KPIs over time.

Helpful tip to clarify:

Don’t confuse Multi-KPI visualization panels with correlation searches—the former is for monitoring, the latter is for alerting.

Summary

• Correlation searches combine multiple signals (KPI or otherwise) to detect patterns and trigger actions

• They use SPL logic, which can include metrics, logs, or custom events

• A real-world correlation search often references data in itsi_summary and uses mstats or stats

• Once triggered, they can initiate Notable Events, tickets, or automations

• Be sure to distinguish them from Multi-KPI visualizations, which are for monitoring—not alerting