SPLK-3002 Introducing ITSI

Introducing ITSI Detailed Explanation

1. Understanding the Basics — What is ITSI?

What does ITSI stand for?

ITSI means IT Service Intelligence. It is a special application built by Splunk. Its goal is to help organizations monitor and understand their IT systems in a smarter, more connected way.

Why is ITSI important?

In a modern company, many systems work together to provide services. For example:

• A website might depend on a web server, a database, and a payment service.

• If one part (like the database) slows down or crashes, the whole website might break.

Traditional monitoring tools often look at just one thing at a time:

• CPU usage

• Memory usage

• Error logs

But they don’t connect the dots to show how everything affects the actual business service. That’s where ITSI is different.

What does ITSI do?

Think of ITSI as a "central brain" that:

• Collects data from many systems

• Analyzes it in real time

• Finds out if something is wrong

• Shows how problems in technical parts affect entire business services like "Online Store" or "Email Platform"

2. The Big Picture — What Makes ITSI Special?

a. Service-Centric Monitoring

ITSI doesn’t just show numbers or graphs. It shows how services are doing.

For example:

• Instead of just showing “Server CPU is 95%”, ITSI shows “Order Checkout Service is in a Critical State because CPU on payment server is too high.”

This helps teams:

• Focus on what matters most

• Fix problems faster

• Avoid business downtime

b. Real-Time Data Analysis

ITSI doesn’t wait hours to analyze logs. It:

• Collects data in real time (as it happens)

• Updates dashboards and alerts immediately

• Finds patterns or problems instantly

This helps your IT team react quickly, sometimes even before users notice a problem.

c. Predictive Intelligence

ITSI can use machine learning to:

• Learn what “normal” performance looks like

• Detect anomalies (sudden strange behavior)

• Warn teams before a system fully fails

This turns ITSI into not just a monitor, but a predictive tool.

3. Core Components – Introduction

ITSI is made of several main parts that work together. These parts are:

1. Services

2. KPIs (Key Performance Indicators)

3. Glass Tables

4. Notable Events

5. Deep Dives

a. Services

What is a "Service" in ITSI?

In everyday life, a service is something that provides a function or result.
For example:

• A food delivery service delivers meals.

• A banking service lets you manage your money.

In ITSI, a service is a collection of IT components (like servers, databases, apps) that work together to provide a business function.

Examples of Services in ITSI:

• “Login System”

• “Search Feature”

• “Checkout Process”

• “Email Notification Service”

These aren’t just servers or tools—they represent the whole function, from end to end.

Why are Services Important?

Let’s say your website goes down. A simple monitoring tool might say:

• "Server X has high CPU usage"

But ITSI goes further. It tells you:

• "The Checkout Service is in trouble because the database response time is too slow, which is caused by Server X being overloaded."

So, ITSI helps you:

• Focus on services, not just raw numbers.

• Understand why something is broken.

• Prioritize what to fix first based on business impact.

What is Inside a Service?

Each service in ITSI has:

1. KPIs (Key Performance Indicators) – to measure how the service is doing.

2. Thresholds – to define what is "Normal", "Warning", or "Critical".

3. Dependencies – connections to other services it relies on.

4. Health Score – a number between 0 and 100 that shows how healthy the service is overall.

Visual Example (Imagine This)

Service: “Order Checkout Service”

Component	KPI Measured	Threshold
Payment API	Response time	Critical if > 2 sec
Database	Query success rate	Warning if < 98%
App Server	Error rate	Critical if > 5%

If any of these KPIs cross their thresholds, the health score of the Checkout Service drops—and you get alerted!

Summary: What to Remember About Services

• A service is a group of KPIs that represent a real business or technical function.

• ITSI uses services to monitor what matters, not just machines.

• Each service gives you a real-time health score based on its KPIs.

• This makes problem-solving faster and more focused.

b. KPIs (Key Performance Indicators)

What is a KPI?

KPI stands for Key Performance Indicator.
In everyday language, a KPI is a measurement that tells you how something is doing.

For example:

• In school, your GPA is a KPI for your academic performance.

• In business, “monthly sales” might be a KPI for success.

In ITSI, a KPI is a metric that tells you how healthy or unhealthy part of a service is.

Why Are KPIs Important in ITSI?

Remember the “Order Checkout Service” example? To know if that service is healthy, you need to measure things like:

• How fast the payment API responds

• Whether database queries are successful

• If there are errors when people try to check out

Each of these measurements is a KPI.

KPIs help you:

• Quantify performance

• Detect problems as they happen

• Trigger alerts when something goes wrong

• Calculate service health scores

How Are KPIs Built?

KPIs in ITSI are built using searches—specifically Splunk searches (SPL) that gather data from logs, metrics, or other sources.

Example: If you want to measure how fast your payment API responds, your KPI search might look like:

index=payments sourcetype=api_logs | stats avg(response_time) as avg_response

This search will return the average response time for your API.

You can then say:

• If avg_response < 1.5 seconds → Normal

• If 1.5–2 seconds → Warning

• If > 2 seconds → Critical

This is called setting thresholds, and we’ll go deeper into that later.

KPI Features in ITSI

1. Base Search
   The main search that pulls in the data (e.g., error count, load time).

2. Thresholds
   Rules that define KPI states like Normal, High, Critical.

3. Split-By Fields
   Allows you to track the same KPI across multiple servers or applications.
   For example: one KPI for "CPU usage", but split by host.

4. Time Policies
   Use different thresholds for different times (e.g., more relaxed at night).

5. Importance Weight
   KPIs can be more or less important when calculating a service’s health score.

Example: KPI for “Login Errors”

Search Field	Example Setting
KPI Name	"Login Error Rate"
Base Search	`index=auth sourcetype=login_logs
Thresholds	Normal: 0–50, Warning: 51–100, Critical: >100
Importance	High (because login problems affect all users)
Schedule	Every 5 minutes

Summary: What to Remember About KPIs

• A KPI is a measurement that monitors part of a service.

• KPIs use searches to gather data in real time.

• They are the building blocks for alerts, health scores, and visual dashboards.

• Every service in ITSI is made up of multiple KPIs.

c. Glass Tables

What Is a Glass Table?

A Glass Table in ITSI is a custom-built dashboard. It shows live data using shapes, colors, icons, and animations—not just charts or graphs.

Think of it like a control center screen in a movie:

• A map that shows system health

• Icons that light up if something goes wrong

• Color-coded indicators for performance

• Real-time movement and changes

In short: It helps you visualize your services and their health in a way that's easy to understand at a glance.

Why Is It Called a "Glass Table"?

Imagine you have a glass tabletop, and under that glass is a picture of your IT infrastructure—servers, applications, data flows.

Then, on the glass, you place live performance data, indicators, and alerts right on top of those systems.
That’s the idea: a transparent, real-time, layered view of your environment.

What Can You Put on a Glass Table?

1. Shapes and Images
   You can draw boxes, circles, arrows, or upload your own background (like a network diagram or data center layout).

2. Icons and Text
   Add symbols (like server icons) and text labels for easy identification.

3. KPI Widgets
   Link any shape or icon to a KPI. The shape will change color based on the KPI’s status:

   • Green = Normal

   • Yellow = Warning

   • Red = Critical

4. Animations
   Arrows or lines can blink or move to show active traffic or issues.

5. Drilldowns
   Click on an element (like a red server icon), and it can:

   • Open a Deep Dive

   • Show a dashboard

   • Run a search

Real-Life Example

Imagine your Glass Table shows the following:

Background Image: A map of your network
Web Server Icon: Linked to KPI "Web Server CPU"
Database Icon: Linked to KPI "DB Query Success Rate"
API Gateway Icon: Linked to KPI "Error Rate"

When something breaks:

• The affected icon turns red

• You can click it to investigate the problem

• The whole dashboard updates live, with no need to refresh

Who Uses Glass Tables?

• IT Operations Teams: To monitor infrastructure in real time

• NOC (Network Operations Center): For big-screen displays

• Executives: To get a high-level view of business service status

You can create different Glass Tables for different audiences. For example:

• Technical version: more details, logs, and metrics

• Executive version: only high-level health indicators and impact

Best Practices for Beginners

• Start simple – Don’t try to draw your entire system at once.

• Use colors carefully – Green/yellow/red works best for quick understanding.

• Test your links – Make sure each icon actually connects to a working KPI.

• Keep it clean – Too many elements can be confusing.

Summary: What to Remember About Glass Tables

• Glass Tables let you build custom dashboards using real-time data.

• You can display data in interactive, visual ways—beyond normal charts.

• They help teams spot problems quickly and understand how they affect services.

• You can click into problems to investigate right away.

d. Notable Events

What Is a Notable Event?

A Notable Event is an alert created by ITSI when something significant happens—usually when a KPI crosses a threshold like “Critical” or “High”.

But it’s more than just a normal alert.

A Notable Event in ITSI is:

• Smart: It includes context, such as which service or server was affected.

• Actionable: You can assign it, acknowledge it, or even use it to trigger a script or create a ticket.

• Grouped and Prioritized: ITSI can group similar events and assign them a severity level.

Think of it as your to-do list for incidents.

When Does a Notable Event Get Created?

There are two main ways:

1. From a KPI Threshold Breach
   Example: If a KPI like “API Error Rate” goes above its critical threshold, a Notable Event is generated.

2. From a Correlation Search
   You can create special searches that detect patterns (like “high CPU + high memory = possible server overload”) and generate an event when that pattern appears.

What’s Inside a Notable Event?

Each Notable Event includes important information such as:

• Time of the event

• Service or KPI that triggered it

• Severity level (Info, Normal, Warning, High, Critical)

• Affected entities (e.g., hostnames, applications)

• Custom fields (like region, environment, team)

• Event status: e.g., new, acknowledged, resolved

• Actions taken or assigned users

This makes it much easier for teams to understand what happened, where, and what to do next.

How Are Notable Events Managed?

You can manage events in the Episode Review Dashboard in ITSI. This is like a control panel where you can:

• Group related events together

• Suppress events during known downtime or maintenance

• Assign ownership to team members

• Tag and filter events for faster searching

• Run automated actions, like sending a Slack message or opening a ServiceNow ticket

Real-Life Example

Let’s say your “Checkout Service” has a KPI called “Payment API Response Time”.
Its thresholds are:

• Normal: < 1.5 sec

• Warning: 1.5–2.5 sec

• Critical: > 2.5 sec

Suddenly, the KPI goes to 3.1 seconds. ITSI will:

• Mark the KPI as Critical

• Trigger a Notable Event

• Display it in the dashboard

• Possibly group it with similar events (like “high error rate”)

• Let your team act on it right away

Suppression (Very Useful for Beginners!)

Let’s say you have scheduled maintenance every Sunday.
You don’t want alerts firing during this time.

You can set up Event Suppression Policies to tell ITSI:

“If this KPI is critical during Sunday from 2 AM to 4 AM, ignore it.”

This helps reduce alert noise and false positives.

Summary: What to Remember About Notable Events

• Notable Events are intelligent alerts created when KPIs or patterns cross a threshold.

• They are detailed, grouped, and actionable.

• You can manage them from a central dashboard and assign them to team members.

• They help IT teams respond faster and reduce downtime.

e. Deep Dives

What Is a Deep Dive?

A Deep Dive in ITSI is an interactive, time-based dashboard that lets you:

• Visually explore how KPIs behaved over time

• Correlate multiple KPI trends together

• Investigate incidents

• Pinpoint root causes

Think of it as your IT microscope—a focused space to explore what went wrong, when it started, and why.

When Do You Use a Deep Dive?

Let’s say you received a Notable Event that says:

“Checkout Service is in a Critical State – API Error Rate is high.”

You’re not sure if the API is the problem, or if something else is causing the issue. So you:

• Open a Deep Dive for the Checkout Service.

• Add relevant KPIs like:

  • API Error Rate

  • Database Query Time

  • Web Server Response Time

• Look at how each one behaved over the past 60 minutes.

Suddenly you see:

• All KPIs looked normal until 2:15 PM

• Then the Database Query Time spiked

• A minute later, API errors began

That tells you the real problem started in the database, and it caused the API to fail. Problem diagnosed!

Features of a Deep Dive

1. Time-Series Graphs
   Each KPI appears as a chart over time. You can see:

   • When the value started to change

   • How severe the change was

   • Whether other KPIs changed at the same time

2. Event Timeline Panel
   This section shows Notable Events and their timestamps, so you can match them to KPI spikes.

3. Zoom & Pan Tools
   You can zoom in on a 5-minute window or scroll across hours to look at the full context.

4. Layout Editor
   You can drag, resize, or arrange KPIs in a way that makes sense for your investigation.

5. Custom Time Ranges
   Deep Dives are not limited to "right now". You can look back:

   • One hour ago

   • Yesterday during an outage

   • Last week during a known spike

Who Uses Deep Dives?

• NOC (Network Operations Center) teams use them to investigate real-time alerts.

• SREs (Site Reliability Engineers) use them for post-incident analysis.

• App owners use them to monitor patterns or test new deployments.

Real-Life Use Case

Imagine this scenario:

• A mobile app team deploys a new version of the login module.

• 30 minutes later, users start reporting login failures.

You open a Deep Dive and add KPIs:

• Login Success Rate

• App Server CPU

• Authentication API Errors

From the Deep Dive, you notice:

• CPU load spiked during deployment

• Error rates increased right after

• Success rate dropped sharply

You now have a clear timeline of what went wrong, which is perfect for:

• Fixing the problem fast

• Documenting the incident

• Preventing it in the future

Summary: What to Remember About Deep Dives

• Deep Dives help you investigate problems over time using KPI data.

• They show how, when, and where an issue started.

• You can compare multiple KPIs, view events, and find the root cause.

• They are essential tools for real-time troubleshooting and post-incident reviews.

Introducing ITSI (Additional Content)

1. Add a Statement: ITSI Is a Splunk Premium App

Why It Matters:

In the SPLK-3002 exam, it is common for questions to test your understanding of whether ITSI is included by default with Splunk or if it is an additional offering. Clarifying this helps prevent confusion.

Suggested Explanation:

Splunk IT Service Intelligence (ITSI) is a premium app, not included by default with a standard Splunk installation.
It requires separate licensing and installation, typically sourced from Splunkbase or through enterprise agreements.

This distinction is critical in enterprise environments and may appear in exam questions that differentiate between core Splunk features and premium modules.

2. Expand KPI Sources: Logs, Metrics, Traces (including OpenTelemetry)

Why It Matters:

KPI sources are evolving. Modern observability strategies go beyond logs and metrics to include traces, especially with growing adoption of OpenTelemetry.

Suggested Explanation:

KPIs in ITSI can be calculated from various data sources, including:

• Logs: Traditional Splunk log data (e.g., from application logs, system logs)

• Metrics: Time-series numerical data (e.g., CPU usage, memory, request counts)

• Traces: Distributed tracing data, especially from modern microservice environments

You can ingest OpenTelemetry-formatted data into Splunk Observability Cloud or through custom collectors in ITSI to construct trace-based KPIs. This integration supports full-stack monitoring.

Understanding the variety of KPI input types prepares you to work in hybrid or cloud-native environments.

3. Clarify Glass Table Data Binding: Real-Time and Historical Support

Why It Matters:

Glass Tables are often associated with real-time dashboards, but they are capable of much more. Many users don’t realize they can also visualize historical trends for deep-dive analysis.

Suggested Explanation:

Glass Tables in ITSI support both real-time and historical data binding, offering flexible visualization for service health, KPIs, and entity states.

You can:

• Bind panels to real-time search results for live monitoring

• Use historical KPI data for retrospective analysis

• Switch between time windows (e.g., last 5 minutes, past 24 hours)

This dual-mode visualization makes Glass Tables suitable for both active troubleshooting and long-term trend analysis.