SPLK-3003 Search

Search Detailed Explanation

In Splunk, "search" refers to the process of querying indexed data to retrieve useful information. Users can perform searches using SPL (Search Processing Language) to analyze logs, monitor activity, detect anomalies, and generate dashboards or alerts.

1. Search Types

Splunk supports multiple types of searches, each suited to different use cases.

Ad-hoc Search

• A one-time search entered manually in the Search & Reporting app.

• Used for exploratory data analysis, quick troubleshooting, or manual investigation.

• Not scheduled or saved unless explicitly done by the user.

Example:

index=web error OR fail | stats count by status

Scheduled Search

• Configured to run at regular intervals (e.g., every 5 minutes, hourly, daily).

• Often used in:

  • Reports

  • Alerts

  • Dashboards (panel updates)

• Results can be stored in:

  • Lookup files

  • Summary indexes

  • Report acceleration cache

Example: A search that runs every hour to find failed login attempts and sends an alert if the count exceeds 10.

Real-time Search

• Continuously updates as new events are indexed.

• Ideal for live monitoring, such as:

  • Intrusion detection

  • Application performance

  • Critical system errors

• Uses more system resources than scheduled or ad-hoc searches.

• Can be configured with "rolling windows" (e.g., last 30 seconds).

2. SPL (Search Processing Language)

SPL is the custom query language used in Splunk. It is inspired by the UNIX pipe model, where commands are chained together with the | symbol to process data in stages.

Basic Structure

• Start with a search command (or default to search).

• Pipe the result to one or more processing commands.

Examples

1. Count HTTP 500 errors by host:

index=web status=500 | stats count by host

2. Average outgoing bytes per hour from firewall logs:

index=firewall | timechart span=1h avg(bytes_out)

Common commands include:

• stats: Perform statistical operations (sum, avg, count, etc.)

• timechart: Plot time-series data

• table: Display results in tabular format

• eval: Create calculated fields

• where: Filter events using conditions

3. Search Modes

Search modes control how much data is returned and how fast the search runs. You can choose between three modes in the Splunk UI.

Fast Mode

• Prioritizes speed.

• Only essential fields are extracted.

• No event highlighting or field discovery.

• Best for final dashboards or reports.

Verbose Mode

• Extracts all fields from all events.

• Includes full event context.

• Useful during investigation or initial search development.

• Slower than other modes.

Smart Mode

• Automatically chooses between fast and verbose mode depending on the search.

• Offers a balance between performance and detail.

4. Search Optimization Tips

Optimizing your searches is critical for performance, especially in large environments with many users.

Filter by Index and Time

Always start your search with index name and a time constraint:

index=app_logs earliest=-1h

This dramatically reduces the volume of data Splunk has to scan.

Use Indexed Fields Early

Use fields that are indexed (like host, source, sourcetype) early in the search to limit the dataset as soon as possible.

Avoid Wildcards

Avoid leading wildcards in field names or values:

host=*prod*   ← Not efficient

Better to use exact matches or wildcards only at the end:

host=prod*    ← More efficient

Summarize Data Before Further Processing

If your dataset is large, use stats, eventstats, or tstats early to reduce the number of events being passed to later parts of the search.

5. Accelerations

Splunk provides several acceleration techniques to improve performance for commonly used or expensive searches.

Data Model Acceleration (DMA)

• Used with Pivot and CIM-compliant apps (e.g., Enterprise Security).

• Speeds up complex queries by pre-processing and storing results.

• Configured in data model settings.

Report Acceleration

• Automatically stores results of scheduled reports.

• Only applicable to transforming searches (e.g., those using stats, timechart).

• Great for dashboards that don’t need real-time data.

Summary Indexing

• Stores the output of scheduled searches in a new index.

• Allows fast lookups and reduced search load.

• Requires a separate summary index and scheduled job.

Example:

| stats avg(cpu_load) by host

The result can be stored every 5 minutes in a summary index and reused later.

6. Search Job Management

Searches in Splunk are executed as jobs. Splunk provides tools to manage and monitor these jobs.

Activity > Jobs

• Lists all current and historical search jobs.

• Allows you to:

  • Inspect job properties

  • Pause, resume, or cancel jobs

  • Export results

Search Quotas

• Configured in limits.conf.

• Controls:

  • Number of concurrent searches per role

  • Memory usage

  • Priority of different search types (e.g., real-time vs scheduled)

dispatch.ttl (Time to Live)

• Determines how long completed search results are kept on disk.

• After this time, the results are deleted.

• Default is 10 minutes for ad-hoc searches, but it can be extended for dashboards or saved searches.

Example: To keep search results for one hour:

dispatch.ttl = 3600

Search (Additional Content)

1. Transforming vs. Non-Transforming Searches

Understanding the difference between transforming and non-transforming searches is crucial for questions related to report acceleration, dashboards, and search optimization.

Transforming Searches:

• These change the shape of raw events into summarized results (usually tables or charts)

• Often used in dashboards and scheduled reports

• Required for Report Acceleration to apply

• End with commands like:

  • stats

  • timechart

  • top

  • chart

Example:

index=web_logs status=500 | stats count by host

This counts events and transforms them into a table — it’s a transforming search.

Non-Transforming Searches:

• Return raw event data

• Used for detailed investigation or exploratory searches

• Not eligible for report acceleration

Example:

index=web_logs status=500 host=web01

This simply filters events; it does not transform or summarize them.

Exam Relevance:

Questions may ask:

"Which of the following searches qualifies for Report Acceleration?"

Only transforming searches like | stats, | timechart, etc., are valid answers.

2. Example Combining eval and where

Many SPL-based questions test your understanding of intermediate-level SPL logic using eval, where, and conditionals. Here is a real-world-style example:

Scenario:
You want to find events where the calculated response time in milliseconds exceeds 1000 ms, and only show those for a specific host.

index=web_logs host=web01 
| eval response_ms = duration * 1000 
| where response_ms > 1000

Breakdown:

• eval response_ms = duration * 1000: Creates a new field

• where response_ms > 1000: Filters based on calculated value

This combined use of eval + where often appears in questions that say:

"Which search correctly calculates and filters based on a derived field?"

3. tstats vs. datamodel — Differences and Use Cases

In Splunk, tstats and Data Models are tightly linked, especially in CIM-compliant apps like Enterprise Security (ES).

datamodel command:

• Used to explore the structure of a data model

• Slower than tstats because it does not use pre-acceleration

Example:

| datamodel Web Web search

tstats command:

• Faster, uses pre-accelerated summaries from Data Model Acceleration

• Preferred in dashboards, alerts, and large-scale environments

Example:

| tstats count from datamodel=Web.Web where Web.status=500 by Web.host

Key Comparison:

Feature	tstats	datamodel
Speed	Very fast (uses accelerated summaries)	Slower
Common in	Dashboards, ES, summary-based reports	Ad hoc data exploration
Acceleration Used	Yes	No

Exam Relevance:

Questions might ask:

"Which command leverages DMA for faster performance in CIM environments?"

Answer: tstats

Summary

• Transforming searches are required for report acceleration; non-transforming are not.

• Combining eval + where shows advanced filtering logic — often used in real troubleshooting.

• tstats is faster and DMA-compatible; datamodel is more exploratory.