SPLK-1004 Working with Self-Describing Data and Files

Working with Self-Describing Data and Files Detailed Explanation

1. What is Self-Describing Data?

Self-describing data refers to data formats that include information about their own structure. This means that tools like Splunk can automatically determine how the data is organized without needing a separate schema or documentation.

In simpler terms, the data itself explains how to read it.

Example:

{
  "user": "alice",
  "location": {
    "city": "New York",
    "zip": "10001"
  }
}

In this JSON snippet:

• The field names (user, location.city, location.zip) help define the structure.

• Splunk can extract and organize this data automatically.

2. Common Formats in Splunk

Splunk can work with several self-describing data formats. These include:

a) JSON (JavaScript Object Notation)

• Most commonly used in modern APIs, applications, and cloud systems.

• Supports nested structures, arrays, and objects.

• Splunk handles JSON natively with commands like spath.

b) XML (Extensible Markup Language)

• Often used in legacy enterprise systems, SOAP APIs, and config files.

• Based on tags: <fieldname>value</fieldname>

c) Key-Value Pair Logs

• Simple format where each field is a key=value pair.

• Example: user=alice action=login status=success

d) Multi-line Events

• Includes structured data spread across multiple lines.

• Common in CLI output, Java stack traces, or tabular logs.

3. Key Commands and Techniques

To work with self-describing data in Splunk, you use a set of powerful commands and configuration options. Below are the most important ones:

a) spath

The primary command for parsing JSON and XML structures.

Functionality:

• Extracts data from nested structures.

• Can specify a path and assign to an output field.

Example:

... | spath input=payload path=customer.name output=customer_name

This takes the JSON object payload, finds customer.name, and creates a new field customer_name.

If you omit path, spath will extract all fields from the structure.

b) KV Mode (KV_MODE)

KV Mode controls how Splunk automatically extracts fields.

Options:

• KV_MODE=auto: Splunk decides based on the content.

• KV_MODE=json: Forces JSON parsing.

• KV_MODE=xml: Forces XML parsing.

Where to set:

• In props.conf (for persistent config)

• As a field in the event metadata (e.g., via sourcetype settings)

KV Mode is especially useful when dealing with raw logs that embed structured data, like a JSON block inside syslog.

c) multikv

Extracts tabular data (like command-line output or reports with column headers and rows).

Use case examples:

• CLI outputs from commands like df, ps, or netstat

• Tables where the first row is a header, and following rows are values

How it works:

... | multikv fields header_field1, header_field2

It will generate one event per line of data under each header.

d) xmlkv

Auto-parses simple XML into field-value pairs.

Example:

<user>admin</user>

Using xmlkv, this becomes:

user=admin

Note: This works only for flat XML structures. For nested XML, use spath.

e) spath with wildcards

You can use spath to extract from arrays or lists using the {} wildcard.

Example:

spath path=errors{}.message

If you have a JSON object like this:

"errors": [
  {"message": "timeout"},
  {"message": "not found"}
]

The above spath command will extract both messages into a multi-value field.

4. Use Cases

Understanding these tools is essential for parsing logs from modern applications, especially those using structured APIs, cloud-native tools, or SIEM integrations.

a) Parsing APIs

• API responses are usually in JSON format.

• You can use spath to extract specific values from the payload.

Example:

... | spath input=api_response path=results[0].status

b) Syslog and Config Files

• Many syslogs include key=value strings or embed JSON blocks.

• KV_MODE and rex can be used to extract fields cleanly.

c) Nested Response Payloads

• Logs may include nested details such as:

  • user.device.os

  • network.source.ip

• spath lets you access these deeply nested fields.

d) Handling Table Logs and CLI Output

• Logs from firewalls, routers, or operating systems may come in tabular formats.

• Use multikv to break them into rows and extract columns as fields.

Working with Self-Describing Data and Files (Additional Content)

1. Enabling AUTO_KV_JSON for Automated JSON Extraction

Splunk supports automatic field extraction for JSON data. In many cases, this is handled implicitly, but advanced configuration using props.conf can improve consistency and reduce the need for manual parsing.

AUTO_KV_JSON Setting

To automatically extract all top-level fields from JSON payloads (even in non-standard log structures), enable the following setting:

AUTO_KV_JSON = true

Where to set it:

[my_sourcetype]
AUTO_KV_JSON = true

What it does:

• Automatically parses valid JSON content in _raw

• Extracts key-value pairs into searchable fields

• Particularly helpful when JSON is embedded in other log formats

Use Case:

For sourcetypes where JSON appears consistently but is not the only data in the event, enabling AUTO_KV_JSON ensures JSON content is always parsed, even without spath.

Exam Note: This configuration is often tested in admin-level questions where extraction methods or indexing behavior is part of the scenario.

2. Handling Escaped JSON (Double-Encoded JSON Strings)

A common challenge in working with logs is escaped JSON, where the actual JSON content is embedded as a string inside another JSON field or text field.

Example of Escaped JSON:

"message": "{\"status\":\"ok\",\"id\":123}"

At first glance, this is a string. Splunk will treat it as:

message = {"status":"ok","id":123}

How to Extract Properly (Two-Step spath):

Step 1: Extract the field containing the escaped JSON:

| spath output=msg_raw path=message

Step 2: Parse the extracted field again as JSON:

| spath input=msg_raw

This second spath parses the previously escaped string as JSON and creates proper fields like status and id.

Best Practice: Always verify whether the JSON content is nested or stringified. Failing to recognize escaped JSON leads to incomplete field extractions.

Exam Tip: This scenario often appears in logs from cloud APIs, middleware logs, or security platforms.

3. Extracting Specific Elements from JSON Arrays

Splunk’s spath command supports precise path-based access to elements in arrays, which is critical for indexed access to structured logs.

Common Syntax Variants:

Syntax	Meaning
results{}	Extracts all values from the array
results{0}	Extracts the first value only
results{2}.user.id	Extracts the user.id field from the 3rd item

Example Scenario:

Given a JSON payload:

"results": [
  {"id": 1001, "user": {"name": "alice"}},
  {"id": 1002, "user": {"name": "bob"}}
]

You can extract the id of the first result using:

| spath input=payload path=results{0}.id output=first_result_id

Explanation:

• This retrieves a single value, not a multivalue field.

• In contrast, results{}.id would return all id fields as a multivalue array.

Why it matters: Fine-grained access is essential for dashboards and alerts where you want to highlight specific indexed elements without needing to loop through all of them.

Quick Recap of Key Enhancements

• AUTO_KV_JSON = true enables automatic JSON field extraction at indexing or search time via configuration

• Escaped JSON requires two-pass parsing using chained spath commands

• spath path=results{N}.field allows direct access to specific array items, reducing complexity in data processing