SPLK-1002 Using the Common Information Model (CIM) Add-On

Using the Common Information Model (CIM) Add-On Detailed Explanation

The Common Information Model (CIM) Add-On in Splunk is a powerful framework that normalizes data from diverse sources, ensuring consistency and enabling cross-source analysis.

1. What Is the Common Information Model (CIM) Add-On?

Definition

• The CIM Add-On standardizes field names, tags, and event types across different data sources.
• It maps raw data fields to CIM-compliant field names for consistent analysis.

Purpose

• Normalize Data: Ensures data from different sources adheres to a common schema.
• Enable Compatibility: Makes data compatible with Splunk apps like Splunk Enterprise Security (ES) and IT Service Intelligence (ITSI).
• Facilitate Cross-Source Analysis: Provides a unified framework for analyzing data from multiple sources.

2. Core Concepts of the CIM Add-On

2.1. Normalization

Normalization is the process of mapping raw data fields to CIM-compliant fields.

1. Field Aliases:

   • Map original field names to CIM-compliant field names.

   • Example:

     alias client_ip AS src
     

2. Tags:

   • Categorize data for use in CIM data models.

   • Example:

     Add the "web" and "proxy" tags to categorize web proxy logs.
     

2.2. CIM Data Models

CIM includes predefined data models tailored to specific domains (e.g., authentication, network traffic, web). Each model defines standard field names and tags for that domain.

1. Authentication:

   • Fields: src, user, action
   • Use Case: Track login attempts.

2. Network Traffic:

   • Fields: src, dest, bytes_in, bytes_out
   • Use Case: Monitor data flow between systems.

3. Web:

   • Fields: http_method, status, url
   • Use Case: Analyze web traffic and errors.

2.3. Validation

The datamodel command checks whether your data aligns with CIM standards. It helps identify gaps in field mappings or tagging.

Validation Example

1. Use the datamodel command to validate data:

   | datamodel Authentication search
   

2. Review the results to ensure data is properly mapped to the Authentication data model.

3. How to Use the CIM Add-On

3.1. Install the CIM Add-On

1. Go to Splunkbase and download the Splunk Common Information Model Add-On.
2. Install it on your Splunk instance.

3.2. Normalize Data

Step 1: Identify Original Fields

• Analyze your raw data to identify fields that need mapping.
• Example: A web log might have the field client_ip.

Step 2: Map to CIM Fields

• Use field aliases to map raw fields to CIM fields.

• Example:

  alias client_ip AS src
  

Step 3: Add Tags

• Apply tags to categorize data for the appropriate CIM model.

• Example:

  Add tags "web" and "proxy" to web proxy logs.
  

3.3. Validate Data

• Use the datamodel command to test data alignment with CIM standards.

• Example:

  | datamodel Web search
  

4. Example: Normalizing Web Proxy Data

Scenario

You have web proxy logs with the following fields:

• client_ip
• http_status
• url_path

You want to map this data to the CIM Web data model.

Steps

1. Create Field Aliases:

   • Map raw fields to CIM fields:

     alias client_ip AS src
     alias http_status AS status
     alias url_path AS url
     

2. Add Tags:

   • Assign tags to categorize the data:

     tag="web"
     tag="proxy"
     

3. Validate the Data:

   • Run the following command to verify alignment:

     | datamodel Web search
     

Result: Your web proxy logs are now CIM-compliant and can be used with the Web data model.

5. Best Practices for Using the CIM Add-On

1. Validate Regularly:

   • Use the datamodel command to check data alignment with CIM models periodically.

2. Use Consistent Field Names:

   • Always use CIM-compliant field names in dashboards and searches for uniformity.

3. Leverage Tags Effectively:

   • Ensure appropriate tags are applied to all events for accurate categorization.

4. Document Field Mappings:

   • Maintain a record of all field aliases and tag mappings for reference.

6. Practical Exercises

Exercise 1: Map Fields to CIM

1. Identify raw fields in your data:

   • client_ip, response_time, user_name.

2. Map them to CIM fields:

   alias client_ip AS src
   alias response_time AS duration
   alias user_name AS user
   

3. Validate the mappings:

   | datamodel Authentication search
   

Task: Verify that the fields align with the Authentication data model.

Exercise 2: Add Tags

1. Assign tags to categorize network traffic logs:

   • Add the tags network and traffic.

2. Validate the tags:

   | datamodel Network_Traffic search
   

Task: Confirm that the logs are categorized under the Network Traffic data model.

Exercise 3: Normalize Data for Web Analytics

1. Map web log fields:

   • client_ip → src
   • url_path → url
   • http_status → status

2. Add tags:

   • web, analytics.

3. Validate:

   | datamodel Web search
   

Task: Ensure the web log data aligns with the Web data model.

7. Summary of Key Points

1. CIM Add-On Overview:

   • Standardizes data for consistent reporting and analysis.
   • Provides predefined data models for various domains.

2. Core Concepts:

   • Normalization: Map raw fields to CIM fields using aliases.
   • Validation: Ensure data compatibility with CIM using the datamodel command.

3. Best Practices:

   • Regularly validate data.
   • Use CIM-compliant field names for consistency.
   • Apply appropriate tags to categorize data.

8. Advanced Techniques for Using the CIM Add-On

8.1. Advanced Field Aliases

Scenario:

Your data sources have inconsistent field naming conventions (e.g., source_ip, client_ip, src_ip).

Solution:

• Use regex-based field aliases to dynamically normalize fields.

Example: In props.conf:

[web_logs]
FIELDALIAS-ip_address = src_ip AS src, client_ip AS src, source_ip AS src

Effect: All variations of IP fields are normalized to src.

8.2. Using Calculated Fields

Calculated fields are useful when raw data needs transformation before mapping to CIM-compliant fields.

Scenario:

Your logs include response_time_ms (milliseconds), but CIM expects duration (seconds).

Solution:

• Use calculated fields to transform values.

Example: In props.conf:

EVAL-duration = response_time_ms / 1000

Effect: Maps response_time_ms to duration by converting milliseconds to seconds.

8.3. Custom Tagging Rules

Custom tags help categorize events for CIM models, ensuring correct mapping.

Scenario:

You have firewall logs, but they aren't categorized for the Network Traffic data model.

Solution:

• Apply tags for proper categorization.

Example: In props.conf:

[firewall_logs]
TAG-network = enabled
TAG-traffic = enabled

Effect: Tags the logs as network and traffic for the Network Traffic data model.

8.4. Enriching Data with Lookups

Lookups can enrich your data with additional fields required by CIM.

Scenario:

Your raw logs lack location information (dest_country).

Solution:

• Use a GeoIP lookup to enrich logs with country data.

Steps:

1. Add a GeoIP lookup file (e.g., geoip.csv) with mappings for IP to country.

2. Configure the lookup in transforms.conf:

   [geoip_lookup]
   filename = geoip.csv
   

3. Apply the lookup in props.conf:

   LOOKUP-dest_country = geoip_lookup src_ip OUTPUT dest_country
   

Effect: Adds dest_country to your data, making it CIM-compliant.

8.5. Validating Large Datasets

Scenario:

You want to ensure a large dataset aligns with the CIM Authentication model.

Solution:

• Use the datamodel command with field-specific validation.

Example:

| datamodel Authentication search | stats count BY src, user, action

Effect: Displays a summary of how data aligns with key fields in the Authentication model.

9. Troubleshooting CIM Add-On Issues

9.1. Missing Fields in Data Models

Cause:

• Fields are not properly mapped or missing in the raw data.

Solution:

1. Check field aliases in props.conf.

2. Verify field extraction rules.

3. Use the fields command to confirm field presence:

   | fields src, dest, action
   

9.2. Incorrect Tagging

Cause:

• Tags do not match the data model requirements.

Solution:

1. Review tags applied to events.

2. Validate tagging using the search command:

   tag=authentication
   

9.3. Validation Errors

Cause:

• Data does not fully comply with CIM field expectations.

Solution:

1. Use the datamodel command for detailed validation.
2. Investigate missing or misaligned fields and resolve them.

10. Optimization Strategies for CIM

10.1. Regularly Audit Field Mappings

• Periodically review and update field aliases and calculated fields to ensure alignment with CIM updates.

10.2. Use Summary Indexing

• For large datasets, create summary indexes to store pre-normalized data, improving query performance.

10.3. Leverage CIM-Compliant Apps

• Utilize Splunk apps like Enterprise Security and ITSI, which rely on CIM, to maximize compatibility.

10.4. Automate Tagging

• Use consistent rules in props.conf to automate tagging for scalability.

11. Practical Exercises

Exercise 1: Field Normalization

1. Identify the raw fields:

   • client_ip, user_name.

2. Map them to CIM fields:

   alias client_ip AS src
   alias user_name AS user
   

3. Validate:

   | datamodel Authentication search | table src, user
   

Task: Confirm the fields align with the Authentication data model.

Exercise 2: Add Calculated Fields

1. Define a calculated field for duration:

   eval duration = response_time_ms / 1000
   

2. Apply the field to your data model.

3. Validate:

   | datamodel Web search | stats avg(duration)
   

Task: Ensure the duration field is present and accurate.

Exercise 3: Enrich Data with Lookups

1. Create a GeoIP lookup to map src_ip to src_country.

2. Apply the lookup:

   LOOKUP-geoip = geoip_lookup src_ip OUTPUT src_country
   

3. Validate:

   | datamodel Network_Traffic search | table src, src_country
   

Task: Verify that src_country is correctly populated.

Exercise 4: Validate CIM Tags

1. Add the tags authentication and login to logs with action=login.

2. Validate the tagging:

   tag=authentication AND tag=login
   

3. Use the datamodel command:

   | datamodel Authentication search | stats count BY tag
   

Task: Confirm that the tags align with the Authentication model.

12. Summary of Key Points

1. Core CIM Concepts:

   • Normalization: Map raw fields to CIM-compliant fields.
   • Validation: Use the datamodel command to ensure compliance.

2. Advanced Techniques:

   • Use calculated fields and aliases for flexible normalization.
   • Enrich data with lookups to meet CIM requirements.

3. Best Practices:

   • Regularly validate data.
   • Document and automate field mappings and tagging rules.
   • Use summary indexing to improve performance on large datasets.

Using the Common Information Model (CIM) Add-On (Additional Content)

1. CIM Core Field Glossary (Mini-Dictionary)

Understanding CIM-compliant field names is critical for accurate mapping, especially in exam scenarios where field semantics must be interpreted correctly. Below is a brief glossary of commonly used CIM fields, their meanings, and examples of typical use cases.

Field	Meaning	Typical Context
src	Source IP or originator of the event	Firewall logs, authentication attempts
dest	Destination IP or receiving endpoint	Network traffic analysis
user	The user associated with an event	Login/logout logs, privilege changes
action	Describes what type of activity occurred	Authentication (e.g., "login", "logout"), changes
duration	Length of time for an event (usually in sec)	Web response time, session duration
status	Status outcome (often success/failure)	Authentication, HTTP response logs
app	Application generating the event	Proxy, VPN, or web logs
signature	Identifier for rule or alert type	IDS/IPS alerts, threat intelligence

Tip for Exams:

Field aliases may be used to map different vendor-specific fields to these standard CIM fields. You should be able to recognize such mappings both in configuration and in multiple-choice options.

2. Common Pitfalls: “Don’t Do This” Guidance

To help avoid configuration mistakes often tested in certification exams or encountered in real environments, here are high-frequency errors and how to avoid them:

Don't do this:

alias client_ip = src

Why it’s wrong: Field alias syntax is incorrect. The alias keyword is used in SPL searches, not in configuration files.

Correct (in props.conf):

FIELDALIAS-src_ip = client_ip AS src

Don’t apply tags using eval or lookup

... | eval tag="authentication"

Why it’s wrong: Tags are metadata used at search-time, managed via props.conf or the Settings > Tags UI, not through inline search commands.

Correct Method:

In props.conf:

[authentication_logs]
TAG-authentication = enabled

Don’t forget field casing sensitivity

FIELDALIAS-StatusCode = http_status AS StatusCode

Why it’s wrong: Splunk field names are case-sensitive. StatusCode is not the same as statuscode.

Use lowercase consistently:

FIELDALIAS-status_code = http_status AS status_code

Don’t forget to validate your mappings

Simply defining field aliases or tags is not enough. You must validate them against the CIM data models.

Always validate:

| datamodel Web search | table src, status, url

Summary of Enhancements

• Core Field Glossary: Clarifies the meaning of key CIM fields, helping in field mapping questions.

• Pitfall Warnings: "Don't do this" boxes illustrate misconfigurations often encountered in exams and real-world setups.

• Exam Tip: Be ready to identify incorrect field alias syntax, misused tags, or overlooked validation commands.