SPLK-2002 Deployment Problems

Deployment Problems Detailed Explanation

Deployment problems in Splunk can occur in various parts of a distributed environment, including forwarder management, search head clustering, and indexer clustering. Proper deployment ensures configurations, apps, and updates reach all intended Splunk instances efficiently and accurately.

This topic explains the most common deployment issues and how to diagnose and resolve them.

1. Deployment Server (DS) Issues

The Deployment Server is used to manage and push configurations and apps to Universal Forwarders (UFs) and some Heavy Forwarders (HFs). When it fails or misbehaves, forwarders may stop receiving updates or run outdated configurations.

Common Problems and Causes

Clients Not Receiving Updates

Possible reasons:

• Misconfigured deploymentclient.conf on the forwarder

  • If the client doesn’t point to the correct Deployment Server or has a missing stanza, it won’t check in.

• Incorrect serverclass.conf on the DS

  • Server classes define which clients receive which configurations.

  • If server class rules don’t match any clients (based on IP, host, machine type), the client won’t get updates.

How to Fix:

• Validate that each forwarder has the correct settings in:

  • $SPLUNK_HOME/etc/system/local/deploymentclient.conf

• On the DS, verify:

  • Server class stanzas are correctly defined.

  • Apps are assigned to appropriate classes.

Latency or Stale Configurations

Symptoms:

• Clients are connected but not getting updated apps or inputs.

• Changes pushed via Deployment Server do not appear on forwarders.

Possible causes:

• Incorrect or outdated server class matching

• Push did not complete successfully

How to Fix:

• Check the log file:

  • deploymentserver.log located at $SPLUNK_HOME/var/log/splunk/

• Look for errors or failed deployments.

• Use splunk display deploy-client or the Splunk Web UI (Deployment section) to monitor client status.

2. Cluster Deployment Issues

In clustered environments (Indexer Cluster or Search Head Cluster), deployment must be handled carefully to avoid replication errors, sync failures, or cluster misbehavior.

SHC or Indexer Cluster Misalignment

Node Not Joining the Cluster

Possible causes:

• Incorrect pass4SymmKey in server.conf

• Cluster secret mismatch

• Wrong cluster master address or missing clustering stanza

Symptoms:

• Cluster master does not recognize the peer.

• Node appears offline in the Monitoring Console or via CLI.

• Warnings or errors appear in splunkd.log.

How to Fix:

• Check that server.conf has the correct:

  • pass4SymmKey

  • mode (indexer or searchhead)

  • master_uri (for indexers)

• Restart the Splunk service after changes.

Replication Not Working

Possible causes:

• Incorrect or insufficient Replication Factor (RF) and Search Factor (SF) settings.

• One or more peer nodes are down or out of sync.

How to Fix:

• Review clustermaster.log for errors.

• Use the Monitoring Console or the CLI:

  • splunk show cluster-status

• Rebalance or fix missing buckets as needed.

Deployment via Deployer (for Search Head Cluster)

The Deployer is responsible for pushing apps and configurations to all members of a Search Head Cluster (SHC).

Common Issues:

• The shcluster_bundle was not packaged correctly.

• The deployment was not pushed with the correct CLI command.

• One or more search heads are out of sync after deployment.

Best Practices:

• Use this command to push apps from the deployer:
  splunk apply shcluster-bundle -target https://<captain_host>:8089 -auth admin:password

• Ensure the app structure is valid under $SPLUNK_HOME/etc/shcluster/apps/.

• For certain changes (e.g., navigation menus, views), a rolling restart of SHC members is required.

3. Key Logs for Deployment Troubleshooting

Effective troubleshooting relies on the right logs. Here’s where to look based on the component:

splunkd.log (all nodes)

• Logs general errors, warnings, and service startup issues.

• Always check this first on any node reporting issues.

clustermaster.log (Indexer Cluster Master)

• Shows replication issues, peer join/leave events, and RF/SF enforcement.

• Located at: $SPLUNK_HOME/var/log/splunk/clustermaster.log

shclustering.log (Search Head Cluster)

• Tracks deployer pushes, knowledge object replication, and SHC captain elections.

• Review this on both deployer and cluster members.

deploymentserver.log (Deployment Server)

• Shows client check-ins, app delivery, and errors related to server classes or push failures.

• Useful to verify whether clients are checking in and receiving updates.

Deployment Problems (Additional Content)

Deployment issues can occur at various levels of the Splunk environment, particularly during the configuration rollout to forwarders, search head clusters (SHC), or indexer clusters. Troubleshooting these requires a solid grasp of Splunk’s deployment architecture, key configuration files, and CLI commands.

1. Forwarder Deployment Failure due to outputs.conf Misconfiguration

One critical and often-overlooked area is the incorrect deployment or misconfiguration of outputs.conf, which defines how forwarders send data to indexers.

Example Failure Scenario:

• A Universal Forwarder (UF) is added to a server class, and receives a deployment app.

• The app contains an outputs.conf with the wrong target indexer IP or port.

• Result: The UF fails to connect, and indexing stops silently.

Troubleshooting Tips:

• Check $SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder for connection or handshake errors.

• Use splunk list forward-server on the UF to confirm the indexer list and connection state.

• Always verify syntax and stanza nesting in the pushed outputs.conf.

2. SHC Captain Election Issues (Split Brain Scenarios)

In a Search Head Cluster (SHC), one member is elected as Captain, responsible for search job scheduling and knowledge object replication. Election failure can lead to "split-brain" states.

Symptoms of Captain Election Issues:

• Multiple nodes claim to be Captain.

• Searches are not scheduled.

• Dashboards fail to update consistently.

Common Causes:

• Insufficient members online to maintain quorum (less than 50%).

• Network partition or node communication failure.

• Recent deployer push or rolling restart not completed correctly.

Detection and Verification:

Use the following command on any SHC member:

splunk show shcluster-status

It will show:

• The current Captain

• Members' status (Up/Down/Syncing)

• Quorum status (e.g., Has Quorum: true/false)

3. Common serverclass.conf Matching Errors in Deployment Server

The serverclass.conf file defines which clients receive which apps. Errors in filtering logic can prevent deployment.

Example Error:

[serverClass:web_servers]
whitelist.0 = host::web-*

Issue: The host attribute is not correctly matched due to incorrect format.

Correct Usage:

[serverClass:web_servers]
whitelist.0 = web-*

The host:: syntax is invalid here. Always match against the clientName, which typically uses hostname or FQDN.

Tip:

Use the Forwarder Management UI or run:

splunk display deploy-clients

...to view which forwarders matched which server classes.

4. CLI Commands for Cluster and Deployment Diagnostics

Here are some critical CLI tools for managing deployments and verifying status:

• SHC Status:

  splunk show shcluster-status
  

• Push SHC Bundle from Deployer:

  splunk apply shcluster-bundle -target https://<captain>:8089 -auth admin:changeme
  

• Check Deployment Server logs:

  • Located at:

    $SPLUNK_HOME/var/log/splunk/deploymentserver.log
    

• View Connected Clients from DS:

  splunk display deploy-clients