​

Troubleshooting Guide

​

Connection Issues

​

ClickHouse Connection Failures

• Fill out values-external-clickhouse.yaml with your ClickHouse host, port, and connection details
• Ensure traceloop-clickhouse-secret exists with required credentials:
  • CLICKHOUSE_USERNAME
  • CLICKHOUSE_PASSWORD
  • Note: This secret is automatically created if using Traceloop Terraform, otherwise create it manually

# Verify the ClickHouse secret exists
kubectl get secret traceloop-clickhouse-secret -n traceloop

# Check secret keys (without exposing values)
kubectl describe secret traceloop-clickhouse-secret -n traceloop

• Verify the host and port are correct
• Ensure the database exists and the user has appropriate permissions
• Check if SSL is required (secure: true)
• Verify network connectivity and firewall rules
• Ensure traceloop-clickhouse-secret contains correct credentials
• Verify values-external-clickhouse.yaml has correct host and configuration

​

Kafka Connection Issues

• Fill out values-external-kafka.yaml with your Kafka broker addresses, connection details, and credentials
• Ensure the following Kafka topics exist: traces, spans, and metrics
  • Note: These topics are automatically created if using Traceloop Terraform, otherwise create them manually

# Verify your Kafka configuration is properly applied
kubectl get configmap -n traceloop | grep kafka

• Confirm broker addresses are accessible from the Kubernetes cluster
• Ensure required topics exist: traces, spans, and metrics
• For Confluent Cloud:
  • Ensure SASL authentication is properly configured
  • Verify SSL is enabled
• Check topic creation permissions
• Verify network policies allow Kafka connectivity
• Verify values-external-kafka.yaml has correct broker addresses, credentials, and configuration

​

PostgreSQL Connection Problems

• Fill out values-external-postgres.yaml with your PostgreSQL address and connection details
• Ensure traceloop-postgres-secret exists with required credentials:
  • POSTGRES_DATABASE_USERNAME
  • POSTGRES_DATABASE_PASSWORD
  • Note: This secret is automatically created if using Traceloop Terraform, otherwise create it manually

# Verify the PostgreSQL secret exists
kubectl get secret traceloop-postgres-secret -n traceloop

# Check secret keys (without exposing values)
kubectl describe secret traceloop-postgres-secret -n traceloop

• Verify database exists and is accessible
• Check user permissions (needs ability to create schemas and tables)
• For SSL connections, ensure proper SSL mode is set
• Verify PostgreSQL version compatibility (9.6 or higher)
• Ensure traceloop-postgres-secret contains correct credentials
• Verify values-external-postgres.yaml has correct database address and configuration

​

Kubernetes Deployment Issues

​

Pod Startup Failures

# Check pod status
kubectl get pods -n traceloop

# Check pod events
kubectl describe pod -n traceloop <pod-name>

# Check pod logs
kubectl logs -n traceloop <pod-name>

• Verify resource requests and limits are appropriate
• Ensure all secrets are properly created
• Check container image pull permissions

​

Ingress Issues

• Verify the load balancer is properly provisioned
• Check DNS records point to the correct load balancer endpoint
• Verify SSL certificate is properly attached to load balancer for HTTPS termination
• Ensure security groups allow traffic from load balancer to Kubernetes cluster on port 30800
• Check load balancer target group health - targets should be healthy on port 30800

• Verify Kong gateway pod is running and healthy
• Check that Kong is listening on the correct port (default mapped to 30800)
• Ensure Kong domain configuration is correct in values-customer.yaml - verify kong-gateway.kong.domain is properly filled

# Check Kong gateway pod status
kubectl get pods -n traceloop | grep kong

# Check Kong pod logs
kubectl logs -n traceloop -l app=kong

# Verify port mapping and service
kubectl get svc -n traceloop

​

Image Pull Failures

• Create the regcred secret in the traceloop namespace using credentials provided by Traceloop
• Ensure the secret is properly referenced in your Helm values

# Verify the regcred secret exists
kubectl get secret regcred -n traceloop

# Check secret details (without exposing credentials)
kubectl describe secret regcred -n traceloop

• regcred secret not created in the traceloop namespace
• Incorrect Docker Hub credentials in the secret
• Secret not properly referenced in pod specifications
• Network connectivity issues to Docker Hub

​

Common Error Messages

​

”Error: connection refused”

# Test ClickHouse connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-clickhouse-host 9000

# Test Kafka connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-kafka-broker 9092

# Test PostgreSQL connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-postgres-host 5432

​

“Error: authentication failed”

• Fill out values-customer.yaml with PropelAuth settings:
  Copy

  Ask AI

  customerConfig:
    propelauth:
      authURL: "traceloop-provided"
  
  customerSecret:
    propelauth:
      verifierKey: "traceloop-provided"
      apiKey: "traceloop-provided"
  

• Ensure service accounts have necessary permissions
• Verify SSL/TLS configuration if required
• Ensure PropelAuth configuration is correct in values-customer.yaml
• Verify PropelAuth secrets contain the correct keys provided by Traceloop

​

”Error: insufficient permissions”

• Check RBAC configuration
• Verify service account permissions
• Ensure necessary Kubernetes roles are created
• Check database user permissions

​

Quick Validation Checklist

1. Component Connectivity
   • All infrastructure components are reachable
   • Proper ports are open
   • Network policies allow required traffic
2. Authentication & Permissions
   • Database credentials are correct
   • Kubernetes secrets exist and are mounted
   • Service accounts have required permissions
3. Infrastructure Health
   • Kubernetes nodes are healthy
   • Sufficient resources are available
   • Storage is properly configured
4. Logging & Monitoring
   • Check pod logs: kubectl logs -n traceloop -l app=traceloop
   • Monitor resource usage
   • Review infrastructure metrics