> ## Documentation Index
> Fetch the complete documentation index at: https://www.traceloop.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting Guide

> Solutions to common issues when self-hosting Traceloop

# Troubleshooting Guide

This guide helps you diagnose and resolve common issues you might encounter when self-hosting Traceloop.

<Info>
  Need immediate assistance? [Schedule a support
  call](https://calendly.com/d/cq42-93s-kcx) with our team.
</Info>

## Connection Issues

### ClickHouse Connection Failures

The Traceloop Helm chart expects specific configuration for external ClickHouse connections:

**Required Configuration:**

* Fill out `values-external-clickhouse.yaml` with your ClickHouse host, port, and connection details
* Ensure `traceloop-clickhouse-secret` exists with required credentials:
  * `CLICKHOUSE_USERNAME`
  * `CLICKHOUSE_PASSWORD`
  * **Note:** This secret is automatically created if using Traceloop Terraform, otherwise create it manually

```bash theme={null}
# Verify the ClickHouse secret exists
kubectl get secret traceloop-clickhouse-secret -n traceloop

# Check secret keys (without exposing values)
kubectl describe secret traceloop-clickhouse-secret -n traceloop
```

Common issues and their solutions when connecting to ClickHouse:

* Verify the host and port are correct
* Ensure the database exists and the user has appropriate permissions
* Check if SSL is required (`secure: true`)
* Verify network connectivity and firewall rules
* Ensure `traceloop-clickhouse-secret` contains correct credentials
* Verify `values-external-clickhouse.yaml` has correct host and configuration

### Kafka Connection Issues

The Traceloop Helm chart expects specific configuration for external Kafka connections:

**Required Configuration:**

* Fill out `values-external-kafka.yaml` with your Kafka broker addresses, connection details, and credentials
* Ensure the following Kafka topics exist: `traces`, `spans`, and `metrics`
  * **Note:** These topics are automatically created if using Traceloop Terraform, otherwise create them manually

```bash theme={null}
# Verify your Kafka configuration is properly applied
kubectl get configmap -n traceloop | grep kafka
```

If you're having trouble connecting to Kafka:

* Confirm broker addresses are accessible from the Kubernetes cluster
* Ensure required topics exist: `traces`, `spans`, and `metrics`
* For Confluent Cloud:
  * Ensure SASL authentication is properly configured
  * Verify SSL is enabled
* Check topic creation permissions
* Verify network policies allow Kafka connectivity
* Verify `values-external-kafka.yaml` has correct broker addresses, credentials, and configuration

### PostgreSQL Connection Problems

The Traceloop Helm chart expects specific configuration for external PostgreSQL connections:

**Required Configuration:**

* Fill out `values-external-postgres.yaml` with your PostgreSQL address and connection details
* Ensure `traceloop-postgres-secret` exists with required credentials:
  * `POSTGRES_DATABASE_USERNAME`
  * `POSTGRES_DATABASE_PASSWORD`
  * **Note:** This secret is automatically created if using Traceloop Terraform, otherwise create it manually

```bash theme={null}
# Verify the PostgreSQL secret exists
kubectl get secret traceloop-postgres-secret -n traceloop

# Check secret keys (without exposing values)
kubectl describe secret traceloop-postgres-secret -n traceloop
```

Common PostgreSQL connectivity issues:

* Verify database exists and is accessible
* Check user permissions (needs ability to create schemas and tables)
* For SSL connections, ensure proper SSL mode is set
* Verify PostgreSQL version compatibility (9.6 or higher)
* Ensure `traceloop-postgres-secret` contains correct credentials
* Verify `values-external-postgres.yaml` has correct database address and configuration

## Kubernetes Deployment Issues

### Pod Startup Failures

If pods aren't starting properly:

```bash theme={null}
# Check pod status
kubectl get pods -n traceloop

# Check pod events
kubectl describe pod -n traceloop <pod-name>

# Check pod logs
kubectl logs -n traceloop <pod-name>
```

Common solutions:

* Verify resource requests and limits are appropriate
* Ensure all secrets are properly created
* Check container image pull permissions

### Ingress Issues

**Our Ingress Architecture:**
Traceloop uses a custom ingress setup rather than traditional Kubernetes ingress controllers.
A load balancer handles SSL termination and forwards HTTP traffic directly to port 30800 on the Kubernetes cluster.
This port maps to a Kong gateway pod, which serves as the unified ingress point for all HTTP endpoints.

If you can't access Traceloop through your load balancer:

**Load Balancer Configuration:**

* Verify the load balancer is properly provisioned
* Check DNS records point to the correct load balancer endpoint
* Verify SSL certificate is properly attached to load balancer for HTTPS termination
* Ensure security groups allow traffic from load balancer to Kubernetes cluster on port 30800
* Check load balancer target group health - targets should be healthy on port 30800

**Kong Gateway Pod Health:**

* Verify Kong gateway pod is running and healthy
* Check that Kong is listening on the correct port (default mapped to 30800)
* Ensure Kong domain configuration is correct in `values-customer.yaml` - verify `kong-gateway.kong.domain` is properly filled

```bash theme={null}
# Check Kong gateway pod status
kubectl get pods -n traceloop | grep kong

# Check Kong pod logs
kubectl logs -n traceloop -l app=kong

# Verify port mapping and service
kubectl get svc -n traceloop
```

### Image Pull Failures

Traceloop images are stored on Docker Hub and require authentication. A `regcred` secret is provided by Traceloop and must be manually created in the traceloop namespace.

**Setup Requirements:**

* Create the `regcred` secret in the traceloop namespace using credentials provided by Traceloop
* Ensure the secret is properly referenced in your Helm values

```bash theme={null}
# Verify the regcred secret exists
kubectl get secret regcred -n traceloop

# Check secret details (without exposing credentials)
kubectl describe secret regcred -n traceloop
```

**Common Issues:**

* `regcred` secret not created in the traceloop namespace
* Incorrect Docker Hub credentials in the secret
* Secret not properly referenced in pod specifications
* Network connectivity issues to Docker Hub

## Common Error Messages

### "Error: connection refused"

Check component connectivity:

```bash theme={null}
# Test ClickHouse connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-clickhouse-host 9000

# Test Kafka connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-kafka-broker 9092

# Test PostgreSQL connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-postgres-host 5432
```

### "Error: authentication failed"

Traceloop uses PropelAuth as the authentication provider. Ensure proper configuration:

**Required Configuration:**

* Fill out `values-customer.yaml` with PropelAuth settings:
  ```yaml theme={null}
  customerConfig:
    propelauth:
      authURL: "traceloop-provided"

  customerSecret:
    propelauth:
      verifierKey: "traceloop-provided"
      apiKey: "traceloop-provided"
  ```

Authentication issues:

* Ensure service accounts have necessary permissions
* Verify SSL/TLS configuration if required
* Ensure PropelAuth configuration is correct in `values-customer.yaml`
* Verify PropelAuth secrets contain the correct keys provided by Traceloop

### "Error: insufficient permissions"

Permission-related problems:

* Check RBAC configuration
* Verify service account permissions
* Ensure necessary Kubernetes roles are created
* Check database user permissions

## Quick Validation Checklist

1. Component Connectivity

   * All infrastructure components are reachable
   * Proper ports are open
   * Network policies allow required traffic

2. Authentication & Permissions

   * Database credentials are correct
   * Kubernetes secrets exist and are mounted
   * Service accounts have required permissions

3. Infrastructure Health

   * Kubernetes nodes are healthy
   * Sufficient resources are available
   * Storage is properly configured

4. Logging & Monitoring
   * Check pod logs: `kubectl logs -n traceloop -l app=traceloop`
   * Monitor resource usage
   * Review infrastructure metrics

<Note>
  Still having issues? We're here to help: - [Schedule a support
  call](https://calendly.com/d/cq42-93s-kcx) for personalized assistance - Join
  our [community Slack](https://traceloop.com/slack) for discussions and updates
</Note>