Skip to main content

Troubleshoot an On-Premises Deployment

Enterprise on-premises license required

The on-premises installer is available to Enterprise customers with an on-premises license. To get a license, contact Permit.io.

This page is for operators who install or run a self-hosted Permit Platform on Kubernetes. Each section starts with the symptom you see, then gives the commands that find the cause and the fix. Commands use the default namespace, permit-platform. If you installed with --namespace, replace it with your namespace.

For routine operations such as upgrades, backups, and credential lookups, see the management guide. For installer flags and values.yaml keys, see the on-premises reference.

Every installer command on this page shows --gke, the flag for an existing Kubernetes cluster. Replace it with the target flag you installed with: --openshift for OpenShift or --kind for a local Kind cluster. A run with no target flag takes the Kind path and fails on an existing cluster, so keep the target flag on every re-run. See Installer command reference.

Installation issues

Frontend domain not configured

Symptom: The installer stops with this error:

[ERROR] ❌ Frontend domain not configured in values.yaml
Please edit charts/permit-platform/values.yaml and replace:
frontendDomain: "CHANGEME_FRONTEND_DOMAIN"

Fix:

  1. Open charts/permit-platform/values.yaml.
  2. Replace CHANGEME_FRONTEND_DOMAIN with the domain users open to reach Permit Platform:
    global:
    frontendDomain: "permit.yourcompany.com" # Your domain here
  3. Run the installer again.

Docker image loading fails

Symptom: The installer can't load the images from the package:

[ERROR] Failed to load permit-backend-v2.tar after 3 attempts
Error loading image: docker: Error response from daemon

Find the cause. Check the Docker service, free disk space, and the Docker daemon logs:

# Check Docker service status
systemctl status docker

# Check available disk space
df -h

# Check Docker daemon logs
journalctl -u docker --tail=20

# Test Docker manually
docker pull hello-world

Fix the cause:

  1. Disk is full. Remove unused Docker data and package caches:

    # Free up space
    docker system prune -f
    sudo apt-get clean
  2. Docker service isn't running. Restart the service and check its status:

    # Restart Docker
    sudo systemctl restart docker

    # Check Docker is running
    sudo systemctl status docker
  3. Your user can't access the Docker daemon. Add your user to the docker group, then sign out and sign in again:

    # Add user to docker group
    sudo usermod -aG docker $USER
    # Log out and back in, then retry

Kubernetes cluster not reachable

Symptom: The installer can't connect to the cluster:

[ERROR] Kubernetes cluster not accessible
error: couldn't get current server API group list

Find the cause. Check which kubeconfig kubectl uses and whether it reaches the cluster:

# Check kubectl configuration
kubectl cluster-info

# Check kubeconfig
echo $KUBECONFIG
ls -la ~/.kube/config

# Test basic connectivity
kubectl get nodes

Fix the cause:

  1. kubectl has no kubeconfig. Point KUBECONFIG at your cluster's kubeconfig, or copy it to the default location:

    # Set kubeconfig if not configured
    export KUBECONFIG=/path/to/your/kubeconfig

    # Or copy config to default location
    mkdir -p ~/.kube
    cp /path/to/kubeconfig ~/.kube/config
  2. The cluster is down. On a managed cluster (EKS, GKE, AKS), check the cluster status in your cloud console. On a self-managed cluster, check the control plane services:

    # For managed clusters (EKS, GKE, AKS)
    # Ensure cluster is running and accessible

    # For on-premise clusters
    systemctl status kubelet
    systemctl status kube-apiserver

The connection works when kubectl get nodes lists your nodes.

Helm deployment fails

Symptom: Helm returns one of these errors:

Error: failed to install chart: context deadline exceeded
Error: UPGRADE FAILED: another operation is in progress

Find the cause. List the releases, including releases stuck in a pending state:

# Check Helm status
helm list -n permit-platform

# Check pending releases
helm list --pending -n permit-platform

# Check namespace
kubectl get all -n permit-platform

Fix the cause:

  1. A release is stuck after a failed upgrade. Roll the release back, or uninstall the permit-platform release and run the installer again. Uninstalling permit-platform keeps the databases, which belong to the third-party-services release.

    # If upgrade failed
    helm rollback permit-platform -n permit-platform

    # Or uninstall and retry (use the flags you installed with)
    helm uninstall permit-platform -n permit-platform
    ./scripts/install-permit-platform.sh --gke --skip-images
  2. Pods can't start before the Helm timeout. Check node capacity and failed pods:

    # Check cluster resources
    kubectl top nodes
    kubectl describe nodes

    # Check for failed pods
    kubectl get pods -n permit-platform --field-selector=status.phase=Failed

Post-installation issues

Web interface not reachable

Symptom: Your browser can't load the Permit frontend at your domain.

Find the cause. Work from the pods outward to DNS:

  1. Check that the pods run and the ingress exists:

    kubectl get pods -n permit-platform
    kubectl get ingress -n permit-platform
  2. Test the frontend from inside the cluster, bypassing the ingress. The permit-frontend service listens on port 80 and forwards to container port 3000.

    # Check if services are responding
    kubectl port-forward -n permit-platform svc/permit-frontend 3000:80 &
    curl http://localhost:3000
  3. Check the ingress controller:

    kubectl get pods -n ingress-nginx
    kubectl logs -n ingress-nginx deployment/ingress-nginx-controller
  4. Check that your domain resolves:

    # Test domain resolution
    nslookup [your-frontend-domain]

    # Check hosts file (for .local domains)
    cat /etc/hosts

Fix the cause:

  1. The domain doesn't resolve. For a .local development domain, add a hosts file entry. For production, create a DNS record that points the domain to your ingress load balancer.

    # For development (.local domains), add to hosts file
    echo "127.0.0.1 permit-frontend.local" | sudo tee -a /etc/hosts

    # For production, ensure DNS points to server IP
  2. The ingress doesn't route traffic. The chart names the ingress permit-platform-ingress.

    # Check ingress status
    kubectl describe ingress permit-platform-ingress -n permit-platform

    # Restart ingress controller if needed
    kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx
  3. The frontend pod isn't running.

    # Check if frontend pod is running
    kubectl get pods -n permit-platform -l app=permit-frontend

    # Check frontend logs
    kubectl logs -n permit-platform deployment/permit-frontend

The problem is fixed when https://<your-domain> loads the Permit sign-in page.

Services fail to start or crash

Symptom: Pods show CrashLoopBackOff or Error, or restart repeatedly.

Find the cause. Read the logs of the failing service and the namespace events:

# Check all pods status
kubectl get pods -n permit-platform

# Check specific service logs
kubectl logs -n permit-platform deployment/permit-backend-v2
kubectl logs -n permit-platform deployment/celery-general
kubectl logs -n permit-platform deployment/opal-server

# Check events for issues
kubectl get events -n permit-platform --sort-by='.lastTimestamp'

Fix the cause:

  1. A service is stuck after a dependency recovered. Restart the deployment:

    # Restart specific deployment
    kubectl rollout restart deployment/permit-backend-v2 -n permit-platform

    # Restart all deployments
    kubectl rollout restart deployment -n permit-platform
  2. Pods run out of memory. Look for containers that ended with OOMKilled:

    # Check node resources
    kubectl top nodes
    kubectl describe nodes

    # Check for OOMKilled pods
    kubectl get pods -n permit-platform -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}' | grep OOMKilled

    Raise the service's memory limit under permitServices.<service>.resources in values.yaml and run the installer again.

  3. A service needs more capacity. Add replicas. The next installer run resets replicas to the values in values.yaml.

    # Scale backend replicas
    kubectl scale deployment permit-backend-v2 -n permit-platform --replicas=2

    # Scale celery workers
    kubectl scale deployment celery-general -n permit-platform --replicas=1

ImagePullBackOff errors

Symptom: Pods stay in ImagePullBackOff because the cluster can't pull the container images:

kubectl get pods -n permit-platform
NAME READY STATUS RESTARTS AGE
permit-backend-v2-xxx 0/1 ImagePullBackOff 0 2m
permit-frontend-xxx 0/1 ImagePullBackOff 0 2m

Find the cause. Describe a failing pod. The events at the end of the output name the pull error:

# Check pod events for detailed error message
kubectl describe pod permit-backend-v2-xxx -n permit-platform

# Common error messages you'll see:
# - "Failed to pull image... unauthorized: authentication required"
# → Missing or incorrect imagePullSecrets
#
# - "Failed to pull image... not found" or "manifest unknown"
# → Images not pushed to registry, or wrong imageRegistry in values.yaml
#
# - "Failed to pull image... denied: Permission denied"
# → GKE/EKS/AKS node doesn't have permission to pull from registry
Pull errorCauseFix
unauthorized: authentication requiredThe registry needs credentials and no pull secret is configuredAdd a pull secret
not found or manifest unknownThe images aren't in the registry, imageRegistry is wrong, or the installer ran without --skip-imagesCheck the registry setting and push the images
denied: Permission deniedThe cluster nodes can't read the cloud registryGrant GKE, EKS, or AKS pull access

Missing pull secret for Artifactory, Harbor, or a private registry

Create a docker-registry secret, list it in global.imagePullSecrets, and upgrade the release:

# Step 1: Create the Kubernetes secret
kubectl create secret docker-registry registry-credentials \
--docker-server=artifactory.company.com \
--docker-username=YOUR_USERNAME \
--docker-password=YOUR_TOKEN \
--namespace=permit-platform

# Step 2: Verify secret was created
kubectl get secret registry-credentials -n permit-platform

# Step 3: Update values.yaml
vi charts/permit-platform/values.yaml

# Add this to global section:
# global:
# imageRegistry: "artifactory.company.com/permit-platform"
# imagePullSecrets:
# - registry-credentials

# Step 4: Upgrade the Helm release
helm upgrade permit-platform charts/permit-platform -n permit-platform

Installer ran without --skip-images

If you pushed the images to a private registry but ran the installer without --skip-images, run the installer again with the flag:

# Problem: Installer loaded wrong images from local tar files!
# Solution: Re-run installer with --skip-images flag

# For GKE, EKS, AKS, or self-managed Kubernetes with a private registry:
./scripts/install-permit-platform.sh --gke --skip-images

# For OpenShift with private registry:
./scripts/install-permit-platform.sh --openshift --skip-openshift-registry --skip-images

Wrong imageRegistry value

global.imageRegistry in values.yaml must match the registry path you passed to push-images-to-registry.sh:

# View current configuration
kubectl get cm -n permit-platform

# Verify imageRegistry matches where you pushed images
cat charts/permit-platform/values.yaml | grep imageRegistry

# Should match your push command:
# If you ran: ./scripts/push-images-to-registry.sh us-central1-docker.pkg.dev/project/repo
# Then values.yaml must have: imageRegistry: "us-central1-docker.pkg.dev/project/repo"

Images not pushed to the registry

List the images in your registry:

# For GKE/Google Artifact Registry:
gcloud artifacts docker images list us-central1-docker.pkg.dev/PROJECT/REPO

# For Artifactory:
curl -u username:password https://artifactory.company.com/v2/_catalog

# For Harbor:
curl -u username:password https://harbor.company.com/v2/_catalog

# For AWS ECR:
aws ecr describe-repositories --region us-east-1
aws ecr list-images --repository-name permit-platform --region us-east-1

If the images are missing, push them from the installer package:

cd permit-platform-installer
./scripts/push-images-to-registry.sh YOUR_REGISTRY_URL

GKE nodes can't read Artifact Registry

Grant the Artifact Registry Reader role to the service account of your GKE nodes:

# Grant Artifact Registry Reader role to GKE service account
gcloud projects add-iam-policy-binding PROJECT_ID \
--member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
--role="roles/artifactregistry.reader"

# Or for specific node pool service account:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member="serviceAccount:SA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/artifactregistry.reader"

# Verify permissions
gcloud projects get-iam-policy PROJECT_ID \
--flatten="bindings[].members" \
--filter="bindings.role:roles/artifactregistry.reader"

EKS nodes can't read ECR

Attach the AmazonEC2ContainerRegistryReadOnly policy to the IAM role of your EKS nodes:

# Verify node IAM role has ECR pull permissions
aws iam get-role --role-name YOUR_NODE_ROLE_NAME

# Add ECR read policy if missing
aws iam attach-role-policy \
--role-name YOUR_NODE_ROLE_NAME \
--policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

AKS nodes can't read ACR

Attach your Azure Container Registry (ACR) to the AKS cluster, which grants the AcrPull role:

# Get AKS cluster identity
az aks show -g RESOURCE_GROUP -n CLUSTER_NAME --query identityProfile

# Grant AcrPull role to AKS
az aks update -g RESOURCE_GROUP -n CLUSTER_NAME --attach-acr ACR_NAME

# Verify access
az acr check-access --name ACR_NAME

Check image pull settings across all pods

# Check which images are failing
kubectl get pods -n permit-platform -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].image}{"\t"}{.status.containerStatuses[*].state.waiting.reason}{"\n"}{end}' | grep ImagePull

# Check all imagePullSecrets are configured
kubectl get deployment -n permit-platform -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.imagePullSecrets[*].name}{"\n"}{end}'

# Test pulling an image: start a pod from the image with the pull secret
# (remove --overrides when the registry uses node authentication, such as GAR, ECR, or ACR)
kubectl run test-pull --image=YOUR_REGISTRY/permit-backend-v2:TAG --namespace=permit-platform \
--restart=Never --command \
--overrides='{"apiVersion":"v1","spec":{"imagePullSecrets":[{"name":"registry-credentials"}]}}' \
-- sleep 60
kubectl get pod test-pull -n permit-platform # STATUS Running or Completed: the pull works. ErrImagePull or ImagePullBackOff: it fails.
kubectl delete pod test-pull -n permit-platform

The problem is fixed when the first command returns no lines and kubectl get pods shows the pods Running.

Prevent image pull errors

When you use a private registry, follow this order:

  1. Push the images to the registry with push-images-to-registry.sh.
  2. Create imagePullSecrets if the registry needs credentials (Artifactory or Harbor).
  3. Set imageRegistry and imagePullSecrets in values.yaml.
  4. Run the installer with the --skip-images flag.

See push images to a private registry in the installation guide for the full workflow.

Can't sign in with the admin credentials

Symptom: One of these happens when you sign in:

  • The page shows "Invalid username or password".
  • The sign-in form reloads.
  • The browser redirects in a loop.

The installer stores the admin password in the global-infrastructure-secret secret under the KEYCLOAK_ADMIN_PASSWORD key. Retrieve service credentials shows the command.

Find the cause. Check Keycloak, the authentication logs of the backend, and the stored admin password:

# Check Keycloak status
kubectl get pods -n permit-platform -l app=keycloak
kubectl logs -n permit-platform deployment/keycloak

# Check backend authentication logs
kubectl logs -n permit-platform deployment/permit-backend-v2 | grep -i auth

# Verify admin password
kubectl get secret global-infrastructure-secret -n permit-platform -o jsonpath='{.data.KEYCLOAK_ADMIN_PASSWORD}' | base64 -d

Fix the cause:

  1. The password doesn't match. Read the stored password again and check that Keycloak created the admin user on startup:

    # Get current admin password from secret
    kubectl get secret global-infrastructure-secret -n permit-platform -o jsonpath='{.data.KEYCLOAK_ADMIN_PASSWORD}' | base64 -d

    # If password doesn't work, check if Keycloak initialized properly
    kubectl logs -n permit-platform deployment/keycloak | grep -i "admin user"
  2. Keycloak doesn't respond. Forward the Keycloak port and request the realm:

    # Verify Keycloak is accessible
    kubectl port-forward -n permit-platform svc/keycloak 8080:8080 &
    curl http://localhost:8080/auth/realms/permit-platform
  3. The backend authentication settings are wrong. Check the backend environment variables and the cookie messages in the backend logs:

    # Check backend authentication environment variables
    kubectl describe deployment permit-backend-v2 -n permit-platform | grep -A 20 Environment

    # Verify cookie configuration
    kubectl logs -n permit-platform deployment/permit-backend-v2 | grep -i cookie

Services can't connect to the database

Symptom: Service logs show PostgreSQL connection errors.

Find the cause. Check the PostgreSQL pod and test a query from the backend:

# Check PostgreSQL pod
kubectl get pods -n permit-platform -l app=postgres
kubectl logs -n permit-platform deployment/postgres

# Test database connectivity from backend
kubectl exec -n permit-platform deployment/permit-backend-v2 -- psql -h postgres -U permit -d permit -c "SELECT 1;"

Fix the cause:

  1. PostgreSQL is stuck. Restart it and wait until the pod is ready:

    kubectl rollout restart deployment/postgres -n permit-platform

    # Wait for database to be ready
    kubectl wait --for=condition=ready pod -l app=postgres -n permit-platform --timeout=300s
  2. The database didn't initialize. Look for the database system is ready log line and list the databases:

    # Check if database initialized properly
    kubectl logs -n permit-platform deployment/postgres | grep -i "database system is ready"

    # Check database size and connections
    kubectl exec -n permit-platform deployment/postgres -- psql -U permit -c "\l"
  3. The password in the secret is wrong. The installer stores the PostgreSQL password in global-infrastructure-secret under POSTGRES_PASSWORD (see Retrieve service credentials). Read the stored password:

    # Check database password in secret
    kubectl get secret global-infrastructure-secret -n permit-platform -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d

Policy sync fails

Symptom: The permit-policy-sync-v2 pod crashes, or its logs show Git errors.

Policy sync reads the repository URL from permitServices.policySync.policyRepoUrl and the SSH deploy key from permitServices.policySync.sshPrivateKey in values.yaml. The chart passes both values to the deployment as environment variables and fails to render when sshPrivateKey isn't a multi-line block that starts with |.

Find the cause. Check the pod, its logs, and the repository URL:

# Check Policy Sync pod status
kubectl get pods -n permit-platform -l app=permit-policy-sync-v2

# Check Policy Sync logs
kubectl logs -n permit-platform deployment/permit-policy-sync-v2

# Verify the repository URL passed to policy sync
kubectl get deployment permit-policy-sync-v2 -n permit-platform -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="POLICY_REPO_URL")].value}'

Fix the cause:

  1. The key or repository is wrong. Test the key against your Git host from a machine that has the private key. Add the public key to the repository as a deploy key if the test fails. To change the key or URL, follow Change the repository or SSH key.

    # Check the key format in values.yaml (must be a multi-line block)
    grep -A3 "sshPrivateKey:" charts/permit-platform/values.yaml

    # Test SSH connection manually (if possible)
    ssh -T git@github.com -i /path/to/permit-policy-key
  2. The service kept a failed state after you fixed the key. Restart policy sync:

    kubectl rollout restart deployment/permit-policy-sync-v2 -n permit-platform

The problem is fixed when the permit-policy-sync-v2 pod is Running and its logs show no Git errors.

Collect logs and monitor resources

Collect diagnostic information

The commands below save logs, pod state, events, and networking configuration to files. To package them for Permit support, follow Collect a support bundle.

# Get all pod logs
kubectl logs -n permit-platform --all-containers=true --selector=app!=postgres --prefix --tail=-1 > permit-platform-logs.txt

# Get pod status and descriptions
kubectl get pods -n permit-platform -o wide > pod-status.txt
kubectl describe pods -n permit-platform > pod-descriptions.txt

# Get events
kubectl get events -n permit-platform --sort-by='.lastTimestamp' > events.txt

# Get service and ingress info
kubectl get svc,ingress -n permit-platform -o yaml > networking.yaml

# Check resource usage
kubectl top pods -n permit-platform > resource-usage.txt

Check resource consumption

kubectl top needs the Kubernetes Metrics Server in the cluster.

# Monitor pod resources
kubectl top pods -n permit-platform

# Monitor node resources
kubectl top nodes

# Check for resource limits being hit
kubectl describe pods -n permit-platform | grep -A 5 -B 5 "resource\|limit\|request"

If services hit their limits, add replicas. For a permanent change, set replicas in values.yaml (see Scale services).

# Scale backend for more capacity
kubectl scale deployment permit-backend-v2 -n permit-platform --replicas=3

# Scale celery workers
kubectl scale deployment celery-general -n permit-platform --replicas=2

# Check horizontal pod autoscaler (if configured)
kubectl get hpa -n permit-platform

Advanced recovery

Reset the installation

A reset deletes all platform data

The reset deletes the namespace and its persistent volumes. The Permit database, Keycloak users, and audit logs are permanently deleted. Create backups first if you might need the data.

The reset uninstalls the three Helm releases, deletes the namespace and its volumes, and runs the installer again. The uninstall steps match Uninstall Permit Platform.

# 1. Uninstall all 3 Helm releases (in reverse order)
helm uninstall permit-platform -n permit-platform 2>/dev/null || true # Permit application services
helm uninstall migrations -n permit-platform 2>/dev/null || true # Database migrations
helm uninstall third-party-services -n permit-platform 2>/dev/null || true # Infrastructure (PostgreSQL, Redis, etc.)

# 2. Delete namespace (removes all resources)
kubectl delete namespace permit-platform

# 3. Clean up any persistent volumes (careful!)
kubectl get pv | grep permit-platform # Check before deleting
kubectl delete pv $(kubectl get pv -o jsonpath='{.items[?(@.spec.claimRef.namespace=="permit-platform")].metadata.name}')

# 4. Re-run installation (use the flags you installed with)
./scripts/install-permit-platform.sh --gke --skip-images

Recover a single service

Force-delete a pod stuck in Terminating, change a deployment's image, or inspect a volume claim. The next installer run overwrites a patched deployment with the settings in values.yaml. Replace new-image-tag with the full image reference.

# Force delete stuck pods
kubectl delete pod <pod-name> -n permit-platform --force --grace-period=0

# Patch deployment to fix issues
kubectl patch deployment permit-backend-v2 -n permit-platform -p '{"spec":{"template":{"spec":{"containers":[{"name":"permit-backend-v2","image":"new-image-tag"}]}}}}'

# Check and fix persistent volume claims
kubectl get pvc -n permit-platform
kubectl describe pvc <pvc-name> -n permit-platform

Get support

Include this information when you contact Permit support:

  • Kubernetes distribution: EKS, GKE, AKS, OpenShift, Kind, or self-managed.
  • Platform version: the installer package file name.
  • Error description: what you ran, what you expected, and what happened.
  • Timeline: when the issue started, and what changed before it.

Attach a cluster summary. For the full support bundle, follow Collect a support bundle.

# Create comprehensive diagnostic bundle
{
echo "=== CLUSTER INFO ==="
kubectl cluster-info
echo "=== NODES ==="
kubectl get nodes -o wide
echo "=== PERMIT PLATFORM PODS ==="
kubectl get pods -n permit-platform -o wide
echo "=== RECENT EVENTS ==="
kubectl get events -n permit-platform --sort-by='.lastTimestamp' | tail -20
echo "=== INGRESS STATUS ==="
kubectl get ingress -n permit-platform
echo "=== STORAGE ==="
kubectl get pv,pvc -n permit-platform
} > permit-support-info.txt

Contact Permit support:

Next steps