Troubleshoot an On-Premises Deployment
The on-premises installer is available to Enterprise customers with an on-premises license. To get a license, contact Permit.io.
This page is for operators who install or run a self-hosted Permit Platform on Kubernetes. Each section starts with the symptom you see, then gives the commands that find the cause and the fix. Commands use the default namespace, permit-platform. If you installed with --namespace, replace it with your namespace.
For routine operations such as upgrades, backups, and credential lookups, see the management guide. For installer flags and values.yaml keys, see the on-premises reference.
Every installer command on this page shows --gke, the flag for an existing Kubernetes cluster. Replace it with the target flag you installed with: --openshift for OpenShift or --kind for a local Kind cluster. A run with no target flag takes the Kind path and fails on an existing cluster, so keep the target flag on every re-run. See Installer command reference.
Installation issues
Frontend domain not configured
Symptom: The installer stops with this error:
[ERROR] ❌ Frontend domain not configured in values.yaml
Please edit charts/permit-platform/values.yaml and replace:
frontendDomain: "CHANGEME_FRONTEND_DOMAIN"
Fix:
- Open
charts/permit-platform/values.yaml. - Replace
CHANGEME_FRONTEND_DOMAINwith the domain users open to reach Permit Platform:global:frontendDomain: "permit.yourcompany.com" # Your domain here - Run the installer again.
Docker image loading fails
Symptom: The installer can't load the images from the package:
[ERROR] Failed to load permit-backend-v2.tar after 3 attempts
Error loading image: docker: Error response from daemon
Find the cause. Check the Docker service, free disk space, and the Docker daemon logs:
# Check Docker service status
systemctl status docker
# Check available disk space
df -h
# Check Docker daemon logs
journalctl -u docker --tail=20
# Test Docker manually
docker pull hello-world
Fix the cause:
-
Disk is full. Remove unused Docker data and package caches:
# Free up spacedocker system prune -fsudo apt-get clean -
Docker service isn't running. Restart the service and check its status:
# Restart Dockersudo systemctl restart docker# Check Docker is runningsudo systemctl status docker -
Your user can't access the Docker daemon. Add your user to the
dockergroup, then sign out and sign in again:# Add user to docker groupsudo usermod -aG docker $USER# Log out and back in, then retry
Kubernetes cluster not reachable
Symptom: The installer can't connect to the cluster:
[ERROR] Kubernetes cluster not accessible
error: couldn't get current server API group list
Find the cause. Check which kubeconfig kubectl uses and whether it reaches the cluster:
# Check kubectl configuration
kubectl cluster-info
# Check kubeconfig
echo $KUBECONFIG
ls -la ~/.kube/config
# Test basic connectivity
kubectl get nodes
Fix the cause:
-
kubectl has no kubeconfig. Point
KUBECONFIGat your cluster's kubeconfig, or copy it to the default location:# Set kubeconfig if not configuredexport KUBECONFIG=/path/to/your/kubeconfig# Or copy config to default locationmkdir -p ~/.kubecp /path/to/kubeconfig ~/.kube/config -
The cluster is down. On a managed cluster (EKS, GKE, AKS), check the cluster status in your cloud console. On a self-managed cluster, check the control plane services:
# For managed clusters (EKS, GKE, AKS)# Ensure cluster is running and accessible# For on-premise clusterssystemctl status kubeletsystemctl status kube-apiserver
The connection works when kubectl get nodes lists your nodes.
Helm deployment fails
Symptom: Helm returns one of these errors:
Error: failed to install chart: context deadline exceeded
Error: UPGRADE FAILED: another operation is in progress
Find the cause. List the releases, including releases stuck in a pending state:
# Check Helm status
helm list -n permit-platform
# Check pending releases
helm list --pending -n permit-platform
# Check namespace
kubectl get all -n permit-platform
Fix the cause:
-
A release is stuck after a failed upgrade. Roll the release back, or uninstall the
permit-platformrelease and run the installer again. Uninstallingpermit-platformkeeps the databases, which belong to thethird-party-servicesrelease.# If upgrade failedhelm rollback permit-platform -n permit-platform# Or uninstall and retry (use the flags you installed with)helm uninstall permit-platform -n permit-platform./scripts/install-permit-platform.sh --gke --skip-images -
Pods can't start before the Helm timeout. Check node capacity and failed pods:
# Check cluster resourceskubectl top nodeskubectl describe nodes# Check for failed podskubectl get pods -n permit-platform --field-selector=status.phase=Failed
Post-installation issues
Web interface not reachable
Symptom: Your browser can't load the Permit frontend at your domain.
Find the cause. Work from the pods outward to DNS:
-
Check that the pods run and the ingress exists:
kubectl get pods -n permit-platformkubectl get ingress -n permit-platform -
Test the frontend from inside the cluster, bypassing the ingress. The
permit-frontendservice listens on port80and forwards to container port3000.# Check if services are respondingkubectl port-forward -n permit-platform svc/permit-frontend 3000:80 &curl http://localhost:3000 -
Check the ingress controller:
kubectl get pods -n ingress-nginxkubectl logs -n ingress-nginx deployment/ingress-nginx-controller -
Check that your domain resolves:
# Test domain resolutionnslookup [your-frontend-domain]# Check hosts file (for .local domains)cat /etc/hosts
Fix the cause:
-
The domain doesn't resolve. For a
.localdevelopment domain, add a hosts file entry. For production, create a DNS record that points the domain to your ingress load balancer.# For development (.local domains), add to hosts fileecho "127.0.0.1 permit-frontend.local" | sudo tee -a /etc/hosts# For production, ensure DNS points to server IP -
The ingress doesn't route traffic. The chart names the ingress
permit-platform-ingress.# Check ingress statuskubectl describe ingress permit-platform-ingress -n permit-platform# Restart ingress controller if neededkubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx -
The frontend pod isn't running.
# Check if frontend pod is runningkubectl get pods -n permit-platform -l app=permit-frontend# Check frontend logskubectl logs -n permit-platform deployment/permit-frontend
The problem is fixed when https://<your-domain> loads the Permit sign-in page.
Services fail to start or crash
Symptom: Pods show CrashLoopBackOff or Error, or restart repeatedly.
Find the cause. Read the logs of the failing service and the namespace events:
# Check all pods status
kubectl get pods -n permit-platform
# Check specific service logs
kubectl logs -n permit-platform deployment/permit-backend-v2
kubectl logs -n permit-platform deployment/celery-general
kubectl logs -n permit-platform deployment/opal-server
# Check events for issues
kubectl get events -n permit-platform --sort-by='.lastTimestamp'
Fix the cause:
-
A service is stuck after a dependency recovered. Restart the deployment:
# Restart specific deploymentkubectl rollout restart deployment/permit-backend-v2 -n permit-platform# Restart all deploymentskubectl rollout restart deployment -n permit-platform -
Pods run out of memory. Look for containers that ended with
OOMKilled:# Check node resourceskubectl top nodeskubectl describe nodes# Check for OOMKilled podskubectl get pods -n permit-platform -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}' | grep OOMKilledRaise the service's memory limit under
permitServices.<service>.resourcesinvalues.yamland run the installer again. -
A service needs more capacity. Add replicas. The next installer run resets replicas to the values in
values.yaml.# Scale backend replicaskubectl scale deployment permit-backend-v2 -n permit-platform --replicas=2# Scale celery workerskubectl scale deployment celery-general -n permit-platform --replicas=1
ImagePullBackOff errors
Symptom: Pods stay in ImagePullBackOff because the cluster can't pull the container images:
kubectl get pods -n permit-platform
NAME READY STATUS RESTARTS AGE
permit-backend-v2-xxx 0/1 ImagePullBackOff 0 2m
permit-frontend-xxx 0/1 ImagePullBackOff 0 2m
Find the cause. Describe a failing pod. The events at the end of the output name the pull error:
# Check pod events for detailed error message
kubectl describe pod permit-backend-v2-xxx -n permit-platform
# Common error messages you'll see:
# - "Failed to pull image... unauthorized: authentication required"
# → Missing or incorrect imagePullSecrets
#
# - "Failed to pull image... not found" or "manifest unknown"
# → Images not pushed to registry, or wrong imageRegistry in values.yaml
#
# - "Failed to pull image... denied: Permission denied"
# → GKE/EKS/AKS node doesn't have permission to pull from registry
| Pull error | Cause | Fix |
|---|---|---|
unauthorized: authentication required | The registry needs credentials and no pull secret is configured | Add a pull secret |
not found or manifest unknown | The images aren't in the registry, imageRegistry is wrong, or the installer ran without --skip-images | Check the registry setting and push the images |
denied: Permission denied | The cluster nodes can't read the cloud registry | Grant GKE, EKS, or AKS pull access |
Missing pull secret for Artifactory, Harbor, or a private registry
Create a docker-registry secret, list it in global.imagePullSecrets, and upgrade the release:
# Step 1: Create the Kubernetes secret
kubectl create secret docker-registry registry-credentials \
--docker-server=artifactory.company.com \
--docker-username=YOUR_USERNAME \
--docker-password=YOUR_TOKEN \
--namespace=permit-platform
# Step 2: Verify secret was created
kubectl get secret registry-credentials -n permit-platform
# Step 3: Update values.yaml
vi charts/permit-platform/values.yaml
# Add this to global section:
# global:
# imageRegistry: "artifactory.company.com/permit-platform"
# imagePullSecrets:
# - registry-credentials
# Step 4: Upgrade the Helm release
helm upgrade permit-platform charts/permit-platform -n permit-platform
Installer ran without --skip-images
If you pushed the images to a private registry but ran the installer without --skip-images, run the installer again with the flag:
# Problem: Installer loaded wrong images from local tar files!
# Solution: Re-run installer with --skip-images flag
# For GKE, EKS, AKS, or self-managed Kubernetes with a private registry:
./scripts/install-permit-platform.sh --gke --skip-images
# For OpenShift with private registry:
./scripts/install-permit-platform.sh --openshift --skip-openshift-registry --skip-images
Wrong imageRegistry value
global.imageRegistry in values.yaml must match the registry path you passed to push-images-to-registry.sh:
# View current configuration
kubectl get cm -n permit-platform
# Verify imageRegistry matches where you pushed images
cat charts/permit-platform/values.yaml | grep imageRegistry
# Should match your push command:
# If you ran: ./scripts/push-images-to-registry.sh us-central1-docker.pkg.dev/project/repo
# Then values.yaml must have: imageRegistry: "us-central1-docker.pkg.dev/project/repo"
Images not pushed to the registry
List the images in your registry:
# For GKE/Google Artifact Registry:
gcloud artifacts docker images list us-central1-docker.pkg.dev/PROJECT/REPO
# For Artifactory:
curl -u username:password https://artifactory.company.com/v2/_catalog
# For Harbor:
curl -u username:password https://harbor.company.com/v2/_catalog
# For AWS ECR:
aws ecr describe-repositories --region us-east-1
aws ecr list-images --repository-name permit-platform --region us-east-1
If the images are missing, push them from the installer package:
cd permit-platform-installer
./scripts/push-images-to-registry.sh YOUR_REGISTRY_URL
GKE nodes can't read Artifact Registry
Grant the Artifact Registry Reader role to the service account of your GKE nodes:
# Grant Artifact Registry Reader role to GKE service account
gcloud projects add-iam-policy-binding PROJECT_ID \
--member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
--role="roles/artifactregistry.reader"
# Or for specific node pool service account:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member="serviceAccount:SA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/artifactregistry.reader"
# Verify permissions
gcloud projects get-iam-policy PROJECT_ID \
--flatten="bindings[].members" \
--filter="bindings.role:roles/artifactregistry.reader"
EKS nodes can't read ECR
Attach the AmazonEC2ContainerRegistryReadOnly policy to the IAM role of your EKS nodes:
# Verify node IAM role has ECR pull permissions
aws iam get-role --role-name YOUR_NODE_ROLE_NAME
# Add ECR read policy if missing
aws iam attach-role-policy \
--role-name YOUR_NODE_ROLE_NAME \
--policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
AKS nodes can't read ACR
Attach your Azure Container Registry (ACR) to the AKS cluster, which grants the AcrPull role:
# Get AKS cluster identity
az aks show -g RESOURCE_GROUP -n CLUSTER_NAME --query identityProfile
# Grant AcrPull role to AKS
az aks update -g RESOURCE_GROUP -n CLUSTER_NAME --attach-acr ACR_NAME
# Verify access
az acr check-access --name ACR_NAME
Check image pull settings across all pods
# Check which images are failing
kubectl get pods -n permit-platform -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].image}{"\t"}{.status.containerStatuses[*].state.waiting.reason}{"\n"}{end}' | grep ImagePull
# Check all imagePullSecrets are configured
kubectl get deployment -n permit-platform -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.imagePullSecrets[*].name}{"\n"}{end}'
# Test pulling an image: start a pod from the image with the pull secret
# (remove --overrides when the registry uses node authentication, such as GAR, ECR, or ACR)
kubectl run test-pull --image=YOUR_REGISTRY/permit-backend-v2:TAG --namespace=permit-platform \
--restart=Never --command \
--overrides='{"apiVersion":"v1","spec":{"imagePullSecrets":[{"name":"registry-credentials"}]}}' \
-- sleep 60
kubectl get pod test-pull -n permit-platform # STATUS Running or Completed: the pull works. ErrImagePull or ImagePullBackOff: it fails.
kubectl delete pod test-pull -n permit-platform
The problem is fixed when the first command returns no lines and kubectl get pods shows the pods Running.
When you use a private registry, follow this order:
- Push the images to the registry with
push-images-to-registry.sh. - Create
imagePullSecretsif the registry needs credentials (Artifactory or Harbor). - Set
imageRegistryandimagePullSecretsinvalues.yaml. - Run the installer with the
--skip-imagesflag.
See push images to a private registry in the installation guide for the full workflow.
Can't sign in with the admin credentials
Symptom: One of these happens when you sign in:
- The page shows "Invalid username or password".
- The sign-in form reloads.
- The browser redirects in a loop.
The installer stores the admin password in the global-infrastructure-secret secret under the KEYCLOAK_ADMIN_PASSWORD key. Retrieve service credentials shows the command.
Find the cause. Check Keycloak, the authentication logs of the backend, and the stored admin password:
# Check Keycloak status
kubectl get pods -n permit-platform -l app=keycloak
kubectl logs -n permit-platform deployment/keycloak
# Check backend authentication logs
kubectl logs -n permit-platform deployment/permit-backend-v2 | grep -i auth
# Verify admin password
kubectl get secret global-infrastructure-secret -n permit-platform -o jsonpath='{.data.KEYCLOAK_ADMIN_PASSWORD}' | base64 -d
Fix the cause:
-
The password doesn't match. Read the stored password again and check that Keycloak created the admin user on startup:
# Get current admin password from secretkubectl get secret global-infrastructure-secret -n permit-platform -o jsonpath='{.data.KEYCLOAK_ADMIN_PASSWORD}' | base64 -d# If password doesn't work, check if Keycloak initialized properlykubectl logs -n permit-platform deployment/keycloak | grep -i "admin user" -
Keycloak doesn't respond. Forward the Keycloak port and request the realm:
# Verify Keycloak is accessiblekubectl port-forward -n permit-platform svc/keycloak 8080:8080 &curl http://localhost:8080/auth/realms/permit-platform -
The backend authentication settings are wrong. Check the backend environment variables and the cookie messages in the backend logs:
# Check backend authentication environment variableskubectl describe deployment permit-backend-v2 -n permit-platform | grep -A 20 Environment# Verify cookie configurationkubectl logs -n permit-platform deployment/permit-backend-v2 | grep -i cookie
Services can't connect to the database
Symptom: Service logs show PostgreSQL connection errors.
Find the cause. Check the PostgreSQL pod and test a query from the backend:
# Check PostgreSQL pod
kubectl get pods -n permit-platform -l app=postgres
kubectl logs -n permit-platform deployment/postgres
# Test database connectivity from backend
kubectl exec -n permit-platform deployment/permit-backend-v2 -- psql -h postgres -U permit -d permit -c "SELECT 1;"
Fix the cause:
-
PostgreSQL is stuck. Restart it and wait until the pod is ready:
kubectl rollout restart deployment/postgres -n permit-platform# Wait for database to be readykubectl wait --for=condition=ready pod -l app=postgres -n permit-platform --timeout=300s -
The database didn't initialize. Look for the
database system is readylog line and list the databases:# Check if database initialized properlykubectl logs -n permit-platform deployment/postgres | grep -i "database system is ready"# Check database size and connectionskubectl exec -n permit-platform deployment/postgres -- psql -U permit -c "\l" -
The password in the secret is wrong. The installer stores the PostgreSQL password in
global-infrastructure-secretunderPOSTGRES_PASSWORD(see Retrieve service credentials). Read the stored password:# Check database password in secretkubectl get secret global-infrastructure-secret -n permit-platform -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
Policy sync fails
Symptom: The permit-policy-sync-v2 pod crashes, or its logs show Git errors.
Policy sync reads the repository URL from permitServices.policySync.policyRepoUrl and the SSH deploy key from permitServices.policySync.sshPrivateKey in values.yaml. The chart passes both values to the deployment as environment variables and fails to render when sshPrivateKey isn't a multi-line block that starts with |.
Find the cause. Check the pod, its logs, and the repository URL:
# Check Policy Sync pod status
kubectl get pods -n permit-platform -l app=permit-policy-sync-v2
# Check Policy Sync logs
kubectl logs -n permit-platform deployment/permit-policy-sync-v2
# Verify the repository URL passed to policy sync
kubectl get deployment permit-policy-sync-v2 -n permit-platform -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="POLICY_REPO_URL")].value}'
Fix the cause:
-
The key or repository is wrong. Test the key against your Git host from a machine that has the private key. Add the public key to the repository as a deploy key if the test fails. To change the key or URL, follow Change the repository or SSH key.
# Check the key format in values.yaml (must be a multi-line block)grep -A3 "sshPrivateKey:" charts/permit-platform/values.yaml# Test SSH connection manually (if possible)ssh -T git@github.com -i /path/to/permit-policy-key -
The service kept a failed state after you fixed the key. Restart policy sync:
kubectl rollout restart deployment/permit-policy-sync-v2 -n permit-platform
The problem is fixed when the permit-policy-sync-v2 pod is Running and its logs show no Git errors.
Collect logs and monitor resources
Collect diagnostic information
The commands below save logs, pod state, events, and networking configuration to files. To package them for Permit support, follow Collect a support bundle.
# Get all pod logs
kubectl logs -n permit-platform --all-containers=true --selector=app!=postgres --prefix --tail=-1 > permit-platform-logs.txt
# Get pod status and descriptions
kubectl get pods -n permit-platform -o wide > pod-status.txt
kubectl describe pods -n permit-platform > pod-descriptions.txt
# Get events
kubectl get events -n permit-platform --sort-by='.lastTimestamp' > events.txt
# Get service and ingress info
kubectl get svc,ingress -n permit-platform -o yaml > networking.yaml
# Check resource usage
kubectl top pods -n permit-platform > resource-usage.txt
Check resource consumption
kubectl top needs the Kubernetes Metrics Server in the cluster.
# Monitor pod resources
kubectl top pods -n permit-platform
# Monitor node resources
kubectl top nodes
# Check for resource limits being hit
kubectl describe pods -n permit-platform | grep -A 5 -B 5 "resource\|limit\|request"
If services hit their limits, add replicas. For a permanent change, set replicas in values.yaml (see Scale services).
# Scale backend for more capacity
kubectl scale deployment permit-backend-v2 -n permit-platform --replicas=3
# Scale celery workers
kubectl scale deployment celery-general -n permit-platform --replicas=2
# Check horizontal pod autoscaler (if configured)
kubectl get hpa -n permit-platform
Advanced recovery
Reset the installation
The reset deletes the namespace and its persistent volumes. The Permit database, Keycloak users, and audit logs are permanently deleted. Create backups first if you might need the data.
The reset uninstalls the three Helm releases, deletes the namespace and its volumes, and runs the installer again. The uninstall steps match Uninstall Permit Platform.
# 1. Uninstall all 3 Helm releases (in reverse order)
helm uninstall permit-platform -n permit-platform 2>/dev/null || true # Permit application services
helm uninstall migrations -n permit-platform 2>/dev/null || true # Database migrations
helm uninstall third-party-services -n permit-platform 2>/dev/null || true # Infrastructure (PostgreSQL, Redis, etc.)
# 2. Delete namespace (removes all resources)
kubectl delete namespace permit-platform
# 3. Clean up any persistent volumes (careful!)
kubectl get pv | grep permit-platform # Check before deleting
kubectl delete pv $(kubectl get pv -o jsonpath='{.items[?(@.spec.claimRef.namespace=="permit-platform")].metadata.name}')
# 4. Re-run installation (use the flags you installed with)
./scripts/install-permit-platform.sh --gke --skip-images
Recover a single service
Force-delete a pod stuck in Terminating, change a deployment's image, or inspect a volume claim. The next installer run overwrites a patched deployment with the settings in values.yaml. Replace new-image-tag with the full image reference.
# Force delete stuck pods
kubectl delete pod <pod-name> -n permit-platform --force --grace-period=0
# Patch deployment to fix issues
kubectl patch deployment permit-backend-v2 -n permit-platform -p '{"spec":{"template":{"spec":{"containers":[{"name":"permit-backend-v2","image":"new-image-tag"}]}}}}'
# Check and fix persistent volume claims
kubectl get pvc -n permit-platform
kubectl describe pvc <pvc-name> -n permit-platform
Get support
Include this information when you contact Permit support:
- Kubernetes distribution: EKS, GKE, AKS, OpenShift, Kind, or self-managed.
- Platform version: the installer package file name.
- Error description: what you ran, what you expected, and what happened.
- Timeline: when the issue started, and what changed before it.
Attach a cluster summary. For the full support bundle, follow Collect a support bundle.
# Create comprehensive diagnostic bundle
{
echo "=== CLUSTER INFO ==="
kubectl cluster-info
echo "=== NODES ==="
kubectl get nodes -o wide
echo "=== PERMIT PLATFORM PODS ==="
kubectl get pods -n permit-platform -o wide
echo "=== RECENT EVENTS ==="
kubectl get events -n permit-platform --sort-by='.lastTimestamp' | tail -20
echo "=== INGRESS STATUS ==="
kubectl get ingress -n permit-platform
echo "=== STORAGE ==="
kubectl get pv,pvc -n permit-platform
} > permit-support-info.txt
Contact Permit support:
- Email: support@permit.io
- Slack: Permit Slack community