Debug a deployment
This guide covers debugging a stuck or failing Nebari Kubernetes Platform (NKP) deployment. It is scoped to NKP only. For Classic Nebari, see Troubleshooting.
The steps below that use kubectl require the CLI installed and KUBECONFIG set. See Deploy a cluster for how to retrieve the kubeconfig.
nic deploy phase
Reading nic output
nic deploy streams structured progress to stdout. When a deploy fails, the error appears at the end of that output. Capture it for sharing or offline review:
nic deploy -f config.yaml 2>&1 | tee deploy.log
For deeper traces that show exactly which internal step hung, enable the built-in OpenTelemetry console exporter (OTEL_EXPORTER is a nic-specific variable; valid values are none, console, otlp, and both):
OTEL_EXPORTER=console nic deploy -f config.yaml
This emits span data for every internal step, including provider API calls and ArgoCD bootstrap.
Common nic deploy failures
| Symptom | Cause | Fix |
|---|---|---|
| Exits immediately with a config or credential error | Pre-flight validation failed | Run nic validate -f config.yaml and fix the reported fields before re-deploying |
context deadline exceeded | Large cluster or slow network | Re-run with --timeout 1h |
| Cloud API error (quota exceeded, insufficient permissions) | Provider-specific | Check your cloud provider's console; fix IAM permissions or request a quota increase, then re-run (nic deploy is idempotent) |
| Hangs after "ArgoCD installed" | ArgoCD can't reach the GitOps repository | Verify GIT_TOKEN has read/write access to the repo, then check ArgoCD server logs (see ArgoCD applications below) |
Post-deploy phase
After nic deploy returns, the cluster still needs ArgoCD to sync and start all foundational services. This section covers debugging that process in dependency order.
ArgoCD applications
Start here: ArgoCD is the engine that delivers everything else.
List all applications and their sync and health status:
kubectl get applications -n argocd
Inspect a stuck or degraded application:
kubectl describe application <name> -n argocd
What to look for:
SyncStatus:Synced(good),OutOfSync(hasn't applied the latest manifests),Unknown(can't reach the source repo; check the ArgoCD server logs).Health:Healthy(good),Progressing(still starting up),Degraded(something is wrong).- A
Degradedapplication includes a human-readable message inStatus.Conditions. Read it before looking elsewhere.
Check ArgoCD server logs directly:
kubectl logs -n argocd -l app.kubernetes.io/component=server --tail=100
Common causes of stuck ArgoCD applications: GIT_TOKEN expired or missing repo access, or the cluster can't pull images from the OCI registry.
Foundational software pods
Each foundational component runs in its own namespace. Check for pods that are not Running:
| Component | Namespace |
|---|---|
| cert-manager | cert-manager |
| Envoy Gateway | envoy-gateway-system |
| Keycloak | keycloak |
| OpenTelemetry Collector | monitoring |
General pattern for any component:
kubectl get pods -n <namespace>
kubectl logs -n <namespace> <pod-name>
cert-manager
A healthy ArgoCD application for cert-manager does not guarantee that cert-manager is issuing TLS certificates. Also check:
kubectl get certificates -A
kubectl describe certificate <name> -n <namespace>
A False ready status usually means a DNS-01 challenge timeout or a Let's Encrypt rate limit. The describe output includes the challenge URL and the specific error message from the ACME server.
Keycloak
If Keycloak pods are running but the sign-in page is unreachable, check whether the HTTPRoute was created:
kubectl get httproutes -A
Nebari Operator
Check the operator pod:
kubectl get pods -n nebari-operator-system
kubectl logs -n nebari-operator-system -l control-plane=controller-manager --tail=100
The most common failure is the operator starting before Keycloak is healthy and failing to connect. Fix: wait for the keycloak ArgoCD application to reach Healthy, then delete the operator pod to trigger a restart:
kubectl delete pod -n nebari-operator-system -l control-plane=controller-manager
NebariApp status
List all NebariApps across namespaces:
kubectl get nebariapps -A
Inspect the status conditions on a specific NebariApp:
kubectl describe nebariapp <name> -n <namespace>
The four status conditions and what each one tracks:
| Condition | What it tracks |
|---|---|
RoutingReady | HTTPRoute created and accepted by Envoy Gateway |
TLSReady | TLS certificate issued by cert-manager |
AuthReady | OIDC client provisioned in Keycloak |
Ready | All three above are True |
Each condition includes a Message field explaining why it is False. Read that before checking logs.
Symptom index
| Symptom | Where to look |
|---|---|
nic deploy exits with an error immediately | nic deploy common failures |
nic deploy times out | nic deploy common failures, timeout row |
nic deploy hangs after "ArgoCD installed" | nic deploy common failures; ArgoCD applications |
ArgoCD app stuck in Progressing or Degraded | ArgoCD applications |
Pod in CrashLoopBackOff or ImagePullBackOff | Foundational software pods |
| Nebari Operator pod crashing or not connecting to Keycloak | Nebari Operator |
| TLS certificate not issuing | cert-manager |
| Can't sign in / Keycloak error | Keycloak |
NebariApp not Ready | NebariApp status |
Related pages
- NKP architecture: understand how the layers fit together before debugging
- Deploy a cluster: the deploy steps this page assumes you've run
- Providers: provider-specific prerequisites and configuration