Installation Troubleshooting

What deploy’s failure codes mean, and what to do about them

You're viewing a development version of manager, the latest released version is 1.6.2

Page not available in that version

The current page Installation Troubleshooting doesn't exist in version 1.6.2 of the documentation for this product.

We can take you to the closest parent section instead: /docs/acd/components/manager/1.6.2/installation/

Overview

deploy checks the machines and your settings before it changes anything, and checks its own work as it goes. When it stops on a problem it recognizes, the message starts with a code in square brackets, such as [ACD-TLS-HOSTNAME], and names the machine, file or setting involved. The code stays the same when the wording changes, so search for it on this page, or quote it to support.

After fixing the cause, run deploy again. Re-running is safe: it picks up from the state the cluster is in. “Nothing was changed” in the table below means the run stopped before its first change.

Expected Messages

These appear in a successful run, and need no action:

Message	Why
`FAILED - RETRYING: [<machine>]: Wait for this server to become Ready (N retries left).`	`deploy` polls each joining machine until it is ready. Ten or more per machine is normal.
`[WARNING]: Collection ansible.posix does not support Ansible version ...`	The bundled collection works with the `ansible-core` release of Enterprise Linux 9.
`[WARNING]: Could not match supplied host pattern, ignoring: acd_host_network_verify`	No machine’s network needed checking at that point.

Problems Without a Code

Symptom	Cause	Fix
`UNREACHABLE` for a machine	`deploy` cannot log in to it over SSH without a password.	Set up the key: see SSH Access.
`preset.yaml points at ..., which does not exist`	The installation ISO is not mounted at the path `deploy init` used, normally `/mnt/esb3027`.	Mount it at that path, or run `deploy init` again from where it is mounted now.
`ansible-core is not installed`	`deploy init` has not been run on this machine, or `ansible-core` was removed.	Run `/mnt/esb3027/deploy init`. It keeps the preset an existing configuration already uses; a new configuration gets `prod` unless you add `lab`.
`deploy init` cannot install `ansible-core`	`dnf` cannot reach the operating system’s repositories.	Fix the repositories. Without internet access, see Air-Gapped Installation.
Signing in fails, though the pages load	The browser uses a name other than the main manager hostname, such as an IP address.	Browse to the first entry of `manager_hostnames`. Sign in to the GUI as `<user>@zitadel.<manager hostname>`.

Failure Codes

Each code below gives the cause, then what to do.

`[ACD-ANSIBLE-VERSION]`

Cause: The installed ansible-core is older than 2.14.

What to do: Run dnf update ansible-core.

`[ACD-NO-DEFAULT-ROUTE]`

Cause: A machine has no IPv4 default route, so its address cannot be chosen.

What to do: Give the machine a default route.

`[ACD-BACKPLANE-AMBIGUOUS]`

Cause: backplane_network matches more than one interface on a machine.

What to do: Narrow backplane_network, or set node_ip to the address on the backplane interface, on that machine’s line in inventory.ini.

`[ACD-BACKPLANE-NO-MATCH]`

Cause: backplane_network matches no interface on a machine.

What to do: Fix backplane_network, or set node_ip to the address on the backplane interface, on that machine’s line in inventory.ini. Every machine needs an address in backplane_network.

`[ACD-NODE-ADDRESS-NOT-FOUND]`

Cause: No interface on a machine holds the address chosen for it.

What to do: Check node_ip for that machine, or backplane_network.

`[ACD-DEFAULT-ADDRESS-NOT-FOUND]`

Cause: No interface on a machine holds its default route’s source address.

What to do: Give the default route a source address that one of the machine’s interfaces holds.

`[ACD-NM-NOT-RUNNING]`

Cause: NetworkManager is not running on a machine, so the deployment cannot set its interfaces’ zones or routing rules.

What to do: Start it with systemctl enable --now NetworkManager and run deploy again.

`[ACD-NM-CONNECTION-MISSING]`

Cause: An interface the deployment configures has no active NetworkManager connection.

What to do: Let NetworkManager manage it (nmcli device set <interface> managed yes), give it a connection (nmcli device connect <interface>, or nmcli connection add) and run deploy again.

`[ACD-PARTIAL-RUN]`

Cause: The run was limited to some machines, so it could not plan every machine’s network. What each machine trusts depends on all of them.

What to do: Run deploy without --limit.

`[ACD-DIRECTOR-ADDRESS]`

Cause: An entry in external_directors has no address, or one that is not an IPv4 address.

What to do: Give each CDN Director’s IPv4 address, such as 192.0.2.201.

`[ACD-MAXMIND-AMBIGUOUS]`

Cause: More than one file in maxmind/ matches the same MaxMind database.

What to do: Leave one file per database in maxmind/.

`[ACD-MAXMIND-INCOMPLETE]`

Cause: maxmind/ holds some, but not all three, MaxMind databases.

What to do: Provide all three databases, or none.

`[ACD-TLS-COMBINED-FILE]`

Cause: One file in certs/ holds both the certificate and the private key.

What to do: Split them into two PEM files in certs/.

`[ACD-TLS-TOO-MANY]`

Cause: certs/ holds more than one certificate or more than one key.

What to do: Leave exactly one certificate and one key.

`[ACD-TLS-UNPAIRED]`

Cause: certs/ holds a certificate without a key, or a key without a certificate.

What to do: Provide both, or neither for a temporary self-signed certificate.

`[ACD-TLS-NOT-PEM]`

Cause: certs/ holds files, but none is a readable PEM certificate or key.

What to do: Convert DER or PKCS#12 files to PEM.

`[ACD-TLS-CERT-UNREADABLE]`

Cause: The certificate file cannot be read as a PEM certificate.

What to do: Provide a PEM certificate.

`[ACD-TLS-KEY-UNREADABLE]`

Cause: The private key cannot be read: it is not PEM, or it has a passphrase.

What to do: Remove the passphrase with openssl pkey -in <key> -out <key>.plain.

`[ACD-TLS-KEY-MISMATCH]`

Cause: The certificate and the private key are not a pair.

What to do: Provide the key the certificate was requested with.

`[ACD-TLS-CERT-EXPIRED]`

Cause: The certificate has expired.

What to do: Replace it with a current certificate.

`[ACD-TLS-HOSTNAME]`

Cause: The certificate does not cover the first entry of manager_hostnames.

What to do: Reissue it for that name, or for a wildinterface covering it.

`[ACD-INVENTORY-NO-SERVER]`

Cause: The inventory lists no server machine.

What to do: List at least one; the first server is the control machine.

`[ACD-INVENTORY-STRAY-DIRECTOR]`

Cause: A machine is listed under [director]. Adding machines under [director] is not currently supported.

What to do: Leave the [director] section empty. List CDN Directors in external_directors in Site Settings.

`[ACD-INVENTORY-SERVER-AND-AGENT]`

Cause: A machine is listed under both server and agent.

What to do: List it under only one.

`[ACD-INVENTORY-EVEN-SERVERS]`

Cause: The inventory lists an even number of servers.

What to do: List an odd number of servers; move the rest under agent.

`[ACD-INVENTORY-DUPLICATE-HOSTNAME]`

Cause: Two machines share a hostname, ignoring case.

What to do: Give each machine a unique hostname.

`[ACD-INVENTORY-ROLE-CHANGED]`

Cause: A machine runs k3s as a server but is listed as an agent, or the other way round.

What to do: List it under its current role, or follow Changing a Machine’s Role.

`[ACD-NODE-OS]`

Cause: A machine does not run Enterprise Linux 9.4 or a later 9 release.

What to do: Update or reinstall it, or remove it from the inventory.

`[ACD-NODE-KERNEL-MODULES]`

Cause: A machine lacks kernel modules k3s and its storage need.

What to do: Install the kernel modules package for the running kernel.

`[ACD-NODE-RESOURCES]`

Cause: A machine has fewer processor cores or less memory than its role needs.

What to do: Add cores or memory, or make it an agent.

`[ACD-VALUES-EDITED]`

Cause: values.yaml was edited by hand.

What to do: Move the changes to values.override.yaml or group_vars/all.yaml, then delete values.yaml.

`[ACD-CLUSTER-API-DOWN]`

Cause: k3s is installed, but no server’s k3s API answers.

What to do: Get k3s answering on one server. If you are sure the check is wrong, run deploy --skip-tags preflight.

`[ACD-CLUSTER-NODE-RENAMED]`

Cause: A machine’s hostname changed after it joined the cluster.

What to do: Set the hostname back to the name in the message.

`[ACD-CLUSTER-NODE-ADDRESS]`

Cause: A machine’s cluster address differs from the one the settings choose.

What to do: Set the address settings back to what the cluster runs with; see Locked at First Install.

`[ACD-CLUSTER-POD-CIDR]`

Cause: A machine’s pod range lies outside cluster_pod_cidr.

What to do: Set cluster_pod_cidr back to what the cluster runs with.

`[ACD-CLUSTER-SERVICE-CIDR]`

Cause: The cluster’s service range differs from cluster_svc_cidr.

What to do: Set cluster_svc_cidr back to what the cluster runs with.

`[ACD-HEALTH-NODE-NOT-READY]`

Cause: A cluster machine is not Ready.

What to do: Check systemctl status k3s or k3s-agent on it, or remove it from the cluster.

`[ACD-HEALTH-LIST-FAILED]`

Cause: The databases or volumes could not be listed.

What to do: Check the cluster API answers with k3s kubectl get nodes.

`[ACD-HEALTH-DATABASE]`

Cause: A PostgreSQL database cluster is not healthy: it is still being built, is switching its primary, or more than one of its instances is not ready. One instance being rebuilt is accepted, and so is one instance per machine the run is bringing back after a role change.

What to do: Inspect it with the kubectl describe command in the message.

`[ACD-HEALTH-VOLUME-FAULTED]`

Cause: A storage volume has no replica with usable data, and not every replica is on a machine the run joins back.

What to do: Recover it from the Longhorn UI or a backup.

`[ACD-CIDR-OVERLAP]`

Cause: cluster_pod_cidr and cluster_svc_cidr overlap.

What to do: Choose two separate ranges.

`[ACD-BACKPLANE-SPLIT]`

Cause: The machines’ cluster addresses are not all on one network, so they could not reach each other. Either a machine’s address is outside backplane_network, or, without one, a machine using a second interface for cluster traffic has an address on a different network from the others. Nothing was changed.

What to do: The message names the machine and the network. Put every machine’s cluster address inside backplane_network (correct node_ip in inventory.ini if needed), or set backplane_network to the network the machines actually share.

`[ACD-CIDR-IN-USE]`

Cause: A machine already uses an address or route in cluster_pod_cidr or cluster_svc_cidr.

What to do: Choose ranges no machine uses.

`[ACD-ROUTING-CONFLICT]`

Cause: A routing rule sends cluster traffic away from the main routing table.

What to do: Follow the message: give the conflicting rules a priority number above 50, or with manage_routes: false, set it to true or add your own lookup main rules ahead of them.

`[ACD-FIREWALL-SOURCE-CONFLICT]`

Cause: Another firewalld zone on a machine already holds an address range that overlaps one deploy needs to trust: the pod or service network, backplane_network, or another machine’s address. firewalld allows a range in only one zone. Nothing was changed.

What to do: Run the commands the message shows: they remove the overlapping range from the other zone. If the zone it names is deploy’s own zone under an earlier name, set firewall_cluster_zone back instead: it is locked at first install. Or set manage_firewall: false.

`[ACD-FIREWALL-NAME-INVALID]`

Cause: firewall_cluster_zone or firewall_public_policy is the name of one of firewalld’s own zones or policies (such as public, trusted or allow-host-ipv6), is empty or not a string, is longer than firewalld allows (17 characters for the zone, 18 for the policy), has characters other than letters, digits, - and _, or both are the same. Nothing was changed.

What to do: Choose other names in group_vars/all.yaml.

`[ACD-FIREWALL-NAME-IN-USE]`

Cause: A machine already has a firewalld zone or policy named like firewall_cluster_zone or firewall_public_policy, and it holds rules deploy did not create. deploy would delete those rules, so it stops instead. Nothing was changed.

What to do: Choose names in group_vars/all.yaml that nothing on the machines uses, or empty that zone or policy yourself.

`[ACD-FIREWALL-HANDOVER-STOPPED]`

Cause: With manage_firewall: false, deploy needs to empty its own zone and policy, but firewalld is stopped. Rules saved in a stopped firewalld would come back when it starts. If this appears after the network changes, firewalld stopped during the run, and nothing was installed on the cluster.

What to do: Start firewalld (systemctl enable --now firewalld) and run deploy again. Or run the commands the message shows, then run deploy again.

`[ACD-FIREWALL-CLOSED]`

Cause: With manage_firewall: false, a machine’s firewall blocks something the cluster needs: a required port is closed, or the pod or service network is not in a zone that accepts all traffic. Nothing was changed.

What to do: Run the firewall-cmd commands the message shows. See Managing the Firewall Yourself. Or set manage_firewall: true.

`[ACD-BACKUP-FOLDER-CREATE]`

Cause: The folder for this run’s backup could not be created.

What to do: Check that the backup folder’s parent exists and is writable, or set backup_dir in group_vars/all.yaml.

`[ACD-BACKUP-FOLDER-EXISTS]`

Cause: Another run started in the same second and owns the backup folder.

What to do: Run deploy again; nothing was changed.

`[ACD-BACKUP-SNAPSHOT]`

Cause: The etcd snapshot for the backup was not found, or more than one was.

What to do: Check the snapshot folder and the k3s log (journalctl -u k3s) on the machine the message names, then run deploy again.

`[ACD-NM-PENDING-EDITS]`

Cause: A network connection deploy needs to change has edits saved in NetworkManager that are not active yet. Applying deploy’s change would activate them too. Nothing was changed on that machine.

What to do: Check the pending edits with nmcli connection show <uuid>. Either undo them with nmcli connection modify, or activate them in a maintenance window with nmcli connection up <uuid>. Then run deploy again.

`[ACD-CLUSTER-ZONE-MISSING]`

Cause: A machine already running k3s has no acd-cluster zone or no acd-public policy (or the names you set). Creating them needs a firewalld reload, which would remove the rules k3s depends on, so deploy will not do it on a running machine. Nothing was changed.

What to do: If you changed firewall_cluster_zone or firewall_public_policy, set them back: they are locked at first install. Otherwise, in a maintenance window, one machine at a time, create them (firewall-cmd --permanent --new-zone=acd-cluster, --zone=acd-cluster --set-target=ACCEPT, --new-policy=acd-public, --policy=acd-public --add-ingress-zone=public --add-egress-zone=HOST), run firewall-cmd --reload, and restart k3s. Then run deploy again.

`[ACD-NODE-UNREACHABLE]`

Cause: A machine stopped answering over SSH while its host network changed, one machine at a time on a running cluster, so the machines after it were left as they were. The change itself may have cut it off.

What to do: Check that machine’s network from its console, then run deploy again.

`[ACD-VERIFY-ZONE]`

Cause: After changing a machine’s network, an interface is not in the firewalld zone deploy planned for it. Nothing was installed on the cluster.

What to do: Run deploy again. If the message comes back, send it to support.

`[ACD-VERIFY-FIREWALL]`

Cause: After changing a machine’s network, its firewall still blocks something the cluster needs, or another zone holds an address range deploy needs to trust. Nothing was installed on the cluster.

What to do: Run the firewall-cmd commands the message shows, then run deploy again. If the message comes back, send it to support.

`[ACD-VERIFY-LEFTOVER]`

Cause: After changing a machine’s network, deploy’s own zone or policy holds a rule it did not create, or is missing one it planned. Nothing was installed on the cluster.

What to do: Run deploy again, which removes the extra rule. Keep your own rules in other zones. If the message names a zone target or priority, run firewall-cmd --reload in a maintenance window and restart k3s (or k3s-agent) on that machine, then run deploy again. If it comes back otherwise, send it to support.

`[ACD-VERIFY-ROUTING]`

Cause: After changing a machine’s network, its routing rules are not what deploy planned: a cluster rule is missing, a removed rule is still active, a rule was added by hand with ip rule add, or cluster traffic uses a routing table other than main. Nothing was installed on the cluster.

What to do: Run the ip rule del commands the message shows, check with ip rule show, then run deploy again. If the message comes back, send it to support.

`[ACD-VERIFY-SYSCTL]`

Cause: After the network changes, a kernel network setting on a machine has another value than the deployment set.

What to do: Find what sets it (a file under /etc/sysctl.d, or a tuned profile), then run deploy again. No k3s step ran.

`[ACD-JOIN-UNREACHABLE]`

Cause: A joining machine cannot reach the server it joins through.

What to do: Check the firewall on both machines, that their node_ip addresses are on one network, and that reverse path filtering (net.ipv4.conf.all.rp_filter) does not drop the replies; see the message.

`[ACD-GATE-DATABASE-UNHEALTHY]`

Cause: After the install, a PostgreSQL database cluster did not become healthy in time. This message is followed by [ACD-GATE-FAILED], which writes the support bundle.

What to do: Wait until kubectl get clusters.postgresql.cnpg.io -A shows Cluster in healthy state, then run deploy again.

`[ACD-GATE-UNHEALTHY-PODS]`

Cause: After the install, pods in a critical namespace were still not running after 10 minutes. This message is followed by [ACD-GATE-FAILED], which writes the support bundle.

What to do: Find the cause with acd-admin status <target> and acd-admin logs <target> on a server, fix it, then run deploy again.

`[ACD-GATE-RELEASE-STATUS]`

Cause: After the install, the Helm release is not deployed. This message is followed by [ACD-GATE-FAILED], which writes the support bundle.

What to do: Find the cause with the helm status command in the message and acd-admin status, fix it, then run deploy again.

`[ACD-GATE-FAILED]`

Cause: The final health check failed. It follows the message that names the failing check.

What to do: Read that earlier message, and send /etc/acd/deploy/support-bundle-acd.tar.gz to support.

Getting Help

When the final health check fails ([ACD-GATE-FAILED]), deploy writes /etc/acd/deploy/support-bundle-acd.tar.gz, with the run’s log, its decision record and what it read from each machine. Send it to AgileTV support.

When a run stops earlier, send support these files from /etc/acd/deploy instead:

deploy.log
decision-record.json
the probe/ folder