Changing the Cluster

Overview

The machines in the cluster are whatever inventory.ini lists. To change them, edit the inventory and run deploy. This page covers the changes that need more than that.

Before you start, check that the cluster is healthy: every machine Ready and every target ok in acd-admin status.

Adding an Agent

1. Install the new machine as for the others: see Before You Begin. If you use a backplane, give it an address in backplane_network.

2. From the first server, set up SSH to it:

   ssh-copy-id root@192.0.2.14
   ssh -o BatchMode=yes root@192.0.2.14 hostname
   

3. Add it under [agent] in /etc/acd/deploy/inventory.ini:

   [agent]
   manager-4 ansible_host=192.0.2.14
   

4. Run deploy:

   /mnt/esb3027/deploy
   

deploy configures the new machine’s network, joins it to the cluster, and updates every other machine’s firewall to trust it.

Adding Servers

The number of servers must stay odd, so add servers two at a time, for example to go from one to three. Prepare both machines as for an agent, list both under [server], and run deploy once. deploy joins them one at a time.

Changing a Machine’s Role

k3s cannot turn a server into an agent, or an agent into a server, in place: the machine leaves the cluster and joins again in its new role. Because the number of servers must stay odd, a role change always moves two machines: two agents promoted to go from one server to three, two servers demoted to go from three to one, or a server and an agent swapped.

1. Take the first machine out of the cluster, whether it is a server or an agent. From any server, drain and remove it:

   kubectl drain <machine> --ignore-daemonsets --delete-emptydir-data
   kubectl delete node <machine>
   

   The drain moves the machine’s pods elsewhere. It can take several minutes and print eviction retries while storage moves. If it does not finish, see When a Drain Does Not Finish.

   Then, on the machine itself, uninstall k3s: run /usr/local/bin/k3s-uninstall.sh if it was a server, or /usr/local/bin/k3s-agent-uninstall.sh if it was an agent.

2. Only when the first machine is out, do the same for the second. Taking two out at once can leave too few machines for the workloads.

3. List both machines under their new roles in inventory.ini, and run deploy.

With a single server, promoting two agents leaves the server alone to run every workload while both agents are out. Expect pods to stay Pending until the machines rejoin. The deploy that brings them back allows for this, and waits up to 10 minutes for the pods to start.

If a machine is listed under a role other than the one it runs, deploy stops with [ACD-INVENTORY-ROLE-CHANGED] before changing anything.

When a Drain Does Not Finish

Kafka keeps its own copies of its data across its brokers, so each broker’s storage volume has a single replica. The storage layer (Longhorn) does not let the last copy of a volume leave a machine, so draining the machine that holds a broker’s volume retries until you stop it. Either:

• Move the broker first. Give its volume a second replica, wait for the copy, then drain:

  kubectl get pods -o wide | grep kafka     # the broker on this machine
  kubectl get pvc                           # its volume: the VOLUME column
  kubectl -n longhorn-system patch volumes.longhorn.io <volume> \
      --type merge -p '{"spec":{"numberOfReplicas":2}}'
  kubectl -n longhorn-system get volumes.longhorn.io <volume>
  

  Drain when the last command shows the volume healthy. After the machine has rejoined, set numberOfReplicas back to 1 the same way.

• Or add --disable-eviction to the drain. It removes the pods without waiting for their disruption budgets. The broker is then down until the machine rejoins: its volume’s only copy stays on the machine’s disk, comes back when the machine joins again, and the broker then catches up from the others. Until then, messages that only that broker holds are unavailable.

A database’s disruption budget can block a drain the same way, and --disable-eviction gets past it too.

Things You Cannot Change

The cluster is built around a few settings that cannot change once it is installed: backplane_network, cluster_pod_cidr, cluster_svc_cidr, the firewall zone and policy names, each machine’s node_ip and node_external_ip, and the machines’ hostnames. See Locked at First Install.