Immutable Images Stay Immutable Until Someone SSHes In
A Packer template that built the same golden image every time in year one does not always build the same image in year three. A base image reference drifts to a new upstream minor version, or a provisioner role picks up a new default from a collection update, and neither shows up as a diff in the template file, so the pipeline stays green while the artifact quietly stops being reproducible.
Packer HCL defines the base image, Ansible hardens it as the provisioner, and the result gets a version tag and promotes unchanged through dev, staging, and production, disposable rather than patched. That is the guarantee three failures erode, none of them tripping a red build. Reproducibility goes first, because a green build gets treated as proof the artifact still matches what shipped last time, and nothing in the pipeline checks that assumption.
A hardening role that changes a kernel parameter gets validated against the build environment, not against what the workload on top of it needs from the OS underneath. I have seen a CIS-hardened image pass every check and then break container networking in production: the CIS Level 1 network baseline reset net.ipv4.ip_forward to 0, correct for a host that is not a router, wrong for a host running Docker’s bridge network. Containers came up. Outbound traffic did not, and the benchmark never asked what depended on that sysctl staying at 1.
The third failure erodes the guarantee itself, not just the artifact. Immutability holds only as long as nobody logs into a running instance and changes it by hand: a hotfix during an incident, a config tweak nobody wants to wait for the next image build to ship. Each exception is defensible, but the same authority gap that breaks a source of truth breaks an image fleet, and once a server is hand-edited, redeploying it from the golden image stops being routine and becomes a decision someone has to justify.
Treating servers as disposable is a discipline enforced at the moment someone is tempted to SSH in, not a property the Packer template carries on its own. The rebuild stage of a closed-loop drift architecture depends on that discipline holding, because rebuild only replaces drift with a known-good state if the known-good state is still reproducible. The pipelines that never produce an exception got the exception path built into the pipeline instead of routed around it.