This project presents a first working state for live migration of bhyve guests on FreeBSD 15.1. A running amd64 guest can move between two hosts over TCP while continuing to run; it is paused only for the final stop-and-copy phase. The work builds on bhyve's experimental snapshot support and adds dirty-page tracking in the vmm kernel module together with a migration protocol in bhyve and bhyvectl.
I previously discussed the state of bhyve live migration, its historical development, existing limitations, and why it matters for the future of FreeBSD virtualization in my article bhyve on FreeBSD and VM Live Migration – Quo vadis?.
The goal is deliberately practical: make it possible to test, discuss, and improve a real implementation rather than discuss live migration only in the abstract. The current code and test tooling are being published as a WIP RFC so that the next steps can be shaped in the open. A more detailed demonstration of a VM being live migrated can be watched on my YouTube channel.
Why this matters
Live migration is an important building block for enterprise virtualisation. It enables host maintenance, hardware replacement, load balancing, and failure-domain management with far less disruption to workloads. It is also central to the Sylve project, which aims to offer a Proxmox-like virtualisation solution for FreeBSD. Migration is not an optional convenience in that kind of platform; it is one of the capabilities that makes clustered operations practical.
It is equally important for BoxyBSD, where I offer free BSD-based VPS instances to individuals. Without a credible migration path, routine host work can require a guest outage. That makes live migration a current show stopper for the project, and is why this work is being pushed toward a testable, reviewable state.
How the current implementation works
The source starts with a pre-copy phase. It sends populated guest memory, then repeatedly obtains a bitmap of pages changed since the previous scan and sends only those pages. Once the remaining dirty set fits the configured downtime target, stops shrinking, or reaches the configured round limit, bhyve pauses the guest and its devices. It then sends the final dirty pages and the existing snapshot state for the vCPUs, kernel state, and emulated devices.
The destination validates the basic VM geometry, receives memory and state, restores it, and reports that it is ready. Only then does the source send GO, the commit point. Before GO, a failed migration can resume the source guest. The destination does not start the guest if it does not receive that commit message. A warm, stop-and-copy mode is available too, which is useful both as a fallback and as a comparison baseline.
What is implemented today
- Pre-copy and warm migration: live pre-copy is configurable with downtime and maximum-round controls; explicit stop-and-copy migration is also supported.
- Multi-vCPU protocol support: the migration handshake carries the guest vCPU count, the destination refuses a count mismatch, and all guest vCPUs are paused and resumed as part of migration. The published test bed has primarily exercised a one-vCPU guest, so multi-vCPU stress and race testing remain required before making broader claims.
- Memory consistency checks: dirty-page tracking covers guest and mapped emulation writes through page dirty state, while vmm write holds are recorded separately. The test workload rewrites and verifies guest memory and reports any missing or stale page as a mismatch. Repeated ping-pong migrations have been used to check that the workload continues without errors.
- Disk-write validation: a guest-side workload repeatedly writes and fsyncs files, verifies checksums, and is checked again after a cold reboot. This is intended to exercise the device pause, drain, and restore path.
- Failure handling: destination refusal, an unreachable target, and a target disappearing during both pre-copy and stop-and-copy have been tested. Before commit, the source is intended to resume the paused guest after failure.
- Basic destination checks: protocol version, page size, memory layout, and vCPU count are checked before data transfer. The migration socket is opened by bhyvectl and passed into bhyve so that bhyve can remain inside its Capsicum sandbox.
- Supported scope: this is amd64-only and depends on the experimental BHYVE_SNAPSHOT support. It relies on snapshot-capable emulated devices; PCI passthrough is not migratablem, yet.
Observed results
In the current two-host test environment, repeated live migrations, memory-rewrite workloads, and fsync-heavy disk workloads have completed successfully. For a slowly dirtying 512 MiB guest, observed downtime has been roughly 138–171 ms. A rapidly dirtying guest may not converge during pre-copy, but still falls back to a correct stop-and-copy completion with a longer pause. These are encouraging engineering results, not a production support promise or a complete correctness proof.
Blockers before upstreaming can be evaluated further
The following are neutral engineering requirements, not criticism of the value of the prototype. They need to be addressed first before further upstreaming into FreeBSD can be evaluated responsibly.