ZFS vs Ceph on Proxmox: Which Cluster Storage Should You Run?

Every Proxmox cluster eventually runs into the same fork in the road: where do the VM disks actually live? On a single node the answer is easy — put them on local ZFS and move on. The moment you add a second and third node and start caring about high availability, the question changes shape entirely. Now you are choosing between two philosophies. ZFS keeps storage local, fast, and simple, and leans on replication to copy data between nodes. Ceph turns your drives into one shared pool that every node can read and write at the same time, at the cost of a much heavier hardware and network commitment. Neither is "better" in the abstract — they solve different problems. This guide walks through the real tradeoffs so you can pick the one that matches your node count, your network, and how much downtime you can actually tolerate.

Key Takeaways

• ZFS is local, Ceph is shared — ZFS lives on one node and replicates copies elsewhere; Ceph presents a single distributed pool accessible from every node simultaneously.
• Failover behavior differs fundamentally — Ceph failover loses zero data because writes are synchronous; ZFS replication is periodic, so a failover loses everything written since the last sync.
• Ceph demands a fast dedicated network — plan for 10GbE minimum, ideally 25GbE or better; ZFS is happy on whatever network you already have because traffic stays local.
• Hardware cost is real — Ceph's 3x replication means you buy roughly three times the raw capacity for the usable space you get; ZFS mirrors or RAIDZ are far more space-efficient.
• Node count decides it — one to three nodes usually points to ZFS; three or more nodes where true HA matters is where Ceph earns its complexity.

How ZFS and Ceph Actually Store Data

ZFS is a local filesystem and volume manager fused into one. When you create a pool with zpool create on a Proxmox node, that pool belongs to that node and nothing else. VMs whose disks sit on it get excellent performance because every read and write stays on the local PCIe or SATA bus — there is no network in the path. You get copy-on-write snapshots, transparent lz4 compression, and end-to-end checksums for free. The catch is that the data is stranded on one machine. If that node dies, its VMs cannot start elsewhere unless a copy of their disks already exists on another node, which is exactly what ZFS Replication on Proxmox: Sync Data Between Nodes sets up.

Ceph works from the opposite direction. Instead of a filesystem tied to one host, it is a distributed object store that pools the disks (called OSDs) across every node into one logical namespace. Proxmox consumes it as RBD — RADOS Block Devices — so a VM disk is really a set of objects spread and replicated across the cluster. Because the data is already present on multiple nodes, any node can start any VM at any time. When you write a block, Ceph does not acknowledge it until it has landed on the configured number of replicas (three by default). That synchronous, multi-node write is the whole point: it is what makes true high availability possible, and it is also what makes Ceph so sensitive to network latency.

Performance: Local Speed vs Distributed Resilience

On raw numbers, local ZFS on NVMe will almost always beat Ceph for a single VM's latency. There is simply no substitute for keeping the data on the same box as the CPU. If you are running a database, a busy Nextcloud instance, or anything where fsync latency matters, local ZFS on good NVMe with a properly tuned ARC — see Proxmox ZFS Tuning: ARC, L2ARC, and SLOG Guide — is hard to beat.

Ceph's performance is a different shape. A single-threaded, single-VM benchmark will look unimpressive next to local NVMe because every write makes a network round trip to two other nodes before it is acknowledged. But Ceph is built to scale out: aggregate throughput across many VMs and many OSDs on a fast network can be enormous, and it keeps climbing as you add nodes and drives. The lever that matters most is the network. On 1GbE, Ceph will feel painfully slow and latency-bound; on 10GbE it becomes usable; on 25GbE or faster with a dedicated cluster network, it starts to feel like proper shared storage. If you take one thing away, it is this: Ceph performance is a network-engineering problem as much as a storage one.

Hardware and Network Requirements

This is where a lot of homelabbers underestimate Ceph. ZFS is forgiving — it will run on a single disk, a pair of mirrored drives, or a RAIDZ vdev, and it wants RAM for its ARC read cache (budget generously; more RAM directly improves read performance). You can build a perfectly good ZFS node on hardware you already own.

Ceph asks for more, in three specific ways:

• Drive count and capacity. With default 3x replication, every usable terabyte costs you three terabytes of raw disk. A cluster that needs 4 TB usable is really a 12 TB raw purchase spread across nodes. Erasure coding can improve that ratio but adds CPU overhead and complexity that most homelabs skip.
• A dedicated cluster network. Ceph replication and rebalancing traffic should never share a link with your VM traffic or, worse, your management interface. Plan a separate 10GbE+ network — this is not optional advice, it is the difference between a healthy cluster and one that stalls whenever a disk fails and Ceph starts backfilling. Proxmox Network Bonding: Link Aggregation and Failover is a good starting point for building that link.
• A minimum of three nodes. Below three, you cannot satisfy min_size=2 when a node is down and still maintain quorum among monitors. Three is the floor; more is better for both resilience and performance.

ZFS, by contrast, is content on the network you already have because its data path is local. Replication traffic is periodic and easy to schedule off-peak.

High Availability: Where the Real Difference Lives

If HA is on your requirements list, this section is the whole decision. Both approaches can restart a VM on a surviving node, but what the VM finds when it boots is completely different.

With Ceph, the VM's disk is already present and current on the surviving nodes because every write was replicated synchronously. Combined with Proxmox High Availability Setup for Automatic VM Failover, a node failure means the VM restarts elsewhere in seconds and loses nothing — it is exactly as if the machine had been power-cycled. No data is left behind on the dead node because there was never any data only on the dead node.

With ZFS replication, the surviving node holds a copy of the disk as of the last replication run. Proxmox's storage replication runs on a schedule with a one-minute minimum interval. So if the source node dies 50 seconds after the last sync, the VM fails over to a disk that is 50 seconds stale — every write in that window is gone. For many homelab workloads (a Pi-hole, a reverse proxy, a media server) that is completely acceptable. For a database or anything transactional, it is not. Be honest about which category your workloads fall into, because this single behavior is the clearest dividing line between the two.

Setting Each One Up on Proxmox

ZFS you can configure at install time or afterward. To create a mirrored pool from two disks and hand it to Proxmox:

# Create a mirrored ZFS pool
zpool create -o ashift=12 tank mirror /dev/disk/by-id/nvme-A /dev/disk/by-id/nvme-B
zfs set compression=lz4 tank

# Register it with Proxmox as a storage target
pvesm add zfspool zfs-vm --pool tank --content images,rootdir

Always reference disks by their /dev/disk/by-id/ path, never /dev/sdX, so the pool survives a reboot that reorders devices. From there, Setting Up ZFS on Proxmox: Pools, Datasets, and Best Practices covers dataset layout and tuning.

Ceph is best deployed through the Proxmox web UI (Datacenter → Ceph), but the CLI shows what is happening under the hood. On each node:

# Install Ceph packages on every node
pveceph install

# On the first node, initialize the cluster on the dedicated network
pveceph init --network 10.10.10.0/24

# Create a monitor and manager
pveceph mon create
pveceph mgr create

# Turn each spare disk into an OSD
pveceph osd create /dev/nvme1n1

# Create an RBD pool for VM disks (3x replication)
pveceph pool create vm-pool --size 3 --min_size 2

Once the pool is created and added as RBD storage, every node in the cluster can place VM disks on it. The full hyper-converged walkthrough, including CRUSH rules and health monitoring, lives in Proxmox Ceph Storage: Hyper-Converged Cluster Setup.

When to Pick Each One

Choose ZFS when:

• You run one to three nodes, or a cluster where per-VM latency matters more than instant failover.
• Your budget rules out buying 3x raw capacity and a dedicated fast network.
• Your workloads tolerate losing a minute of writes on a failover (most homelab services do).
• You want maximum local performance from NVMe and rich snapshot/compression features with minimal moving parts.

Choose Ceph when:

• You have three or more nodes and genuinely need any VM to run on any node.
• Zero-data-loss failover is a hard requirement, not a nice-to-have.
• You can commit to a dedicated 10GbE+ cluster network and the extra raw disks replication demands.
• You are building toward scale and want storage that grows by adding nodes rather than forklift-upgrading a SAN.

A common and underrated pattern is to run both: local ZFS for the hypervisor's own disks and latency-sensitive VMs, plus a Ceph pool for the workloads that must survive a node loss. Proxmox is perfectly happy hosting both storage types side by side, and the flexibility is worth more than religiously picking one.

Conclusion

The ZFS-versus-Ceph decision is really a question about failure, not features. ZFS gives you fast, cheap, simple local storage and asks you to accept that a node failure costs you a copy operation and possibly a minute of writes. Ceph gives you shared storage that shrugs off a dead node with zero data loss, and asks you to pay for it in drives, network, and complexity. For most one-to-three-node homelabs, ZFS is the pragmatic winner and the reason so many builds start there. Once you cross into three-plus nodes where uptime is the product, Ceph stops being overkill and starts being the point. If you are still mapping out your cluster, How to Set Up a Proxmox Cluster: Complete Two-Node Guide is the natural next step — get the cluster right first, then layer the storage decision on top.