Proxmox SDN Fabrics for Full-Mesh Ceph Cluster Networks

Proxmox's SDN Fabrics stack lets you build a routed full-mesh Ceph network by clicking through Datacenter > SDN > Fabrics — Proxmox generates the FRR routing config for you, so you never hand-edit /etc/frr/frr.conf again. The concrete outcome: a three-node hyper-converged cluster gets an OpenFabric or OSPF underlay with automatic multi-path failover across its NICs, no dedicated switch required. Proxmox VE 9.2 pushes this further with WireGuard as a fabric protocol for encrypted multi-site meshes and BGP with EVPN/VXLAN route-map filtering.

Key Takeaways

• No hand-editing: PVE auto-generates and versions the FRR config from the GUI, then applies it cluster-wide with one action.
• OpenFabric first: IS-IS-based OpenFabric scales more cleanly than OSPF for a growing mesh; OSPF is fine for a static 3-node box.
• Switchless mesh: A routed underlay gives Ceph automatic multi-path and per-link failover without a top-of-rack switch.
• 9.2 encryption: WireGuard fabric protocol brings encrypted underlays for multi-site clusters; BGP adds EVPN/VXLAN with route-map control.
• Verify always: vtysh -c "show openfabric neighbor" (or show ip ospf neighbor) confirms adjacencies before you trust the mesh with Ceph traffic.

Why route the Ceph mesh instead of bridging it?

For years the standard 3-node hyper-converged pattern was a hand-rolled full mesh: three NICs per node, cross-connected, glued together with a broadcast bond or a manual FRR config full of OpenFabric stanzas copied from the wiki. It worked, but every reinstall meant re-typing routing config, and a single typo in /etc/frr/frr.conf would silently break OSDs peering.

SDN Fabrics, shipped in the Proxmox VE 9 SDN stack and extended in 9.2, moves that whole layer into the cluster's config database. You describe the fabric — which nodes, which interfaces, which protocol — in the GUI, and PVE renders the correct FRR configuration on every node and reloads frr.service. The routing state now lives in /etc/pve, so it is cluster-replicated and survives a node rebuild.

The payoff for Ceph is specific. A routed underlay gives you equal-cost multi-path (ECMP) across the mesh links, so Ceph replication traffic uses every NIC and reroutes around a dead cable in seconds instead of stalling. If you have already stood up storage the traditional way in Proxmox Ceph Storage: Hyper-Converged Cluster Setup, Fabrics is the networking upgrade that removes the fragile hand-built mesh underneath it.

Building a fabric under Datacenter > SDN > Fabrics

The workflow is deliberately boring, which is the point. In the GUI:

1. Go to Datacenter > SDN > Fabrics and click Add to create a fabric, giving it a name and picking the protocol (OpenFabric, OSPF, or in 9.2 BGP/WireGuard).
2. Add each cluster node to the fabric and assign the interfaces that carry the mesh — the physical NICs cross-connected between nodes.
3. Assign a router-id / loopback per node (Fabrics manages a dummy loopback used as the routing identity).
4. Click Apply on the SDN panel to push the generated FRR config to every node.

That "Apply" step is the one people forget. Fabric edits stage as pending config; nothing touches FRR until you apply the SDN configuration, exactly like VNets and zones. If you have worked through Proxmox SDN: Configure Zones, VNets, and Subnets, this is the same pending/apply model you already know.

Under the hood, the apply writes per-node FRR config and reloads it. You can confirm the daemon is happy:

systemctl status frr.service
vtysh -c "show running-config"

The running-config there is machine-generated. Resist the urge to edit /etc/frr/frr.conf by hand — the next apply overwrites it, and the wiki explicitly warns that hand edits to a Fabric-managed FRR config are not supported.

OpenFabric or OSPF: which protocol for a small cluster?

Fabrics gives you both link-state options, and the choice matters more than it looks.

Factor	OpenFabric (IS-IS)	OSPF
Basis	IS-IS, runs directly on layer 2	OSPFv2, runs over IP
Scaling	Cleaner as node/link count grows	Fine for small, static meshes
Addressing	Address-family agnostic (v4/v6 alike)	IPv4 only in the Fabrics implementation
Config surface	Minimal, area-free for a flat mesh	Areas/costs to reason about
Best for	Growing homelab, future multi-node	A fixed 3-node box you won't expand

For a full-mesh Ceph underlay I reach for OpenFabric by default. IS-IS was built for exactly this kind of routed fabric, it treats IPv4 and IPv6 identically, and it stays quiet as you add a fourth or fifth node. OSPF is not wrong for a static three-node cluster — it is well understood and the neighbor tooling is familiar — but Proxmox's OSPF fabric is OSPFv2 (IPv4 only), and it carries more conceptual baggage (areas, per-interface cost) than a flat storage mesh needs.

The honest tradeoff: OpenFabric is less familiar to sysadmins who cut their teeth on OSPF, so troubleshooting means learning show openfabric output instead of the OSPF commands you already have muscle memory for. If your team only knows OSPF and the cluster will never grow past three nodes, OSPF will cost you less on day two.

How to wire a routed full-mesh Ceph underlay

The physical layout is unchanged from the classic hand-rolled mesh: on a 3-node cluster, each node gets two mesh NICs (say enp1s0 and enp2s0) cross-connected so every node has a direct link to the other two. No switch sits in the storage path.

What changes is that you stop configuring routing by hand. Add all three nodes to an OpenFabric fabric, assign those two NICs on each node as fabric interfaces, and apply. Each node advertises its loopback, and OpenFabric computes paths across both physical links. Because there are two disjoint paths between any pair of nodes, ECMP load-balances Ceph traffic and, when a cable dies, IS-IS reconverges over the survivor — typically inside a couple of seconds, well under Ceph's default OSD heartbeat grace.

Then point Ceph at the fabric. Keep public and cluster networks separate: the Ceph public network (client/MON traffic) can ride your regular LAN, while the Ceph cluster network (OSD replication and backfill) rides the routed mesh loopback subnet. In /etc/pve/ceph.conf:

[global]
    public_network = 10.10.0.0/24
    cluster_network = 10.255.255.0/24

Here 10.255.255.0/24 is the loopback range the fabric routes. OSD replication now flows over the multi-path mesh, isolated from client I/O — the separation that keeps a rebalance from starving VM storage.

A realistic gotcha from doing this on real hardware: MTU. If you run jumbo frames for Ceph (9000 MTU is common and worth it for a 10GbE+ mesh), the MTU must be set consistently on every mesh NIC and match what Ceph expects. A single interface left at 1500 will not fail loudly — instead OSDs will show intermittent slow ops under load while small packets pass fine. Set it in the interface config and verify with ip link show enp1s0 on all three nodes before you trust the mesh.

For a purely switchless two-link mesh, Fabrics is a cleaner successor to the bond-based approach in Proxmox Network Bonding: Link Aggregation and Failover: instead of a broadcast bond faking redundancy at layer 2, you get real routed multi-path at layer 3.

VE 9.2: WireGuard and BGP fabric protocols

Proxmox VE 9.2 is where Fabrics grows past the single-site storage mesh.

WireGuard as a fabric protocol means the underlay links between nodes can be encrypted tunnels rather than raw NIC-to-NIC connections. Cryptographic keys are generated and managed per node, so a two-site cluster — or a stretched cluster reaching a remote node over the internet — becomes an encrypted mesh without you standing up and babysitting WireGuard interfaces by hand. Fabrics manages the tunnel config alongside the routing, and the routing process runs over the encrypted transport. For a homelab spanning two locations, this is the difference between exposing Ceph replication in cleartext and never doing that.

BGP with EVPN/VXLAN arrives for larger or more segmented deployments. The BGP fabric uses an eBGP unnumbered underlay — each node gets a unique ASN and peers over the physical interfaces without IP addresses on the fabric links. BGP as a fabric protocol lets you filter route exchange with route maps, and it underpins EVPN/VXLAN overlays where you want tenant isolation on top of the routed underlay. This is where Fabrics connects to the overlay networking you may already run with Proxmox Open vSwitch Setup for Advanced VM Networking — the fabric is the routed foundation the VXLAN overlay rides on.

Controlling route exchange with prefix lists and route maps

Once BGP or a multi-fabric setup enters the picture, you rarely want every node advertising every prefix. In 9.2, route maps and prefix lists are first-class SDN objects, so you can decide exactly which routes cross the fabric — for example, advertising only the Ceph cluster subnet into the storage fabric and keeping management prefixes out of it.

A prefix list that permits only the Ceph cluster network looks like this in the generated FRR config:

ip prefix-list CEPH-CLUSTER seq 10 permit 10.255.255.0/24
route-map FABRIC-OUT permit 10
 match ip address prefix-list CEPH-CLUSTER

You configure the intent in the GUI; PVE renders the FRR stanzas. After any change, apply the SDN config so the route-map propagates cluster-wide — the same one-action apply that pushes the rest of the fabric. The value here is keeping the storage fabric's routing table tight: fewer prefixes means faster convergence and no accidental path where OSD replication leaks onto a management link.

A two-layer spine-leaf example

For clusters past three or four nodes, a flat mesh stops being practical — the cross-connect NIC count explodes. Fabrics supports a two-layer spine-leaf topology instead:

• Leaf layer: each Proxmox/Ceph node connects up to the spines. Leaves originate the node loopbacks and the Ceph cluster subnet.
• Spine layer: dedicated routing nodes (or switches speaking OpenFabric/BGP) that every leaf peers with. Spines carry transit only, no endpoints.

Every leaf has an equal-cost path to every other leaf through each spine, so ECMP spreads Ceph replication across all uplinks and any single spine or uplink failure just removes one path. You keep the same Ceph public/cluster split — cluster traffic stays on the routed fabric loopbacks, public traffic on the client network — but now it scales horizontally by adding leaves rather than re-cabling a mesh. OpenFabric's clean scaling is exactly why it is the better protocol pick the moment spine-leaf is on your roadmap.

Verify adjacencies before trusting the mesh

Applying the fabric is not the finish line — confirm the routing actually formed. Drop into FRR's shell with vtysh on each node.

For OpenFabric:

vtysh -c "show openfabric neighbor"
vtysh -c "show openfabric topology"

For OSPF:

vtysh -c "show ip ospf neighbor"
vtysh -c "show ip route ospf"

On a healthy 3-node OpenFabric mesh, every node will list two neighbors in the Up state and the route table will show two equal-cost next-hops toward each remote loopback. If a neighbor is missing, the usual culprits are a mismatched MTU, an interface not actually assigned to the fabric, or a forgotten SDN apply. Pull one mesh cable and re-run the command: you will watch the neighbor drop and traffic reconverge onto the surviving path, which is the whole reason you built a routed underlay instead of a static one.

Conclusion

SDN Fabrics turns the fragile, hand-edited full-mesh Ceph underlay into a GUI-managed, cluster-replicated routing layer that heals itself when a NIC dies — and in VE 9.2 it stretches encrypted across sites with WireGuard and BGP. Pick OpenFabric for anything you might grow, keep the Ceph cluster network on the routed loopbacks, and always confirm adjacencies with vtysh before you cut storage over. Next step: stand up the fabric on a test 3-node cluster, then move your Ceph cluster_network onto it and watch a cable-pull reconverge without an OSD flapping.