https://proxmoxpulse.com 2026-05-06T01:47:11.942Z https://github.com/jpmonette/feed Proxmox Pulse hello@proxmoxpulse.com https://proxmoxpulse.com In-depth Proxmox VE tutorials, tips, and best practices for homelab enthusiasts and system administrators. Covers installation, VMs, LXC containers, storage, networking, and more. https://proxmoxpulse.com/images/og-default.png https://proxmoxpulse.com/favicon-32x32.png © 2026 Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-ldap-active-directory-authentication/ 2026-05-06T00:00:00.000Z

> /var/log/proxmox-ldap-sync.log 2>&1 ``` With `--purge false`, users removed from AD won't lose Proxmox access until you enable purge or manually disable them. In environments with strict offboarding requirements, run a weekly purge job as well: ```bash # Weekly purge — removes Proxmox users no longer in LDAP 0 5 * * 0 root /usr/bin/pveum realm sync corp --purge true --scope users >> /var/log/proxmox-ldap-sync.log 2>&1 ``` Pair this with [SSH hardening and fail2ban](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) to make sure stale accounts left behind by a delayed purge cycle can't be exploited. ## Mapping AD Groups to Proxmox Roles This is where centralized auth pays off. Instead of assigning roles to individual users, assign them to groups — Proxmox respects LDAP group membership for permission decisions. First, confirm groups appeared after the sync: ```bash pveum group list ``` Then assign roles to groups at the appropriate resource path: ```bash # 'infra-admins' AD group gets full Administrator access cluster-wide pveum acl modify / --group 'infra-admins' --role Administrator # 'dev-team' gets VM admin rights only in the dev resource pool pveum acl modify /pool/dev-pool --group 'dev-team' --role PVEVMAdmin # Read-only auditors can view everything, change nothing pveum acl modify / --group 'proxmox-readonly' --role PVEAuditor ``` The built-in roles worth knowing: | Role | What It Allows | |------|---------------| | `Administrator` | Full cluster access — treat like root for Proxmox operations | | `PVEAdmin` | Manage VMs, storage, networks — no user or realm management | | `PVEVMAdmin` | Full VM lifecycle — no node or storage administration | | `PVEVMUser` | Start, stop, and open console — no config changes | | `PVEAuditor` | Read-only view of the entire cluster | | `PVEDatastoreAdmin` | Manage backups and storage pools — useful for dedicated backup operators | If none of these fit, create a custom role: ```bash pveum role add StorageOperator \ --privs "Datastore.Audit,Datastore.AllocateSpace,Datastore.AllocateTemplate" ``` ## Testing and Verifying Authentication End-to-End Before announcing the change to your team, verify the full chain works from the Proxmox host itself: ```bash # Test the LDAP bind directly (install if missing: apt-get install -y ldap-utils) ldapsearch -H ldaps://dc01.corp.example.com \ -D "svc-proxmox@corp.example.com" \ -w "YourLongRandomPassword!" \ -b "DC=corp,DC=example,DC=com" \ "(sAMAccountName=testuser)" cn mail ``` A successful bind returns `cn` and `mail` attributes for the test user. Common errors: - `ldap_bind: Invalid credentials (49)` — wrong bind DN or password - `Can't contact LDAP server` — firewall blocking port 636 or DNS failure; test reachability with: ```bash nc -zv dc01.corp.example.com 636 ``` - `TLS: hostname does not match` — the server cert CN or SAN doesn't match the hostname you configured; use the FQDN that matches the certificate Then test interactively: log out of the Proxmox GUI, select the `corp` realm in the login dropdown, and authenticate as a known AD user. If you're using [Ansible playbooks to manage your Proxmox cluster](/articles/automate-proxmox-ansible-vm-playbooks/), add an LDAP connectivity check task to your pre-upgrade playbook — LDAP failures discovered mid-upgrade are painful. If realm config changes aren't taking effect, restart `pvedaemon`: ```bash systemctl restart pvedaemon ``` ## What to Do When LDAP Auth Breaks The most common failure scenario: the LDAP server becomes unreachable, or the bind account password expires, and suddenly nobody can log in. This is exactly why you keep a local admin account active and tested. ```bash # Log in as root@pam via console, iDRAC, iLO, or IPMI # Then check configured realms pveum realm list # Disable the broken realm while you investigate # (existing sessions continue; new logins to this realm fail with a clean error) pveum realm modify corp --disable true # Re-enable once the LDAP issue is resolved pveum realm modify corp --disable false ``` Common root causes when LDAP breaks suddenly: 1. **Bind account password expired** — check in AD: ```powershell Search-ADAccount -PasswordExpired -UsersOnly | Select-Object Name, PasswordExpiredAt ``` 2. **Certificate expired** — check the LDAPS cert expiry from the Proxmox host: ```bash echo | openssl s_client -connect dc01.corp.example.com:636 2>/dev/null \ | openssl x509 -noout -dates ``` 3. **Domain controller unreachable** — DNS change, firewall rule update, or DC maintenance window 4. **Realm config corrupted after a Proxmox upgrade** — inspect `/etc/pve/domains.cfg` directly for garbled entries; the file is plain text and editable Active sessions are unaffected when LDAP goes down — Proxmox session tokens aren't re-validated against LDAP on every request. Anyone already logged in continues working until their session expires (default timeout is 2 hours). New logins fail. ## Conclusion Proxmox VE's LDAP and Active Directory integration eliminates the overhead of managing Proxmox-specific passwords for every admin — the setup takes under 30 minutes once you have the bind account and base DN in hand. The discipline that makes it reliable over the long term is three things: a tested local admin account as a fallback, LDAPS with verified certificates in production, and scheduled sync jobs rather than ad-hoc manual runs. Once AD groups map to Proxmox roles, onboarding a new team member is as simple as adding them to the right security group — Proxmox picks it up on the next sync. To close down the remaining attack surface after enabling centralized auth, work through [SSH hardening, fail2ban, and the Proxmox firewall](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) as your next step. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-api-tokens-secure-automation/ 2026-05-05T00:00:00.000Z

0.66" } } } provider "proxmox" { endpoint = "https://192.168.1.10:8006/" api_token = var.proxmox_api_token insecure = false } ``` In `terraform.tfvars` — add this file to `.gitignore` before your first commit: ```hcl proxmox_api_token = "automation@pve!terraform=a1b2c3d4-e5f6-7890-abcd-ef1234567890" ``` Or export it as an environment variable so it never touches the filesystem at all: ```bash export TF_VAR_proxmox_api_token="automation@pve!terraform=a1b2c3d4-e5f6-7890-abcd-ef1234567890" ``` ## Using the Token in Ansible The `community.general.proxmox_kvm` module accepts token credentials directly: ```yaml - name: Provision a VM community.general.proxmox_kvm: api_host: 192.168.1.10 api_user: automation@pve api_token_id: terraform api_token_secret: "{{ proxmox_token_secret }}" node: pve name: my-vm cores: 2 memory: 2048 state: present ``` Store `proxmox_token_secret` in an Ansible Vault file, not in a plaintext `group_vars` file. Use `ansible-vault encrypt_string` to inline-encrypt the value if you prefer a single-file setup over a separate vault. ## Rotating and Revoking Tokens List all tokens for a user: ```bash pveum user token list automation@pve ``` Revoke a token immediately — effective in seconds, no grace period: ```bash pveum user token remove automation@pve terraform ``` For a zero-downtime rotation, create the replacement first, then remove the old one: ```bash # 1. Create replacement token pveum user token add automation@pve terraform-v2 --privsep 1 --comment "Rotated 2026-05-05" # 2. Assign ACLs to the new token pveum acl modify /vms \ --users automation@pve \ --tokens automation@pve!terraform-v2 \ --roles PVEVMAdmin pveum acl modify /storage/local-lvm \ --users automation@pve \ --tokens automation@pve!terraform-v2 \ --roles PVEDatastoreUser # 3. Update secrets manager, test new token, update all consumers # 4. Remove old token pveum user token remove automation@pve terraform ``` The gotcha here: token ACLs are attached to the token ID, not the user. When you delete and recreate a token — even with the identical name — all ACLs are gone and must be reassigned from scratch. Keep a documented list of which ACLs each token needs, or script the reassignment as part of your rotation runbook so you do not rediscover this at 2am. ## Why the PVE Realm Is Safer for Service Accounts A `automation@pve` user has no Linux system account. There is no `/etc/passwd` entry, no shadow file entry, and no ability to SSH into the Proxmox host using those credentials directly. If the API token is compromised, the attacker has API access scoped to the token's ACLs — they cannot pivot to an interactive shell on the hypervisor. A `automation@pam` user maps to a real Linux account that does have a shadow entry. Depending on your SSH configuration, that user may be able to authenticate to the host directly. The [LOLPROX analysis of Proxmox hypervisor exploit paths](/articles/lolprox-protecting-proxmox-from-hypervisor-exploits/) makes this point clearly: reducing the identity attack surface at the API layer is one of the highest-leverage hardening steps available. Use the PVE realm for every service account. Reserve PAM accounts for humans who need both shell access and web UI access. ## Conclusion Replacing root credentials with scoped API tokens is a 20-minute change that pays back every time you rotate a secret, offboard a team member, or recover from a credential leak without touching your root account. Create `automation@pve`, assign narrowly-scoped ACLs, enable privilege separation on the token, store the secret in a proper secrets manager, and document the ACL assignments for your rotation runbook. The natural next step is putting this token to work in a structured Ansible workflow — the [Ansible VM automation guide](/articles/automate-proxmox-ansible-vm-playbooks/) walks through a production-ready inventory approach that pairs directly with what you set up here. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-lxc-resource-limits/ 2026-05-04T00:00:00.000Z

` shows real-time consumption; tune from data, not guesses. ## How Proxmox LXC Resource Controls Work Under the Hood LXC containers on Proxmox are namespaced processes sharing the host kernel — no hypervisor overhead, but also no hardware isolation. Resource limits are enforced entirely by Linux cgroup v2. When you run `pct set 101 --cpulimit 2`, Proxmox writes to `/sys/fs/cgroup/lxc/101/cpu.max` and the kernel scheduler does the rest. Each container gets its own cgroup slice. On Proxmox VE 9.1 you can inspect the full hierarchy: ```bash # Inspect the cgroup tree for container 101 systemd-cgls /sys/fs/cgroup/lxc/101 ``` Three resource classes matter for most workloads: 1. **CPU** — absolute quota and relative scheduling weight 2. **Memory** — hard ceiling, swap budget, and soft pressure hints 3. **Block I/O** — bandwidth and IOPS caps per device All three can be configured persistently via `pct set`. Changes survive reboots because Proxmox writes them back to `/etc/pve/lxc/.conf`. The one exception is I/O throttling — direct cgroup writes don't survive container restarts, which is why hook scripts matter. ## How to Set CPU Limits on LXC Containers ### The Difference Between Cores, CPU Limit, and CPU Units These three settings look similar in the Proxmox UI but control completely different scheduler knobs: | Option | cgroup v2 knob | What it actually does | |---|---|---| | `--cores N` | `cpuset.cpus` | Container sees N vCPU threads | | `--cpulimit N` | `cpu.max` | Hard cap: N × 100% of one physical core | | `--cpuunits N` | `cpu.weight` | Relative scheduling priority (default: 1024) | `--cpulimit` is the ceiling that actually prevents CPU saturation. Setting it to `2.0` means the container can consume at most 200% CPU — two full core-equivalents of wall-clock time — regardless of how many cores the host has or how many the container can see via `--cores`. ```bash # Cap container 101 to 1.5 cores worth of CPU — applies immediately pct set 101 --cpulimit 1.5 # Lower scheduling priority of a bulk-processing background container pct set 102 --cpuunits 256 # Pin a latency-sensitive container to 2 threads with a matching hard cap pct set 103 --cores 2 --cpulimit 2 ``` **Gotcha from experience**: On a 16-core host with four containers each set to `--cores 4`, all four can simultaneously peg all their cores if no `--cpulimit` is configured. I watched a Jellyfin container doing 4K transcodes bring a database container to its knees this way. Always pair `--cores` with `--cpulimit` for workloads you don't fully trust. ### Verifying the CPU Limit Took Effect ```bash # cpu.max format is: quota period (in microseconds) # 150000 100000 = 150ms per 100ms window = 1.5 cores cat /sys/fs/cgroup/lxc/101/cpu.max # Or read directly from the Proxmox config pct config 101 | grep cpu ``` The `cpu.stat` file shows cumulative throttled time — useful for confirming a container is actually hitting its limit: ```bash cat /sys/fs/cgroup/lxc/101/cpu.stat | grep throttled # throttled_usec 14230891 ← non-zero means the cap is being enforced ``` ## Configuring Memory and Swap in Proxmox LXC Memory configuration uses two knobs that work together: ```bash # Give container 105 2 GB RAM and 512 MB swap pct set 105 --memory 2048 --swap 512 ``` The part that trips people up: `--swap` is **additive**, not a total budget. The container above gets 2048 MB RAM **plus** 512 MB of swap space on top — 2560 MB of total virtual memory. If you set `--memory 2048 --swap 0`, the container has exactly 2048 MB and no swap at all. For latency-sensitive workloads like databases, disable swap entirely: ```bash # Database container: 4 GB hard limit, no swap, no latency spikes from swapping pct set 106 --memory 4096 --swap 0 ``` ### Why the OOM-Killer Fires Instead of Throttling When a container hits its memory ceiling, the Linux kernel doesn't pause it or throttle allocations — it runs the OOM-killer and terminates the process with the highest `oom_score` inside that cgroup. For stateless services (nginx, Redis with `maxmemory` set, Prometheus), this is usually survivable. For PostgreSQL or any workload with a write-ahead log, it can corrupt data mid-write. The practical rule: set `--memory` at least 20-30% above your measured working set, and monitor `memory.current` before tightening: ```bash # Current RSS for container 106, human-readable cat /sys/fs/cgroup/lxc/106/memory.current | numfmt --to=iec # Example output: 1.8G ``` ### Soft Memory Pressure with memory.low Proxmox doesn't expose a soft memory limit in the UI, but cgroup v2's `memory.low` knob is available directly. Writing to it tells the kernel to evict other containers' pages before touching this container's working set under host memory pressure: ```bash # Protect container 105's working set below 1.5 GB from host reclaim echo $((1536 * 1024 * 1024)) > /sys/fs/cgroup/lxc/105/memory.low ``` This is a hint, not a guarantee — but it meaningfully improves behavior when you're running a mix of critical services and background batch jobs on the same host. ## Disk I/O Throttling: The Feature the Proxmox UI Skips Per-container I/O throttling is absent from the Proxmox VE 9.1 web interface, but cgroup v2's `io.max` interface is fully functional. Without it, a container running a bulk `rsync` or a backup agent can saturate your storage bus and cause latency spikes in every other container and VM — the kind of thing that's hard to diagnose after the fact. ```bash # Find the major:minor device number of your storage pool's block device lsblk -no MAJ:MIN /dev/nvme0n1 # Example output: 259:0 # Cap container 101 to 50 MB/s read, 30 MB/s write, 3000 read IOPS, 1000 write IOPS echo "259:0 rbps=52428800 wbps=31457280 riops=3000 wiops=1000" \ > /sys/fs/cgroup/lxc/101/io.max ``` Direct cgroup writes vanish on container restart. Persist them with a hook script: ```bash # Add this line to /etc/pve/lxc/101.conf hookscript: local:snippets/iolimit-101.sh ``` ```bash #!/bin/bash # /var/lib/vz/snippets/iolimit-101.sh CTID=$1 PHASE=$2 if [ "$PHASE" = "post-start" ]; then sleep 1 # cgroup needs a moment to initialize DEVNO=$(lsblk -no MAJ:MIN /dev/pve/vm-${CTID}-disk-0 2>/dev/null || echo "259:0") echo "${DEVNO} rbps=52428800 wbps=31457280 riops=3000 wiops=1000" \ > /sys/fs/cgroup/lxc/${CTID}/io.max fi ``` ```bash chmod +x /var/lib/vz/snippets/iolimit-101.sh ``` **Important gotcha**: `io.max` applies to all block I/O the container generates — including reads and writes that go through bind mounts from the host filesystem. If you're running [Docker inside an LXC container on Proxmox](/articles/docker-inside-lxc-containers-proxmox/), all Docker layer pulls and container writes count against the same limit. A 30 MB/s write cap will throttle your Docker image builds too. Size your limits with that in mind. **Older tutorials use the wrong paths**: cgroups v1 used `blkio.throttle.read_bps_device` and `blkio.throttle.write_bps_device`. Those paths don't exist on Proxmox VE 9.x. If a guide shows those paths, it was written for Proxmox VE 7 or older. ## Monitoring Real-Time Resource Usage Before setting any limits, spend a few minutes under real workload watching actual consumption. The Proxmox web UI averages over 60-second windows and will miss short bursts entirely. ```bash # Real-time stats for container 101, refreshes every second pct monitor 101 ``` For a host-wide snapshot of all running containers: ```bash for CTID in $(pct list | awk 'NR>1 && $2=="running" {print $1}'); do echo "=== CT ${CTID} ===" printf " CPU throttled_usec: " awk '/throttled_usec/ {print $2}' /sys/fs/cgroup/lxc/${CTID}/cpu.stat 2>/dev/null printf " Memory current: " cat /sys/fs/cgroup/lxc/${CTID}/memory.current 2>/dev/null | numfmt --to=iec done ``` For I/O accounting, `io.stat` gives cumulative bytes and operations per device: ```bash cat /sys/fs/cgroup/lxc/101/io.stat # 259:0 rbytes=2048000000 wbytes=512000000 rios=150000 wios=40000 ... ``` A non-zero and growing `throttled_usec` in `cpu.stat` confirms a CPU limit is actively being enforced. If you're not seeing throttling but the container still feels slow, the bottleneck is elsewhere — check I/O wait with `iostat -x 1` on the host. ## Applying Limits in Bulk via the Proxmox API Managing limits one container at a time with `pct set` is fine for a handful of containers. For a homelab with a dozen LXCs, or a production cluster with many more, `pvesh` handles it cleanly: ```bash # Apply a "low-priority background" profile to containers 200 through 209 for CTID in $(seq 200 209); do pvesh set /nodes/pve/lxc/${CTID}/config \ --cpulimit 0.5 \ --cpuunits 256 \ --memory 512 \ --swap 256 done ``` `pvesh` takes the same parameters as `pct set` and works identically — the difference is that `pvesh` talks to the Proxmox REST API directly, so you can run it remotely or embed it in CI pipelines. If your infrastructure is already managed as code with [Ansible playbooks for Proxmox](/articles/automate-proxmox-ansible-vm-playbooks/), the `community.general.proxmox` module accepts `cpus`, `cpuunits`, `memory`, and `swap` as task parameters — same idempotent workflow, no custom scripting required. ## Common Pitfalls When Setting LXC Resource Limits **`--cpulimit` throttles even on an idle host.** Once set, the kernel enforces the CPU cap regardless of whether other containers are competing. If you have a weekly report generator that needs to run fast, use `--cpuunits` to raise its priority instead — that only activates under contention, not when the host is idle. **Swap on ZFS volumes doubles ARC pressure.** If your LXC root disk is a zvol on a ZFS pool, container swap I/O goes through ZFS, which creates its own ARC churn. For ZFS-backed containers, `--swap 0` is almost always correct — compensate with sufficient `--memory` instead. **Limits don't cover in-kernel work done on behalf of the container.** If your container generates heavy NFS traffic or triggers ZFS prefetch, those kernel threads run outside the container's cgroup. You can see a container's CPU limit enforced at 1.0 while the host shows high `%sys` from ksoftirqd or nfsd. This is expected kernel behavior with no simple workaround — it's a characteristic of the container model, not a bug in your configuration. **Resource limits are also a security boundary.** An unprivileged LXC with no CPU or memory cap can run a trivial fork bomb and degrade every other tenant on the host. Setting conservative defaults — even for containers you trust — is a meaningful layer of defense that works alongside the network and access controls covered in [Hardening Proxmox VE: Firewall, fail2ban, and SSH Security](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/). This is worth it any time you run more than two or three containers on a host. ## Recommended Baseline Settings by Workload Start from these values and adjust after a week of `pct monitor` observation: | Workload | `--cpulimit` | `--cpuunits` | `--memory` | `--swap` | |---|---|---|---|---| | Web server (nginx/Caddy) | 1.0 | 1024 | 512 MB | 256 MB | | Database (Postgres/MariaDB) | 2.0 | 2048 | 2048 MB | 0 | | Monitoring (Prometheus) | 1.0 | 768 | 1024 MB | 512 MB | | Media server (Jellyfin) | 4.0 | 512 | 2048 MB | 1024 MB | | Backup agent (restic/borgmatic) | 0.5 | 128 | 256 MB | 256 MB | | Dev/test (low priority) | 0.5 | 256 | 512 MB | 512 MB | The `--cpuunits` values are relative — only their ratios matter. A container at 2048 gets twice the scheduler slices of one at 1024 during CPU contention. On an idle host, both run unrestricted regardless of their `--cpuunits` value. The backup agent row deserves special attention: backup containers are the most common culprit for host-wide slowdowns. Capping a restic or borgmatic container to 0.5 cores and slow I/O means [your automated Proxmox Backup Server jobs](/articles/automated-backups-proxmox-backup-server/) finish a bit later — but your production containers stay responsive throughout the backup window. ## Conclusion With CPU limits, memory ceilings, and I/O throttling in place, LXC containers become proper tenants on your Proxmox host rather than free-range processes competing for the same resources. The combination of `pct set` for persistent CPU and memory configuration and hook scripts for I/O throttling covers everything the web UI doesn't expose — and most of it applies live without touching a running workload. The logical next step is measuring the impact over time: wire up per-container cgroup metrics in your monitoring stack so you can see when a container is chronically hitting its CPU quota and needs its limit raised, or when it's consistently well under and the headroom can be reclaimed. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/automate-proxmox-ansible-vm-playbooks/ 2026-05-03T00:00:00.000Z

Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-backup-server-s3-storage-backend/ 2026-05-02T00:00:00.000Z

.r2.cloudflarestorage.com ``` ### Wasabi Wasabi matches B2 at $6.99/TB/month but enforces a **90-day minimum storage policy**. Deleting objects stored less than 90 days still bills for the full 90 days. Do not use Wasabi with prune schedules shorter than 90 days or you will pay for data you no longer hold. Wasabi regional endpoint format: ``` https://s3.us-east-1.wasabisys.com ``` ## How to Add an S3 Remote in PBS 4.2 ### Via the Web UI 1. Open the PBS web UI at `https://:8007` 2. Navigate to **Configuration → Remotes** 3. Click **Add** and select **Type: S3** 4. Fill in the fields: - **ID**: a short name (e.g., `b2-offsite`) - **Endpoint**: your full S3 endpoint URL - **Bucket**: your bucket name - **Region**: the bucket's region string - **Access Key**: your `keyID` - **Secret Key**: your `applicationKey` 5. Click **Test Connection** — PBS runs a list operation against the bucket and surfaces auth errors immediately before you save ### Via CLI ```bash proxmox-backup-manager remote create b2-offsite \ --type s3 \ --endpoint "https://s3.us-west-004.backblazeb2.com" \ --bucket "pbs-offsite-2026" \ --region "us-west-004" \ --access-key "your-keyID-here" \ --secret-key "your-applicationKey-here" ``` Verify the remote saved correctly: ```bash proxmox-backup-manager remote list ``` Expected output: ``` ┌─────────────┬──────┬───────────────────────────────────────────────┐ │ name │ type │ endpoint │ ╞═════════════╪══════╪═══════════════════════════════════════════════╡ │ b2-offsite │ s3 │ https://s3.us-west-004.backblazeb2.com │ └─────────────┴──────┴───────────────────────────────────────────────┘ ``` ## Setting Up Sync Jobs and Retention ### Creating the Sync Job Via the web UI: 1. Go to **Datastore → \ → Sync Jobs** 2. Click **Add Sync Job** 3. Set **Remote** to your S3 remote 4. **Remote Store**: the namespace path to use in the S3 bucket — use your datastore name (e.g., `backups`) 5. **Schedule**: `daily` or a cron expression like `0 2 * * *` for 2 AM daily 6. **Remove Vanished**: enable this — it deletes S3 objects for snapshots that have been pruned locally Via CLI: ```bash proxmox-backup-manager sync-job create daily-s3-sync \ --store backups \ --remote b2-offsite \ --remote-store backups \ --schedule "0 2 * * *" \ --remove-vanished true ``` Trigger the first sync manually before relying on the schedule: ```bash proxmox-backup-manager sync-job run daily-s3-sync ``` Watch progress under **Administration → Task History**. A 500 GB datastore over a 100 Mbit uplink expects the initial upload to complete in 2–4 hours. Incremental syncs after that transfer only new chunks — typically 2–8 GB/day for a homelab with 3–4 VMs running daily backups. ### How Retention Propagates to S3 PBS does not enforce retention directly on S3. Pruning happens locally first, then the `remove-vanished` flag propagates deletions on the next sync cycle. The sequence: 1. Local prune job runs and removes old snapshots from the local datastore 2. Next sync job runs with `remove-vanished: true` 3. PBS compares the S3 object list against local state and deletes orphaned chunk files Your S3 bucket will lag local retention by up to one sync cycle — 24 hours at a daily schedule. On Wasabi specifically: a snapshot pruned at day 89 still triggers the full 90-day minimum charge because Wasabi sees an early deletion. Set your Wasabi prune schedule to keep at least 90 days minimum. ## What PBS Actually Uploads to S3 This is where the most common misconception lives. PBS stores backups as content-addressed 4 MB chunk files. When syncing to S3, it uploads: - **Chunk files** (`.blob`) — deduplicated backup data, already compressed - **Snapshot manifests** (`.manifest`) — links each snapshot to its chunk list - **Index files** (`.fidx`, `.didx`) — mapping tables for file-level and block-device backups It does **not** sync the task log or the searchable catalog. This means you can restore data from S3 on a fresh PBS instance by registering the S3 remote as a source datastore, but you will need to rebuild the catalog afterward: ```bash proxmox-backup-client catalog rebuild --repository @:backups ``` The 15–25% storage overhead estimate comes from these metadata files. A local datastore holding 800 GB of deduplicated backup data will occupy roughly 920 GB to 1 TB on S3. ## Provider Cost Comparison | Provider | Storage/TB/month | Egress | Min. term | Best for | |---|---|---|---|---| | Backblaze B2 | $6.00 | Free (3x stored/day) | None | Low-restore-frequency homelabs | | Wasabi | $6.99 | Free | 90 days | Long-retention cold storage | | Cloudflare R2 | $15.00 | Free | None | Frequent restores, zero surprise bills | | MinIO (self-hosted) | Hardware only | Free | None | Air-gapped or LAN backup copies | | AWS S3 Standard | $23.00 | $0.09/TB | None | Avoid for homelab-scale volumes | For a homelab with 2 TB stored, daily incrementals, and 30-day retention: B2 costs ~$12/month, Wasabi ~$14/month, R2 ~$30/month. R2's zero-egress advantage only makes financial sense if you are pulling hundreds of GB in restores per month. Most homelabs run few enough restores that B2 wins on total cost. ## Securing Credentials and Encrypting the Datastore PBS stores S3 credentials in `/etc/proxmox-backup/remotes.cfg`, readable only by root. For most setups that is sufficient. If your PBS runs as a VM on a shared Proxmox host with other administrators, enabling datastore-level encryption means data pushed to S3 is client-side encrypted before it leaves your network — your S3 provider stores only ciphertext. Enable encryption in PBS: **Datastore → \ → Encryption → Generate Key**. Store the key file somewhere other than the PBS host itself — a password manager or a dedicated secrets store. Lose the key and your S3 backups become permanently unrecoverable, so treat it with the same discipline as your SSH private keys. The [Build a Private Cloud at Home with Proxmox VE](/articles/build-private-cloud-home-proxmox-ve/) guide covers the broader architecture behind layered backup strategies on Proxmox, including where PBS fits alongside Proxmox's native VM snapshot tooling. For the host-level hardening that protects PBS credentials in the first place, the [Hardening Proxmox VE: Firewall, fail2ban, and SSH Security](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) guide is the companion read. ## Troubleshooting Common Sync Errors **`AuthorizationHeaderMalformed`** The region string does not match what the provider expects. B2 regions look like `us-west-004`, not `us-west-1`. Copy the exact region string directly from the bucket detail page in the B2 console. **`403 Forbidden` on test connection** Your API key lacks list permission on the bucket. For B2, verify the application key includes `listBuckets`, `readFiles`, and `writeFiles` capabilities. For R2, confirm the API token is scoped to the correct bucket. **Sync shows 0 bytes transferred** Normal if no new backups have been created since the last sync. PBS only transfers chunks absent from the remote. Confirm backups are running: ```bash proxmox-backup-client snapshots --repository @:backups ``` **Initial sync is stuck at very low throughput** Check for a rate limit under **Configuration → Bandwidth Limits** in the PBS web UI. Also check whether B2's free API tier cap (2,500 Class B operations per day) has been hit — the client applies exponential backoff when rate-limited. For large datastores, upgrade to a paid B2 API tier or spread the initial sync over several days using bandwidth throttling. ## Conclusion PBS 4.2's S3 backend turns any S3-compatible bucket into a fault-tolerant off-site backup copy for $6–7/month per TB with no additional hardware. Set up the remote, create a daily sync job with `remove-vanished` enabled, and your local prune policy propagates to S3 automatically. The step most people skip: do a test restore from S3 before you need it — register your S3 bucket as a source on a temporary PBS instance or spare VM, browse the snapshots, and pull one back. That is the only confirmation that your offsite copy actually works. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-ve-small-business-vmware-alternative/ 2026-05-01T00:00:00.000Z

Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-open-vswitch-vm-networking/ 2026-04-30T00:00:00.000Z

Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-homelab-essential-lxc-containers/ 2026-04-29T00:00:00.000Z

> .env echo "AUTHENTIK_SECRET_KEY=$(openssl rand -base64 60 | tr -d '=+/')" >> .env echo "AUTHENTIK_ERROR_REPORTING__ENABLED=false" >> .env ``` ```bash docker compose pull && docker compose up -d ``` First startup takes 2–3 minutes while Authentik runs database migrations. Complete setup at `http://192.168.1.15:9000/if/flow/initial-setup/`. **Gotcha**: The default Authentik compose file uses `latest` image tags. Pin to a specific release (e.g., `ghcr.io/goauthentik/server:2024.12.3`) before your first pull. Authentik occasionally ships breaking API changes between minor versions, and an unattended `docker compose pull` on the wrong day will break SSO for every proxied service simultaneously. **Worth the complexity?** Only if you have five or more services to protect. For a two-container setup, HTTP Basic Auth through NPM is sufficient. Authentik pays off when you want audit logs, TOTP enforcement, and a single logout that propagates across all services at once. ## Resource Planning: Running All Six on One Node | Container | RAM Allocated | RAM at Idle | vCPUs | Disk | |---|---|---|---|---| | Pi-hole | 256 MB | 70 MB | 1 | 4 GB | | Nginx Proxy Manager | 512 MB | 180 MB | 1 | 8 GB | | Vaultwarden | 256 MB | 25 MB | 1 | 4 GB | | Uptime Kuma | 256 MB | 90 MB | 1 | 4 GB | | Jellyfin | 1024 MB | 220 MB | 2 | 8 GB + media | | Authentik | 1024 MB | 450 MB | 2 | 10 GB | | **Total** | **3328 MB** | **~1035 MB** | **8** | **38 GB** | An 8 GB node handles the full stack with headroom. On a 4 GB node, drop Authentik — it's the heaviest container and the least essential for a basic homelab. For guidance on node selection, storage layout, and networking to support this kind of infrastructure, the guide on [building a private cloud at home with Proxmox VE](/articles/build-private-cloud-home-proxmox-ve/) covers the hardware decisions that set the foundation. ## Security Hardening for the Container Stack All six containers handle sensitive data. A few non-negotiable steps before you consider this stack production-ready: - **Restrict admin ports**: Lock ports 81 (NPM admin), 3001 (Uptime Kuma), and 9000 (Authentik) to your LAN subnet using Proxmox firewall rules. The [Proxmox firewall and SSH hardening guide](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) covers datacenter-level rules that apply at the container level without touching iptables manually. - **Schedule backups**: All six containers store persistent state on disk. Configure a Proxmox backup job for each CT in Datacenter → Backup, targeting PBS or an NFS share. Weekly retention of two backups is the minimum viable safety net. - **Pin versions**: Vaultwarden and Authentik are security-critical. Never run `latest` tags — pin to a specific version and update deliberately after reading the changelog. - **Enable TOTP**: Pi-hole, NPM, Uptime Kuma, and Authentik all support TOTP. Enable it on each admin account before exposing any service through NPM to the internet. ## Conclusion These six LXC containers — Pi-hole, Nginx Proxy Manager, Vaultwarden, Uptime Kuma, Jellyfin, and Authentik — give you a complete homelab services layer that runs comfortably on a single 8 GB Proxmox VE 9.1 node. Deploy them in order: DNS first, reverse proxy second, then everything else behind it. Once Authentik is in place, the natural next step is wiring its forward-auth middleware into each NPM proxy host — a 10-minute configuration that replaces per-service login prompts with a single SSO portal covering your entire homelab. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/migrate-bare-metal-truenas-proxmox-zfs/ 2026-04-28T00:00:00.000Z

/dev/null # Output: /sys/kernel/iommu_groups/14/devices/0000:02:00.0 ls /sys/kernel/iommu_groups/14/devices/ ``` If the group contains only the HBA (a co-located PCIe root port is fine), add it to the VM after creation: ```bash qm set 110 --hostpci0 02:00.0,pcie=1,rombar=0 ``` `pcie=1` is required for modern HBAs. `rombar=0` prevents a boot hang seen with some LSI cards inside QEMU. ## Option 2: Pass Through Individual Disks When the SATA controller is part of the motherboard chipset and shared with the Proxmox boot drive, pass individual drives instead. Never use `/dev/sdX` — those letters reassign at boot. Use stable by-id paths: ```bash ls -la /dev/disk/by-id/ | grep -v part | grep ata # ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1234567 -> ../../sdb # ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7654321 -> ../../sdc ``` After exporting the pool, attach each drive: ```bash qm set 110 --scsi1 /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1234567 qm set 110 --scsi2 /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7654321 ``` Repeat for every drive. A 6-drive RAIDZ2 means six `qm set` commands. This is worth it only if you can't add a dedicated HBA. For long-term use, a used IT-mode card at $20–$40 on eBay is the cleaner investment. ## Create the TrueNAS VM ```bash qm create 110 \ --name truenas-scale \ --memory 32768 \ --balloon 0 \ --cores 4 \ --sockets 1 \ --cpu host \ --machine q35 \ --bios ovmf \ --net0 virtio,bridge=vmbr0 \ --ostype l26 ``` `--balloon 0` prevents Proxmox from reclaiming RAM that TrueNAS is actively using as ZFS ARC cache. Add the supporting disks and installer: ```bash # EFI disk for OVMF boot qm set 110 --efidisk0 local-lvm:1,efitype=4m,pre-enroll-keys=0 # OS boot disk — TrueNAS uses a mirrored boot pool, 32 GB minimum qm set 110 --scsi0 local-lvm:32,format=raw,iothread=1,ssd=1 # Installer ISO qm set 110 --ide2 local:iso/TrueNAS-SCALE-24.10.2.iso,media=cdrom # Boot order: ISO first, then OS disk qm set 110 --boot order="ide2;scsi0" ``` The IOMMU mechanics are identical to [GPU passthrough on Proxmox](/articles/gpu-passthrough-proxmox-complete-guide/). If you've done GPU passthrough before, the `hostpci` setup will feel familiar — you're just passing storage instead of video. Now attach the HBA or individual disks from the previous sections. ## Install TrueNAS and Import Your Existing Pools Boot the VM. The TrueNAS SCALE 24.10.2 installer completes in under five minutes on NVMe. Select the OS boot disk (`scsi0`) as the installation target — not the pool drives. After TrueNAS reboots into the dashboard, go to **Storage → Import Pool**. TrueNAS scans the attached drives and surfaces your existing ZFS pool. Select it and click **Import**. For a 20 TB pool this finishes in under 30 seconds — no data moves, only pool metadata is recognized. Verify from the TrueNAS shell or SSH: ```bash zpool status zfs list -r zpool history tank | tail -20 ``` All vdevs should show `ONLINE`. If your most recent scrub event appears in the history output, the pool imported from the correct state. ## Reconfigure Shares and Verify Before Decommissioning The IP and interface name change because TrueNAS now has a `virtio` NIC. In the TrueNAS UI, go to **Network → Interfaces**, set a static IP, then update any SMB or NFS bindings that reference the old interface. Re-enable periodic scrub schedules under **Data Protection** — they don't carry over with a pool import. If you have [Proxmox Backup Server](/articles/automated-backups-proxmox-backup-server/) pointing at a TrueNAS share, update the storage definition with the new IP. The datastore contents are unchanged. Before wiping the old bare-metal install, run a full scrub inside the VM: ```bash zpool scrub tank watch zpool status ``` Expect roughly 1 hour per 10 TB on spinning disks. Also check SMART data on all drives to confirm nothing new appeared during the migration: ```bash for disk in /dev/sdb /dev/sdc /dev/sdd /dev/sde; do echo "=== $disk ===" smartctl -a "$disk" | grep -E "Reallocated|Pending|Uncorrectable" done ``` Keep the bare-metal server powered off but intact for at least a week. If a missing share or misconfigured permission surfaces, you'll want that fallback. ## Conclusion With an HBA or pool drives passed through directly, TrueNAS imports existing ZFS 2.2.x pools intact and your data never leaves the disks. The critical steps are exporting pools from the Proxmox host before attaching them to the VM, using `/dev/disk/by-id/` for individual disk passthrough, and running a post-import scrub to confirm clean pool state. Next, consider [hardening the Proxmox host with firewall rules and fail2ban](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) — the NAS is only as secure as the hypervisor it runs on. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-cpu-pinning-low-latency-vms/ 2026-04-27T00:00:00.000Z

--cpuaffinity 4-7` to bind a QEMU process and all its threads to specific physical cores - **isolcpus matters**: Without kernel-level core isolation, other processes can still preempt your pinned VM threads and cause jitter - **NUMA alignment**: On EPYC or multi-socket systems, pin memory and CPUs to the same NUMA node or every memory access crosses the interconnect - **Live migration caveat**: A VM with `cpu: host` set will not migrate to nodes with a different CPU microarchitecture — design for this upfront - **Not always worth it**: General-purpose VMs and idle containers get no benefit; reserve pinning for sustained latency-sensitive workloads ## What CPU Pinning Actually Does Under the Hood Without pinning, the Linux Completely Fair Scheduler (CFS) migrates QEMU threads across physical cores continuously. This is efficient for throughput, but every thread migration causes L1 and L2 cache evictions on the core the thread just left and cold-cache misses on the core it lands on. For a 4K gaming session or a Postgres database handling sub-100 ms queries, those cache evictions translate to visible stutters or latency spikes. `cpuaffinity` in Proxmox works by setting the cpuset cgroup for the entire QEMU process at VM start. Every thread QEMU spawns — vCPU threads, the I/O thread, device emulation threads — is constrained to the cores you specify. This is a process-level constraint rather than per-vCPU-thread pinning, but for the overwhelming majority of workloads it produces the same result with far less configuration overhead. The kernel parameter `isolcpus` goes one step further: it removes specified cores from the scheduler's pool entirely. Nothing lands on an isolated core unless it explicitly requests affinity for that core. This is the difference between "I want my VM on cores 4-7" and "nothing else touches cores 4-7, ever." ## How to Check Your NUMA Topology First Understanding your hardware layout before picking cores is non-negotiable. Pinning to sibling hyperthreads on a saturated physical core, or crossing NUMA nodes, will make latency *worse* than no pinning. ```bash lscpu --extended ``` Look at the `CPU`, `CORE`, `SOCKET`, and `NODE` columns. On an Intel Core i9-13900K, the 8 P-cores appear in hyperthreaded pairs — logical CPUs 0 and 16 share physical core 0, logical CPUs 1 and 17 share physical core 1, and so on. Always pin both hyperthreads of the same physical core together. For NUMA topology on EPYC, Threadripper, or any dual-socket system: ```bash numactl --hardware ``` On a single-socket consumer CPU this shows one node. On an EPYC 7302P, memory latency between the two NUMA nodes differs by roughly 30 ns — enough to add measurable jitter to database workloads. For the most detailed view, install `hwloc` and render the full topology tree: ```bash apt install hwloc -y lstopo --no-io ``` This shows L3 cache domains. On AMD CPUs each CCX shares its own L3, so keep all pinned cores within a single CCX. On a Ryzen 9 7950X with two 8-core CCDs, keeping the VM to one CCD means all 8 cores share the same 32 MB L3 — a meaningful cache advantage for workloads with hot working sets. ## How to Isolate Cores from the Host Kernel This step is optional for lightly loaded hosts but strongly recommended for gaming or real-time workloads. Open the GRUB config: ```bash nano /etc/default/grub ``` Append to `GRUB_CMDLINE_LINUX_DEFAULT`: ```ini GRUB_CMDLINE_LINUX_DEFAULT="quiet isolcpus=4-7,12-15 nohz_full=4-7,12-15 rcu_nocbs=4-7,12-15" ``` `nohz_full` stops the kernel from sending periodic timer interrupts to isolated cores, reducing jitter from hundreds of microseconds down to under 5 µs. `rcu_nocbs` offloads RCU callbacks off those same cores. All three options must reference the same core list. Apply and reboot: ```bash update-grub && reboot ``` After reboot, confirm isolation is active: ```bash cat /sys/devices/system/cpu/isolated # Expected output: 4-7,12-15 ``` Leave at least 2 physical cores (plus their hyperthreads) for the Proxmox host OS. On a 12-core i9-12900K I keep cores 0-3 for Proxmox — the web UI, storage daemons, PBS sync jobs, and any lightweight LXC containers all run there without touching the VM's dedicated P-cores. ## How to Configure cpuaffinity on a Proxmox VM The `cpuaffinity` option accepts a comma-separated list of CPU IDs or ranges matching the logical CPU numbers from `lscpu`. ```bash qm set 100 --cpuaffinity 4-7 ``` Or edit the config file directly: ```bash nano /etc/pve/qemu-server/100.conf ``` ```ini cpuaffinity: 4-7 ``` The number of vCPUs in the VM does not need to exactly match the pinned range. If you pin to `4-7` (4 logical CPUs) and the VM has `cores: 4,sockets: 1`, each vCPU thread lands on one core. QEMU's I/O thread and device emulation threads are also constrained to that range, but they are lightweight and do not meaningfully compete with the vCPU threads. For a gaming VM with 8 vCPUs across 4 physical P-cores with hyperthreading: ```bash qm set 100 --cpuaffinity 4-7,12-15 qm set 100 --cores 8 --sockets 1 qm set 100 --cpu host ``` The `cpu: host` flag passes all physical CPU features directly to the guest, including AVX-512 and TSC invariant mode. Pair this with [the full GPU passthrough configuration](/articles/gpu-passthrough-proxmox-complete-guide/) to get consistent frame times from a dedicated gaming or AI VM. The tradeoff: `cpu: host` makes the VM non-migratable to hosts with a different CPU microarchitecture. If you're running [a K3s Kubernetes cluster on Proxmox](/articles/k3s-kubernetes-cluster-proxmox-vms/) and need live migration across heterogeneous nodes, stick with `cpu: x86-64-v3` and skip `cpuaffinity` on those VMs — portability matters more than the marginal latency gain. ## NUMA Alignment: Bind Memory to the Right Node On any multi-NUMA system, memory accesses that cross NUMA nodes add 20-40 ns per access. If your pinned cores are on NUMA node 0, bind the VM's RAM to node 0 as well. Confirm which NUMA node owns your target cores: ```bash numactl --hardware # Look for "node 0 cpus:" and "node 1 cpus:" lines ``` Then add NUMA binding to the VM config: ```bash nano /etc/pve/qemu-server/100.conf ``` ```ini cpuaffinity: 4-7 numa: 1 numa0: cpus=4-7,hostnodes=0,memory=16384,policy=bind ``` `policy=bind` hard-allocates guest RAM from NUMA node 0 only. If node 0 runs out of free memory the VM will fail to start rather than silently falling back to slower remote memory. `policy=preferred` is softer but can mask a memory budget mistake. Use `bind` so you know immediately if something is wrong. On a properly aligned Proxmox EPYC system, STREAM TRIAD benchmark scores inside the guest improve 20-35% compared to unaligned placement. On real-world workloads like Postgres or ffmpeg encoding, expect 8-15% throughput improvement with p99 latency 30-50% lower. ## Verifying the Pinning Is Actually Working Start the VM and locate the QEMU process ID: ```bash qm start 100 pgrep -a qemu-system-x86 | grep " 100 " ``` Check process-level affinity: ```bash taskset -cp # pid 12345's current affinity list: 4-7,12-15 ``` Inspect individual thread placement: ```bash ps -eLo pid,tid,psr,comm | grep qemu | head -20 ``` The `PSR` column shows which physical CPU each thread is currently scheduled on. Every thread should show a value within your pinned range. If any land outside it, the `isolcpus` parameter was not applied — run `cat /proc/cmdline` to confirm it appears in the active boot parameters. For a latency measurement inside a Linux guest, install and run `cyclictest` from the `rt-tests` package: ```bash apt install rt-tests -y cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0 --loops=10000 ``` On a pinned, isolated 4-core setup with `nohz_full` on a 13th-gen Intel host, worst-case latency consistently lands under 80 µs. Without pinning on the same hardware under moderate host load, worst-case values hit 3-8 ms. That is a 40x difference at the p99.99 — the exact improvement that eliminates audio dropouts in a DAW or micro-stutters in a GPU passthrough gaming VM. ## When CPU Pinning Is Worth the Effort (and When It Isn't) Worth pinning: - **GPU passthrough gaming VMs**: frame time consistency depends on stable vCPU scheduling; even 2 ms jitter shows up as stutters at 144 Hz - **Audio production workstations**: real-time audio at 48 kHz with a 128-sample buffer needs scheduling latency well under 3 ms - **Low-latency databases**: Postgres, Redis, or ClickHouse under sustained query load benefit measurably from cache-warm cores - **[Home Assistant with heavy integrations](/articles/home-assistant-os-on-proxmox-2026-setup-guide/)**: fast polling loops on a loaded host miss intervals without dedicated cores Not worth pinning: - **General-purpose web servers**: CFS handles bursty traffic better than static affinity; pinning forfeits burst headroom - **LXC containers**: use `cpulimit` cgroup throttling instead — the overhead-to-benefit ratio for LXC pinning is rarely justified - **Lightly loaded single-VM hosts**: CFS already gives a near-idle VM the run of all available cores - **Batch compute VMs**: a VM that runs a 30-second ffmpeg transcode every few hours will not notice pinning The gotcha I ran into: pinning a VM to cores 0-3 on a system where the Proxmox host was using core 0 for NVMe interrupt handling caused *more* latency variance than no pinning at all. Always check `cat /proc/irq/*/smp_affinity_list` after setting up pinning, and move storage and network IRQs off your isolated cores with `echo > /proc/irq//smp_affinity`. ## Conclusion CPU pinning on Proxmox VE 9 is a 20-minute configuration change with immediate, measurable results for latency-sensitive workloads: `isolcpus` in GRUB, `cpuaffinity` in the VM config, and a `taskset` check to confirm. The logical next step is interrupt affinity — disable `irqbalance` and manually assign NVMe and PCIe interrupts to your reserved host cores, or run `irqbalance` with a `--banirq` list for your storage and network devices. That is where the last few microseconds of jitter typically hide on a well-tuned Proxmox host. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-lxc-bind-mounts-host-storage/ 2026-04-26T00:00:00.000Z

.conf` — for example `mp0: /host/path,mp=/container/path`. - **Privileged containers skip the shift**: A privileged container avoids UID remapping but trades namespace isolation for convenience. - **Exclude from backups**: Add `backup=0` to any large mount point entry to keep media libraries out of `vzdump` archives. ## What Are LXC Bind Mounts and When Do You Need Them LXC containers share the Proxmox host kernel but are isolated by namespaces and cgroups. That makes them fast to start and cheap on RAM, and it also means they can safely access host directories if you configure mount points correctly. You reach for a bind mount when: - Multiple containers need access to the same dataset — a media library read by Jellyfin and written by Sonarr simultaneously - You want application data outside the container rootfs so it survives `pct restore` or `pct destroy` - You are running Docker inside an LXC container and want the Docker volume data on a host path you can snapshot — exactly the setup described in [Running Docker Inside LXC Containers on Proxmox](/articles/docker-inside-lxc-containers-proxmox/) - You want to snapshot application data independently via ZFS without snapshotting the whole container disk image A bind mount is **not** a copy. Write from the container, the host sees the change immediately. Delete a file from the host side, the container loses it. Plan your data layout before you start. ## How to Add a Bind Mount from the Proxmox UI In Proxmox VE 9.1, mount points live in the container's **Resources** tab. 1. Select the container in the left pane, click **Resources** → **Add** → **Mount Point**. 2. Set **Storage** to `Directory` and enter the **Host Path** — an absolute path to an existing directory on the Proxmox host (e.g., `/mnt/data/media`). 3. Set **Mount Point** to the path where it should appear inside the container (e.g., `/media`). 4. Optionally tick **Read-only** to prevent container writes to the host path. 5. Click **Add**, then restart the container: **More** → **Reboot** or run `pct reboot ` from the shell. > **Gotcha**: The UI will not create the host directory for you. If the path does not exist on disk, Proxmox will accept the config but the container will fail to start with a vague `lxc-start` error. Create it first: ```bash mkdir -p /mnt/data/media ``` ## Configuring Bind Mounts via the LXC Config File For scripted deployments, editing the config directly is faster and easier to put under version control. ```bash nano /etc/pve/lxc/101.conf ``` Add a mount point entry at the bottom: ```ini mp0: /mnt/data/media,mp=/media ``` Multiple mount points use `mp0` through `mp9`: ```ini mp0: /mnt/data/media,mp=/media mp1: /mnt/data/config,mp=/config,ro=1 ``` Full set of options supported in Proxmox VE 9.1: | Option | Example | Effect | |--------|---------|--------| | `mp=` | `mp=/data` | Container-side mount path (required) | | `ro=1` | `ro=1` | Mount read-only inside the container | | `backup=0` | `backup=0` | Exclude this path from `vzdump` backups | | `replicate=0` | `replicate=0` | Skip during ZFS or PBS replication | | `shared=1` | `shared=1` | Mark as cluster-shared storage | After editing the config, restart the container and verify the mount: ```bash pct reboot 101 pct exec 101 -- df -h /media ``` ## The UID/GID Remapping Problem in Unprivileged Containers This is where almost everyone gets burned the first time. Unprivileged LXC containers use a UID/GID mapping defined in `/etc/subuid` and `/etc/subgid` on the host. Proxmox ships with: ```bash cat /etc/subuid # root:100000:65536 ``` This means container UID 0 (root) maps to host UID 100000, container UID 1000 maps to host UID 101000, and so on. When a container process writes to a bind-mounted host directory, the host kernel sees host UID 101000, not UID 1000. The directory permission check happens against the shifted UID. ### Calculating the Mapped Host UID ```bash # Inside the container, check the running user: id # uid=1000(ubuntu) gid=1000(ubuntu) # On the host, that container UID maps to: # host_uid = 100000 + container_uid = 101000 ``` ### Option 1: Chown the Host Directory to the Shifted UID The cleanest fix — change ownership on the host to the mapped UID: ```bash chown -R 101000:101000 /mnt/data/media ``` Container processes running as UID 1000 can now read and write the directory transparently, with no special config beyond the mount point entry itself. ### Option 2: Use POSIX ACLs for Shared Paths ACLs are more surgical when multiple containers or host users need access to the same path: ```bash # Install acl if not already present (Proxmox host is Debian-based) apt install acl # Grant the container's mapped UID read/write/execute access setfacl -m u:101000:rwx /mnt/data/media # Default ACL so new files and subdirectories inherit the rule setfacl -d -m u:101000:rwx /mnt/data/media ``` ### Option 3: Map Container Root to Host Root For containers where the container's root user needs to own host files, add a custom UID map to the container config. This is a targeted override, not a global switch: ```ini lxc.idmap: u 0 0 1 lxc.idmap: u 1 100001 65535 lxc.idmap: g 0 0 1 lxc.idmap: g 1 100001 65535 ``` This maps container UID 0 to host UID 0 while keeping all other UIDs shifted. The host directory just needs to be owned by root: ```bash chown root:root /mnt/data/special chmod 755 /mnt/data/special ``` > **Security note**: Mapping container root to host root reduces namespace isolation — a compromised container root can affect any root-owned bind-mounted path on the host. This is acceptable for trusted internal workloads. For the broader security picture on LXC and Proxmox isolation, see [Hardening Proxmox VE: Firewall, fail2ban, and SSH Security](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/). ## Bind Mounts in Privileged Containers Privileged containers (`unprivileged: 0` in the config) have no UID remapping — container UID 1000 is host UID 1000. Standard Unix permissions apply directly: ```bash chown -R 1000:1000 /mnt/data/media ``` Privileged containers are the simpler path when you are running Docker inside LXC. Docker's overlay2 storage driver needs real root access, so the LXC container must be privileged anyway. In that setup — like the Portainer and Dockge workflow described in [Managing Docker on Proxmox with Portainer and Dockge](/articles/managing-docker-on-proxmox-with-portainer-and-dockge/) — bind-mounting Docker volume directories from the host just works without any UID arithmetic. ## Real-World Configs That Work ### Media Library Shared Across Two Containers Host ZFS dataset `/mnt/tank/media` mounted read-only into Jellyfin, read-write into Sonarr, with both excluded from `vzdump`: ```ini # /etc/pve/lxc/200.conf (Jellyfin — consumer) mp0: /mnt/tank/media,mp=/media,ro=1,backup=0 # /etc/pve/lxc/201.conf (Sonarr — writer) mp0: /mnt/tank/media,mp=/media,backup=0 ``` ```bash # Both containers run as UID 1000 internally chown -R 101000:101000 /mnt/tank/media ``` ### Docker Data Directory on a Host ZFS Dataset Run Docker inside a privileged LXC but keep `/var/lib/docker` on a ZFS dataset you can snapshot independently: ```ini # /etc/pve/lxc/300.conf (privileged container: unprivileged: 0) mp0: /mnt/ssd/docker-data,mp=/var/lib/docker ``` ```bash chown root:root /mnt/ssd/docker-data chmod 710 /mnt/ssd/docker-data ``` Expect Docker to initialize its overlay2 storage driver the first time the container starts — about 10 to 15 seconds on a fresh ZFS dataset before the daemon comes up. ### Read-Only Config Injection Manage application configs centrally on the host; containers pick up changes on next restart: ```ini mp0: /mnt/configs/nginx,mp=/etc/nginx,ro=1 mp1: /mnt/configs/app,mp=/app/config,ro=1 ``` This pattern works well for CI/CD pipelines where Ansible or a deploy script writes the host directory and container restarts pull in the new config. ## Troubleshooting Bind Mount Failures **Container fails to start, errors in the journal** ```bash journalctl -u pve-container@101.service --no-pager | tail -40 ``` The most common cause is the host directory not existing or a typo in the config path. Verify: ```bash ls -la /mnt/data/media ``` **Permission denied inside the container** Check the effective ownership from the host: ```bash ls -lan /mnt/data/media # Owner UID should match 100000 + container_uid ``` Fix it: ```bash chown -R 101000:101000 /mnt/data/media ``` **Files created inside the container have large UIDs when viewed from the host** Expected behavior for unprivileged containers. UID 101000 on the host is UID 1000 inside the container. Use `chown` with the mapped UID when you need to manipulate these files from the host side. **Mount not present after `pct restore`** `pct restore` rebuilds the container config from the backup archive. Mount point entries added after the backup was taken are not included. Re-add the `mp` lines to the config manually, or ensure you capture the config file as part of your backup procedure. For a robust backup strategy that covers both containers and host datasets, [Automated Backups with Proxmox Backup Server](/articles/automated-backups-proxmox-backup-server/) lays out a production-grade approach. **`vzdump` backups are enormous** A bind-mounted media library can balloon a container backup from 2 GB to 2 TB. Add `backup=0` to the mount point line: ```ini mp0: /mnt/tank/media,mp=/media,ro=1,backup=0 ``` Back up the host dataset separately via PBS or ZFS send. ## Conclusion Bind mounts in Proxmox LXC are straightforward once you have the UID shift internalized: for unprivileged containers, `chown 101000:101000` on the host directory is the fix for nearly every permission error you will encounter. Add `mp0: /host/path,mp=/container/path` to the container config, restart, and verify with `pct exec`. The pattern scales cleanly to a dozen containers sharing the same ZFS datasets. Your next step: put those host directories on a ZFS dataset with hourly snapshots — five minutes of work that gives you point-in-time recovery for all your container data without touching the containers themselves. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-notifications-email-webhooks/ 2026-04-25T00:00:00.000Z

Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-nfs-storage-vm-disks-backups/ 2026-04-24T00:00:00.000Z

:8006` 2. Navigate to **Datacenter → Storage → Add → NFS** 3. Fill in the form: - **ID**: A short identifier with no spaces (e.g., `nas-proxmox`) - **Server**: The NAS IP or hostname (e.g., `192.168.10.50`) - **Export**: Click the dropdown — Proxmox runs `showmount` against the server and lists available exports automatically - **Content**: Check all types this share will serve: `Disk image`, `ISO image`, `Container template`, `VZDump backup file`, `Snippets` - **Max Backups**: Set a per-VM retention limit if using this as a VZDump target 4. Click **Add** The share mounts on all cluster nodes within seconds. Check the **Tasks** pane at the bottom of the UI to confirm there are no mount errors before creating any VMs against the new storage. ### What the Content Types Actually Do | Content Type | File Format | Typical Use | |---|---|---| | Disk image | `.raw`, `.qcow2` | VM disk images | | ISO image | `.iso` | OS install media | | Container template | `.tar.zst` | LXC base images | | VZDump backup file | `.vma.zst` | VM and container backups | | Snippets | YAML/JSON | Cloud-init user-data configs | ## How to Add NFS Storage via the CLI The `pvesm` tool is the right approach when scripting Proxmox node setup or managing storage from Ansible: ```bash pvesm add nfs nas-proxmox \ --server 192.168.10.50 \ --export /mnt/tank/proxmox-nfs \ --content images,iso,vztmpl,backup,snippets \ --options vers=4.1 ``` Verify the storage was added and check capacity: ```bash pvesm status ``` To list the contents of the storage: ```bash pvesm list nas-proxmox ``` Proxmox writes the storage definition to `/etc/pve/storage.cfg`, which `pmxcfs` replicates to all cluster nodes: ```ini nfs: nas-proxmox path /mnt/pve/nas-proxmox server 192.168.10.50 export /mnt/tank/proxmox-nfs content images,iso,vztmpl,backup,snippets options vers=4.1 ``` ## NFS Mount Options That Actually Improve Performance Proxmox passes mount options directly through the `options` field. These are the ones worth setting: ```bash pvesm set nas-proxmox --options vers=4.1,hard,timeo=600,retrans=2,nconnect=4 ``` | Option | Effect | Recommendation | |---|---|---| | `vers=4.1` | Forces NFSv4.1 with session trunking | Always prefer v4.1 over v3 for cluster use | | `hard` | Retries indefinitely if the server becomes unreachable | Required for VM disks — `soft` will corrupt data | | `timeo=600` | 60-second timeout before retry (units: 0.1 second) | Increase on networks with occasional latency spikes | | `retrans=2` | Retries before reporting an error | 2 is fine on stable LANs; default is 3 | | `nconnect=4` | Opens 4 parallel TCP connections to the NFS server | 30–50% throughput increase on 10 GbE (kernel 5.15+) | | `noatime` | Skips access-time write on reads | Minor write reduction on ISO and backup shares | `nconnect=4` is the highest-impact single option if you are on 10 GbE. Benchmark before and after with: ```bash dd if=/dev/zero of=/mnt/pve/nas-proxmox/test.img bs=1M count=1024 oflag=direct rm /mnt/pve/nas-proxmox/test.img ``` On a TrueNAS SCALE system with an NVMe-backed pool and a direct 10 GbE connection, `nconnect=4` typically moves sequential write throughput from around 350 MB/s to 700–900 MB/s. On 1 GbE, the difference is negligible — the link is already saturated long before the connection count matters. ## Using NFS as a Proxmox Backup Target Once NFS storage is added with the `backup` content type enabled, pointing a backup job at it is straightforward: 1. Go to **Datacenter → Backup → Add** 2. Set **Storage** to `nas-proxmox` 3. Choose your **Schedule** (daily, weekly, or a cron expression) 4. Configure **Retention** (keep last N backups per VM) 5. Save — Proxmox handles the rest, writing `.vma.zst` archives directly to the NFS share For deduplication, encryption, and server-side integrity verification, [Proxmox Backup Server](/articles/automated-backups-proxmox-backup-server/) is the better tool — it can also use an NFS share as its datastore backing, though local NVMe or a ZFS dataset gives better PBS performance for dedup index operations. A practical combination that works well: VZDump to NFS for rapid daily snapshots, and PBS on a separate host for deduplicated, encrypted long-term retention with offsite replication. ## Common NFS Gotchas on Proxmox ### root_squash Blocks Disk Image Creation This is the most common first-time issue. If you get a `Permission denied` error when creating a VM disk on NFS storage, the export is almost certainly using `root_squash`. Proxmox writes disk images as root; `root_squash` maps root to `nobody`, which has no write access. Fix on TrueNAS: uncheck **Enable Root Squash** in the NFS share settings. Fix on Linux: change `root_squash` to `no_root_squash` in `/etc/exports` and run: ```bash exportfs -ra ``` ### NFSv3 File Locking in a Cluster NFSv3 uses a separate statd/lockd protocol for file locking. Under high concurrency — two Proxmox nodes creating disk images simultaneously — stale lock files can accumulate and cause `flock` failures. NFSv4.1 (`vers=4.1`) handles locking natively and eliminates this class of problem entirely. ### NFS Storage Shows as Unavailable After a NAS Reboot Proxmox marks NFS storage unavailable if the mount times out during node boot. After the NAS comes back online, re-trigger the mount: ```bash mount /mnt/pve/nas-proxmox ``` Or rescan via the UI: **Datacenter → Storage → nas-proxmox → More → Rescan**. If your NAS boots slower than your Proxmox nodes, add `_netdev` to the mount options so the kernel waits for network availability before attempting the mount. ### QCOW2 and LXC Container Root Filesystems QCOW2 disk images on NFS work fine for KVM VMs, but LXC containers cannot use QCOW2 for their root filesystems — LXC needs raw block devices or directory-backed storage. If you want LXC container data on NFS, use the `dir` storage plugin pointing at the NFS mountpoint rather than the native `nfs` plugin. This distinction is worth knowing before you try to migrate a container and get an unexpected error. ### ISO Upload Permissions When uploading an ISO through the web UI, Proxmox writes it as `www-data` (UID 33). If you also mount the same NFS share from another client and try to delete or modify those files directly, you will hit permission errors. The clean rule: manage ISO files exclusively through the Proxmox UI or `pvesm` — do not mix access methods on the same export. ## Conclusion NFS is the lowest-friction way to give every Proxmox cluster node shared access to ISO libraries, container templates, and VZDump archives without deploying Ceph. Add the storage once at the Datacenter level, set `no_root_squash` on the export, and include `vers=4.1,hard,nconnect=4` in your mount options for solid performance on modern hardware. The natural next step is setting up a scheduled VZDump job targeting this storage, then layering [Proxmox Backup Server](/articles/automated-backups-proxmox-backup-server/) on top for deduplication and encryption. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/migrate-hyper-v-vms-proxmox-ve/ 2026-04-23T00:00:00.000Z

Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-lets-encrypt-acme-certificate/ 2026-04-22T00:00:00.000Z

ACME > Challenge Plugins`) 2. Your Proxmox node has outbound internet access to `acme-v02.api.letsencrypt.org` on port 443 3. Your node's hostname is a fully qualified domain name Check the current hostname: ```bash hostname --fqdn ``` If the result is just `pve` with no domain suffix, fix it before proceeding. The [Proxmox installation guide](/articles/install-proxmox-ve-on-any-hardware/) covers hostname configuration as part of its initial setup checklist, but the quick fix is: ```bash hostnamectl set-hostname pve.yourdomain.com ``` Then update `/etc/hosts` so the node's IP maps to the FQDN: ```ini 192.168.1.10 pve.yourdomain.com pve ``` ## How to Register an ACME Account Proxmox's ACME client needs a Let's Encrypt account to issue certificates. Register once per node. **Via the Web UI**: 1. Navigate to `Datacenter > ACME` 2. Under "Accounts", click **Add** 3. Enter your email address and accept the Terms of Service 4. Select the **Staging** directory first — it has no rate limits and lets you validate the full flow without burning production quota **Via the CLI**: ```bash # Staging — use this first pvenode acme account register staging your@email.com \ --directory https://acme-staging-v02.api.letsencrypt.org/directory # Production — switch to this after staging succeeds pvenode acme account register default your@email.com \ --directory https://acme-v02.api.letsencrypt.org/directory ``` List registered accounts at any point: ```bash pvenode acme account list ``` ## HTTP-01 vs DNS-01: Which Challenge Type to Use | Challenge | Requirement | Best for | |-----------|-------------|----------| | HTTP-01 | Port 80 publicly reachable on your domain's IP | Public-facing hosts | | DNS-01 | API access to your DNS provider | Homelabs, private IPs, wildcards | | TLS-ALPN-01 | Port 443 publicly reachable | Rarely needed with Proxmox | For a typical homelab Proxmox node, DNS-01 is the right call. Your host stays completely internal — only the Proxmox node needs outbound HTTPS access to Let's Encrypt's servers and your DNS provider's API. No inbound port forwarding required. ## Setting Up the Cloudflare DNS Plugin First, create a scoped API token in Cloudflare: 1. Log into Cloudflare > **My Profile > API Tokens > Create Token** 2. Select the **Edit zone DNS** template 3. Scope it to your specific zone (domain) — not "All zones" 4. Copy the token immediately — it won't be shown again Add the plugin in Proxmox: **Web UI**: `Datacenter > ACME > Challenge Plugins > Add` - Plugin ID: `cloudflare` - Plugin type: `Cloudflare Managed DNS` - API Token: paste your token **CLI**: ```bash pvenode acme plugin add dns cloudflare \ --api cf \ --data "CF_Token=your_cloudflare_api_token_here" ``` Verify the plugin was saved: ```bash pvenode acme plugin list ``` ## Configuring the Domain on Your Node Attach a domain and the DNS plugin to your node's certificate configuration. **Web UI**: `Node > Certificates > ACME > Add` - Domain: `pve.yourdomain.com` - Challenge type: `DNS` - Plugin: `cloudflare` **CLI**: ```bash # Set the ACME account for this node pvenode config set --acme "account=default" # Set the domain with the DNS plugin pvenode config set --acmedomain0 "pve.yourdomain.com,plugin=cloudflare" ``` Verify the config was written correctly: ```bash grep -E "^acme" /etc/pve/nodes/$(hostname)/config ``` Expected output: ```ini acme: account=default acmedomain0: pve.yourdomain.com,plugin=cloudflare ``` ## Ordering the Certificate With the account and domain configured, request the certificate: **Web UI**: `Node > Certificates > ACME` > **Order Certificates Now** A task log shows the challenge flow in real time. DNS-01 validation against Cloudflare typically takes 30-60 seconds — Cloudflare's API propagation is fast enough that the challenge succeeds on the first validation attempt. **CLI**: ```bash pvenode acme cert order ``` If you're using the staging account, the certificate issuer will be "Fake LE Intermediate X1" — that's correct. Once staging succeeds, switch to the production account and re-order: ```bash pvenode config set --acme "account=default" pvenode acme cert order --force ``` After a successful order, the Proxmox web UI immediately starts serving the new certificate. Reload your browser — the padlock should now show a valid issuer. ## How Auto-Renewal Works Proxmox does not use a separate cron job for certificate renewal. The `pve-daily-update.service` systemd unit runs once per day and checks whether any node certificates expire within 30 days. If they do, it renews automatically via the same ACME config. Check the timer status: ```bash systemctl status pve-daily-update.timer ``` Trigger a manual renewal check: ```bash pvenode acme cert renew ``` If a renewal fails silently, check the update log: ```bash grep -i acme /var/log/pveupdate.log | tail -20 ``` Expect to see lines confirming the renewal check ran and either skipped (cert still valid) or completed successfully. ## Gotchas and Pitfalls From Real Use **Rate limits hit fast during testing**: Let's Encrypt's production CA allows 5 failed certificate orders per domain per hour. If your plugin config is wrong and you retry quickly, you'll burn the limit. Always test with the staging CA first — it has no rate limits. **DNS propagation timing**: The Cloudflare plugin inserts the TXT record and then waits before signaling ACME to validate. Cloudflare is typically 5-15 seconds. Slower DNS providers can take 2-5 minutes, and the Proxmox ACME client's built-in propagation wait may time out before slower providers finish. If validation fails consistently with a non-Cloudflare provider, look for a `sleep` or `propagation_seconds` setting in its plugin script under `/usr/share/proxmox-acme/dnsapi/`. **Wildcard certificates require DNS-01**: Let's Encrypt issues wildcard certs only via DNS-01. To get `*.yourdomain.com`, set: ```bash pvenode config set --acmedomain0 "*.yourdomain.com,plugin=cloudflare" ``` Note that `*.yourdomain.com` covers `pve.yourdomain.com` but not the apex `yourdomain.com`. Add a second domain entry (`--acmedomain1`) for the apex if you need it. **Cluster nodes each need separate certs**: In a three-node cluster, configure ACME independently on `pve1`, `pve2`, and `pve3`. Each node's config lives at `/etc/pve/nodes//config`. There is no mechanism to push a certificate from the cluster view. Once you've invested the time setting up a [full Proxmox private cloud](/articles/build-private-cloud-home-proxmox-ve/), budget an extra 10 minutes per additional node for certificate setup. **Port 8006 vs port 80**: HTTP-01 challenge needs port 80 forwarded to the Proxmox host — not port 8006. Forwarding 80 → 8006 via NAT will fail the challenge because the ACME token is served on port 80, not the web UI port. ## Verifying the Certificate After ordering, confirm the cert details: **Web UI**: `Node > Certificates` — you'll see the Let's Encrypt cert listed alongside the original self-signed CA cert. **CLI**: ```bash openssl x509 \ -in /etc/pve/local/pveproxy-ssl.pem \ -text -noout \ | grep -E "(Subject:|Issuer:|Not After)" ``` Expected output: ``` Subject: CN=pve.yourdomain.com Issuer: C=US, O=Let's Encrypt, CN=R10 Not After : Jul 23 12:00:00 2026 GMT ``` The certificate is valid for 90 days. Proxmox renews it when fewer than 30 days remain, so in steady state you should never see expiry-related downtime. ## Using a Different DNS Provider Proxmox ships ACME DNS plugins for 30+ providers. List all available APIs: ```bash ls /usr/share/proxmox-acme/dnsapi/ ``` The naming convention: `dns_cf.sh` → pass `--api cf` to `pvenode acme plugin add dns`. For AWS Route 53: ```bash pvenode acme plugin add dns myroute53 \ --api route53 \ --data "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE&AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" ``` If your DNS provider isn't in the list, the `acme-dns` delegation approach works universally — you create a CNAME from `_acme-challenge.yourdomain.com` to a subdomain on a separate acme-dns server you control. More setup, but compatible with any registrar. ## Conclusion Setting up Let's Encrypt on Proxmox takes about 15 minutes and permanently eliminates the certificate warning on your management interface. Register a staging account, add your DNS plugin, configure the domain, confirm the staging cert orders cleanly, then switch to production — the systemd timer handles every renewal from that point forward. This pairs directly with the steps in the [Proxmox firewall, fail2ban, and SSH hardening guide](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) to give your management interface a properly secured baseline from day one. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-high-availability-vm-failover/ 2026-04-21T00:00:00.000Z

> /etc/modules ``` ## Configure the Hardware Watchdog with watchdog-mux Proxmox ships `watchdog-mux`, a daemon that multiplexes the watchdog device so multiple HA processes can share it safely. It must be running on every cluster node. Check and enable it: ```bash systemctl status watchdog-mux systemctl enable --now watchdog-mux ``` Verify the LRM connected to it: ```bash journalctl -u pve-ha-lrm --since "5 minutes ago" | grep -i watchdog ``` You should see a line confirming the LRM opened `/run/watchdog-mux.sock`. Errors here mean fencing is broken and recovery will hang indefinitely. The watchdog timeout is configurable: ```ini # /etc/default/pve-ha-manager HA_WATCHDOG_TIMEOUT=60 ``` The 60-second default is appropriate for most setups. Shorter values increase sensitivity to transient network blips; longer values delay recovery. ### Setting Up IPMI Fencing for Bare-Metal Nodes For bare-metal servers with IPMI — which covers most enterprise hardware and many homelab boards — IPMI fencing is more reliable than a software watchdog alone. It gives you hard power control even when the OS is completely unresponsive. Install the fence agents package on all nodes: ```bash apt install fence-agents ``` Test your BMC credentials before configuring anything: ```bash fence_ipmilan -a 192.168.1.52 -l admin -p yourpassword -o status ``` Expected output: `Status: ON`. If this fails, fix IPMI access first — there is no point configuring HA fencing around a broken BMC connection. While you're securing IPMI access, make sure it's restricted to your management VLAN; the [Proxmox hardening guide](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) has practical firewall rules for exactly this scenario. Configure the fence agent per-node via the Proxmox API: ```bash pvesh set /nodes/pve2/config \ --fence-plugin ipmilan \ --fence-ipmi-ip 192.168.1.52 \ --fence-ipmi-user admin \ --fence-ipmi-password yourpassword ``` ## How to Create HA Groups and Enroll VMs ### Create an HA Group HA groups control which nodes are eligible to run a set of VMs and in what priority order. Navigate to **Datacenter → HA → Groups → Add**, or use the API: ```bash pvesh create /cluster/ha/groups \ --group critical-vms \ --nodes "pve1:3,pve2:2,pve3:1" ``` The trailing number is priority — higher wins. Equal priority means Proxmox picks the surviving node arbitrarily. | Option | Effect | |--------|--------| | `restricted` | VMs only ever run on nodes listed in this group | | `nofailback` | VMs don't migrate back when the preferred node recovers | | Node priority | Determines which surviving node receives the VM first | Start with one group containing all nodes at equal priority. Tune after watching real failovers. ### Add VMs and Containers to the HA Group In the web UI: select a VM, click **More → Manage HA**. Or with the CLI: ```bash pvesh create /cluster/ha/resources \ --sid vm:101 \ --group critical-vms \ --state started \ --max_restart 3 \ --max_relocate 3 ``` - `--state started`: the desired state HA will actively maintain - `--max_restart`: restart attempts on the current node before escalating to relocation - `--max_relocate`: relocation attempts across nodes before marking the resource failed LXC containers use `--sid ct:102`. Confirm all enrolled resources: ```bash pvesh get /cluster/ha/resources ``` Before adding a VM, always verify its disk is on shared storage: ```bash qm config 101 | grep -E "^(scsi|virtio|ide|sata)" # You want output like: # scsi0: ceph-pool:vm-101-disk-0,size=32G # Not: # scsi0: local-lvm:vm-101-disk-0,size=32G ``` A VM on `local-lvm` appears enrolled and healthy in the HA panel, then silently fails to recover when you need it most. There is no warning at enrollment time. ## How to Test HA Failover the Right Way Do not use `systemctl poweroff` to test failover. A clean shutdown lets the node announce its departure to the cluster, which changes how the CRM handles the transition — it's not a realistic crash simulation. Use a hard power-off instead. From a machine with IPMI access: ```bash ipmitool -H 192.168.1.51 -U admin -P yourpassword chassis power off ``` Alternatively, on a dedicated test node, force a kernel panic: ```bash # WARNING: This immediately crashes the system. Test nodes only. echo c > /proc/sysrq-trigger ``` Watch recovery in real time from a surviving node: ```bash watch -n2 "pvesh get /cluster/ha/status/current" ``` Expected timeline: - **0-30s**: Corosync detects the absent node, CRM initiates fencing - **30-60s**: Watchdog resets the failed node, or IPMI confirms power-off - **60-90s**: CRM issues relocation commands; LRM brings VMs online on the surviving node If the status stays in `recovery` past 90 seconds, the CRM is waiting on a fencing confirmation that never arrived: ```bash journalctl -u pve-ha-crm -f ``` The log will tell you exactly which fence operation stalled. It's almost always either `watchdog-mux` not running on every node after a reboot, or stale IPMI credentials. ## Common HA Mistakes to Avoid **VM on local storage.** Enrolled in HA, appears healthy, fails silently on recovery. Verify storage before adding any resource. **Skipping the IPMI fence test.** `fence_ipmilan ... -o status` takes 10 seconds to run. Skipping it takes hours to debug when HA stalls at 3am. **Two nodes without a qdevice.** One failure, no quorum, HA freezes. Either add a third node or deploy `corosync-qnetd` on a lightweight device before relying on HA for anything real. **NTP drift.** Corosync is sensitive to clock skew. Offset over a few hundred milliseconds triggers spurious node-unreachable events. Run `timedatectl status` on each node and confirm NTP is active and synced. **max_restart set to 1.** A VM that needs 45 seconds to complete its startup health check will relocate unnecessarily on the first failed check. Set `max_restart` to at least 3 for non-trivial workloads. **No N-1 capacity planning.** HA restarts VMs, but if surviving nodes are already at 90% RAM utilization, the VMs fail to start anyway. For a three-node cluster with 128 GB per node, plan as though any single node may be absent — cap total allocated RAM at 256 GB. ## Conclusion With `watchdog-mux` confirmed running, shared storage in place, and VMs enrolled in HA groups, Proxmox automatically recovers critical workloads within 90 seconds of a node failure. Fencing isn't bureaucratic overhead — it's the safety mechanism that makes corruption-free restarts possible. Run the hard power-off test before you declare success. Once HA is protecting your VMs at the infrastructure level, add point-in-time recovery at the data level: schedule regular backups via [Proxmox Backup Server](/articles/automated-backups-proxmox-backup-server/) so that even a storage failure has a fallback beyond the last snapshot. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-lvm-thin-pools-vm-snapshots/ 2026-04-20T00:00:00.000Z

Proxmox Pulse https://proxmoxpulse.com/articles/k3s-kubernetes-cluster-proxmox-vms/ 2026-04-19T00:00:00.000Z

` with the token from the previous step: ```bash curl -sfL https://get.k3s.io | \ K3S_URL=https://192.168.1.10:6443 \ K3S_TOKEN= \ sh -s - agent \ --node-name k3s-worker-1 ``` Repeat on `k3s-worker-2` with `--node-name k3s-worker-2`. Each agent join completes in under 30 seconds. From your local machine: ```bash kubectl get nodes -o wide ``` Expected output: ``` NAME STATUS ROLES AGE VERSION k3s-control Ready control-plane,etcd,master 5m v1.32.3+k3s1 k3s-worker-1 Ready 2m v1.32.3+k3s1 k3s-worker-2 Ready 1m v1.32.3+k3s1 ``` ## Adding Persistent Storage with Longhorn K3s includes a `local-path` provisioner that creates node-local volumes — fine for stateless workloads, useless for anything that needs to survive pod rescheduling to a different node. Longhorn replicates block storage across your worker nodes and fixes this. ### Add a Dedicated Disk to Each Worker From the Proxmox host (VMs stay running — virtio hotplug works on Linux 5.x+ guests): ```bash qm set 102 --virtio1 local-lvm:50,discard=on qm set 103 --virtio1 local-lvm:50,discard=on ``` The disks appear immediately as `/dev/vdb` inside the VMs. Do not partition or format them — Longhorn manages the raw block device directly. ### Install Longhorn Prerequisites on Each Worker ```bash sudo apt-get install -y open-iscsi nfs-common sudo systemctl enable --now iscsid ``` Skipping `open-iscsi` is the single most common reason Longhorn volumes get stuck in `Attaching`. The iSCSI initiator failure is silent — the pod just hangs. ### Deploy Longhorn v1.7.1 ```bash kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/deploy/longhorn.yaml ``` Watch the rollout (takes 3–4 minutes on first deploy): ```bash kubectl -n longhorn-system get pods --watch ``` Once all pods are running, set Longhorn as the default storage class: ```bash kubectl patch storageclass local-path \ -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' kubectl patch storageclass longhorn \ -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` Now any PVC without an explicit storage class gets a Longhorn volume replicated across both workers. ## Common Gotchas and How to Fix Them **Nodes stuck in `NotReady` after join**: Almost always a firewall issue. K3s needs ports 6443 (API), 8472/UDP (Flannel VXLAN), and 10250 (kubelet) open between nodes. If `ufw` is active on your VMs, it will silently drop VXLAN traffic. Either disable it or open those ports explicitly: ```bash sudo ufw allow 6443/tcp sudo ufw allow 8472/udp sudo ufw allow 10250/tcp ``` **Node IPs change after a DHCP lease renewal**: This is why the static IP step matters. If you skipped it and used DHCP, the flannel overlay breaks when the IP changes. Fix by setting static IPs via `qm set` cloud-init config and reprovisioning the affected node. **`kubectl get nodes` shows the node as `NotReady` after a Proxmox host reboot**: Check that the `k3s` and `k3s-agent` services started correctly. Cloud-init sometimes races with systemd on first boot after a snapshot restore. ```bash sudo systemctl status k3s # control plane sudo journalctl -u k3s-agent -f # workers ``` **Template disk shows as `unused0` after import**: Re-attach it manually: ```bash qm set 9000 --scsi0 local-lvm:vm-9000-disk-0 ``` This happens when the storage name used in `qm importdisk` doesn't exactly match the storage pool ID shown in the Proxmox UI. ## Securing the Cluster The default K3s kubeconfig at `/etc/rancher/k3s/k3s.yaml` is world-readable on the control plane — fix that immediately: ```bash sudo chmod 600 /etc/rancher/k3s/k3s.yaml ``` The node token in `/var/lib/rancher/k3s/server/node-token` grants full cluster join rights. Treat it like a root password and rotate it after initial setup. For the Proxmox host layer, [hardening Proxmox VE with firewall, fail2ban, and SSH security](/articles/hardening-proxmox-firewall-fail2ban-ssh-security/) covers host-level lockdown you should do in parallel with cluster setup. For disaster recovery: [Proxmox Backup Server](/articles/automated-backups-proxmox-backup-server/) can snapshot all three K3s VMs at the hypervisor level, giving you a clean restore point before cluster upgrades or Kubernetes version bumps. Pair hypervisor snapshots with Longhorn's built-in snapshot support for application-level recovery. ## Conclusion You now have a three-node K3s v1.32 cluster on Proxmox VE 9.1: control plane with embedded etcd, two workers with Longhorn persistent storage, and a local `kubeconfig` ready for `kubectl`. The cloud-init template approach means adding a fourth node is a `qm clone` and a 30-second agent join — no manual OS setup. The logical next step is deploying an ingress controller (ingress-nginx requires two `kubectl apply` commands) and exposing your first service outside the cluster. ]]> Proxmox Pulse https://proxmoxpulse.com/articles/self-host-openclaw-ai-assistant-proxmox-lxc/ 2026-04-18T00:00:00.000Z

Proxmox Pulse https://proxmoxpulse.com/articles/proxmox-two-factor-authentication-totp-webauthn/ 2026-04-17T00:00:00.000Z

**Important**: Enroll 2FA for `root@pam` before enabling this policy. Enabling it first locks out any account that hasn't enrolled — including your own. ### Checking Per-User Enrollment Status From the web UI, **Datacenter → Permissions → Two Factor** lists all enrolled factors. From the CLI: ```bash # List all Proxmox users pveum user list ``` # Inspect the raw TFA config (contains hashed secrets — handle with care) cat /etc/pve/priv/tfa.cfg Never share or expose `tfa.cfg` — it contains the TOTP secrets and WebAuthn credential data for all users. ## API Tokens and 2FA API tokens (`user@pam!tokenname`) don't support 2FA by design — they're meant for automation. This means token hygiene matters even more once 2FA is enforced for interactive logins. Best practices for API tokens: - Grant tokens **minimal permissions** — avoid `Administrator` or `PVEAdmin` roles - Enable privilege separation so a token can't exceed its own grants - Store tokens in environment variables or a secrets manager, never in plaintext config files - Rotate tokens periodically and audit usage in `/var/log/pve/tasks/` ```bash # Create a scoped token with privilege separation enabled pveum user token add automation@pve backup-token --privsep 1 ``` # Grant only the specific permission needed pveum acl modify /storage/backups \ --user automation@pve!backup-token \ --role PVEDatastoreUser With `--privsep 1` active, the token can only use permissions explicitly assigned to it — it cannot inherit everything from the parent user account. ## Recovery: Regaining Access After Losing Your Authenticator This is the scenario most people worry about. Proxmox gives you three recovery paths in order of preference. ### Option 1: Use Your Recovery Keys When you enrolled TOTP, Proxmox generated single-use recovery codes. If you saved them: 1. On the 2FA login prompt, click **Use recovery key** 2. Enter one of your saved codes 3. Once inside, immediately re-enroll a new TOTP device and generate fresh recovery codes Store recovery codes in a password manager or printed in a physically secure location — not in the same place as the device you're recovering from. ### Option 2: Remove 2FA via CLI If you have SSH or console access to the node: ```bash # Remove all 2FA factors for a user pveum user tfa delete root@pam ``` After running this, the account can log in with password only. Re-enroll immediately. ### Option 3: Edit the TFA Config Directly As a last resort with physical console access: ```bash # Back up first cp /etc/pve/priv/tfa.cfg /root/tfa.cfg.bak ``` # Edit and remove the affected user's entry nano /etc/pve/priv/tfa.cfg The `pveum` CLI method is safer and should be tried before manually editing config files. ## Additional Security Layers to Stack with 2FA Two-factor authentication is most effective as part of a layered approach: **Restrict port 8006 by source IP** Even with 2FA enabled, there's no reason to leave the management interface open to all subnets. Scope access using the Proxmox firewall: ```bash # Allow web UI access only from your management network pvesh create /nodes/pve/firewall/rules \ --action ACCEPT --type in --proto tcp \ --dport 8006 --source 192.168.1.0/24 ``` pvesh create /nodes/pve/firewall/rules \ --action DROP --type in --proto tcp --dport 8006 **Use a dedicated non-root daily account** Create a Proxmox-realm admin account for routine work and reserve `root@pam` for break-glass access only: ```bash pveum user add admin@pve --comment "Daily admin" pveum acl modify / --user admin@pve --role Administrator ``` Enroll 2FA on the daily account and stop interactive root logins. **Monitor failed logins** ```bash # Tail failed authentication attempts in the proxy log grep -i "authentication failure" /var/log/pveproxy/access.log | tail -20 ``` Consider shipping this log to a central SIEM or at minimum checking it weekly. ## Conclusion Two-factor authentication is one of the highest-value security changes you can make to a Proxmox VE installation. TOTP takes five minutes to set up and works with any authenticator app — there's no valid reason to leave it disabled. WebAuthn raises the bar further with phishing-resistant credentials for anyone handling production infrastructure. The combination of 2FA, SSH key authentication, scoped API tokens, and Proxmox firewall rules closes the most common attack paths against the management plane. Enable 2FA today, save your recovery codes somewhere physically secure, then layer on the remaining controls. Your future self will appreciate it the next time a password shows up in a breach notification. ]]> Proxmox Pulse