I have a Proxmox server for my HA install which I built through following Whiskerz install (so I’m no whizz at this).
Everything has been running fine then I was running out of disk space so I simply increased the space. All seemed fine. In the past week however after a Proxmox backup my HA is’nt running. Looking at the log in Proxmox I get the following:
INFO: starting new backup job: vzdump 100 --storage local --quiet 1 --mailnotification always --mailto david@swallowmead.com --mode snapshot --compress lzo
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-01-11 00:00:03
INFO: status = running
INFO: VM Name: hassosova-3.12
INFO: include disk 'sata0' 'local-lvm:vm-100-disk-1' 96G
INFO: include disk 'efidisk0' 'local-lvm:vm-100-disk-0' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-100-2021_01_11-00_00_03.vma.lzo'
INFO: started backup task 'b10e2ebd-3bd1-4c89-9318-c1423db0c376'
INFO: resuming VM again
INFO: 0% (83.2 MiB of 96.0 GiB) in 3s, read: 27.7 MiB/s, write: 13.1 MiB/s
INFO: 1% (989.5 MiB of 96.0 GiB) in 23s, read: 45.3 MiB/s, write: 27.4 MiB/s
INFO: 2% (2.0 GiB of 96.0 GiB) in 47s, read: 42.2 MiB/s, write: 42.0 MiB/s
INFO: 3% (2.9 GiB of 96.0 GiB) in 1m 20s, read: 30.2 MiB/s, write: 28.8 MiB/s
INFO: 4% (3.8 GiB of 96.0 GiB) in 1m 43s, read: 40.8 MiB/s, write: 38.2 MiB/s
INFO: 5% (4.8 GiB of 96.0 GiB) in 2m 15s, read: 31.2 MiB/s, write: 31.1 MiB/s
INFO: 6% (5.8 GiB of 96.0 GiB) in 2m 43s, read: 34.4 MiB/s, write: 34.4 MiB/s
INFO: 7% (6.8 GiB of 96.0 GiB) in 3m 30s, read: 22.2 MiB/s, write: 22.0 MiB/s
INFO: 8% (7.7 GiB of 96.0 GiB) in 4m 5s, read: 27.7 MiB/s, write: 27.5 MiB/s
INFO: 9% (8.6 GiB of 96.0 GiB) in 4m 17s, read: 78.1 MiB/s, write: 69.8 MiB/s
INFO: 10% (9.6 GiB of 96.0 GiB) in 4m 39s, read: 46.2 MiB/s, write: 45.5 MiB/s
INFO: 11% (10.6 GiB of 96.0 GiB) in 5m 5s, read: 37.0 MiB/s, write: 36.6 MiB/s
INFO: 12% (11.6 GiB of 96.0 GiB) in 5m 34s, read: 35.1 MiB/s, write: 34.1 MiB/s
INFO: 13% (12.5 GiB of 96.0 GiB) in 5m 59s, read: 39.5 MiB/s, write: 39.2 MiB/s
INFO: 14% (13.4 GiB of 96.0 GiB) in 6m 24s, read: 37.5 MiB/s, write: 35.3 MiB/s
INFO: 15% (14.4 GiB of 96.0 GiB) in 6m 48s, read: 41.0 MiB/s, write: 40.5 MiB/s
INFO: 16% (15.4 GiB of 96.0 GiB) in 7m 13s, read: 41.3 MiB/s, write: 40.2 MiB/s
INFO: 17% (16.3 GiB of 96.0 GiB) in 7m 41s, read: 33.3 MiB/s, write: 33.1 MiB/s
INFO: 18% (17.3 GiB of 96.0 GiB) in 8m 7s, read: 37.7 MiB/s, write: 36.4 MiB/s
INFO: 19% (18.3 GiB of 96.0 GiB) in 8m 34s, read: 36.7 MiB/s, write: 36.2 MiB/s
INFO: 20% (19.2 GiB of 96.0 GiB) in 9m 10s, read: 27.0 MiB/s, write: 26.3 MiB/s
INFO: 21% (20.2 GiB of 96.0 GiB) in 9m 35s, read: 40.4 MiB/s, write: 39.8 MiB/s
INFO: 22% (21.1 GiB of 96.0 GiB) in 10m 7s, read: 30.5 MiB/s, write: 29.6 MiB/s
INFO: 23% (22.1 GiB of 96.0 GiB) in 10m 33s, read: 37.4 MiB/s, write: 37.2 MiB/s
INFO: 24% (23.0 GiB of 96.0 GiB) in 10m 58s, read: 39.1 MiB/s, write: 37.6 MiB/s
INFO: 25% (24.0 GiB of 96.0 GiB) in 11m 23s, read: 39.9 MiB/s, write: 39.6 MiB/s
INFO: 26% (25.0 GiB of 96.0 GiB) in 12m 0s, read: 27.1 MiB/s, write: 26.3 MiB/s
INFO: 27% (25.9 GiB of 96.0 GiB) in 12m 26s, read: 37.4 MiB/s, write: 36.4 MiB/s
INFO: 28% (26.9 GiB of 96.0 GiB) in 12m 55s, read: 32.9 MiB/s, write: 31.7 MiB/s
INFO: 29% (27.9 GiB of 96.0 GiB) in 13m 18s, read: 44.7 MiB/s, write: 43.6 MiB/s
INFO: 30% (28.8 GiB of 96.0 GiB) in 13m 47s, read: 33.8 MiB/s, write: 32.3 MiB/s
INFO: 31% (29.8 GiB of 96.0 GiB) in 14m 18s, read: 31.2 MiB/s, write: 31.1 MiB/s
INFO: 32% (30.8 GiB of 96.0 GiB) in 14m 51s, read: 31.3 MiB/s, write: 30.8 MiB/s
INFO: 33% (31.7 GiB of 96.0 GiB) in 15m 24s, read: 29.1 MiB/s, write: 28.8 MiB/s
INFO: 34% (32.7 GiB of 96.0 GiB) in 15m 49s, read: 37.8 MiB/s, write: 36.5 MiB/s
INFO: 35% (33.6 GiB of 96.0 GiB) in 16m 10s, read: 47.9 MiB/s, write: 41.5 MiB/s
INFO: 36% (34.6 GiB of 96.0 GiB) in 16m 33s, read: 42.7 MiB/s, write: 42.0 MiB/s
INFO: 37% (35.5 GiB of 96.0 GiB) in 16m 51s, read: 53.7 MiB/s, write: 47.1 MiB/s
INFO: 38% (36.6 GiB of 96.0 GiB) in 17m 16s, read: 42.7 MiB/s, write: 42.2 MiB/s
INFO: 39% (37.5 GiB of 96.0 GiB) in 17m 32s, read: 57.7 MiB/s, write: 50.1 MiB/s
INFO: 40% (38.4 GiB of 96.0 GiB) in 18m 1s, read: 32.8 MiB/s, write: 32.6 MiB/s
INFO: 41% (39.4 GiB of 96.0 GiB) in 18m 24s, read: 43.5 MiB/s, write: 37.2 MiB/s
INFO: 42% (40.4 GiB of 96.0 GiB) in 19m 2s, read: 27.2 MiB/s, write: 26.9 MiB/s
INFO: 43% (41.3 GiB of 96.0 GiB) in 19m 29s, read: 35.2 MiB/s, write: 30.8 MiB/s
INFO: 44% (42.3 GiB of 96.0 GiB) in 20m 9s, read: 25.0 MiB/s, write: 25.0 MiB/s
INFO: 45% (43.2 GiB of 96.0 GiB) in 20m 35s, read: 35.5 MiB/s, write: 30.9 MiB/s
INFO: 46% (44.2 GiB of 96.0 GiB) in 21m 5s, read: 33.5 MiB/s, write: 32.4 MiB/s
INFO: 47% (45.1 GiB of 96.0 GiB) in 21m 32s, read: 35.7 MiB/s, write: 30.3 MiB/s
lzop: No space left on device: <stdout>
ERROR: VM 100 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: VM 100 qmp command 'backup-cancel' failed - unable to connect to VM 100 qmp socket - timeout after 5974 retries
ERROR: Backup of VM 100 failed - VM 100 qmp command 'query-backup' failed - got timeout
INFO: Failed at 2021-01-11 00:42:00
INFO: Backup job finished with errors
TASK ERROR: job errors
I assume it says I have no space, but I already increased the VM space so have plenty. I must be missing something. Should this kill HA? Advice really appreciated.
1 post - 1 participant