Proxmox VE: RAW, QCOW2 or ZVOL?

5 min readMar 17, 2021

In our previous article, we compared the two virtualization technologies available in Proxmox; LXC and KVM. After analysis, we find that both technologies deliver good CPU performance, similar to the host. On the other hand, disc reading and writing performance are far from advantageous for KVM. This article will delve deeper into our analysis to see how the different storage formats available for KVM, namely ZVOL, RAW and QCOW2, compare with the default configurations. Although we analyze only three formats, Proxmox supports several others such as NFS, GluserFS, LVM, iSCSI, Ceph, etc. The full list is available on the Proxmox wiki “Storage” page.

Choose your storage format

In Promox VE 4.4, it is possible to use different Virtual Machine storage formats. During pre-configuration, it is possible to select the storage format when creating a Virtual Machine.

Storage formats

Local / RAW

This storage format is probably the least sophisticated. The Virtual Machine disk is represented by a flat file. If your virtual drive is 8GiB in size, then this file will be 8GiB. Please note that this storage format does not allow “ snapshot” creation. One of the RAW format advantages is that it is easy to save and copy because it is only a file.

Local / QCOW2

This storage format is more sophisticated than the RAW format. The virtual disk will always be presented as a file. On the other hand, QCOW2 allows you to create a “thin provisioning” disc; that is, you can create a virtual disk of 8GiB, but its actual size will not be 8GiB. Its exact size will increase as data is added to the virtual disk. Also, this format allows the creation of “snapshot”. However, the time required to do a rollback is a bit longer compared to ZVOL.

ZVOL

This storage format is only available if you use ZFS. You also need to set up a ZPOOL in Proxmox. Therefore, a ZVOL volume can be used directly by KVM with all the benefits of ZFS: data integrity, snapshots, clone, compression, deduplication, etc. Proxmox gives you the possibility to create a ZVOL in “thin provisioning”.

Architectural difference

The following diagram shows the architectural differences between the three formats we want to test.

From this diagram, it should be understood that RAW and QCOW2 are superimposed on the VFS and Local File layers. At the same time, ZVOL can be used directly by QEMU, thus avoiding the Local Files, VFS and ZFS Posix Layerlayers. In all likelihood, ZVOL should outperform RAW and QCOW2. That’s what we’re going to check with our tests.

Before presenting our test results, you will find below a table that summarizes each storage format’s distinctions.

Analyse des résultats

Local / RAW

Surprisingly, the RAW format outperforms ZVOL in some tests. In other tests, the RAW format provides satisfactory results. However, it does not provide enough advantages over other storage formats (clone, snapshot, compression, etc.). It should be understood that this format is a good option but that there are better alternatives.

Local / QCOW2

QCOW2 is definitely to be avoided since it was impossible to obtain a measurable result in 3 different tests. During the AIO Stress test, KVM became completely unstable and stopped responding. And, during the tiobench test, the test had still not finished after more than 48 hours. The other formats produce a result in less than 3 hours.

Note that QCOW2 provides reasonable performance when the file is written in an ext4 format. We concluded that the combination with ZFS is problematic.

ZVOL

This format provides similar performance to RAW, at times a little slower. However, it comes with a set of features that are not negligible in virtual machine management: thin provisioning, compression, snapshot, clone, migration, etc.

Reading tests provide better results. It is obviously in writing that the performances are less good.

Conclusion

In conclusion, it would appear that the ZVOL format is a good choice compared to RAW and QCOW2. A little slower in writing but provides significant functionality. However, you should consider ZFS being very greedy in terms of machine resources (memory and I/O). A server with ZFS needs better technical specifications if you expect good performance; this was apparent with our old servers.

That said, ZFS offers several configuration settings to accommodate different workloads. The few tests we conducted show that Promox’s default settings are not optimal for our virtual machines. In our next articles, we will test some of these parameters to optimize ZFS/ZVOL performance.

How do you convert your drives to ZVOL?

With Proxmox, it’s effortless to switch from one storage format to another. Just select the disk to convert and use the “Move Disk” action on the web interface. You will then be able to select the new storage format.