tabs ↹ over ␣ ␣ ␣ spaces

by Jiří {x2} Činčura

Comparing disk speed of various disks and VMs in Azure

5 Mar 2020 Azure, Azure VMs, Cloud

For an estimation and planned move of some resources to Azure, I had to do comparison of various VMs and disks. It’s not meant to be exhaustive with all possibilities. Just some numbers one can compare.

The move is considered mostly because the VMs can be placed closer to the customer compared to on-premise datacenter, in case you’d like to ask.

Setup

The measurement was done using CrystalDiskMark version 7.0.0 x64 running as an Administrator. Default profile was used with this test: 4 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>. CrystalDiskMark was selected because on-prem machines had it already installed.

I tested Standard D8s v3 (8 vcpus, 32 GiB memory), Standard L8s_v2 (8 vcpus, 64 GiB memory) and Standard L8s (8 vcpus, 64 GiB memory) all running Windows Server 2019 Datacenter from the default image in Azure with Premium SSDs (always fresh VM). P20 (D8sv3 only), P30, P40 (D8sv3 only), OS disk (because why not), Temp disk and NVMe disk (only available in L8sv2) were tested. No further VM modification was done except for attaching disks and turning on read caching where possible.

Read

Read (MB/s) Sequential 1MiB (Q=8, T=1) Sequential 1MiB (Q=1, T=1) Random 4KiB (Q=32, T=16) Random 4KiB (Q=1, T=1)
L8s – P30 disk (no caching) 203.859 78.228 20.910 0.994
L8sv2 – P30 disk (no caching) 167.754 66.484 21.042 0.911
D8sv3 – P30 disk (R caching) 135.068 135.060 67.520 27.121
D8sv3 – P20 disk (R caching) 135.067 135.069 67.516 26.741
D8sv3 – P40 disk (R caching) 135.065 135.059 67.513 26.760
L8sv2 – NVMe disk 1837.758 1840.725 355.024 48.016
L8s – OS disk (RW caching) 417.126 417.154 169.536 52.680
L8sv2 – OS disk (RW caching) 89.965 89.961 17.703 17.702
D8sv3 – OS disk (RW caching) 140.113 140.090 68.338 27.494
L8s – Temp disk 412.129 412.084 168.776 17.344
L8sv2 – Temp disk 84.724 84.736 16.879 16.496
D8sv3 – Temp disk 135.059 135.073 67.519 16.566
Read (IOPS) Sequential 1MiB (Q=8, T=1) Sequential 1MiB (Q=1, T=1) Random 4KiB (Q=32, T=16) Random 4KiB (Q=1, T=1)
L8s – P30 disk (no caching) 194.4 74.6 5105.0 242.7
L8sv2 – P30 disk (no caching) 160.0 63.4 5137.2 222.4
D8sv3 – P30 disk (R caching) 128.8 128.8 16484.4 6621.3
D8sv3 – P20 disk (R caching) 128.8 128.8 16483.4 6528.6
D8sv3 – P40 disk (R caching) 128.8 128.8 16482.7 6533.2
L8sv2 – NVMe disk 1752.6 1755.5 86675.8 11722.7
L8s – OS disk (RW caching) 397.8 397.8 41390.6 12861.3
L8sv2 – OS disk (RW caching) 85.8 85.8 4322.0 4321.8
D8sv3 – OS disk (RW caching) 133.6 133.6 16684.1 6712.4
L8s – Temp disk 393.0 393.0 41205.1 4234.4
L8sv2 – Temp disk 80.8 80.8 4120.8 4027.3
D8sv3 – Temp disk 128.8 128.8 16484.1 4044.4
Read (us) Sequential 1MiB (Q=8, T=1) Sequential 1MiB (Q=1, T=1) Random 4KiB (Q=32, T=16) Random 4KiB (Q=1, T=1)
L8s – P30 disk (no caching) 40817.78 13385.39 98649.36 4115.48
L8sv2 – P30 disk (no caching) 49612.29 15723.51 98600.99 4477.90
D8sv3 – P30 disk (R caching) 61571.46 7712.98 30938.71 150.24
D8sv3 – P20 disk (R caching) 61626.01 7690.78 30897.07 152.44
D8sv3 – P40 disk (R caching) 61461.37 7719.54 30853.62 152.34
L8sv2 – NVMe disk 3990.00 569.32 3883.10 85.12
L8s – OS disk (RW caching) 20000.46 2498.25 12340.33 77.16
L8sv2 – OS disk (RW caching) 92281.99 11597.10 116845.03 230.29
D8sv3 – OS disk (RW caching) 59395.36 7428.99 30443.21 148.17
L8s – Temp disk 20200.87 2534.45 12374.76 235.33
L8sv2 – Temp disk 97826.63 12307.93 122198.56 247.96
D8sv3 – Temp disk 61528.75 7709.53 30871.49 246.41

As expected, the NVMe disk is a clear winner with impressive numbers. Disks P20, P30 and P40 have similar performance in D8sv3 no matter the sequential vs random IO or other parameters. I suppose it’s the limit of D8sv3 VM itself and hence bigger/faster disks do not make sense unless you also get bigger VM.

Write

Write (MB/s) Sequential 1MiB (Q=8, T=1) Sequential 1MiB (Q=1, T=1) Random 4KiB (Q=32, T=16) Random 4KiB (Q=1, T=1)
L8s – P30 disk (no caching) 204.068 40.052 21.093 1.082
L8sv2 – P30 disk (no caching) 167.793 43.201 20.993 1.068
D8sv3 – P30 disk (R caching) 133.368 45.303 21.093 1.392
D8sv3 – P20 disk (R caching) 133.166 44.668 9.704 1.386
D8sv3 – P40 disk (R caching) 132.965 46.768 31.476 1.410
L8sv2 – NVMe disk 1300.216 1303.379 1206.941 202.007
L8s – OS disk (RW caching) 417.296 413.544 169.629 59.352
L8sv2 – OS disk (RW caching) 89.971 89.968 9.357 11.926
D8sv3 – OS disk (RW caching) 140.084 140.092 28.910 42.610
L8s – Temp disk 412.075 412.138 168.797 36.305
L8sv2 – Temp disk 84.729 84.730 16.878 16.772
D8sv3 – Temp disk 135.069 135.069 67.517 27.387
Write (IOPS) Sequential 1MiB (Q=8, T=1) Sequential 1MiB (Q=1, T=1) Random 4KiB (Q=32, T=16) Random 4KiB (Q=1, T=1)
L8s – P30 disk (no caching) 194.6 38.2 5149.7 264.2
L8sv2 – P30 disk (no caching) 160.0 41.2 5125.2 260.7
D8sv3 – P30 disk (R caching) 127.2 43.2 5149.7 339.8
D8sv3 – P20 disk (R caching) 127.0 42.6 2369.1 338.4
D8sv3 – P40 disk (R caching) 126.8 44.6 7684.6 344.2
L8sv2 – NVMe disk 1240.0 1243.0 294663.3 49318.1
L8s – OS disk (RW caching) 398.0 394.4 41413.3 14490.2
L8sv2 – OS disk (RW caching) 85.8 85.8 2284.4 2911.6
D8sv3 – OS disk (RW caching) 133.6 133.6 7058.1 10402.8
L8s – Temp disk 393.0 393.0 41210.2 8863.5
L8sv2 – Temp disk 80.8 80.8 4120.6 4094.7
D8sv3 – Temp disk 128.8 128.8 16483.6 6686.3
Write (us) Sequential 1MiB (Q=8, T=1) Sequential 1MiB (Q=1, T=1) Random 4KiB (Q=32, T=16) Random 4KiB (Q=1, T=1)
L8s – P30 disk (no caching) 40799.23 26029.26 98372.25 3777.46
L8sv2 – P30 disk (no caching) 49617.07 24181.19 98992.59 3829.51
D8sv3 – P30 disk (R caching) 62353.77 23019.10 98263.99 2936.44
D8sv3 – P20 disk (R caching) 62433.34 23427.75 204262.98 2949.40
D8sv3 – P40 disk (R caching) 62371.02 22387.84 66148.71 2900.10
L8sv2 – NVMe disk 5623.68 803.43 1577.99 20.11
L8s – OS disk (RW caching) 19946.69 2512.65 12277.98 68.47
L8sv2 – OS disk (RW caching) 91918.53 11602.18 116659.01 234.15
D8sv3 – OS disk (RW caching) 59136.99 7424.76 29183.79 95.48
L8s – Temp disk 20195.58 2532.72 12364.83 112.18
L8sv2 – Temp disk 97673.81 12279.27 122200.62 242.85
D8sv3 – Temp disk 61271.61 7692.77 30760.47 148.73

Same story, first place for NVMe. Who would have thought, right? 😃 Now at least for the Random 4KiB (Q=32, T=16) (which matches the best the workload the machines are doing now) workload the P20, P30 and P40 in D8sv3 show different performance and match the expectation of P20 < P30 < P40.

Summary

Different workloads have different needs, so measure yourself. Frankly switching between VMs isn’t that difficult, so you can change your mind later if needed. And if you relies on best IO subsystem possible, get the Lsv2 and use the NVMe disk available.

Profile Picture Jiří Činčura is an independent developer focusing on data and business layers, language constructs, parallelism and databases. Specifically Entity Framework, asynchronous and parallel programming, cloud and Azure. He's Microsoft Most Valuable Professional and you can read his articles, guides, tips and tricks at www.tabsoverspaces.com.