I’ve benchmarked sequential NFS performance in our ESX VMs lately to find out how different virtualization layers interact. Using gigabit ethernet a modern physical machine easily saturates the network link so you can expect approximately 100 MB/s. It this also possible from inside a VM?
Why is NFS in a virtual machine useful? All the usual reasons for using NFS come to mind: for example multiple web-servers sharing common data on an external NAS appliance. If we mount a NFS server in a VM, NFS traffic will use the “data” interface in the ESX infrastructure.
My quick-and-dirty-test in our VMWare environment was based a Suse Linux Enterprise Server (SLES) 10 based, 64-bit template with current VMWare tools installed. The physical server was a Sun 4150 with dual-quadcore CPUs and 40 GB of RAM. There are 6 GigE ports dedicated to client data (but remember that VMware will use only one of them for a particular VM). The counterpart was a Netapp 3050 cluster running OnTAP GX 10.0.2 - again connected using gigabit ethernet with a volume on a 16 disk RAID DP aggregate.
I used our usual battle-tested NFS mount options (rsize=32768, wsize=32768, hard, tcp, timeo=60) to mount the export inside the virtual machine and then used dd to write and read 10 GB of data. Here are the results:
- write: 85 MB/s
- read: 85 MB/s
After increasing /proc/sys/sunrpc/tcp_table_slot_entries from the default of 16 to 128 performance improved a bit:
- write: 90 MB/s
- read: 98 MB/s
Actually I didn’t check for other traffic using the network links but in general I’m very satisfied with these results considering this is “only” gigabit ethernet.
The scenario I’m interested in is the virtualization of out Linux-based mail servers: one of them is based on Postfix/Dovecot and uses NFS to an Ontap 7-based 3050 to store ~1 TB of emails in ~30k accounts. During the day it maintains approximately 1600 IMAP connections and processes lots of POP3 requests - all this with a load of 4-6. At night the machine is almost idle and of course there is also a second one configured in a heartbeat HA-cluster so we waste lots of resources here.
Of course sequential throughput is not really meaningful for maildir-based mail servers so I’ll have to follow up with a couple of metadata benchmarks…
Comments welcome (moderated)
If you want to know how performance of a virtual disk in a NFS datastore is you might want to check this post.
2 comments ↓
Hi! Whe I’ve read the last part, I have thought:”Hey, there is someone who is trying the same as us!!!”.
Two summers ago, we tried to convert our mail servers into a HA(Active / Active, Load Balanced) cluster. We used Dovecot and GFS without success, the load was too much for the nodes.
The past summer we tried with Dovecot+OCFS2 in three physical nodes and it was ok for a week but when the summer ended it was very difficult to mantain due to the load in the servers, high IOwait, etc.(mainly due to our list server spaming our own mailservers, we tunned the list server but it was effortless)
It was much better than GFS but still innacceptable for a normal user.
Now we think to use NFS, first with physical nodes and If it performs well, with VMs.(which is the final idea because it is very flexible and scalable )
For the moment we dont have a NetApp(we plan to purchase a 3050 next september), so we will do the tests in an NFS server…
Why did you use a node to failover instead of balancing the traffic to both (it is NFS). It is related to the locking mechaninsm?
Do yu think several VMs(3,4) will perform well in this type of service, i mean the IMAP and POP3 connections, locking,indexes…etc…
Do you have any recommendation? NFS Tunning, ideas… Any Dovect, postfix recommendation? Dovecot recommends the indexes in a different disk…,
Thanks for your blog!
Jon
Most cluster file systems are optimized for data throughput and not metadata performance so I wouldn’t expect any miracles
Netapp handles small files and metadata operations (create/delete) very well. Just avoid too many files (>50000) in a directory.
If you want to find out how a filesystem performs with maildir based email I’d recommend the Postmark benchmark or (better) a parallelized version.
Our HA setup is like this because we do not need load balancing (the single server handles the load easiliy) and yes, having two active nodes is different (Dovecot docs recommend splitting users between multiple servers -> active-active cluster but again we don’t need that). K.I.S.S.
I also would not recommend a cluster file system for mail because recovery (say a fsck) in case something fails is usually a nightmare. Additionally there is often no fast replication mechanism which can cope with millions of files (=emails) in a maildir scenario. Netapp + volume SnapMirror is easy, fast and absolutely reliable and if your primary system breaks just switch to the replica.
You must log in to post a comment.