Update!!! – 2/06/2009
If your Linux server is a Ubuntu VM, enable the avoidance of persistent MAC bindings for VMware NICs, as per this post
Update!!! – 12/11/2008
VMware doesn’t really provide best practices for VMs because it’s a matter of taste. So I’ll let you know how you can build templates that optimise your virtual environment deployment performance, disk I/O performance, conserve space and reduce your backup footprint.
Part 1: Creating and optimising your Linux VM template.
First of all, most people aren’t aware of this, but VMware can create thin disks. That means that even though you’ve allocated 20GB, VMware only uses as much space on disk as there is data in your VMDK. Unfortunately the VI client is unable to do this by default, so when creating a template, do not create a disk in VI client.
1. Login to the Service Console of the server
2. Run: vmkfstools -c 20G -d thin /vmfs/volumes/datastore/virtualmachine_name/virtualmachine_disk1.vmdk
3. Login to the VI client and add the existing disk you just created to the VM template.
4. Install your Linux flavour (you don’t need to offset Linux partitions as they automatically align to 64k)
—
As a result of point (2), you will see that doing an ll -h in the Service Console in the directory where the VMDK is, reveals a 20GB VMDK file. However, when you run a du -hs in the Service Console, in the directory where the VMDK is, you will see that disk usage is nowhere near the 20GB mark.
This means that your templates will consume less space and will be quicker to deploy. Furthermore, when you deploy from a template, the normal disk (thick) type will be deployed so there will be no performance issues – a win-win situation.
Part 2: Configuring the Linux VM template with internal smarts…
The second point has to do with backup via VCB. I know not all people are sold on VCB, but at some point in time it comes in damned useful. Furthermore, very few people understand VCB snap-shotting, or at least the few people I’ve talked to, know very little about it.
Let’s talk about this from the Linux side of things:
When Linux deletes a file, it doesn’t actually delete it – it doesn’t scrub the sectors on which the file resided. In fact, all that initially happens when you delete a file is Linux makes note that the space occupied by the file is listed as available space for storing new information. Until something is written to the sectors where that old file was stored the file is still there.
Do you see where I’m going with this?
VCB works below the VM layer – it captures all the data in the VMDK file – which means any space that’s held deleted files that wasn’t zeroed out, gets backed up too – and this is especially a problem on Oracle/MySQL servers where backups consume heaps of space.
So here’s what you do:
1. Create your template
2. Create a script called /usr/bin/zero-out:
#!/bin/bash
FileSystem=`grep ext /etc/mtab| awk -F" " '{ print $2 }'`
for i in $FileSystem
do
number=`df -B 512 $i | awk -F" " '{print $4}'`
percent=$(echo "scale=0; $number * 95 / 100" | bc )
dd count=`echo $percent` if=/dev/zero of=`echo $i`/zf
rm -f $i/zf
done
3. Run: chmod a+x /usr/bin/zero-out
4. Run: echo “0 11 * * * /usr/bin/zero-out” > file && crontab -u root file && rm -f file - change the time of the command runtime to suit your applications.
What the script in (4) does, is to zero out 95% of all the white-space (by creating a file full of zeroes). This is the updated bit – it will prevent a disk full situation. It’s effectively the exact same thing as the VMware Shrink Disk function in your VMware tools (except with defrag and temp files cleaning) – with the problem being that you can’t run a Shrink Disk as a scheduled task, as far as I know (though I’m searching)
By running this script in your entire infrastructure, you will reduce the size of your VCB/VSS backups across the board, for a very small performance penalty out of hours.
And your template is done
Cheers,
Leo
[...] active on the VMware blogoshere. One of the articles that really caught my attention was the “VM Template best practices (Linux)” by Leo [...]
We normally use NFS which thin provisions by default so presumably there is no advantage in handling disk creation this way? Or if we explicitly thin provision a disk at creation time does that mean that the templating process can be smarter about copying the template to the deployed VM…. sounds like an experiment worth trying.
Uh, that’s an interesting one to try Nigel. I haven’t thought about that. My own little D530 farm at home uses iSCSI – I may give it a shot with NFS – if you try it, let me know.
First, make sure you copy the ENTIRE for/dd line above. My screen cut off the rm -f command. (which means it wouldn’t run anyways missing the done, but also that the zf files would not be deleted, filling up my mount points).
Second, I personally would not do this (this way) on a live server. My databases are always adding data, my log files are always growing, and my users are always receiving EMail. With this scenario there will be a brief moment in time when each local mount point is full. Many applications behave badly when they run out of disk space.
I have done this before manually, and I dd about 90% of the available space, so the file systems don’t fill up.
Even though you are writing zeros, it still takes a long time to write Gigs of zeros to a disk, and then it takes (less) time to delete the file. A local IDE drive can take 1-10 seconds to delete a 5Gig file.
Hi Ben,
Well picked up – I did not think of that.
The post has been updated with a function that writes 95% of the space on each etx1/2/3 filesystem mount with zeroes. Obviously that percentage is adjustable.
Either way, the new script should prevent issues like disk-full situations.
Cheers,
Leo
A quick experiment with separate creation of an explicitly thin VM disk on an NFS datastore, as opposed to just using the VI defaults is not significantly different in terms of subsequent template deployment time.
This make sense, thinking about it, because there is no reasonable way for an NFS client to discover which blocks are not provisioned in a sparse file, and there is no other external record of which blocks are used within the vmdk. The vmdk file does include the ddb.thinProvisioned = “1″ entry but this appears to be decorative :-/
So chalk this up as one advantage of VMFS over NFS.
It would be fairly simple to make the clone operation skip *writing* the all zero blocks when copying, thus making the destination sparse (ie thin provisioned), and speeding up the copy if you are bandwidth limited (as we are).
Last I checked VMWare Shrink Disk didn’t work on LVM filesystem layouts anyways, so this may be the only option, thanks for the scripts. Even doing it to 70-80% of the filesystem will be of benefit for VCB.
I think we’re going to have to wait until ESX 4 the ability to clone to thin provisioned vmdks without having to make the disk first (without use of NFS or backend hardware thin luns).
[...] spent in OS configuration. We have also followed most of the recommendations made by Leo Raikhman (Linux, Windows) aside from the thin disk [...]
[...] easy to deduplicate, like a huge file full of zeros. On Linux you’d do something like what Leo Raikhman suggests as his zero-out script, and on Windows you can use sdelete to do the same. SHARETHIS.addEntry({ title: “How File [...]
[...] VM Template best practices (Linux) | Leo's Ramblings VMware doesn’t really provide best practices for VMs because it’s a matter of taste. So I’ll let you know how you can build templates that optimise your virtual environment deployment performance, disk I/O performance, conserve space and reduce your backup footprint. (tags: vmware howto) no comments yet. « links for 2009-03-18 Leave a Comment [...]
I’ve done similar… actually DD’s from /dev/zero to a file on the filesystem with control-Z of nohup (to push to background) then delete the file as it is being written to… when it reaches disk full.. DD ends the file handle is closed and the diskspace is released instantly back to the filesystem…. YES you can READ/WRITE to a file that is deleted (as long as an active file handle is open to it).
i.e. (from memory):
# dd if=/dev/zero of=/path/myzero bs=1048576 ( enter + control-Z )
[1]+ Stopped dd if=/dev/zero of=/path/myzero bs=1048576
# bg
[1]+ dd if=/dev/zero of=/path/myzero bs=1048576 &
# rm /path/myzero
# [... time passes ...]
[1]+ Done dd if=/dev/zero of=/path/myzero bs=1048576
#
increasing the bs usually speeds things up somewhat.
but yeah, picking 70-95% of the space means no actual running out of disk space for that super short period of time.
–Doug
updated the script w/ tail -1 (as df spits out two lines) and pushed dd to background and put the percent up to 98% as well as increasing block size for improved speed and lastly.. put in some “randomness” in the temporary filename:
#!/bin/bash
filename=ZeroOut${PPID}${RANDOM}
FileSystem=`grep ext /etc/mtab| awk -F” ” ‘{ print $2 }’`
for i in $FileSystem
do
number=`df -B 8192 $i | tail -1 | awk -F” ” ‘{print $4}’`
percent=$(echo “scale=0; $number * 98 / 100″ | bc )
dd count=${percent} bs=8192 if=/dev/zero of=${i}/${filename} &
# pause for 1/2 seconds
sleep 0.5
# delete the file as it is being written to
rm -f ${i}/${filename}
done
[...] This is a partial update to this post [...]