Tag Archives: Virtualization

Storage Benchmarking (part 1 of…)

This will be a series of blog posts that will try to help you establish a consistent storage benchmarking methodology.

Storage is an area that I have focused on for much of my career, I’ve been fortunate to be involved in a lot of very challenging and fun storage projects over the years.  I spent a good period of time in the trenches, either for storage manufacturers or a reseller, performing professional services for custom storage implementations.  It is common place for apprehension on any changes in storage systems, if that is vendor, disk types, RAID type and grouping of disks, and more.  Storage is a complicated (and expensive) beast, it is still one of the most expensive (and profitable) components within the entire datacenter.

Virtualization changed the world for IT, it is one of the most disruptive single concepts that truly changed how IT does business.  I know I don’t have to give history lessons on the impact this has had for the hardware vendors, but it is apparent it was mostly negative for server and network vendors and extremely positive for storage vendors. Prior to virtualization being common place storage area networks (SANs), or other shared storage, was not overly common…when it did exist it was for niche applications that represented a very small part of the services an IT department managed.  With virtualization, and really all of the great things that came with VMware’s virtualization (e.g. high availability, Vmotion, etc), shared storage became a fundamental component in all datacenters.  Shared storage, if it be fiber channel, iSCSI, or NFS, went from representing a fraction of the data storage in a company to addressing essentially all data stored.

Storage has become more critical today than it ever was, we have optimized and streamlined all other aspects of the IT platform…however most storage vendors are still building products that are based on ancient principles, changes are complex and high risk.  It is critical that you have a methodology to actually assess any changes to your storage environment to determine if it is a net gain or loss for your business, and storage performance directly impacts the success of the business.  The challenge is that storage benchmarking is really hard, it is a monumental task to take on and actually do it well.  Various tools have been used to try and do point performance tests, however they are not adequate for assessing replacing an entire storage platform or changing the configuration of an existing one.

Now that I have a long winded introduction lets start looking at what it takes to be successful in storage benchmarking.  Like any initiative it needs to be an actual project with a process, while you can run and download Iometer and run a test within minutes…does that test actually mean anything?  Does it correlate to anything other than another similar run of Iometer?  Likely not.

I happen to focus on storage for a growing public cloud provider, I have spent a significant portion of time over the past 3+ years benchmarking storage platforms.  I have tried various tools to try to assess a rating for storage systems that are under evaluation, and what is important to me may not be important to you.  You need to determine what are the critical aspects for your business, here are the primary areas that I score systems on, in no particular order:

  • Reliability
  • Availability
  • Durability & Security
  • Sustainability
  • Scalability
  • Performance

That is a lot of abilities, so I will break down what I mean in each one into what I am referring to.  You will need to determine what are the specific requirements within each of the scoring areas are for your use case, as nothing is valid if it isn’t within the context of your use case.

Reliability

Reliability is a pretty critical for my use case, and this extends to how predictable are all of the other areas that we are evaluating.  When a component fails, do you have a consistent outcome?  Does performance become unpredictable?  Does the “useable” capacity that is reported change after a failure?  There are a lot of gotchas, and the key is to know and be able to predict them as you must factor them into your implementation plan for the solution.  This is an area that isn’t entirely objective as you just can’t test all conditions and context is everything.

Availability

This is a measure of resiliency, fault tolerance, survivability or in other words how consistent can you access my data..even during failure conditions.  This is where storage vendors fit their dog-and-pony show in the datacenter in, they pull this disk and that cable to “prove” that the data is available even after these failures.  I won’t go into the specific tests that I use for this, you may trust your vendor or you may not…how this is assessed must be within the context of each specific storage system.  How you test/prove availability for legacy architecture, scale-out architecture, or hyper converged architecture type storage systems can vary greatly.

Durability & Security

How confident are you that when written is the data that returned and, perhaps even more critical, that the data returned is the actual data that was written.  Security is grouped with durability as there are two aspects of security, confidence that an unauthorized entity cannot modify or otherwise access my data.  This primarily relates to checksums on written data (to set confident point of reference), scrubbing on stored data (to compare to point of reference), and repair of data (create new copy from parity or mirroring).  This is nearly impossible to actually test for, it is something you must query your vendor about.  There are many ways faults can be introduced that can cause loss of data integrity: bit rot, medium read error, controller/cable failures, and more.  Most vendors address this through checksums, in fact it wasn’t long ago that many vendors used this as the primary differentiator between “enterprise” storage systems or not…and it is often assumed to be present in modern enterprise storage systems, but you may be surprised to find that it doesn’t exist in many popular solutions.

Security is often addressed through encryption, some vendors may claim that their self encrypting disks (SED) are all that is necessary, however if the keys are stored on the disk then the encryption does nothing more than provide rapid-erase (if the disk is still operational), such as before you send it for replacement or otherwise decommission it.

In proprietary systems you really have to trust your vendor, and ideally the vendor has 3rd party validation of their offering so that you aren’t just taking the word of an individual that may be more interested in closing the deal than you keeping your job or your company surviving the “what if”, when it does happen.

Sustainability

Another area that may be more subjective than not, as it is difficult to assess this without a great amount of actual experience using the particular system…so if you are looking at a new product you have to be subjective based on your exposure while evaluating, or try to find IDC, or other, rankings comparing the offering.

Ultimately, is it operational within your environmental boundaries, both physical, staffing (expertise), etc.  Does it integrate seamlessly with any existing processes or tools that you depend on (e.g. monitoring and alerting systems)?  Can you or your staff adequately manage the system during a time of crisis (you know, 6am on a holiday when something horrific happens)?  Having logical and intuitive interfaces is a big differentiator here, or do you refer to documentation anytime you try to manage the system?

Scalability

Scalability is absolutely critical for my use case, and it actually is a comprehensive topic that addresses all of the other abilities and performance.  Does reliability of the system decrease, maintain or increase with scale?  Does the risk for data loss increase or decrease with scale?  Can you sustainably operate 100s or 1000s of these systems?  Do you need 100 operators to manage 1000 systems?

Performance

This is the area that I will devote other posts to, as this is where things get more complex.  Vendors typically have comparison between their solution and others that cover the other topics, however reference performance benchmarks are just that, a reference.  Any benchmark is only valid as a comparison to another benchmark executed the same way using the same workload, if that is a synthetic benchmark or not…and if all instances being compared were setup consistently to the respective vendors “best” practices.

To be continued…

Tagged , , ,

vCloud Director – Using Guest Customization Scripts (Linux)

The intent of this article is to cover the steps for leveraging scripting within guest customization. A vCloud user may wish to peruse this as an avenue of automatically installing additional software that is hostname specific, e.g. security management software that integrates a Linux OS to Active Directory.

I am going to assume the reader knows how to login to vCloud Director, either within an organization or within the system context. I also assume that an existing virtual machine exists that we will work with, in my example I will use Linux (CentOS).

  1. Stop the vApp if it is currently running (we cannot edit the properties of a running VM)
  2. Open the vApp so that we can see the individual virtual machines

    wpid-voila_capture569-2012-03-15-18-24.png

  3. Right click the virtual machine (or use the action menu) to access the Properties
  4. Switch to the Guest OS Customization tab
  5. Select the option to “Enable guest customization”

    wpid-voila_capture582-2012-03-15-18-24.png

  6. This enables basic guest customization, such as configuring the guest OS hostname, setting the root password and network configuration.
  7. Scroll down within the guest customization tab
  8. You will see a text box, we can input script content within this text box. Alternatively you can upload the script that will be injected into the guest OS during the customization process. I will first start with a simple script that calls an existing shell script within the guest OS. Please also notice that we have specific sections for “precustomization” and “postcustomization”, pre-customization is before the standard vCloud Director customization process and the other is post this process. If the script that you wish to use is dependent upon the hostname or network connectivity, then you would be best served by using a post-customization script. 

In my example I am calling out to two scripts myscript-pre.sh and myscript-post.sh — these scripts must be in place within the OS file system before it can be ran

    .wpid-voila_capture581-2012-03-15-18-24.png

    NOTE: If you wish to upload a script using the Browse button it must be a text only script, it cannot be an executable binary.

  9. Click OK to save those changes
  10. Power on the virtual machine as usual
  11. Create your script within the guest OS in the path you specified
  12. My test script is quite lame, so don’t laugh. The goals are to answer questions that I’ve seen, such as if the network is available and which user context the script runs under.
    • Pre-customization:

      wpid-voila_capture587-2012-03-15-18-24.png

    • Post-customization:

      wpid-voila_capture588-2012-03-15-18-24.png

  13. Shutdown your virtual machine

    wpid-voila_capture577-2012-03-15-18-24.png

  14. Right click and select to Power On and Force Recustomization
  15. After customization completes, login and verify that your script ran.
    • Pre-customization:
      wpid-voila_capture589-2012-03-15-18-24.png
    • Post-customization:

      wpid-voila_capture585-2012-03-15-18-24.png

Observations:

There seems to be little documentation from VMware on “when” exactly a pre-customization script is ran vs a post-customization script. The time is only 23 seconds apart, so what exactly occurs during those 23 seconds? Logging services (syslogd) and most other system services do not start until after the pre-customization script has ran, so little output exists for what occurs during that window (or prior). It appears that pre-customization occurs at the time that vmware-tools start, on my system that is S03…which is the 2nd service to start (after microcode_ctl). You can also compare your time stamps to /var/log/messages in order to see what events are occurring.

In looking at the /var/log/vmware-imc/customization.log we can see a bit more detail as to timing.

wpid-voila_capture586-2012-03-15-18-24.png

Pre-customization occurs before the default vCloud Director customization scripts set execute, which set hostname and network configs (and generate SID or join an AD domain on Windows).

Post-customization is likely the area that most scripts will need to be executed, after the network configuration is set. In testing I encountered a situation that a script that was dependent on additional network services (e.g. to support NFS) would fail if executed directly as a post-customization script, a work around that resolved this was just adding a “sleep 30” prior to the script execution.

An area of challenge is troubleshooting these scripts as there is no way to run customization in an interactive form. The easiest way to confirm things are going to work is making sure the script can run as root if you execute it directly from a login shell. Next you can insert it into the post-customization process and assume that it will work. VMware has published a couple of KB articles that discuss which log files are relevant to the process, you can review those logs for any errors. Ideally your script itself will have error logging capability.

If you wish for advanced customization capabilities, then your best bet is probably to not use the vCloud Director customization at all…or at least only use it to configure the networking. vCenter Orchestrator is far more feature rich and extensible, the limitations on what can be done in vCenter are most likely only constrained by the amount of effort you put into developing your workflows. The customization process used within vCloud Director is more similar to that of Lab Manager than of vCenter, so if you run into trouble you may try searching under Lab Manager discussion groups.

References:

Tagged , ,

Managing Windows 2008 Server Core

In the interest of reducing overhead within my lab environment I decided to try and use Windows 2008 Server Core (R2/x64). If you’ve ever installed Server Core, the first thing you notice is you are only presented with a CMD shell at login, there is no full GUI. You can launch applications, installers, etc. however there is no Start Menu to assist you on your way.

I decided I’d chase this rabbit for a bit, as I am researching the use of Server Core in conjunction with vCloud Directory deployed vApp services…as a bloated UI heavy OS isn’t the most practical when it comes to “scale of cloud”, sure we have magical Transparent Page Sharing (for more info on TPS see: PDF written by Carl Warldspurger)…but just like energy efficiency, the easiest WATT to save is the one you never used. I know many of my fellow vGeeks have home labs, and very few of us have the host resources we’d like to have so we are all chasing efficiency, especially in regard to storage and host memory. In order to adapt to using Server Core you need to figure out how to manage it, so I thought I would write an article about what I have learned.

Sconfig.cmd

The first option Microsoft offers is the shell tool, Sconfig, “Server Configuration”, you can access this by running Sconfig.cmd from the CMD prompt.

wpid-voila_capture448-2011-06-3-12-55.png

We can navigate this tool easily by inputing the number of the section we wish to view or modify. If we select #4 (Configure Remote Management) we are given the following subset of options:

wpid-voila_capture449-2011-06-3-12-55.png

If you recall, by default almost all of the remote management tools are disabled on Windows. Likewise, the firewall is enabled and fairly restrictive. Being able to turn on these remote management options quickly so that we can move away from the console is always a benefit.

I went ahead and selected #1 Allow MMC Remote Management, remote MMC is pretty useful as it allows me to consolidate the management tasks between multiple target servers into one place. The window immediately indicated that it was configuring the firewall and enabling required services. It then gave a popup indicating the final status.

wpid-voila_capture450-2011-06-3-12-55.png

I would personally prefer to not have the popup window that I then having to use a mouse to navigate to and select OK, my preference would have been for that status update and acknowledgement to be provided within the textual interface; I suppose I shouldn’t be surprised that a company that is bad at UI is even worse at shell. I will give them some credit, as Sconfig seems to have more intelligence than many of their GUI based wizards do, which often leave some tasks incomplete that must be manually finished in other wizards. Since this is my template I personally went ahead and enabled all of the remote management options to avoid having to do so later.

Core Configurator

The next tool which I found is Core Configurator, which can be downloaded from CodePlex. This tool is delivered as an ISO, one option within vCloud Director would be to upload this ISO and attach it to your Windows 2008 Server Core virtual machine as needed, this may provide some additional security however it isn’t necessarily convenient. Personally, I opted to copy the contents of the ISO to a directory within my Windows 2008 Server Core template so that it is readily available.

Microsoft states that Core Configurator supports the following tasks:

  • Product Licensing
  • Networking Features
  • DCPromo Tool
  • ISCSI Settings
  • Server Roles and Features
  • User and Group Permissions
  • Share Creation and Deletion
  • Dynamic Firewall settings
  • Display | Screensaver Settings
  • Add & Remove Drivers
  • Proxy settings
  • Windows Updates (Including WSUS)
  • Multipath I/O
  • Hyper-V including virtual machine thumbnails
  • JoinDomain and Computer rename
  • Add/remove programs
  • Services
  • WinRM
  • Complete logging of all commands executed

In order to launch Core Configurator, you simply navigate to the directory that contains it and run Start_Coreconfig.wsf (default for attached ISO this would likely be D:\Start_Coreconfig.wsf), which presents this interface:

wpid-voila_capture452-2011-06-3-12-55.png

Selecting the small expansion arrow at the bottom reveals a few more convenient options:

wpid-voila_capture453-2011-06-3-12-55.png

Control Panel view:

wpid-voila_capture454-2011-06-3-12-55.png

I was able to use this tool to install all of the latest Microsoft hotfix packages, the interface could use a “select all” option…but then again, you generally want to review what you are installing and this encourages you to do so. You must select the hotfix to be installed one at a time.

wpid-voila_capture455-2011-06-3-12-55.png

Firewall Settings

You can easily view and modify the Firewall Settings:

wpid-voila_capture456-2011-06-3-12-55.png

wpid-voila_capture457-2011-06-3-12-55.png

wpid-voila_capture458-2011-06-3-12-55.png

Conclusion

Without the GUI we are accustomed to, even the most basic tasks become challenging. Perhaps you know how to configure interfaces, join an Active Directory domain, or even change the computer name from command line on Windows; this task was previously foreign to me. Just how much do we save by using Core instead of a full version of Windows 2008 Server? Here is a screen shot taken from vCenter on resource utilization for this particular Windows 2008 Server Core (R2 x64):

wpid-voila_capture460-2011-06-3-12-55.png

Here is the same resource accounting for a similar (base) config for Windows 2008 R2 Standard:

wpid-voila_capture461-2011-06-3-12-55.png

Notice the Active Guest Memory of each of the above? With only default installation + VMware Tools installed thats a 47% decrease in active memory, it is also a ~decrease of ~20% decrease in storage capacity to support the base OS. While this isn’t much in a large production environment, however I don’t have the luxury of a Cisco UCS B230 with 32-DIMM slots for my lab…when my host only has 16GB of RAM, that increases the number of base OS I can support…again, the easiest unit of X to conserve is the one you don’t use.

Tagged , , ,

Home Lab – Storage Performance Test (Part 1)

This is a continuation of my Home Lab Build – Overview

In order to help out my fellow vGeeks I thought I should keep with my “comparison” to the hardware storage appliances. While I personally won’t be running my system as a 4-disk configuration, I realize that some of them may. I ran some tests using Bonnie++ benchmarking and dd from /dev/zero to provide some benchmarks, I realize that these will not be 100% representative of the performance that would be experienced with protocols in place however it should provide a relative comparison between the disk configuration options.

I have chosen to use Bonnie++ for my benchmarks as it is a far faster setup, it operates from within the management console of Nexenta. If you are not familiar with Bonnie++ and how it performs testing you can find more info here: http://www.coker.com.au/bonnie++/readme.html

I will run three tests using Bonnie++, only varying the block size between 4KB, 8KB, and 32KB.

  • run benchmark bonnie-benchmark -p2 -b 4k
  • run benchmark bonnie-benchmark -p2 -b 8k
  • run benchmark bonnie-benchmark -p2 -b 32k

Each test will be performed against each of the following storage pool configurations:

  • RAID0 (no disk failure protection)
  • RAIDZ1 (single disk failure protection, similar to RAID5)
  • RAIDZ2 (double disk failure protection, similar to RAID6) *this configuration will only be tested with 4K block sizes to display the parity tax*

I will run a few “hardware” variations, my target configuration with 2 vCPU and 4-6GB RAM as well as a reduced configuration with 2vCPU and 2GB of RAM. I expect the decrease in RAM to mostly decrease read performance as it will reduce the working cache size.

I intended to have the time to create some lovely graphs to simplify the process of comparing the results of each test, however I could either wait another week or two before finding time or I should share the results in the output format from Bonnie++. In order to get this info to my fellow vGeeks, I have decided to publish the less-than-pretty format, after all, any real geek prefers unformatted text to PowerPoint and glossy sales docs.

Hardware Variation 1 (2 vCPU/6GB RAM) / 4-disk RAID0

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
157MB/s 42% 84MB/s 32% 211MB/s 28% 1417/sec
156MB/s 41% 83MB/s 32% 208MB/s 28% 1579/sec
——— —- ——— —- ——— —- ———
314MB/s 41% 168MB/s 32% 420MB/s 28% 1498/sec

================== 8k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
148MB/s 22% 92MB/s 20% 212MB/s 20% 685/sec
147MB/s 21% 90MB/s 20% 212MB/s 21% 690/sec
——— —- ——— —- ——— —- ———
295MB/s 21% 182MB/s 20% 424MB/s 20% 688/sec

================== 32k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
144MB/s 12% 90MB/s 11% 210MB/s 14% 297/sec
153MB/s 12% 92MB/s 12% 210MB/s 15% 295/sec
——— —- ——— —- ——— —- ———
298MB/s 12% 183MB/s 11% 420MB/s 14% 296/sec

Hardware Variation 2 (2 vCPU/2GB RAM) / 4-disk RAID0

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
113MB/s 21% 75MB/s 22% 216MB/s 31% 980/sec
113MB/s 21% 74MB/s 22% 217MB/s 31% 936/sec
——— —- ——— —- ——— —- ———
227MB/s 21% 150MB/s 22% 434MB/s 31% 958/sec

================== 8k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
110MB/s 13% 80MB/s 15% 209MB/s 22% 521/sec
110MB/s 13% 80MB/s 15% 210MB/s 23% 524/sec
——— —- ——— —- ——— —- ———
220MB/s 13% 161MB/s 15% 420MB/s 22% 523/sec

================== 32k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
114MB/s 8% 81MB/s 9% 218MB/s 13% 297/sec
113MB/s 8% 79MB/s 9% 218MB/s 12% 294/sec
——— —- ——— —- ——— —- ———
228MB/s 8% 161MB/s 9% 436MB/s 12% 296/sec

Hardware Variation 1 (2 vCPU/6GB RAM) / 4-disk RAID1+0 (2 x 1+1 mirrors)

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
89MB/s 27% 53MB/s 19% 143MB/s 22% 1657/sec
89MB/s 27% 53MB/s 19% 144MB/s 22% 1423/sec
——— —- ——— —- ——— —- ———
178MB/s 27% 106MB/s 19% 288MB/s 22% 1540/sec

================== 8k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
83MB/s 13% 53MB/s 12% 147MB/s 16% 800/sec
83MB/s 12% 54MB/s 12% 147MB/s 16% 752/sec
——— —- ——— —- ——— —- ———
167MB/s 12% 107MB/s 12% 294MB/s 16% 776/sec

================== 32k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
85MB/s 7% 55MB/s 7% 141MB/s 9% 277/sec
82MB/s 7% 53MB/s 7% 135MB/s 9% 266/sec
——— —- ——— —- ——— —- ———
167MB/s 7% 109MB/s 7% 276MB/s 9% 271/sec

Hardware Variation 2 (2 vCPU/2GB RAM) / 4-disk RAID1+0 (2 x 1+1 mirrors)

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
65MB/s 11% 48MB/s 14% 154MB/s 22% 892/sec
65MB/s 11% 48MB/s 13% 152MB/s 22% 786/sec
——— —- ——— —- ——— —- ———
130MB/s 11% 97MB/s 13% 306MB/s 22% 839/sec

================== 8k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
67MB/s 7% 47MB/s 9% 157MB/s 14% 669/sec
67MB/s 7% 47MB/s 9% 155MB/s 14% 637/sec
——— —- ——— —- ——— —- ———
135MB/s 7% 94MB/s 9% 313MB/s 14% 653/sec

================== 32k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
68MB/s 5% 31MB/s 3% 153MB/s 8% 338/sec
68MB/s 5% 31MB/s 3% 151MB/s 8% 342/sec
——— —- ——— —- ——— —- ———
136MB/s 5% 62MB/s 3% 304MB/s 8% 340/sec

Hardware Variation 1 (2 vCPU/6GB RAM) / 4-disk RAIDZ1 (RAID5)

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
109MB/s 30% 54MB/s 22% 133MB/s 21% 813/sec
108MB/s 32% 54MB/s 22% 131MB/s 20% 708/sec
——— —- ——— —- ——— —- ———
218MB/s 31% 108MB/s 22% 265MB/s 20% 761/sec

================== 8k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
114MB/s 25% 60MB/s 17% 131MB/s 14% 525/sec
118MB/s 24% 60MB/s 18% 133MB/s 14% 517/sec
——— —- ——— —- ——— —- ———
232MB/s 24% 121MB/s 17% 265MB/s 14% 521/sec

================== 32k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
107MB/s 12% 60MB/s 8% 138MB/s 9% 163/sec
111MB/s 11% 60MB/s 8% 138MB/s 9% 172/sec
——— —- ——— —- ——— —- ———
218MB/s 11% 121MB/s 8% 276MB/s 9% 167/sec

Hardware Variation 2 (2 vCPU/2GB RAM) / 4-disk RAIDZ1 (RAID5)

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
74MB/s 15% 40MB/s 12% 160MB/s 18% 715/sec
76MB/s 15% 41MB/s 13% 165MB/s 19% 651/sec
——— —- ——— —- ——— —- ———
151MB/s 15% 82MB/s 12% 325MB/s 18% 683/sec

================== 8k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
75MB/s 9% 42MB/s 8% 167MB/s 21% 384/sec
73MB/s 8% 42MB/s 8% 166MB/s 20% 387/sec
——— —- ——— —- ——— —- ———
149MB/s 8% 85MB/s 8% 333MB/s 20% 386/sec

================== 32k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
73MB/s 5% 41MB/s 4% 168MB/s 11% 182/sec
71MB/s 5% 40MB/s 4% 168MB/s 11% 183/sec
——— —- ——— —- ——— —- ———
144MB/s 5% 82MB/s 4% 337MB/s 11% 182/sec

Hardware Variation 3 (2vCPU/8GB RAM) / 4-disk RAIDZ1 (RAID5)

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
114MB/s 34% 58MB/s 22% 146MB/s 22% 872/sec
114MB/s 34% 59MB/s 23% 147MB/s 21% 693/sec
——— —- ——— —- ——— —- ———
228MB/s 34% 118MB/s 22% 293MB/s 21% 783/sec

Hardware Variation 1 (2 vCPU/6GB RAM) / 4-disk RAIDZ2 (RAID6)

================== 4k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
71MB/s 20% 43MB/s 16% 111MB/s 20% 716/sec
71MB/s 20% 43MB/s 16% 110MB/s 20% 677/sec
——— —- ——— —- ——— —- ———
143MB/s 20% 86MB/s 16% 221MB/s 20% 696/sec

================== 8k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
75MB/s 13% 42MB/s 10% 110MB/s 12% 540/sec
74MB/s 16% 42MB/s 11% 104MB/s 11% 491/sec
——— —- ——— —- ——— —- ———
149MB/s 14% 84MB/s 10% 215MB/s 11% 515/sec

================== 32k Blocks ==================
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
70MB/s 7% 43MB/s 6% 109MB/s 8% 202/sec
70MB/s 7% 42MB/s 6% 109MB/s 8% 203/sec
——— —- ——— —- ——— —- ———
140MB/s 7% 85MB/s 6% 218MB/s 8% 203/sec

Summary
I have to admit, I was incorrect in my prediction that the RAM size would more directly correlate to read performance…it actually seems that increasing the RAM somehow leads to a slight decrease in read performance, while improving write performance. I am going to speculate this has to do with poor caching algorithms, or at least poor for this workload, as well as ZIL being performed in RAM. The larger RAM leads to increased L2ARC (cache) side, this does improve random-seeks significantly but decreases max read throughout (large block) due to the L2ARC leading to inaccurate predictive reads (speculation).

Much like a NetApp storage systems, writes are always attempted to be done in large chunks…if you actually were to watch the iostat output for the physical devices you would see that it is very much peaks and valleys for writes to the physical media, even though the incoming workload is steady state. NetApp and ZFS both attempt to play a form of tetris in order to make the most efficient write possible, the more RAM available the better it can stage these writes to complete efficiently.

One key measure is the actual throughput per given disk, RAID0 is a good way to determine this. If we look at the results we have the following metrics, as expected re-writes always suffer as they do in any file system. We will focus on writes, reads and random-seeks and I will use the numbers from the lowest memory configuration for RAID0:

  • Writes: 227MB/s
  • Reads: 434MB/s
  • Random Seeks: 958/sec

Now we need to break this into per-disk statistics, which is simply dividing the above value by the number of physical disk.

  • Writes: 56.75MB/sec/disk
  • Reads: 108.5MB/sec/disk
  • Random Seeks: 239.5 IOPS/disk

Of course, we can see that the one flaw in Bonnie++ is that we do not have latency statistics. We normally expect a 7200 RPM SATA disk to offer 40-60 IOPS with sub-20 millisecond response, I have no measure of the response time being experienced during this test or how much of the random seeks were against cache. I selected the lowest RAM (cache) configuration to try and minimize that in our equation.

We can then use this as a baseline to measure the degradation in each protection scheme on a per-data disk basis. In a RAID1+0 configuration we have 2 disks supporting writes, and 4-disks supporting reads and this leads to our reasonable performance for reads. The reason my lab is operating in a RAID1+0 configuration is that my environment is heavily read oriented, and with the low number of physical disks I did not want the parity write-tax in addition with 6 1TB SATA drives I am not capacity restricted.

I almost went into a full interpretation of my results, however I stumbled upon this site in my research: http://blogs.sun.com/roch/entry/when_to_and_not_to You will find a detailed description into the performance expectations of each RAID configuration, the telling portion is this:

Blocks Available
Random FS Blocks / sec
RAID-Z
(N – 1) * dev
1 * dev
Mirror        
(N / 2) * dev
N * dev
Stripe
N * dev
N * dev

The key item to interpret is that with RAID-Z, the random IOPS are limited to a single device. You will see in the referenced blog posting that a configuration of multiple small RAID-Z groups performs better than a large RAID-Z group, as each group would have 1-device supporting the random workload. This may not be 100% in correlation with RAID5, or whatever RAID scheme your storage platform uses as they are not all created equal.

Tagged ,

Home Lab Build – Overview

I realize I’m not alone in this process, it seems many of my fellow VMware enthusiasts are putting together home labs. This will be the first of a few blog posts regarding my home lab, I am working on planning out the requirements and my overall goals. Primary requirements are to be able to operate the majority of the VMware products in order to advance my understanding and satisfy curiosity of the growing portfolio, and hopefully to help me obtain more advanced certifications.

I have always had a lab of sorts on my laptop, though the new corporate issued laptop isn’t quite as beefy as the one from my previous employer. I previously had a MacBook Pro 17” with the i7 processor, the new laptop is a 15” with the i5. In order to make the most out of the laptop hardware both have 8GB of RAM and a dual drive configuration, including SSD. I will go into specifics of my MBP configuration in a separate entry. However I needed something more powerful, 8GB of RAM just doesn’t go very far to supporting multiple ESX hosts, vCenter, database servers and the other infrastructure requirements for hosting a vCloud environment. I had contemplated going for a larger virtual host environment, perhaps a Mac Pro 12-core with 32GB of RAM…until I did the math and compared the results to my “budget”. I decided to go for a lower cost route.

Hosts – So far I have determined the following requirements:

  • Minimum of 2 hosts capable of operating ESXi 4.1
  • Each host must provide at least 2 Gigabits of connectivity
  • Ideally is listed on VMware compatibility list

Network – I have set the following network hardware requirements

  • 8 Gigabit Ethernet ports
  • Switch must support 802.1Q VLAN tagging
  • Should support LACP
  • No proprietary software to manage
  • Support for jumbo frames

Storage – As we all know, shared storage is essential. Yes, we can operate without shared storage but every advanced feature requires shared storage. Since this is a home lab my performance requirements are minimal.

  • Provide iSCSI and NFS storage
  • Provide RAID capabilities to increase performance and resiliency
  • Performance scalability
  • Flexibility

With a little bit of time and creativity I believe I found solutions for each of the requirements. I will detail the hardware selected for each area above.

Hosts:
I have selected to use Dell T110 servers, these servers feature the entry quad core Xeon processors (X3400). I settled on these after looking at several options, including those from HP and home built from bare components. The T110 won out in large part on price, the base price with the Xeon X3430 was $379 but I opted for the upgrade to the X3440 with Hyper-Threading for $90. I couldn’t find any VMware specific benchmarks on either of these processors, however the PassMark score for the X3440 was 5303 vs the X3430 with 3638 which represents a 45% improvement. This is in part due to the Hyper-Threading, which is a debate in of itself regarding hypervisor benefits.

EDITED Feb-16-2010 – NOTE: after purchasing my first 2 hosts Dell decreased the pricing to $329 + $90 for $419 per host, Dell does not provide price guarantee but AMEX does…

The server is listed on the VMware CL and the Xeon 34xx processor includes Fault Tolerance support. Additionally, this server and motherboard support both ECC and non-ECC memory which allows for selecting lower priced non-ECC memory. Due to this I was able to max out the RAM on each host to 16GB for a reasonable price.

The servers are scheduled to arrive early next week.

Network:
I considered Netgear, D-Link, HP, Linksys and Cisco switches in trying to pick which was the best value. I would have loved to have a Catalyst switch due the proven track record, however that price alone would have exceeded what I now spent on my 2 ESX hosts. I settled on the Cisco SLM2008, it offers LACP (for when VMware gets around to it), static link aggregation (802.3ad – 2 group limit), jumbo frames and VLANs. Additionally it has a built in management web interface that works from any browser, not requiring any software to be installed is a bonus in my book. If you have a PoE switch to connect it to (or a power injector) it can run from PoE on port 1, otherwise a power brick is included. While I don’t see any value in jumbo frames for IP storage, being able to support MTU sizes larger than 1500 is critical in using Layer2 tunneling options, such as private vCloud Network Isolation features.

The switch arrived today.

EDITED Feb-16-2010
Storage:
As a storage guy I would have loved to have a NetApp FAS3210 with Flash Cache (a.k.a. PAM, or Performance Acceleration Module) but this would neither fit into my budget nor my wife’s noise tolerance. I have selected to go with a software based solution which I haven’t found many using it judging by the blog posts I’ve read. I have decided to use Nexenta Community Edition for my storage build out, I have advised former customers about this as an option for labs but haven’t actually worked with it myself. In a lab environment it can be self contained, in an enterprise solution it should be combined with an enterprise FC SAN.

While an Iomega, Synology, Drobo, or other storage appliance may be simpler to setup I am certain the option I am going with will smoke the competition at a lower price…we’ll see if I can stay on “budget”. For “budget” comparison sake I am going to work with Amazon pricing for devices that I may have considered:

  • Iomega IX4-200d 4TB (4x1TB)        $593.98
  • Thecus 4-bay N4200                         $779.21 + disk 4 x $64.99 = $1039.17
  • Synology DS411+                                 $639.99 + disk 4 x $64.99 = $899.95
  • Drobo FS (5-bay)                                $695.00 + disk 5 x $64.99 = $1019.95

I already had a server purchased that I am adding this role onto, but I also ended up adding a 3rd host to my configuration as the physical server hosting my storage system is clearly taken out of the ability to be “flexible” on maintenance and configuration changes…so I will show both totals. I already had a few parts that I am going to use, however I will try to add a price for those in to keep a fair tally.

The hardware that is added to the ESX host specific to storage is:

  • SAS Controller: Intel SASUC8I PCIe (OEM’d LSI SAS3801 for a big savings) + breakout cable: $154.99 + $19.99
  • 4-bay external SAS/SATA enclosure and cables        $179.00 + $29.50 + $27.50
  • SATA HDDs: 4 x Seagate 1TB 32MB Cache, 2 x Hitachi 1TB – 4 x $64.99 + 2 x $60
  • SSD for cache: OCZ 90GB SandForce controller drive $129.00
  • Dual-port Intel Gigabit ET NIC        $162.99
  • Dell T110 + 16GB RAM                $419.00 + $129.00

So the rough total here is ~$1630 including my ESX host and ~$1082 without the host, which gives me a full 6 disk storage system that I can expand pretty easily with dual GigE and it serves as my management host for the rest of my environment. Now I realize this is over the price of the other systems, but I believe this will provide more flexibility and far better performance than what those other systems are capable of.

EDITED Feb-23-2010

Update:
I’ve had a few questions about the pricing break down, so I thought I would try to make this a more reasonable comparison. In reality the storage I ended up with is far greater than any of the NAS appliances, I have more drive bays (8) and can actually increase to 16-bays for the price of the disk, enclosure and SAS/SATA controller.

In order to keep this an actual comparison to the appliance based options I thought I should show the pricing that is added to my first host.

  • SAS Controller + Breakout cable                        $154.99 + $19.99
  • 4 x Seagate 1TB 7200.12 Drives                        $259.96
  • 1 x 3.5” to 5.25” drive bay adapter (for ESX DASD) ~$5
  • TOTAL =                                                        $439.94

Those are truly the only additional hardware pieces needed, this would give you a 4-disk storage appliance that shares your first host. You can allocate as much or little vCPU and RAM as you wish, realizing that most of those hardware appliance options only have a low end desktop processor and 512MB of RAM.

Edit: You can find more info in the continuation of my lab build here:
Home Lab – Storage Performance Test (Part 1)

Tagged , , , ,

Another satisfied UCS customer

I spent much of this week deploying hardware to support a customer’s Cisco Unified Communications (UC).  In this case the customer is deploying a new telecommunications solution to add new capabilities to support the growing business, as well as move away from a legacy system that is becoming costly to maintain.

In order to support their overall strategy of maintaining efficiency in the data center through consolidation and virtualization, the design criteria included using Cisco UCS to facilitate consolidating the servers that support the UC environment.  Previously this deployment would have started with at least 3 physical systems, through virtualization we were able to reduce this down to 2 servers.  While this is only a 3:2 reduction, as this project moves from a proof of concept (POC) to a final production rollout the number of servers would have increased if using traditional hosting methods, however though the use of virtualization the customer can add availability and services without requiring additional hardware.

The overall solution included leveraging their existing NetApp storage system, however we needed to add fiber channel protocol support in order to meet Cisco specifications.  The solution included adding a Cisco UCS blade system comprised of a UCS6120 and a UCS 5108, as well as a Nexus 5000.  We are currently running with 2 B200 blades, each with dual 2.53Ghz 4-core CPUs and 32GB of RAM.  Additional blades will be added to this environment in the near future to support the virtualization of the remaining physical application servers in the data center.

As is typical with the Cisco UCS and NetApp deployments the project has gone pretty smoothly, more complexity is generated by the physical environment which we install these systems than are generally present within the product themselves.  In the end it makes for another happy Cisco UCS, VMware vSphere, and NetApp storage customer.  Hopefully their Cisco UC rollout goes just as smooth, thankfully that is in someone else’s hands.

Tagged , ,