Thin VMDK’s : Sdelete and vmkfstools to help zero out

I was working in my lab to create a new Windows 2012R2 Template that I can use. Since it is in my Lab and only have so much SSD space (lucky to have my Lab on SSD Drives šŸ™‚ ) I wanted to makes sure that after the all the patches for Windows 2012R2 is loaded and I did a disk cleanup that I could have the least amount of space used by my new Template. I scratched around to find a util that I could use to do a “Zero out of free space” and found Sdelete. After reading a few blogs on how other people have used sdelete I was ready to use the following command:

  • sdelete.exe -c

Well that did not work to well for me as there was a change in the way sdelete -c worked. So -c would write out the free space with random data. Anyway…so I saw that -z was the correct option. But now my thin disk was a thick disk. So I followed the following way to get my newly created Thick VMDK Thin again.

As you can see I started below with a 40Gb vmdk (42Gb -2Gb memory swap file) that was using 18Gb space:Space01

Then I ran sdelete.exe -c from within the OS and the VMDK basically inflated to full size:



Next was to run sdelete.exe -z to Zero Out all the free space inside the OS:


After this the VMDK is still at 40Gb ..only the free space have been Zeroed out. No we need to power down the VM and from the ESXi cmd line we need to run vmkfstools -KĀ virtualmachinevmdk.vmdk.Ā In my case the command was : vmkfstools -K Test.vmdk. Note the K is caps. -K will “Punch Holes” in the free spaces that have been Zeroed out.


After all this you can power on the VM and your VMDK will be thin.


In my case I saved about 3Gb on my template.

Freenas 9.3 and VAAI Enhancements

I use Freenas in my lab for all my shared storage. Just find it easy to install and work with. I noticed some good updates in version 9.3 of Freenas that is worth sharing.


From the Release notes, full VAAI for all 7 primitives (you can find them here) have been added:

Kernel iSCSI has completely replaced the old iSCSI code, adding support for VMWare VAAI (all 7 primitives), MS ODX and Windows 2012 Clustering as well as much higher performance and space efficiency (zero’d blocks can now be reclaimed). Support for STUN and pool storage thresholds also makes VMWare behavior far more robust when ZFS pools fill up.

MinĀ 8Gb disk

With Version 9.3 the recommendation is now a 8Gb disk (or Memory Stick, I use them in my lab)

Just a bit more on my storage in my lab that I use. The configuration is as follow :

  • 2x HP Micro Servers
  • 6Gb memory
  • 4Gb (will be 8Gb now) memory stick
  • Hard Drive configurations :
    • 2x 1Tb SSHD (they have 8Gb SSD on the Disk)
    • 3x 500Gb 7200 rpm SATA
    • 2x 250Gb 7200 rpm SATA
    • 1x 2Tb 7200 RPM SATA
  • Each Micro Server only have 1 onboard Nic that I use
  • No Jumbo Frames ans the inboard nic do not support it

Deep Security & vROPS : Importing the DSM certificate into vROPS

This is an overview on how to import the Deep Security Certificate into vROPS. This is needed to ensure the Deep Security Adapter will work correctly. Note that this process must be done before the DSM Adapter is installed.

The process involves the following :

  • Copy the Import-Cert script to the Analytic’s VM
  • Copy the DSM Certificate to the the Analytic’s VM
  • Run the Import Script to import the Certificate into the Analytic’s VM keystore file

Copy the Import Script and DSM Certificate to Analytic’s VM

Using WinSCP, login at the root user copy the following two files to the /root folder of the Analytic’s VM:

Login to the console of the Analytic’s VM (via VM Console or Putty) as the root user. Import the DSM Certificate as follow:

  • from the /root folder
  • chmod +x import-cert
  • ./import-cert
  • When asked for the full path, enter as follow:Ā /root/dsmcert.cer Ā (Where dsmcert.cer is the name of the DSMĀ Certificate)

The import should complete success fully. Once this process is complete the DSM vCOPS Adapter can be installed.

Deep Security and vROPS Adaptor : Some Use Cases

I tweeted about a vROPS Adapter that is available. To my knowledge, Trend Micro is the first Security vendor to release an adapter for collecting Security Events into vROPS. I have the Adapter installed in my lab and thought to give some use cases on how I was using the Adapter to display security events. First just a quick overview of the adapter.


The Adapter is supported on Deep Security version 9.0 SP1 and 9.5. You also need vROPS 5.8.x. On the Deep Security side you can have integration with vCenter and thus have the Adapter pull in all the Virtual Machines in your Environment and also Physical. Stats about Security Events is collected every 5min. The Adapter collect stats on the following modules :

  • Anti-Malware
  • Web Reputation
  • Firewall
  • IPS
  • Log Inspection
  • Integrity Monitoring
  • Total Event Count

Use Case 1 : Performance Impact of Security Events

The first use case is to look at the impact on a VM performance if there are a lot of security events. Security events can be Anti-Malware inside the VM, Web Reputation…or any of the above modules. In my use case I have a SQL server that was working at about 10%. As seen below there was an initial spike in CPU Demand but this was not due to Security Events. 3 Hours later there was another spike and this can be clearly seed due to Security Events (Firewall Events).


Use Case 2 : Heat Map to display which Computers have Security EventsĀ 

If you have a large VDI Deployment..say 1000-2000 VM’s and you want to quicly see which VM’s have Malware on them, or even any security events collected by Deep Security Manager, the Heat Map is for you. For this Heat Map I set the configuration to display all my VM’s in my lab and show Total Events with Range 1-13. Thus 1= Green and 13=Red. As you can see there is a “red” VM and some “Brown” ones.


If I click on the “Red” VM I am then shown a Metric Graph for each of the modules(Using Interactions in the Dashboard). Thus a Metric Graph for Total Events, Malware, Web Rep, ect. From here I can see which Security Module that is being exploited.


Use Case 3 : Ā TopN of all VM’s and Security Events

This is a very useful dashboard. It will show you the TopN VM’s for each Security Module. thus quickly seeing for each Module the top Exploited VM.


Use Case 4 : Creating Application Groups for Security EventsĀ 

How about only looking at your DMZ VM’s for Security Events, or any application. vROPS have the ability to create custom application groups and tiers. I create a custom Application called HR Application with 4 tiers:

  • HR Workstations
  • HR Database Servers
  • HR Application Servers
  • HR Mail Servers

Below the Heat Map that I have created showing only this application and all Security Related Events:


As we can see there is some workstations that have reached the threshold and is showing Red.


Trend Micro Deep Security was the first products to integrate Agent Less protection into ESXi. Trend Micro is also the first Vendor to use an Adapter for vROPS to collect Security Related Events into vROPS and be able to overlay Security Events over VM Performance and the ability to create meaningful vROPS Dashboards to display Security related information. I think a bit of creativity and this Adaptor can be really good use in any environment that is using VMWare and Deep Security.

Deep Security : Exporting the current SSL Certificate

I had to scratch around to find the way to export the SSL key for Deep security Manager. The trick is to find the password for the keystore file that is generated during the Deep Security installation. The rest is then pretty easy. Here is the way I did this :

Obtaining Keystore Password

On the DSM Server, navigate to the following directory :

  • c:\Program Files\Trend Micro\Deep Security Manager\installfiles\

In this folder find the following file :

  • genkey.bat

Open the file and look for the “-storepass” parameter. The syntax next to the -storepass is the keystore password. Note this password down.Keystorepass

Exporting the DSM Certificate

The Keystore file is located in the following directory :

  • c:\Program Files\Trend Micro\Deep Security Manager

The Keystore file name is :

  • .keystore

The keytool.exe file is located in the following folder :

  • c:\Program Files\Trend Micro\Deep Security Manager\jre\bin\

To export the current SSL certificate execute the following command from the keystore file location:

  • jre\bin\keytool.exe -export -alias tomcat -keystore .keystore -file dsm.crt

When prompted enter the keystore password.

Once competed you will have the Deep Security Manager Certificate.DSMCRT



Exchange 2013 on VMware : Some Considerations

I was busy with a Exchange 2013 Design that included a DAG. My initial setup I did on my home desktop using VMware Workstation. The setup was as follow :

  • 1x AD Server – Windows 2012R2 Server
  • 2x Desktops – Windows 8.1
  • 1x PKI Server
  • 2x Exchange 2013 Servers

I had to take this and do a design taking into account the client already have a Cluster and this DAG VM’s would be added. Here is some of the Design Considerations I took into account:

  • Virtual Machine Configuration
    • Used VMXNET 3 Adapter
    • Used “Virtual Socket” to allocate vCPU’s and not “Cores per Socket”
    • Used Memory reservations
      • Note the impact to HA when using Reservations…design your HA configuration around/for this
    • Note NUMA configurations. If you have to gave the VM 10 vCPU and you only have 6 pCPU’s per socket(2 sockets =12 pCPU’s) you the VM will not be NUMA optimized.
      • For this I had a look at the CPU Contention on the Cluster and it was low
    • Created multiple vDisk as follow:
      • OS drive (Exchange was also installed here)
      • Database Drive – DB 1 from Server 1
      • Database Drive – DB 2 From Server 2 (basically the remote server DAG Replicated DB would be on this drive for both servers)
  • ESXi Host Configuration
    • This is a tricky one as the “Best way ” to configure the ESXi CPU’s is to disable Hyper-Threading. Now if you have a Cluster with 10 hosts and you will only have 2 VM’s that need Hyper-Treading disabled…makes no sense to disable the whole Cluster. Thus consider the “Resource -> Advance CPU -> HT Sharing -> None” option in the VM configuration. Make sure the number of vCPU’s is not more than the number of pCPU’s on the processor(NUMA comes to play here)
  • Networking
    • I used an additional vLan for the Replication network and had no routing for this vLan
    • One have to evaluate the Replication traffic needed, taking into account the ESXi host network card speeds, number of network cards in the server and load balancing that might be needed on the pNic’s. I wanted to ensure the replication traffic had enough bandwidth without impact to other VM traffic.
      • The best option here is to have a VDS switch and ensure that “Load Balance on pNic” is enabled (in my case the client DID NOT have a VDS…)
      • What I did not want is to have the Replication traffic “flat line” the pNic on the vSwitch. Thus I created a Port Group and enabled Traffic Shaping on the Port Group. Limit the Traffic on this Replication Port Group to 750Mb (The servers had 1Gb Nic’s). Thus there “should” always be bandwidth available on a pNic for other VM traffic. If you have vCOPS in the environment you can always evaluate this setting and adjust as need later.
      • I also had a look at the pNic usage on the servers and they were in the low 50-100Mb usage at the time.
    • Don’t be fooled by the VMXNET 3 in guest indication that the speed if 10Gb…if you have 1Gb nic’s in the server the in guest will still state 10Gb. The two have nothing to do with each other. Thus your speed between ESXi hosts will be at 1Gb and not 10Gb.
  • Storage
    • All the documents that one read states the huge IO improvement in Exchange 2013. But you still need to make sure you will have enough IO. Also in my case I already had a storage unit…so have to make do with that.
    • I placed the Database vDisks on different Raid Groups(not just Lun’s…ensured they also on diff Raid Groups)
    • The Hosts already have multi pathing enabled
  • Cluster Settings
    • DRS
      • Created DRS Rules to “Keep the VM’s Appart” that was part of the DAG group and the Witness Server. Thus there is 3 Servers that is part of the rule.
    • HA
      • Disabled Guested Monitoring for the DAG VM’s
      • I disabled HA for the DAG Servers. I did not want the Server to auto start in case of the a Host Failure.
        • Since we have a DAG the DB would fail over to the other Exchange server.
        • If there is any issues on the “failed” VM when starting up we did not want to to have any impact on the Exchange Servers DB’s.
        • We added the following process:
          • After Host failure ensure that all DB’s are mounted on the remaining DAG member
          • Ensure users hare connected to new DB on reaming DAG Server
          • Make sure backups were successful on remaining Dag Server
          • Power up Failed DAG Server
          • Make sure Replication is working
          • Activate the DB on its original Server
  • DRP and Backup/Restore
    • Day to day backups was already in place
    • Daily Backups was being replicated of site
    • Point here is to ensure that this topic is not left out of the design

I suppose there is many ways to skin a cat. This was the way I did it for this client given the infrastructure that I had.

  • Can this be done differently – yes…as long as you have reasons for your decisions
  • Explain in your design what other options you looked at (like using in guest iSCSI perhaps)
  • State where you got the information from
  • A “Best Practice Guide” is only give me guidelines for my design. I need to design for the client what is their Best Practice to implement this solution using external resource that would still validate the design but within the clients guidelines/framework/limits that I was given (in my case: existing storage, vSwitch’s..ect)

Here is some of the documents that I used for my design:

VMtools : vShield Driver renamed to Guest Introspection Driver

When ESXi 5.5 Update 2 was released I read in the release notes that the vShield Driver in VMtools was renamed. During an installation I did this week I thought to take a screenshot of the new driver. The new name is called: Guest Introspection Drivers. As per the release notes it is just Ā a name change, noting more. ReadĀ here about this change in the release notes. Below is a screenshot of the VMtools with the new driver name:


Deep Security : Supported Agent Features by Platform

I was busy downloading Deep Security 9.5 for a project I am working on when I noticed a new document with the Title : Supported Features by Platform. Must say that this document was really good to read!

So what is this about ? Well there is a few types of Agents:

  • Windows Agents
  • Windows with Agent Less (Using NSX or vShield Manager on ESXi)
  • Linux Agents
  • Linux with Agent Less

Then there is the Features of the Deep Security Agent that is supported on each of these. For Anti-Malware here is the list:

  • File Scan
  • Registry Scan
  • Memory Scan
  • Smart Scan
  • Real Time Scan

Now with Agent Less protection not all of the above is possible. (Note that you can run Agent Protection with Agent Less Protection called Coordinated Approach)

Here is the Deep Security 9.5 Document that gives a great overview of what is supported by which Agent.

VM Snapshots : Migration vs Clone

I was busy consolidating some standalone ESXi hosts today. Basicly adding them to my new vCenter and then Migrating the VM’s (Powered off) to my new Cluster and Storage. Easy stuff. I noticed the the one VM had about 13 Snapshots (yea..its the developers VM…). I thought well it will consolidate the snapshots anyway on the target. Well after the migration all the snapshots was still there. This make me think about the difference in Cloning and Migrating a VM with Snapshots.

In my lab I tested this. I created a VM with some Snapshots:


SoI did some tests with this VM.

After a Clone of the VM all the Snapshots were consolidated and thus no more snapshots present.

After a Migration of the VM all the snapshots was still present.

I am sure I knew this at some point far far back but always good to just do a test and then you know the correct expected result.

VM Security Best Practices – List of all the settings from the Security Guide

I am busy with a internal project where I needed to create some VM’s that comply to the VMware Best Practice Guide. I created a VM and then added all the below setting to the VMX file. This post is just for anyone that needs to copy all the settings instead of typing them out šŸ™‚ = “true” = “true” = “true” = “false” = “true” = “true” = “true” = “true”
isolation.monitor.control.disable = “true” = “true”
isolation.bios.bbs.disable = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true” = “true”
RemoteDisplay.maxConnections = “1”
log.keepOld = “10”
log.rotateSize = “100000”
tools.setInfo.sizeLimit = “1048576”
isolation.device.connectable.disable = “true”
isolation.device.edit.disable = “true”
tools.guestlib.enableHostInfo = “false”
RemoteDisplay.vnc.enabled = “false”