vC Ops : How to use a Heat Map and create a new Dashboard

Heat Maps are one the most cool wigets to use. One can use it to quickly display “hot spots” in your Virtual or Physical Environment( I state Physical as vC Ops can be used to monitor Physical devices also by using Hyperic…more on this later).

As an example if you want to know the Luns in a cluster that has the highest Read time, hosts with the highest CPU Ready Time, vNic’s that has workload over a certain percentage…the list is long. For the post I want to explain how to configure a Heat Map. The first time I had to configure a Heat Map it took me a while to understand the thresholds and settings that can be used. So here goes.

First thing I do is to create a new Dashboard call Heat Map Test and drag a Heat Map widget on to the Dashboard. See below screenshots.

1. Create a new Dashboard as follow. Click on the small “+” icon on the end of the Dashboard.

2. From here you can either drag a Template Dashboard on to create a new Dashboard. Click on the small icon next to the “Manage Templates” to create a new Dashboard.

3.  Now we have a new Dashboard…give it a name, adjust the layout, select the number of column’s ( wide slection of 1-3, but change it so that u can see how the Layout on the Right hand side change and then use your mouse to ajust the width of the column’s), “Mark as Default” will set this Heat Map as the defaul Heatmap when you login to vC Ops and then Drap and Drop the Heat Map Widget onto the Right. Then click OK. Screenshot below. (Click on the below image to see a larger version)

4. Now we have a New Dashboard called Heat Map Demo with one Heat Map Widget on it. Just some note’s on how to ajust the Widget. Top right hand side is all the Widget Control’s.

  • Single Arrow : First one on the left. Minimise or Maximise the Widget
  • Double Arrow Down : Make’s the Widget larger Vertical (Horizontal was set  when we edit/create the Dashboard on the “Drag to Change Layout”…top Right)
  • Double Arrow Up : Make’s the Widget Smaller Vertical
  • Little Gear Icon : This is the Icon we will use to configure the Widget
  • X Icon : Deletes the Widget from the Dashboard

5. Click on the “Little Gear Icon” to configure the Widget. We will be creating a simple Heat Map showing All Datastores in our environment with high Read and Write Latency. Configure the Heat Map as follow :

  • Widget Title : Datastore Stats
  • Refresh Widget Content : On
  • Widget Refresh Interval : 300 sec (5 min…no use you make this less as we only collect stats every 5 min.)
  • Group By : For now we will leave this as Not Selected (Later more on this)
  • Then By : Leave on Not Selected
  • Mode : Instance
  • Resource Kinds : Select Datastore
  • Attribute Kinds : Select Datastore -> Read Latency (ms)
  • Select Widget tage to filter : Dataceter -> “Your Datacenter” (NOTE on this…if you have selected to many or you dont get results…there is a small “x” Icon….click this and all your sellections will be reset. Then select again..I use that icon a lot!)
  • Top Right : Configuration : Click on the small “Green +” to add a configuration. Call this Datastore Read Latency. (We will add another config called Datastore Write Latency)
  • Colors : Green : 0 (Min Value), Red 25 (Max Value)(Adjust to your needs…this is the high and low settings)
  • Now Click on the Blue Floppy Icon to safe this configuration. (Dont click on Ok yet)

Should look like this(Click on the images to see a larger version) :

6. Now we need to create a New Configuration of this Heat Map with Write Latency. Only thing to do is to Click on the “Green +” icon and add a new Configuration called : “Datastore Write Latency”. Then select the Attribute Kind called Wite Latency (ms). Then save the Configuration. Now click on OK.

7. The completed Wiget looks as follow:

Note the Drop down box have two selections based on the configurations. You can click on any of the Datastore blocks and then on the small  Icon indicated by the Green Arrow. This will take you to a detailed view of the Datastore for trouble shooting.

vC Ops : Using Tags to group objects

Tags are useful when you need to group certain object together. Lets use an example. U have a client that want to see only certain datastores stats. Thus we can create a Tag and assign a tag value and associate those individual datastores to the Tag. When we create a dashboard we can use the tags then to display only these datastore stats.

Lets look at how to create a tag and then use it. For our example we will create a Tag and assign two datastores to the Tag. Then we will create a heatmap showing free space availible on the Tag.

First we need to create a Tag Group. This is done as follow:

  1. Click on Environment -> Environment Overview
  2. Then click on the small “Spanner” icon. See image below. This will open the “Manage Resource Tags” screen
  3. Then click on the Tag Green plus icon and add a Tag called ” Storage Tags”
  4. Next click on the Green Plus in the Tags Value and create a Tag Value named “ClientX Datastores
  5. Click Ok

Once done you will now see a “Storage Tags” option in the list of resource on the left hand side of the Environment Overview screen. Expand this and you will see the “ClientX Datastores” Tag Value. Now we need to Drag and Drop the two datastores onto this “ClientX Datastores” Tag Value. On the top tight hand side there is a search bar. I have entered the first two letter of the Datastores names and a list of those datastores is displayed. Next is to Drag and Drop the two Datastores onto the Tag Value.

Once this is done it the Tag Value should have a number “(2) ” next to it. Now click on the “ClientX Datastores”. The two Datastores is now displayed on the Right Hand side plane. U can also edit the Tag Value by clicking on the Tag Value and then on the top click on the “Edit” Icon. U can change the DT / Super Metric Package / Attribute Package / and more. See below image.

Next is to  create a new Dashboard and add a Heatmap Widget. Below is the configuration for the Heatmap. Arrows indicate the settings I have changed/selected.

The completed Dashboard looks like this:

This is just one use case for useing Tags, but gives you the idea on how to create them.

vC Ops : How Hyperthreading is handle in CPU Stats

I was creating a CPU Dashboard in my lab and noted that my stats on CPU Core Utilization % was not showing “correctly”. Below is a screenshot of my lab enviroment with the 3 hosts.

So lets explain the colors first. My sliding scale is set to 0 being Green and 20 being Red. Brown is about 9-11. As you can see most of my pCPU’s are Brown (9-11%). For Each Host we see a Red block in the bottom right hand side corner. This is the avg CPU Utlization as a %…..and it is above 20% but the rest of my Cores is at 9-11% … how is this possible ?

Well simple…Hypertreading. vC Ops keep track of the Core with its Hypertreaded “Partner” and adds these two metric values up to give the Full core Utilization a value. Thus if I add these two values up I will get a 20% value for the Core and its Hypertreaded partner.

The same for other CPU Metric’s will apply. Keep this in mind when looking at Heatmap stats.

vC Ops : vSphere Replication Dashboard

One of the most asked question that I get is around the bandwidth requirements that is needed for vSphere Replication. If you did not know with the new vSphere Kits that was announced, one of the new futures that was included is vSpere Replication in all the ESXi editions from Standard and up including the Essentials Plus Kit.

Anyway…back to the the bandwidth. While this question depends on a few factors, what we can do is to build a dashboard that can show the stats. What I created was a dashboard in vC Ops Custom UI that shows 2 Sites that is replicating to each other…thus an Active/Active Site. While I am using only vSphere Replication without SRM, this will also work with SRM.

The completed Dashboard looks like this :

The Dashboard have been configured as follow:

  1. Left and Righ hand side I have a Scoreboard with total VM count that is being replicated, Total Replicaed Data in Kbps, Tx and Rx Replication data in Kpbs. Each side represents a Site (In my case Site A and B). I used a few Super Metric’s to create the sum of the VM’s and Total Replation Data.
  2. In the Middle I have a Total of Site A & B. This Scorebord indicates the Total VM’s being replicated and also the Total Kbps being used between both sites by vSphere Replication. This is a good indication of the total Bandwidth needed and used for replication.
  3. At the bottom of each scoreboard I have a Graph representing that Total Kbps per site (Left and Right) and the Total for all the Sites in the middle. This is also done with a super metric.

vC Ops : Heatmap for Power ON/OFF VM’s using tags

Based on an internal discussion around VM sending alerts when powered off, I though of making a Custom Heatmap showing Red/Green if a VM is powered Off/On. The use case I was thinking off was for a Management Cluster where you want to see if you have your Management VM Powered On (Yes I know if they go down you will know immediately as your services will not be availible, but this is a simple use case). Using Tags and a Heatmap I created a simple dashboard showing Green for VM’s that is Powered On and Red if they are Powered Off. You can also send an Email if the state of the VM changes but you might just get a few emails.

Firstly I created the Tags. This is done as follow:

  1. Click on Environment -> Environment Overview
  2. Then click on the little “spanner” icon (it’s the Tags icon). This will open the “Manage Resource Tags” window.
  3. Click on the Green + icon to add a Tag called “Power State”.
  4. Click on the Green + icon to add a Tag Value called “Managment VM’s”.
  5. Click on “OK”.
  6. Screenshots below.

Next is to add the Management VM’s from the List of Objects. You can search for them also. You need to drag and drop the VM’s onto the Tag Value.  Tag Value is by default 0 and as soon as objects is dropped onto the Tag Value it will increment. I added the following VM’s into the Tag Value:

  1. Virtual Center
  2. vCloud Director Servers
  3. vCNS Manager
  4. SQL Server
  5. DynamicOps (This will be my VM that is powered Off)

Next we create a New Dashboard and add a Heatmap to the Dashboard. Configuration of the Heatmap as follow:

  1. Refresh Wiget Content = On
  2. Widget Refresh Interval = 300 (Seconds)
  3. Group by = Virtual Machines
  4. Resource Kinds = Virtual Machines
  5. Colors : click on the small blocks (Indicated with the black arrows) and change the colors like in the screenshot(0 means VM is powered Off thus we need to make it Red, 1 means On and need to be Green).
  6. Set Min = 0 and Max =1
  7. Atrribute Kind = Select: System -> Powered ON
  8. Select Tags to Filter = This is where you select the new Tag, Power State, and Tag Value, Management VM’s.
  9. Configuration = Click on the “+” icon to add a configuration called “MGMT Power”
  10. Click Ok. Screenshots below.

Your completed Widget should look like this. In my example I have one VM powered Off and is shown in Red here.

vC Ops : Planning & Reports > Summary, View & Reports Explained

I was doing a vC OPs training session with a VMware Partner today and one of the questions that came up was to explain the Summary, View and Reports tabs. What happens when those values are change? What is the impact? And where can we see the differance in the UI when we change those settings. Here is the explanations on what is effected when the values is changed.

The Summary, View and Report tab changes how we see the time line in the UI display and reports.The screenshot below show the location of the Summary, Views and Reports configuration screen.


By changing the summary view time line (Daily, Weekly, Monthly, Quarterly, Yearly) we can change the timeline of the displayed report in the UI.Lets look at how by changing this we can manipulate the displayed reports. If we click on the Planning tab, then Summary we can see the report is in a weekly format. The screenshot  shows this above the Purple line.

When we change the Summary view to a Daily view the UI refreshes and displays the report using a Daily trend and not Weekly. The screenshot below shows this new trend line above the purple line.

View – Trend View

Changing the Trend view we can adjust the Planning Views. For our example we have used the following values:

Default Interval =   Daily      : This will display the current view in a daily time line                Default Time Window =  8   : This will display 8 Intervals. In our view it will be 8 days as we have set Default Interval to Daily. If Default Interval was set to Weekly we would have seen 8 weeklies as the time line.                                                                                                 Forecast Horizon =          4    : This will show 4 Days of forecasting (Based on the       Default Interval set to Daily).

Below a screenshot of the settings as they have been set as above example (Daily,8,4).

View – Non-Trend View 

Non-Trend view includes all List and Summaries that combines all values into a single value. Thus if we look at Planning View where the Type is Summary, the values in those views is defined as a Non-Trend View.  By changing the values of the “Interval to Use” and “Number of Intervals” we ajust the number of datapoint to use. Below is a screenshot of an example. When changing the above settings the value of “Total Capacity(VMs)” should change. Note that the more datapoints that is used the more accurate the summary value.(weekly with 10 weeks =  840 datapoints vs Montly with 5 months = 3600 datapoints)

Distrubution Buckets

When u set the Bucket you get more/less distribution point in your graphs. This is useful to see more granular distribution of resource utilization. As an example you have 32 hosts in a cluster and you want to see down to 5% the CPU utilization of the hosts. Setting the Bucket vale to 20 will show this CPU to host utilization in more granular detail. Max value for this is 20. Below is a screenshot of an example with Buckets set to 20.


These settings affects the reports time line and intervals that is generated.

Report Period : This affects the report time line. Thus if a Monthly is selected the report will run over one month.

Interval Start : Set the offset interval based on the Report Period. Thus if it is October and we set this value to 2, the report will show details for August.

Report Intervals :  This defines the data points that will be shown. If set to Daily and Report Period is set to Monthly we will see a daily interval on the report. Note that Report Interval cannot be greater that Report Period time interval.

Below a screenshot of using a the Virtual Machine Capacity Overview Report with a Report Period of one Month, no Interval offset and Report Interval of Daily.

vC Ops : Oversized VM’s Explained

I have done a few POC’s on vC Ops and the one question everyone asked is : How does the calculation work to determine an oversized VM. I suppose this is much easier to explain on a whiteboard but lets give this a go on a blog.

Once you have logged onto the vSphere UI (https://ui-ipaddress), click on “Configuration” and then on “Oversized & Undersized VM’s” as below screen shot

First thing is to know that there is two calculations that is done before a VM is considered Oversized. First is to see if the VM will violate the Oversize settings. If that is true than the next check is so see if the VM will violate the Undersized settings. If that is false then the VM will be listed as Oversize.

Now the settings. Lets start with the easy ones. “CPU/Memory demand less than” is the percentage that defines the upper limit(Below graph it is “U”). Any values below this percentage value will be used to determine is the VM is oversized. With this there is the option for “Any” or “Both” of the thresholds. Thus is “Both” is selected both the CPU and Memory MUST be oversized (explained below). If “Any” is selected then any of the 2 metric’s can be oversized  for the VM to be considered for oversized.

Next is the “% Oversized Threshold”. This is the tricky one. It does NOT define the TIME that the value is under the Threshold but rather the % of area covered of the graph that is under the Threshold (Threshold being the CPU/Memory thresholds, “U”). Looking at the graph below, the red area is what is used. If this area value (as a percentage of area) is more than the “% Oversized Threshold” value, the VM can be considered as oversized (remember the second check must still be done..Undersized).

Now we need a time line defined (“T” in the above graph). This is set under : Summary, View & Report. Then set the Report Period to Weekly(if you want a weekly Time Interval). See below graph for the location.

The last check is to see it the VM will violate Undersized Thresholds. Basically the same calculations but using the Undersized Values for the calculations. If the VM do not violate the Undersized thresholds the VM will be consider Oversized and displayed in the report Oversized Virtual Machines.


vCNS : Configuring SSL VPN-Plus

One of the new futures of vCNS(vCloud Networking and Security) is SSL VPN-Plus. This allow clients to setup VPN access thru an Edge Firewall to networks behind the Edge device. Below is how I did my first installation and configuration in my Test environment.

I have already configured my Edge Firewall with External IP’s that is internet facing. From here you need to login to the vShield Manager, click on you Datacenter Object, then on Network Vitualization, then on Edges and double click on your Edge device. From here click on VPN. Screenshot below:

Next steps is as follow:

  1. Click on Server Settings. The IP Address is the list of IP’s that have been configured on the Edge on the “Internet” interface. Select the IP that u want to use for the client to connect. Port 443 is the default. Click on OK. (you can configure Cipher List and Certificate’s if you want to)
  2. Next is the IP Pool. This is the IP address that the clients will get when they login via the VPN. I used an IP range of as this is not in used on my network. The Gateway I have set to You can also set the DNS and DNS Suffix. Ensure the Status is set to Enabled. (We will setup a firewall rule later to allow this range access to you internal network.)
  3. Authentication. I used Local with the default setting. (You can configure AD/LDAP/Radius/RSA-ACE also)
  4. Installation Package. This is used to create and Installation package that the client can use to install the VPN client onto their PC. Windows/Linux and Mac is supported. The Profile name is the name that the client will see, so make it descriptive. Select Linux and Max is you want to enable installation to those OS’s. Ensure the Status is set to Enabled. Set the Installation Parameters also. For my test environment I only set to “Create a Desktop icon”.
  5. Users. This is where you can create Local users for VPN access. (Based on point 3 as we selected Local Authentication). I created one test account called VPNTest. Ensure the Status is set to Enabled.
  6. Client Configuration I set Tunneling mode to Full. I used my Edge Internal GW’s IP address for the “Default gateway”. (“In split tunnel mode, only the VPN flows through the vShield Edge gateway. In full tunnel, the vShield Edge gateway becomes the remote user’s default gateway and all traffic (VPN, local, and internet) flows through this gateway.”)
  7. Portal Customization. Used to change the look and feel of the VPN web page.
  8. Then go to the top and click on the Dashboard and set the service to Enabled.
  9. Next is to create a Firewall Rule to allow any source(Or if you know from where you clients will connect you can enter a range) to connect to the IP Address (Point 1 IP address) on port 443 and Allow this. (Make sure to Publish this rule)
  10. We also have to create a Firewall rule to allow Pool) to (This is my Internal Network range).

Now that we have done all this we can connect to the external IP from Point 1: https://ip.address.vpn. Make sure it is httpS.

From here you can use the User account that u created in Point 5 to login. Once login in you can click on the Installation Package that you created in Point 4 and install the client. When the Installation is done you will have an Icon on the desktop that you can use to connect to the VPN.

vC OpS : How often is Dynamic Threshold calculated

A few clients have asked me how often does vC OPS calculate Dynamic Threshold. For those who don’t know, Dynamic Threshold(DT) is the way in which vC OPS calculate what is normal for a metric. Metric can be any of the Performance stats that is collected form vCenter or any other installed adapter (ie Hyperic) excluding system attributes.

The example I use when explaining DT is backups is to take a cluster that starts it backups in the morning at 3am. For that time it does the backups CPU might be running above 80%. If you have conventional monitoring applications u might have set a hard threshold that will send u a alert when CPU is above 80%. Thus every night u will get a 3am alert that the cluster CPU is above 80%. Yes you can set a time line not to monitor…but all manual.

Now take vC Ops and DT. vC OPS will “learn” that the normal behavior for the CPU metric is to spike at 3am…so if that is normal why send an alert at 80% ? However vC OPS will send u an alert if the Cluster CPU is at 30% as this not normal. Thus the backups have not run. Consider that the conventional monitoring application would not have send u an alert.

Back to when DT is calculated. This process happens on the Analytic’s VM every day at 1am.

If for some reason you need this to run now you can “force” a run. This can be done as follow:

  • Logon to Custom UI (http://ui-vm_ip/vcops-custom) as an Administrator
  • Then click on Admin -> Support
  • On the Dashboard menu click on Info
  • Click on the small icon as indicated below