VMware vExpert 2016: NetApp Honorees

Last Friday VMware released the official list of the honorees for the VMware vExpert 2016 program. I’m proud to have been chosen for this award for the third year, and even prouder to see how many other NetApp employees, including our new Solidfire brethren, and “extended family” are on the list:

  • Chris Gebhardt (@chrisgeb), vTME and Dr. Desktop, Lord of EUC at NetApp
  • Henry Vail, Senior Architect for Converged Infrastructures at NetApp
  • Joel Kaufman (@thejoelk), TME Director for manageability at NetApp
  • Kyle Murley (@kylemurley), Systems Engineer for Solidfire at NetApp
  • Melissa Palmer (@vmiss33 and vmiss.net), TME for Converged Infrastructures at NetApp
  • Shawn Lieu (@ShawnLieu), Solutions Architect at Veeam and NetApp A-Team member

If there’s anyone that I’ve missed in the above list, please let me know and I’ll be happy to update & make sure that you’re included.


Tech Smorgasbord #5

An on-going reference series for interesting technology or projects which deserve further investigation, or for technical documentation (of one media format or another) that looks to be especially good reference material.

Free tech ebooks

Let’s start with something everybody loves – freebies! The New Stack has launched a new series of books on Docker and they’re giving them away. The first book is out now with four more books planned to be released over the next six months:

  1. Book 1: The Docker & Container Ecosystem
  2. Book 2: Applications & Microservices with Docker & Containers (coming in January)
  3. Book 3: Automation & Orchestration with Docker & Containers (coming in March)
  4. Book 4: Networking, Security & Storage with Docker & Containers (coming in May)
  5. Book 5: Monitoring & Management with Docker & Containers (coming in June)


SDN under Ravello

Ravello Systems has some truly great tech enabling nested virtualization in the cloud, and many people have jumped on the bandwagon of running some – or in some cases all – of their home labs using Ravello rather than on their own equipment. It helps, of course, that Ravello have a very active presence in the VMware and OpenStack communities, provide free trials of their product, and even offer free accounts to VMware vExperts. Thanks to this, we’ve seen an explosion of blogs detailing how to run various software using Ravello’s Smart Labs – even software defined networking (SDN) technology.


Thomas Beaumont (@tleej) has a great series on running VMware’s NSX under Ravello – which lead to him being chosen as one of the three winners in Ravello’s recent blog writing contest.




Cumulus Networks

If you’d rather play with Cumulus Linux instead, Christian Elsen (@ChristianElsen) has you covered with a great post on getting it working with Ravello:


Network automation

Speaking of networking, O’Reilly has just published an Early Release edition of the upcoming Network Programmabiility and Automation book by Jason Edelman (@jedelman8), Scott Lowe (@scott_lowe), and Matt Oswalt (@Mierdin). With this authorial lineup the book is practically guaranteed to be a must-read for those inclined towards either networking or automation.

In the meantime, you can check out a couple recent blog posts by Jason on the same subject:

OpenConfig, Data Models, and APIs

Network Automation with Ansible – Dynamically Configuring Interface Descriptions

Clustering with Red Hat Enterprise Linux 7

UnixArena (@UnixArena) has a highly detailed 8-part (so far, at least) series covering clustering under RHEL7 with Pacemaker. Pacemaker is one of the critical software components providing cluster high availability for both RHEL and OpenStack.

  1. http://www.unixarena.com/2015/12/compare-redhat-cluster-releases-rhel-7-ha-vs-rhel-6-ha.html
  2. http://www.unixarena.com/2015/12/rhel-7-redhat-cluster-with-pacemaker-overview.html
  3. http://www.unixarena.com/2015/12/rhel-7-installing-redhat-cluster-software-corosync-pacemaker.html
  4. http://www.unixarena.com/2015/12/rhel-7-configuring-pacemaker-corosync-redhat-cluster-part-4.html
  5. http://www.unixarena.com/2015/12/rhel-7-pacemaker-cluster-resource-agents-overview.html
  6. http://www.unixarena.com/2015/12/rhel-7-pacemaker-cluster-resource-group-management.html
  7. http://www.unixarena.com/2015/12/rhel-7-pacemaker-configuring-ha-kvm-guest.html
  8. http://www.unixarena.com/2016/01/rhel-7-pacemaker-cluster-node-management.html

Mac OS X Hypervisor Framework

With the release of Mac OS 10.10 (Yosemite), Apple added an intriguing new feature to the operating system with very little fan fare. The release notes only offered this brief paragraph:

Hypervisor (Hypervisor.framework). The Hypervisor framework allows virtualization vendors to build virtualization solutions on top of OS X without needing to deploy third-party kernel extensions (KEXTs). Included is a lightweight hypervisor that enables virtualization of the host CPUs.

Since then, there hasn’t been a lot of further discussion on the topic, either – except for the fine folks at pagetable.com. First there was a fascinating article in January of last year on using the framework to run a DOS emulator (hvdos), and then in June came the announcement of xhyve, a port of FreeBSD’s bhyve hypervisor.

(Interesting aside: bhyve was initially developed and open-sourced by NetApp back in 2011, and you can find more information, including numerous conference presentations and recordings on the FreeBSD site.)

And now Veertu Labs has launched their new virtualization product for the Mac based on Apple’s hypervisor framework. Maish Saidel-Keesing (@maishk) has a good write up here:


I haven’t played with it yet myself, but I’m looking forwad to giving it a spin, while still keeping an eye on xhyve’s future.

All CLI all the time

If you’ve perused much of my prior posts, you’ll know that I enjoy using the CLI quite a bit – whether it’s for the operating system, an application, or an infrastructure device, textual interfaces just seem more fun and (usually) more efficient to me. Sadly, despite the UNIX power of Mac OS X, its rich CLI is often overlooked so it was a nice surprise to stumble across Herb Bischoff’s Awesome OS X Command Line. It’s by no means exhaustive, but there’s quite a few little tips, tricks, and hints captured of which I wasn’t previously aware.

I also came across a nice study guide for PowerCLI put together by Christophe Calvet which includes a good conceptual introduction and links to a number of additional resources for both PowerCLI and PowerShell.

Attack Methods for Gaining Domain Admin Rights in Active Directory

Earlier in my IT career I spent a large amount of time on the job dealing with security issues: physical security systems, firewalls, operating system hardening, corporate security policies, etc.  While it’s been a few years since I’ve  had any real security responsibilities, infosec remains an area of significant interest to me. This article by Sean Metcalf (@PyroTek3) is a nicely detailed examination of some of the common vulnerabilities in Microsoft’s Active Directory today and how to mitigate against them. Lots of references and backing sources provides a treasure trove of related reading.




Seasonal Learning Opportunities 2015

This a brief update to last year’s post on good deals for our continuing quest to Always Be Learning.

The following are simply in alphabetical order, and I’m sure only represent the tip of the iceberg of good deals. Please share any others you hear about in the comments.

Books and Videos


Apress are offering a Cyber Monday sale of $10 each for any of their ebooks ($20 each for any of the Spring ebooks) until 11:59pm tonight, November 30th. The  site normally offers an Apress ebook Deal of the Day as well as a Springer Daily Deal.


Cisco Press are having a Cyber Monday Sale with 55% off eligible items using code CM2015. This includes books, ebooks, video training, practice exams, and more. They also consistently offer eBook and Video Deals of the Week (from their home page).

Manning Publications

This year they’re again having a “Green Tuesday” sale until the end of November where all eBook purchases under $50 are 40% off (codegt112415acc) and purchases over $50 are 50% off (code gt112415bcc). They’re also having a “Countdown to 2016” sale during December with a different discount deal each day, a chance to win a free ebook each day, and a chance to win an Apple iPad Pro.

And of course they have a Deal of the Day selected from across their entire catalog (eBooks, physical/printed books, or “MEAPs” – Manning Early Access Program books similar to O’Reilly’s Rough Cuts where books are made available as chapters are completed).


O’Reilly have been one of the premier publishers of IT books for decades, and in recent years have of course added ebooks, videos, and other media to their output. While O’Reilly have Ebook Deals of the Day (usually two) and Video Deal of the Week, their biggest sales are at this time of year.

Their Cyber Monday Sale is running again this year (till December 1st 5am PST) and you can score 50% off any ebook or video, or 60% off when ordering $100 or more. And remember – O’Reilly sell/distribute books by other publishers as well including No Starch Press, Wrox, Wiley, Sybex, and many others.

Even better, O’Reilly are the force behind SafariOnline, the premier tech ebook/video subscription service which is having its own Cyber Monday sale today: 50% off the normal annual subscription price of $399! So for $199 you can get an all-you-can eat tech buffet – this is one of the best deals out there if you consume (or want to consume) a large amount of tech content. And yes – it includes offline (tablet/laptop/etc.) access!

Pearson IT Certification

Pearson are having the same Cyber Monday Sale as Cisco Press with 55% off any digital items. On their site you’ll find not only tech books from Cisco Press but also VMware Press,  and others, and of course video training, practice exams, and more.  If you miss this sale, they also consistently offer eBook and Video Deals of the Week (different from the ones on the Cisco Press site).


GNS3 Academy

You may know GNS3 as the best vendor-neutral networking simulator around, but did you realize they also offer training? This year they’re offering a Black Friday sale where all of their courses (normally priced between $19 – $49) are only $15 using coupon code BLACKFRIDAY. They also have several free courses to check out at any time.


If you’re ready to take the leap for your CCIE, check out INE‘s Cyber Monday Sale.  They’re offering 3 deals: 33% off their Routing & Switching Everything Bundle, 50% off rack rental tokens, and 20% off INE apparel.


The company offering the best catalog of tech video training – by some of the best, most knowledgeable instructors – are going to be offering a Cyber Monday sale, too. What is it? They still haven’t announced it, but given that a Pluralsight subscription (as low as $299/year)  is tied with SafariOnline for the best tech learning value per dollar, you’re going to want to check back to see what deals they’re offering!



Yes, this is a double listing, but only because Safari does include a number of video training classes in their inventory – and because at $199 a Safari subscription is an incredible deal!

Unrelated Good Deal


My favorite backup/synch service is offering their unlimited plan for only $149/year as a Cyber Monday deal (until December 1st). SpiderOak combines the granual backup of products like CrashPlan with the synch capabilities of Dropbox, but with a core focus on zero-knowledge security and privacy. If you value your data – and the privacy of that data – you should seriously think about using them.

Take advantage of everything that’s out there, and get prepared for annother year of learning and growth!

Kicking the Tires: VMware vCloud Air OnDemand

Before We Begin

Cloud Computing. The Cloud. Private Cloud. Public Cloud. Hybrid Cloud.Cloud apps. Cloud platforms. Cloud automation. Cloud bursting. Intercloud. Multicloud. Cloudcloudcloudcloudcloudcloudcloudcloud


Ok, enough of that, then.

Private and Public

VMware, as everyone know, is the 800 pound gorilla of infrastructure virtualization, of Infrastructure-as-a-Service, of private cloud. Despite this,  and despite VMware’s aggressive pursuit of being more than just the King of Virtualization, it has not historically been focused on, or a leader in, public cloud.

Clearly, Amazon Web Services is the 800 pound – nay, 800 ton – gorilla of public cloud.  Why? A number of reasons: early market entry, parent brand recognition, aggressive pricing, sustained investment,  and many others. But the biggest reasons are almost certainly the simplest: speed and ease. A relative novice can quickly setup an AWS account and within a few minutes have a virtual machine running on the Internet – all payment taken care of easily via credit card and all charges based on a simple utility usage model. Pay for what you use when you use it, and by how much of it you use. If you use it less, you pay less, and if you don’t use it at all then you don’t pay anything.

If you’re reading this, you’ve undoubtedly used AWS yourself and understand this well. If you haven’t actually gotten your hands wet, you should and you can – just go sign up for the AWS Free Tier for a year and see for yourself: http://aws.amazon.com/free/.

A Step In the Right Direction

VMware’s initial foray into public cloud with vCloud Hybrid Service, now renamed as vCloud Air, was fairly self-explanatory and focused on offering hybrid cloud services where a customer’s private cloud (already running on vSphere) could be expanded and extended into a cloud provider’s infrastructure (in this case VMware’s). Built on vSphere and vCloud Director, it was a powerful platform that leveraged a company’s existing familiarity and trust with VMware’s products to (relatively) easily work with off-prem infrastructure services. It just wasn’t really public cloud , nor were the acquisition & usage models the same: you couldn’t just go sign up with a credit card, and your billing would follow a subscription model for blocks of resources (like your cable TV bill) rather than only actual resources used (like your utility bill).

Now VMware is expanding to provide a true public cloud experience with the development of vCloud Air OnDemand: a pay-as-you-go cloud utility service aimed more squarely at AWS and other public cloud service providers. I was fortunate enough to be selected as an early access participant through the vCloud Air OnDemand Ambassador program, and got to play a bit with it.


Clouds in the Air

In a nutshell: it’s pretty darn good. It certainly looks a lot nicer, cleaner, and more professional than AWS, and it provides a much needed simplified UI in front of vCloud Director (though direct access to the vCD UI is also available throughout the vCloud Air UI for those more comfortable with it or who need to use some of the more advanced features and configurations).

The initial setup is straightforward: login, click on “Virtual Private Cloud OnDemand”, choose your first datacenter to place your workload, and begin provisioning virtual machines.

The wizard for creating your VMs is simple and straight forward, and provides detailed visibility into the costs associated with your configuration choices. You can look at either the per-hour or  the per-month cost for the VM, and tweak the setting to your heart’s – and bank account’s – content.

You can also create a VM “from scratch” outside of this simple wizard, which will bounce you out to the vCloud Directory UI instead where you can build new vApps as custom as you like, or import existing vApps and OVFs.

The on-going management dashboard is divided into Resource Usage (i.e. what you’re using & how much it’s costing you), Virtual Machines (the default tab you are taken to), Gateways, and Networks. The majority of your daily VM operations are easily done from here.


Quibbles and Nits

It’s the little things that trip you up: the untied shoelace, the toy in the wrong place, the step that you didn’t see. Despite the polished look of the vCloud Air interface, there were a number of things that proved annoying or made the product more difficult to work with than it needed to be:

  • Root passwords: If you use any of the pre-created operating systems in the catalog, the automation will set a new random root password (and provide that password to you in the interface) which you then need to change upon login. Seems reasonable, right? Except that every VM I created this way would never take the new password but would instead return to the login prompt without the change taking effect. Over and over. I tried this with different choices (CentOS 6.3 64-bit, CentOS 6.4 32-bit, Ubuntu Server 12.04 64-bit, Ubuntu Server 12.04 32-bit) across different datacenter locations (US Virginia 14, US California 13) with no difference in behavior. An easy workaround is to simply boot into single user mode, enter the  random password, and then manually change using the ‘passwd’ command. Easy, but annoying (and not an issue I’ve encountered with images from AWS, Digital Ocean, etc.).
  • Intermittently, the web UI would display incorrectly – usually either calculated fields wouldn’t update automatically or in some cases fields simply wouldn’t display, like this one:


  • Help and Support: Choosing the Help option from the upper-right menu takes you to the vCloud Air Documentation Center, which is very good. Choosing the Support Center option takes you to the vCloud Air Support Center – which looks good, but any searches are run against the entire VMware support site and not filtered by (or at least sorted for) vCloud Air. Worse, there’s no  option for vCloud Air in the product list on the left for the user to filter their own results.
  • Internet access:  I’ll go out on a limb & say that in the vast majority of cases, users will need a newly-created VM to be able to access the Internet (for OS updates or software installation if nothing else). For most public clouds, including AWS, the default state of a new VM is Internet-accessible, including inbound access. Yet here it’s not enabled by default, there’s nothing presented in the UI to configure it simply (it’s actually a three step process), and how to do so is semi-buried in the documentation where it is not really clearly described.

In the end, the above are just unnecessary friction for an otherwise slick and powerful product.


VMware has done a good job of moving into the utility cloud space with OnDemand. If you’re looking for a cloud service built on the most enterprise-class virtualization products available, run by the company that built those products, and which can allow seamless import/export between the cloud service and your existing private cloud – you want vCloud Air. If you want the power of vCloud Director yet with a simpler interface, you want vCloud Air. And if you want the speed and ease of AWS with the same technology you’re used to in your datacenter, you want vCloud Air OnDemand.

And like AWS, it’s easy to try – just go sign up for an account and get $300 in service credits for the first 90 days!



Tours of the Black Prompt: NetApp FAS Service Processors

The Tours of the Black Prompt series so far:

Over the course of this series, we’ve focused on the command line interface available for the operating systems that run on NetApp FAS storage array controllers: Data ONTAP 7-mode and clustered Data ONTAP. In this post, we’ll focus on a CLI that is not part of the operating system: the Service Processor shell.

Service Processor Shell

NetApp FAS array controllers have had built-in out-of-band management for many years. Depending on the series, older FAS models have used either baseboard management controllers (BMC) or remote LAN management (RLM) ports for this functionality. The newer FAS models, including the 2200, 3200, 6200, and 8000 series, all use a service processor (SP) for out-of-band management. BMCs, RLMs, and SPs offer similar base functionality, but SPs provide the most capabilities and features. The SP CLI behavior described below is the same regardless of whether the controller connected to the SP is running 7-mode or clustered Data ONTAP.

Commands and Privilege Levels

Logging in via SSH (telnet is not supported) you are provided a simple administrative-level prompt:


The prompt is very minimal and only indicates that you are connected to a Service Processor (the “SP” in the prompt) at the normal administrative privilege level (the “>” in the prompt). This is of course very similar to the Data ONTAP shell prompts but without the cluster or hostname being designated.

From here, you can see the available command structure by simply typing either “?” or help followed by [Enter] :

SP> ?
 date - print date and time
 exit - exit from the SP command line interface
 events - print system events and event information
 help - print command help
 priv - show and set user mode
 sp - commands to control the SP
 rsa - commands for Remote Support Agent
 system - commands to control the system
 version - print Service Processor version
SP> help
 date - print date and time
 exit - exit from the SP command line interface
 events - print system events and event information
 help - print command help
 priv - show and set user mode
 sp - commands to control the SP
 rsa - commands for Remote Support Agent
 system - commands to control the system
 version - print Service Processor version

As you can see, there are far fewer commands available for the SP than there are for either version of Data ONTAP. The SP CLI is limited to functionality necessary or useful for situations that require out-of-band access.

For the vast majority of times that an administrator will be connecting to the Service Processor, they will be using it for the most basic functionality: serial console access using the system console command.

SP> system console
 Type Ctrl-D to exit.
 SP-login: admin
 * This is a SP/RLM console session. Output from the *
 * serial console is also mirrored on this session.  *

Connecting to the system console does require a secondary authentication. While the built-in admin or root user (depending on the version of Data ONTAP) are allowed to login to the SP by default, it is possible for other users to be configured for access to the SP who may or may not be allowed console access to Data ONTAP.

At this point, the SP session will be able to see all output visible to the physical serial port, as well as being able to provide any input to it. Access via system console is not restricted or limited in any way; access and capabilities are only limited by the configuration of the user.

While the SP console session and the physical serial console session do display some of the same information, they still have separate and independent shell environments. If, while an SP session is connected to the system console, there is a concurrent connection to the physical serial port, any input or output from that console session would be mirrored to the SP session. The inverse, however, is not true: any input or output initiated from the SP session will not be visible to the physical console session.

Pressing Ctrl+d from the SP session will end the system console access and return the administrator to the SP CLI prompt.

cluster1::> SP>

The SP itself can also be accessed from the physical serial port by pressing Ctrl+g. This is useful where an administrator is using either a console/terminal server for centralized out-of-band management, or when connected directly to the console (such as during initial setup). The administrator can then return to the serial console by pressing Ctrl+d.


Switching console to Service Processor
Service Processor Login:


Just like Data ONTAP, there are two additional privilege levels available: advanced and diag. You can change to these levels using the priv set command.

SP> priv set advanced
 Warning: These advanced commands are potentially dangerous; use them only when directed to do so by support personnel.

The asterisk between the “SP” and “>” indicates that you are in either the advanced or diag privilege level.  There is unfortunately no visual distinction between these two levels, but you can run the priv command with no modifiers to display the current privilege level. This is again just like with Data ONTAP.

SP*> priv

More commands are available within the higher privilege levels than in the normal admin level, though they are not necessarily obvious from the top-level output.

SP*> ?
 date - print date and time
 exit - exit from the SP command line interface
 events - print system events and event information
 help - print command help
 priv - show and set user mode
 sp - commands to control the SP
 rsa - commands for Remote Support Agent
 system - commands to control the system
 version - print Service Processor version

There are several commands available in Advanced level that aren’t in the normal Admin level, with most being for the display of additional information:

  • sp log audit to display the command history of the SP
  • sp log debug to display the debug information of the SP
  • sp log messages to display the contents of the messages file for the SP
  • system battery auto_update status to display the current setting for the battery firmware automatic updates
  • system fru log show to display the history log related to FRU data

There are also several commands to modify or verify the SP configuration:

  • system battery auto_update [enable|disable] to configure the setting for the battery firmware automatic updates
  • system battery verify [URL] to compare the current battery firmware image with another image available at the specified URL
  • system nvram flash clear to erase the NVRAM flash content (only available when the system is powered on)
SP*> priv set diag
 Warning: These diagnostic commands are for use by support personnel only.
SP*> ?
 date - print date and time
 exit - exit from the SP command line interface
 events - print system events and event information
 gdb - commands to control GDB pass-through
 help - print command help
 priv - show and set user mode
 sp - commands to control the SP
 rsa - commands for Remote Support Agent
 system - commands to control the system
 version - print Service Processor version
 ping - send ICMP ECHO_REQUEST packets to network hosts
 ping6 - send ICMPv6 ECHO_REQUEST packets to network hosts
 traceroute - trace route to HOST
 nslookup - query the nameserver for the IP address of the given HOST optionally using a specified DNS server

The most useful commands at the diag privilege level may be the most basic for troubleshooting network connectivity:

  • ping and ping6
  • traceroute
  • nslookup

Command Syntax and Help

You can see the syntax for a given command by passing it the “-?” or “?” flag, or by using the help command:

SP> events ?
 events all - print all system events
 events info - print system event log information
 events newest - print newest system events
 events oldest - print oldest system events
 events search - search for and print system events
SP> events -?
 events all - print all system events
 events info - print system event log information
 events newest - print newest system events
 events oldest - print oldest system events
 events search - search for and print system events
SP> help events
 events all - print all system events
 events info - print system event log information
 events newest - print newest system events
 events oldest - print oldest system events
 events search - search for and print system events

The information available for the SP CLI commands is not as verbose and detailed as for Data ONTAP, and manual pages are unfortunately not available. The best source of more information for SP commands will be found in the System Administration Guide for the appropriate Data ONTAP release.

Command Completion

Tab completion is not available for the SP CLI, nor can you abbreviate commands. All commands must be fully entered in order for them to be recognized.

Navigation and Editing

Command-line editing and navigation utilizes the standard keystrokes and combination previously discussed in CLI Efficiency: Common Basics

You can navigate through your previously entered commands using the up and down arrows, or by using Ctrl+n and Ctrl+p, but there is no history command for the SP CLI. It is also worth noting that SP commands entered prior to accessing a system console session will not be displayed after returning to the SP CLI prompt.

Just like with Data ONTAP, you can enter multiple commands on the same command line by separating each command with a semi-colon. The commands will then be executed in order of entry.

SP*> priv; date
 Sun Nov  30 02:10:02 GMT 2014

As you’ll have noticed, the Service Processor shell has an interface similar to and consistent with the Data ONTAP 7-mode shell despite the different use cases for each.

In a future article, I’ll go into more details around SP setup, configuration and usage beyond the basics described in this post.