VMware released an update for vRealize Automation (vRA) at the start of this month. This was a slight increment from 7.5 to 7.6. At a high level, two key areas of change are NSX integration and the vRealize Orchestrator (vRO) user experience. The Release Notes goes into a bit more detail and I’ll be using those more detailed items as a guide for the content in this post. I will be skipping over the new NSX pieces as I don’t have those available to me.
VMware
vRealize Automation – Dealing with a disconnected/orphaned IAAS Server
Following the installation of a replacement UPS I made a horrible discovery – there was problems with my vRealize Automation setup. The first sign of what was to come was when I reviewed the console of the vRealize Automation (vRA) Virtual Applications.
Anyone who has had to setup vRealize Automation would probably know the sinking feeling I started feeling. Having to setup vRA again from scratch was not something I really wanted to do….
Managing F5 Load Balancers with vRealize Orchestrator
F5 Load Balancers (LB) have been a common feature across a number of environments I’ve worked at. While administration of these devices is generally performed via the web interface, F5s also have a REST API that allows the same management tasks to be performed. This opens the possibility of using VMware’s vRealize Orchestrator (vRO) to manage F5 Load Balancers via the same REST API.
vRealize Orchestrator – PowerShell Hosts
PowerShell Hosts are one of the types of endpoint available in vRealize Orchestrator’s Inventory. By having a PowerShell Host, you can leverage the breadth of PowerShell functionality from within your vRealize Orchestrator workflows. In this article, I’ll run through adding a PowerShell Host as well as some considerations from a technical and security point of view.
Adding A PowerShell Host
vRealize Orchestrator has a built-in Workflow for adding a Host under Library > PowerShell > Configuration. Run the “Add a PowerShell host” Workflow to start it. The opening interface is below:
Creating Service Accounts with vRealize Orchestrator
vRealize Orchestrator (vRO) has a lot of plugins that allow it to integrate with other systems and services. One of such plugin is for Active Directory. This plugin allows you to perform a number of standard AD activities, like creating users. vRO already has built in workflows to create and manipulate users. In this post, I’m going to run through what you might end up implementing if you wanted to be able to create Service Accounts via vRO.
Improving the vRA Admin Experience – Reservation Alerts to Slack
The Reservation system in vRealize Automation (vRA) provides a bucket of resources to a team or business unit via Business Group. A risk with Reservations comes about with how I think VMware intended them to be used vs how some organisations may use them. I suspect VMware’s intention was that Reservations should be self-managed by the Business Group associated with it. This makes sense if each individual team has a Business Group as the scope of what’s in the Reservation is “their stuff”. It would mean if a Reservation reached capacity, it would be up to that team to manage the situation.
What if the Business Group was being used differently, where it covers multiple teams? In the event of the Reservation becoming full, the scope is larger than one team. In this situation, it might be good to get a heads up on when Reservations are running low on resources. Email alerts can be setup and yes, sent through to Slack, the formatting in Slack is less than desirable. So I decided to look at a way of doing it better.
Improving the vRA Customer Experience – Send Chef errors to Slack
One of the issues that can be amplified by automation is logging. Some logs have an ephemeral nature, having a short lifespan due to various factors. This can be especially painful if the logs relate to failures and contain information that could assist in fixing the problem.
This was the issue I was seeing when vRealize Automation (vRA) requests would fail when Chef attempted to apply settings. If Chef failed critically, vRA would be made aware of it and fail the entire request. Of course, vRA would then delete the virtual machine and the local Chef logs. In many cases, there was a gap of only a minute or two between the Chef failure and the vRA cleanup tasks.
Installing ElasticStack Beats on vCenter 6.7
I recently deployed a vCenter appliance to 6.7 after a power outage corrupted the 6.5 instance. A followup task for the virtual appliance was getting the ElasticStack Beats (MetricBeat, Filebeat) installed again. In this post, I will go through the process of installing the Beats and some of the minor issues I ran into.
VMware vRealize Suite Lifecycle Manager 1.2 – First Impressions
When VMware created the vRealize brand, they grouped together some of their most complex products under one banner. vRealize Automation (vRA) required the deployment and configuration of two components – a virtual appliance and a Windows server. The Windows server had a long list of prerequisites. In terms of operational management, using products like vRA meant ongoing work on scripts, workflows and other artifacts. The logical response to this is to create a non-production instance to protect your production instance. Moving updates to production could be achieved manually or via VMware’s Codestream product, but both approaches left a lot to be desired. vRealize Suite Life Cycle Manager (vRSLCM or just LCM) is a new approach to this set of problems.
Getting LCM Running
LCM comes supplied as a “Virtual Application” where a few configuration options are required to provision it. One of the LCM-specific settings is whether you want to enable the vaguely named “Content Management”. Enabling this will cause the appliance to use 4 processors instead of 2. Once the appliance is deployed and started, the rest of the configuration happens via the web interface.
VCSA 6.5 Root Account Password – Reset and Cause Investigation
One of the more frustrating experiences one can experience with VMware’s vCenter Server Appliance (VCSA) is having the root account locked out or forgetting the password for it. I recently experienced this after I rebuilt the VCSA in my home lab from scratch.
How to Reset the VCSA Root Password
VMware have a short process on how to reset the password for the root account, detailed in KB2147144. THe process is:
- Backup the VCSA (via snapshot or other means)
- Reboot the VCSA
- During the boot process, when the photon splash screen appears press the e key to get into the boot menu
- In the text box that appears, go to the line starting with “linux”. Go to the end of the line (which is right after the text “consoleblank=0”) and enter the text rw init=/bin/bash This will cause the boot process to jump right into the bash shell without needing credentials
- Press F10 to continue booting
- At the command prompt, run the passwd command to reset the password
- Unmount the file system by running umount /
- Reboot the VCSA using reboot -f
- Following reboot, confirm the new password works
- If you took a snapshot in step 1 remove it
Cause Investigation
After resetting the password and restarting, I still couldn’t login. One thing I noticed was there had already been 20 login failures. In my situation, the VCSA was working one evening and the following morning the login issues happened. Something had to be causing these issues. Using the password reset process to get to the bash shell again, I looked around in some logs. First I tried checking the /var/log/messages log.
Unfortunately, using ‘FAILED’ only showed 2 login attempts on the console which were caused by myself after the lockout happened. Failed SSH login attempts are logged under a text string that uses ‘Failed”. The second search attempt used ‘Failed’ and yielded better results.
The log had numerous entries for 192.168.1.55 trying to login using root and other accounts. The system on 192.168.1.55 was the trial of Nexpose. Even though I didn’t have credentials set in Nexpose to logon to the VCSA, it still was trying to logon using root and was causing the failures. This hadn’t been an issue prior to the VCSA rebuild.
I excluded the VCSA from the scanning that Nexpose performs and did the password reset process again. I was now able to login successfully.