Richard and I have been busy this week. It started with a conversation about automating the installation of new build servers. Richard was looking at writing PowerShell to install and configure the TFS build agent, along with all the various SDKs that we use across all out projects. Our current array of build servers have all been built by hand and each has a different set of SDKs to build specific project types. Richard’s aim is to make a single, homogenous build server configuration so we can then scale out for capacity much more quickly than before.
Enter, stage left, SCVMM. For my part, I’ve been looking at what can be done with VM Templates and, more importantly, service templates. It seemed to me that creating a Build Server service in SCVMM with a standard template would allow us to quickly and easily add and remove servers to the group.
There isn’t much written about the application/script side of SCVMM server templates, so I thought I’d write up my part.
Note: I’m not a System Center specialist. We use Config Manager, Virtual Machine Manager and Data Protection Manager at Black Marble for our own services rather than being a System Center partner.
Dividing up the problem space
Our final template uses a single PowerShell script to perform the configuration and installation work. Yes, I could have created steps in the service template to install each of the items Richard’s script deployed, but we decided against that. The reasoning is relatively simple: It’s much easier to modify the PowerShell script to add, remove or change the stuff that gets deployed. It’s hard to do that with SCVMM, as far as I can tell.
However, during testing I discovered that if I added windows roles and features through the template it was faster than when the various installers Richard called in his script triggered the feature addition.
The division of labour, then, became the following:
SCVMM Tasks:
- VM Template is created for the target OS. The VM template is configured to automatically join the new machine to our domain and place the machine in the correct OU. It also sets the language correctly. More on that later.
- Service Template is created for a Build Servers service. It’s a single tier service that has a minimum of one machine and a maximum of twenty (that maximum is a bit on an arbitrary value on my part). The service template adds the roles and features to the machine definition and runs two script application blocks:
- The first simply runs xcopy to pull the contents of a folder on a share to the local PC. I do this because running a power shell PowerShell script from a network share doesn’t work – in an interactive session you are prompted before the script executes because it’s from an untrusted location and I haven’t worked out how to suppress the prompt yet.
- The second application executes powershell.exe and feeds in the full path to Richard’s PowerShell script, newly copied onto the local disk.
Step 1: VM Template
There’s a wealth of information about creating VM templates in the internet, so I’m not going to cover this in depth. I did, however, want to pull out a couple of things I discovered along the way.
I installed my base VM with the UK English regional settings. When SCVMM converted that into a template via sysprep, the resulting machine comes up in English US, which is really annoying. Had I been paying attention, I would have noticed that we already had an unattend.xml file to correct this, which I could have referenced in the VM template settings. However, I found a much more interesting way to address the issue (which of course led me down another rabbit hole).
A bit of research led me to a very interesting post by Gunter Danzeisen. In it he shows how to use powershell to modify an unattendsettings property of the VM template within System Center. This is at the same time both irritating and enlightening.
It’s irritating, because I am truly fed up of ‘hidden’ functionality in products that causes me pain. The VM Template clearly allows me to specify an unattend.xml file, so why have an internal one as an object property. Moreover, why not simply document it’s existence and let me modify that property – why do I need two different methods which then makes me constantly wonder which gets priority.
It’s enlightening, because I can modify that property really easily – it’s simply a collection of name/value pairs that marry against the unattend.xml settings.
There is a bit of snag with this approach, however, which I’ll come onto in a little while.
Anyway, back to the plot. I followed Gunter’s advice and used PowerShell to set the language values of the internal unattend. I then decided to use the same approach to see if I could add other settings – specifically the destination OU for the server when added to AD.
The PowerShell for the region settings is below:
$template = Get-SCVMtemplate | where {$\_.Name -eq "My VM Template"}
$settings = $template.UnattendSettings
$settings.add("oobeSystem/Microsoft-Windows-International-Core/UserLocale","en-GB")
$settings.add("oobeSystem/Microsoft-Windows-International-Core/SystemLocale","en-GB")
$settings.add("oobeSystem/Microsoft-Windows-International-Core/UILanguage","en-GB")
$settings.add("oobeSystem/Microsoft-Windows-International-Core/InputLocale","0809:00000809")
Set-SCVMTemplate -VMTemplate $template -UnattendSettings $settings
For reference, removing a setting is easy – simply reference the name of the setting when calling the remove method:
$settings.remove("oobeSystem/Microsoft-Windows-International-Core/UserLocale")
I then set the destination OU with the following setting:
$settings.add("specialize/Microsoft-Windows-UnattendedJoin/Identification/MachineObjectOU","OU=FileServers,DC=mydomain,DC=local")
The Snag
There is a problem with this approach. If you use an unattend.xml file, you can override that setting when you add the VM template to your service template. However, whilst I could find the unattendsettings property of the VM when referenced by the template, I couldn’t modify it.
If we access the Service Template object with:
$svctemplate = Get-SCServiceTemplate | where {$_.name -eq "TFS Build Service"}
We get an object that contains one or more ComputerTierTemplates (depending on how many tiers you gave your service). Each of those has a VMTemplate object that holds the information from our original VMTemplate, and therefor has our unattendsettings.
$svctemplate.ComputerTierTemplates\[0\].VMTemplate.UnattendSettings
So, we can grab those settings and modify them. Great. The trouble is, I haven’t found a way to update the stored configuration. Set-SCServiceTemplate doesn’t let me stuff the settings back in the same was as Set-SCVMTemplate does, and you can’t use the latter with a reference to the VMTemplate child of our template.
Now, I decided that I would create a copy of my original VM template just for Build Servers, so I could set a different target OU for the servers. In hindsight, I’m not sure whether this is better than overlaying unattend.xml files, and I haven’t experimented with how the unattend.xml might interact with the unattendsettings yet either. If you try, please let me know how you get on.
Step 2: Service Template
Once I’d got my VM Template sorted, the next step was to create a service. There’s a pretty nice design surface for these that allows you to pick a ‘starter’ template with the right number of tiers, although it’s dead easy to add a new tier.
I started with the Single Machine template, which gave me a single tier. You then need to drag a VM template from a list of available templates onto the tier. The screenshot below shows my single tier Service. The VM template has a single NIC and is configured to connect to my Black Marble network.
The light blue border on the large box (the service tier) indicates it’s selected. That will show the tier properties at the bottom of the design window. In here I have set a minimum and maximum number of servers that can be deployed in the tier.
Notice also the availability set option – if I needed to ensure that VMs in this service were spread across multiple hosts for resilience I could tick this option. I don’t care where build servers get deployed (they go onto our Lab VM hosts and are effectively ‘disposable’) so I have left this alone.
Open the properties of the tier (right-click or choose View All Properties in the property pan) and a dialog opens with machine properties. In here I have configured the roles and features for the build server (I deliberately haven’t set these in the VM Template so I can have fewer, more general VM templates).
Also in here are the Application Configuration settings that cause the VM to run Richard’s PowerShell. The first is a simple one that references cmd.exe to run xcopy. All the settings on this are default.
The biggest point here is the timeout setting. Richard’s script can take an hour or so to run, so the timeout is a BIG number. The first few times I deployed the script task failed because of this, although in reality the script itself was still running happily and completed fine.
I have changed the advanced settings for this script, though. To make debugging a little easier (the VM is a black box whilst deploying, so it’s tricky to see what’s going on) I have directed standard output and standard errors to a file. I’ve also turned off the options to automatically ‘detect’ failures through watching output, error and exit codes. Richard’s script can be run repeatedly with no ill-effect, so I’ve left the restart option to restart the script. That means that if I restart the deployment job from SCVMM if it fails, the script will be run.
Once the service template is created, I deployed the service with a single server. We then added a second server by scaling out the tier.
We can do this via the SCVMM console, or using the virtualmachinemanager PowerShell module:
$serviceInstance = Get-SCService -Name "TFS Build Servers"
$computerTier = Get-SCComputerTier -Service $serviceInstance | where { $_.Name -eq "TFS Build Server" }
New-SCVirtualMachine -ComputerTier $computerTier -Name "Build03" -Description "" -ReturnImmediately -ComputerName "Build03" -StartAction "NeverAutoTurnOnVM" -StopAction "SaveVM"
Lessons Learned
It’s been an interesting process overall. I think we’ve made the right choice in using a single, easily modifiable powershell script to do the heavy lifting here. Yes, I could have created Application Profiles in SCVMM for each of the items Richard installed, but it would have been harder to make changes (and things like the Azure SDK are updated faster than I can blink!).
I’m still considering whether my choice of adding the domain location to the unattendsettings in the VM template object was a good choice or not. I’m happy that the language settings should go in there – we never change those. I need to experiment with how adding an unattend.xml file affects the settings in the template object.
Service Templates are a great way to go. When you think about it, most of our IT systems tend to be service-focused. Using service templates for things like SharePoint or CRM are a no brainer, but also things like web servers, where we have a number of relatively heterogeneous VMs that host internal web services or sites. Services in SCVMM collect those VMs into manageable groups that can be easily spread across multiple hosts for resilience if required. It’s also much easier to find VMs in services than in a very long list of hosts!
In terms of futures, I’m interested in where Desired State Configuration will take us. Crafting the necessary elements for this project would be extremely complicated with DSC right now, but when all the ducks are in order it should make life much, much easier, and DSC is certainly on my learning list.