Microsoft Azure Stack is a truly unique beast in terms of the capabilities it can bring to an organisation, and the efficiencies it can bring to a project that spans Public Cloud and on-premises infrastructure through it’s consistency with public Azure.
We’ve been using Stack in anger for a customer project for a year now and have learned several things about development and testing, and how to configure the ASDK to be an effective tool to support the project. This post will summarise those learnings and how I deploy the ASDK so you may mirror our approach for your own projects.
Learning 1: Cloud-consistent, but not quite
Azure Stack is great in that you use the exact same technologies to develop and deploy as its big brother. However, there are some nuances that we’ve hit. Stack lags behind Azure in API versions and provides a subset of Resource Providers. Whilst it is possible to configure your Azure subscription to the same API versions as Stack using policies, that’s not a guarantee of compatibility. In our project a partner org has been using Azure with those policies applied, but they still managed to update the language-specific SDKs for Storage beyond those supported by Stack. You need to be very watchful about version support, and our experience is that there is no replacement for using Stack as part of your pre-production testing effort.
Learning 2: Performance may vary
This one is obvious, if you think about it. Stack is built on different infrastructure than Public Azure because we don’t get things like FPGA-accelerated networking, for example. That means you need to performance test your stuff on Stack or you may be upset that it doesn’t meet the stats you got from Azure. Don’t get me wrong – we’ve put some very big load on both single-box ASDK and multi-node MAS during our project, but stuff like VM provisioning times, network performance and storage read/write times are different than production Azure and if your solution relies on this stuff you need to load test on your target environment.
Learning 3: Identity, identity, identity
Stack can be configured in connected or disconnected modes and that is a big, fundamental different you need to be aware of. In connected mode you use Azure AD for a single source of identity across your Stack and Azure subscriptions. With disconnected mode you are using ADFS to connect Stack to an Active Directory. The key difference is if you are using Service Principals – in disconnected mode you can only do certificate authentication. This is supported on Azure but not the default. Our approach has been (since our project is disconnected) to do all our dev in an ASDK for absolute confidence of compatibility but with careful governance you could use public Azure. The key difference is that with the ASDK you create a service principal using PowerShell and the certificate gets created at that point for you to extract. With public Azure you need to create the cert first and provide it when creating and service principal.
Learning 4: ‘D’ is for Demo, not Dev; at least out-of-the-box
The ASDK is a wonderful thing. It’s easy to download and deploy a single-node version of Stack for testing. However, the installer delivers the system in a bubble. To use it you have to RDP into the host to access the portal and mess around with VPNs. Connecting to your CI/CD tool of choice (we use VSTS) is hard verging on impossible and at the very least active use for development is impractical.
You also have to content with the fact that the ASDK is built on evaluation media, and that you can’t upgrade it from version to version. I have no argument as to why this is – you shouldn’t be able to use the dev kit as a cheap way of getting Stack into your prod environment, but it means that if you want to use the ASDK for development you need to roll your sleeves up a bit… More on that in a moment.
Learning 5: Matt McSpirit Rocks!
For a long time I kept thinking that I should collect the ragtag bunch of PowerShell I used to deploy our ASDK each month into something resembling a usable solution and publish it. Then Matt McSpirit released ConfigASDK and I no longer needed to! If you use ASDK at all, then you need Matt’s Script because it makes post-install configuration an absolute breeze!
Effective Deployment 1: Hacking the Installer
We now have more than one ASDK and I need to rebuild each one pretty much monthly. That means streamlining the process. To do that, I convert the downloaded VHD to a WIM and we import it into our Config Manager deployment so I can use PXE to easily reimage the servers. Once that’s done I can clear down the storage spaces left over from the previous install and rebuild. I need the ASDKs to be on our network, so I assign a /16 network range for each Stack as well as assigning an IP on our main subnet to the host. I also modify the deployment scripts so I can use a custom region and external DNS suffix which allows us to access the ASDKs over DirectAccess – handy for when we’re working on site with our customer.
To do all this I loosely follow the documentation to deploy using PowerShell.
Reconfigure Network Ranges
The networking for the ASDK is all configured from the OneNodeCustomerConfigTemplate.xml file that is in c:\CloudDeployment\Configuration. I simply open that file and replace all occurances of 192.168. with my own /16 IPv4 range (e.g. 10.1.). That will give you an ASDK with several address ranges and importantly, a full /24 block of IP addresses for services (storage endpoints, Public IP Addresses, etc) that can sit on your network.
You’ll need to configure your own network to publish the route into that /16 address range, with an IP address on your own network the gateway address – we’ll assign this to the VM in stack that does the networking during the installation.
Reconfigure Region and ExternalDomainSuffix
When you deploy AzureStack it will create an AD called Azurestack.local that the infrastructure VMs will join, including the host. You will also find that the portal, admin portal and services are published on a DNS domain of local.azurestack.external. In that domain, ‘local’ is the region and ‘azurestack.external’ is the external domain suffix. Understandably, for a dev/demo deployment these are not exposed to you as things you can change. However, changing them is actually pretty easy when you know where to look.
When you run the InstallAzureStackPOC.ps1 command, it actually calls the DeploySingleNode.ps1 script that is in c:\clouddeployment\setup and if you open that script you’ll find that there are parameters called RegionName and ExternalDomainSuffix and both these params have default values. Simply open that file in an editor and change them. Line 143 contains the $RegionName parameter with a default of ‘local’ so I change that to be the name of my ASDK host; line 130 contains the $ExternalDomainSuffix parameter with a default of ‘azurestack.local’ so I change that to stack..
This approach means that I end up with all the ASDKs within stack.mydomain.local, with the name of the host being the region – that’s a good logical approach as far as I and my team are concerned!
Now, doing that isn’t quite all you need to do. Just like with the network routing requirement, you have to get the DNS requests to the Azure Stack DNS or nothing will work. If you follow my approach for the network address range, then the next step is to create some delegated subdomains in your internal DNS. We are using Windows Server and our internal DNS is linked to our AD, so the steps are as follows:
- Open the DNS console on your DC.
- Within your internal domain, create a HOST record for the stack DNS. I normally call mine stackdns. because I have more than one. Assuming you’re done the neworking changes like me, the IP address for that is <your /16 subnet>.200.67 (e.g. 10.1.200.67).
- Within your internal domain create a subdomain of your internal domain called ‘stack’.
- Within that stack subdomain, right-click and select New Delegation. In the dialog that pops up enter the name you specified as the RegionName and then when asked for a DNS FQDN enter the hostname you just registered.
- If you are using App Services you also need to create a delegation for their domain – .cloudapp.. Simply create another subdomain within stack called cloudapp and then repeat the previous step to register the delegation.
Once you’ve configured your internal network and DNS, then modified the config file and script you can perform the installation. When doing so, make sure that in the parameters of the InstallAzureStackPOC.ps1 you specify the NatIPv4Address parameter as the IP address you set in your routing configuration to be the IP address you used in your network router configuration.
Effective Deployment 2: Removing the NAT
This bit hasn’t changed since the preview releases and I follow the notes on the AzureStack blog post. Once the ASDK install completes I simply run a short script that contains the powershell at the end of that post to remove the NAT, at which point that VM sits on my internal network with the IP address I set in the deployment and have configured in my router and traffic just flows.
Effective Deployment 3: ConfigASDK
Matt’s ConfigASDK script is excellent, an I use this to configure our ASDKs post-install. The script will deploy additional resource providers for you (SQL, MySQL and AppServices), install handy utils on the host and pull down a bunch of marketplace offerings you’ll find helpful. For our project we are not using the SQL and MySQL RPs so I skip those, along with the host customisation part.
Now, Matt’s script is built with the Stack team and as such doesn’t currently support the hacks for region and external DNS. I have a fork of the repo in my GitHub where I have added the RegionName and ExternalDomainSuffix params to Matt’s script, but that may not be up to date so you should look at what I’ve done and proceed with caution. I had to put a bunch of code in to replace what were understandably-fixed URLs. However, once I’d done that all Matt’s code worked as expected.
Usual disclaimers apply – use my version of Matt’s script at your own risk. make sure you understand the implications of what I’m describing here before you do this on your own network!
Post-Deployment Configuration Tips
Once you’ve got your ASDK up and running there are still a couple of things you should tweak for a happier time, particularly if you batter the thing as much as we do in terms of creation and deletion of resources and load testing.
Xrp VM RAM
Something you can do with ASDK but not with MAS is get at the VMs that form the fabric of Stack. In an ASDK these are not resilient and are smaller in config than their production counterparts. Our experience is that one of them does not have enough RAM configured by default. The AzS-Xrp01 VM is create with 8192MB of RAM. We see it’s demand climb well beyond this in use and as it does, you may see storage failure messages when trying to create accounts, access the storage APIS or blades in the portal. I give it another 4GB – increasing it to 12288MB in total and that seems to sort it. You don’t need to turn off the VM to do this – just go into Hyper-V Manager and increase the amount.
App Services ScaleSets
As of 1808 you can do this in the portal rather than through PowerShell. When App Services is installed there are a number of VM scalesets created that contain the VMs that host the App Hosting Plans. The thing is, the App Services team don’t know what you need, so the deploy with a single Shared worker tier VM and none in the other tiers. You need to go into the portal and change the instance count to an appropriate number for your project. I normally scale back Shared to zero and set the Small scaleset to a reasonable number based on the project. I also set the Management scaleset to to two instances. I’ve found that makes for a more reliable system, again, because we tend to batter our ASDK during development.
Parting thoughts
Please remember that the approach I describe here is not the way the Stack team expect you to deploy an ASDK. You need to have a reasonable understanding of networking and DNS to know the implications of what I change in my approach to deployment. Also bear in mind that Stack releases monthly updates so if you’re reading this in a year’s time, stuff might have changed so be careful.
Azure Stack is a very powerful solution for a very specific set of customer requirements – it’s not designed to be something everybody should use and you need to remember that. If you do have requirements that fit, hopefully this post will help you configure your ASDK to be a useful component in your development approach.