Bicep is a great improvement over ARM Templates but doesn’t remove the need to validate our code at build time. I could continue to use the ARM-TTK and validate the generated template, but bicep has it’s own built in rules. Getting build errors in a way that can provide meaningful information in my CI/CD tooling is an interesting challenge.
A quick problem description
We are using Azure DevOps for build and release automation. It supports ingestion of test results in a variet of formats such as NUnit and JUnit. With ARM Templates we can use the ARM-TTK, and Sam Cogan’s great task that creates the right formatted output to ingest using the PublishTestResults task.
Bicep now has a ’lint’ option that we can use to validate our templates before we build them, and it supports the SARIF standard format for those results. Whilst there is an extension for Azure DevOps to display SARIF files, I would prefer to see those test results in the same place as things like unit tests.
To get what I want we’ll need to do three things:
- Run bicep lint in our pipeline and capture the SARIF format results to a file.
- Transform the SARIF file to a format Azure DevOps understands like NUnit or JUnit
- Publish the test results file during the pipeline.
Simple, right?
Better yet, somebody has already done it! John Reilly has an excellent post that covers both Azure DevOps pipelines and GitHub Actions.
Well, we wouldn’t be here if everything worked how I wanted it to… I followed John’s post in my pipeline and got errors. This post is about what those errors were and how I solved them.
Step 1: Running bicep lint in our pipeline
Problem one: Text format
John is using the AzureCLI task in his pipeline and the az bicep lint command with output simply piped to a file. He the uses an NPM package called sarif-junit to convert the file to xml.
When I tried the same thing, however, the conversion failed with the error: Unexpected token '�', "��{
I spotted that John’s pipeline was using a linux cloud agent; I was using Windows. Could it be text formatting? Examination of the file showed that it was UTF-16, and apparently the node task didn’t like that. The fix, then, was to capture the output into a PowerShell var rather than pipe straigh to file, and then specify the text formatting when I saved it:
$output = az bicep lint --file ${{parameters.bicepFile}} --diagnostics-format sarif
$output | Out-File -FilePath $(Build.SourcesDirectory)\bicep.sarif -encoding ascii
Problem two: Errors give no output
By default the bicep linter treats the issues it can handle as warnings. That means the bicep build command doesn’t fail, so the pipeline doesn’t fail. It also means that the transformed output from bicep lint actually gets imported as a passing test, not a warning. That’s not terribly helpful.
The solution to that problem is to use the bicepconfig.json file that the CLI supports and change the warnings to errors:
{
"analyzers": {
"core": {
"enabled": true,
"rules": {
"adminusername-should-not-be-literal": {
"level": "error"
}
}
}
}
}
When I did that, however, I got no output from the command at all! It turns out that (at time of writing) there is a documented bug in AZ CLI which results in no output from the bicep linter being displayed.
The solution to that turned out to be pretty simple: Use the Bicep CLI instead of AZ CLI. Including the transformation, my pipeline task looks like this:
- task: AzureCLI@2
displayName: 'Lint Bicep files'
inputs:
azureSubscription: $(azureServiceConnection)
scriptType: 'ps'
scriptLocation: 'inlineScript'
useGlobalConfig: true
inlineScript: |
#az bicep install
$output = bicep lint ${{parameters.bicepFile}} --diagnostics-format sarif
$output | Out-File -FilePath $(Build.SourcesDirectory)\bicep.sarif -encoding ascii
npx -y sarif-junit -i $(Build.SourcesDirectory)\bicep.sarif -o $(Build.SourcesDirectory)\bicep.xml
Followed by the PublishTestResults task to ingest the JUnit format file:
- task: PublishTestResults@2
displayName: 'Publish Bicep test results'
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(Build.SourcesDirectory)\bicep.xml'
testRunTitle: 'Run_$(Build.BuildNumber)'
failTaskOnFailedTests: true
condition: always()
Step 2: Presenting test results
Problem: Results not imported correctly
The image above shows how the test results are displayed when the JUnit file is imported. As you can see, it’s actually not very helpful:
- The column where we would expect to see the name of the test (e.g. adminusername-should-not-be-literal) actually contains the error message, including the line number.
- Selecting the test to view more details actually shows less: There’#s no error message and nothing tells me what file the error is present in.
The PublishTestResults task has excellent documentation, including details of how the JUnit files are imported and what field value goes where.
Here is a sample output file, using the no-hardcoded-location test, as generated by the sarif-junit module.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="2" failures="2" errors="0" skipped="0">
<testsuite name="My suite" tests="2" failures="2" errors="0" skipped="0">
<testcase classname="no-hardcoded-location" name="Ln 39 A resource location should not use a hard-coded string or variable value. Please use a parameter value, an expression, or the string 'global'. Found: 'uksouth' [https://aka.ms/bicep/linter/no-hardcoded-location]" file="//D:/repos/tuServFakeExternalSystems/Bicep/Shared/Modules/KeyVault.azuredeploy.bicep">
<failure/>
</testcase>
<testcase classname="BCP104" name="Ln 53 The referenced module has errors." file="//D:/myproject/Bicep/Shared/./Main-Shared.azuredeploy.bicep">
<failure/>
</testcase>
</testsuite>
</testsuites>
Looking at the task documentation, we can see several problems:
- The task imports the
nameattribute as the name of the test. - It doesn’t understand the
fileattribute of thetestcaseelement - It needs the error message to be a
messageattribute on thefailureelement.
We need to transform the XML generated by the node task - moving the data around. It needs to look like this:
<?xml version="1.0" encoding="utf-8"?><testsuites tests="2" failures="2" errors="0" skipped="0">
<testsuite name="My suite" tests="2" failures="2" errors="0" skipped="0">
<testcase name="no-hardcoded-location">
<failure message="Ln 39 A resource location should not use a hard-coded string or variable value. Please use a parameter value, an expression, or the string 'global'. Found: 'uksouth' [https://aka.ms/bicep/linter/no-hardcoded-location] //D:/repos/tuServFakeExternalSystems/Bicep/Shared/Modules/KeyVault.azuredeploy.bicep" />
</testcase>
<testcase name="BCP104">
<failure message="Ln 53 The referenced module has errors." />
</testcase>
</testsuite>
The failure message combines the values of what were name and file, and what was classname is now name.
To get the desired output, I decided to use XSLT.
I actually asked Bing Copilot to generate the XSLT for me, because I hate writing XSLT. Not relevant to this article, but pretty cool - it got it right on the first go!
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<!-- Copy all nodes and attributes by default -->
<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()" />
</xsl:copy>
</xsl:template>
<!-- If a failure element exists within a testcase element, set the value of the message
attribute of the failure element to the contents of the name attribute of the testcase element concatenated with the file attribute of the testcase element -->
<xsl:template match="testcase/failure">
<xsl:copy>
<xsl:attribute name="message">
<xsl:value-of select="concat(../@name, ' ', ../@file)" />
</xsl:attribute>
</xsl:copy>
</xsl:template>
<!-- Set the value of the name attribute of the testcase element to the value of the classname
element of that testcase element -->
<xsl:template match="testcase/@name">
<xsl:attribute name="name">
<xsl:value-of select="../@classname" />
</xsl:attribute>
</xsl:template>
<!-- Remove the classname attribute from the testcase element -->
<xsl:template match="testcase/@classname" />
<!-- Remove the file attribute from the testcase element -->
<xsl:template match="testcase/@file" />
</xsl:stylesheet>
A few lines of PowerShell apply the transform to the xml file created by the node module. My task now looks like this:
- task: AzureCLI@2
displayName: 'Lint Bicep files'
inputs:
azureSubscription: $(azureServiceConnection)
scriptType: 'ps'
scriptLocation: 'inlineScript'
useGlobalConfig: true
inlineScript: |
#az bicep install
$output = bicep lint ${{parameters.bicepFile}} --diagnostics-format sarif
$output | Out-File -FilePath $(Build.SourcesDirectory)\bicep.sarif -encoding ascii
npx -y sarif-junit -i $(Build.SourcesDirectory)\bicep.sarif -o $(Build.SourcesDirectory)\bicep-tmp.xml
$xslt_file = new-object xml
$xslt_file.Load("$(Build.SourcesDirectory)\DevOpsPipelines\junit.xsl")
$xslt = new-object System.Xml.Xsl.XslCompiledTransform
$xslt.Load($xslt_file)
$xslt.Transform("$(Build.SourcesDirectory)\bicep-tmp.xml", "$(Build.SourcesDirectory)\bicep.xml")
Our imported test results now look much better:
Conclusion
What I thought was going to be a quick job turned into a series of mysteries to solve, but the end result is what I need:
- Use Bicep linting to validate my files and import the results into my Azure DevOps pipeline run as Test Results.
- Work on a Windows build agent (I also needed to run on a private agent) with no text format issues.
- Avoid the AZ CLI bug that results in no output from bicep lint.
- Transform the results to display in a meaningful way in Azure DevOps.
Hopefully this post will save others time and pain!