Neural Flow Blog

Enabling GitHub Copilot Coding Agents to Access Azure

2025-08-15T00:00:00Z

Before I start with the details, I want to make something extremely clear: Do not give coding agents access to production. We’ve all read/seen the stories on the internet about AI’s potential for causing chaos if not properly controlled. This is not specific to AI, but it’s a general rule: Don’t give developers/anyone permanent, ungoverned access to production resources without strict controls in place. I could spend the rest of this blog post lecturing about this, but then we’d never actually get to the point of this article. So warnings aside, lets get to it.

Background

I’ve been experimenting with GitHub Copilot coding agents, assigning them tasks almost every day (and night) and they’re pretty impressive. But on occasion I actually need coding agents to interact with Azure resources, usually as part of diagnosing an issue/fixing a bug that requires real-time access to those resources (only in dev/test, of course). This is also sometimes useful if I want the coding agent to be able to verify some infrastructure it’s creating Bicep or Terraform files for.

So, this blog post shows you how to configure your coding agents to be able to use a federated identity to access Azure resources securely, without needing to store any secrets in your repository. As well as set up RBAC permissions so that they only get the level of access they need.

Why This Matters

Here’s what I can do now with my Copilot agents:

Diagnose issues that might be occurring in dev/test environments
Access key vault secrets that might be required to access dev/test databases or other resources
Validate or obtain specific information about the Azure environment that is being worked with
Automatically provision (and of course cleanup) Azure resources as part of coding agent development process. But remember, deploying resources in Azure usually has a cost associated with it.
Enable your Coding Agents to use the Azure MCP Server.

The most important thing to note however, is the principle of least privilege: If your coding agent doesn’t need access to Azure, don’t provide it access. If just needs to be able to read diagnostic logs from a Log Analytics workspace, then that’s all it should be allowed to do. It’s very easy to over-provision access, so always err on the side of caution.

Azure Costs

If you give coding agents permission to deploy resources, be aware of the potential costs involved. Because coding agents now read the copilot-instructions.md in your repository, it will be critical to provide cost management guidance within that file - including the fact that they should delete any resources they create once they are done with them.

My advice is to avoid giving them write/deploy access unless absolutely necessary, or at the very least have a method of tracking and auditing their resource usage. It would be quite simple to create a janitor (a GitHub Actions workflow) that deletes any resources that were created in a resource group that the Coding Agent has permission to create in.

Getting the Security Right

Before we dive into the setup, let me be clear about the security principles I follow. I’ve seen too many “quick demos” that skip these fundamentals:

Least privilege always: Give only the minimum permissions needed. You can expand later.
Federated credentials only: Workload identity federation is the right way to give your coding agents access to Azure.
Environment isolation: Dev/test, staging need separate identities. But I shouldn’t have to repeat: don’t give coding agents access to production.
Check what your agents are doing: You need to know what your agents are doing in Azure.
Your agent, your cost: Coding agents don’t care about your budget (unless you tell them to and what to do about it).

These aren’t optional “best practices”—they’re requirements if you want to sleep well at night.

How to Set This Up

The process is as follows:

Create a GitHub environment called copilot for your coding agent to use in the GitHub repository.
Create an Entra ID app registration for your coding agent.
Create a federated credential for the app registration that is usable by the coding actions GitHub workflow.
Assign the least access RBAC roles to the app registration that coding agent will need.
Create the secrets and variables for the copilot environment in the GitHub repository.
Configure or create the copilot-setup-steps.yml workflow to use the credential to authenticate to Azure.

The next sections show the process in detail.

1. Create a GitHub Environment

It is best to create an environment in GitHub that your coding agent has access to. This allows you to provide environment specific variables and secrets that your coding agent will get access to. This is required for the federated credentials to work.

I have called my environment copilot, but you can name yours anything you like. Make sure to adjust any steps accordingly.

Go to your GitHub repository
Click on Settings
Scroll down to the Environments section
Click New environment
Name it something like copilot
Click Configure environment

Create a new GitHub Environment for copilot to use

2. Create the Entra ID App Registration

Next, you need an Entra ID application registration for your coding agents to use. You should create a dedicated app registration for each repository so that you can scope the permissions for each coding agent to prevent over-privileging.

Head to the Azure Portal
Navigate to Microsoft Entra ID (using the Azure search box)
Select the App registrations blade under Manage
Click New registration
Name it something meaningful like gitHub-coding-agent-yourreponame
Leave the Redirect URI blank and the Supported Account Types as Accounts in this organizational directory only
Click Register

Create Entra ID App Registration for Coding Agent

Once it’s created, grab these values from the Overview page:

Application (client) ID - You’ll need this for GitHub secrets
Directory (tenant) ID - Also required for GitHub secrets

Copy Entra ID App Registration Details

# Example values (yours will be different)
AZURE_CLIENT_ID="12345678-1234-1234-1234-123456789012"
AZURE_TENANT_ID="87654321-4321-4321-4321-210987654321"

3. Create a Federated Credential

This is where the magic happens. Federated credentials create a trust relationship between GitHub and Azure without storing any secrets. It’s way more secure than cramming service principal secrets into your repo.

Here’s the crucial part that tripped me up initially:

In your app registration, go to Certificates & secrets
Click the Federated credentials tab
Click Add credential
Select GitHub Actions deploying Azure resources
Fill in the details:
- Organization: Your GitHub username or organization
- Repository: Your repository name
- Entity type: Environment
- Environment name: copilot (this is crucial. It must match exactly the name of the GitHub environment you created)
- Name: Something descriptive like gitHub-coding-agent-yourreponame-copilot-environment

Create GitHub Federated Credential for Coding Agent

The environment name must be exactly copilot with lowercase ‘c’. I learned this the hard way when I kept getting AADSTS7002138 errors. Azure federated identity credentials are case-sensitive, so Copilot or COPILOT will fail.

4. Set Up Azure RBAC Roles

Now for the permissions. The specific roles depend on what your agents need to do, but always start with minimal permissions and expand as needed. Just remember the golden rule:

You can grant the RBAC roles through the Azure Portal just as you would any other application registration, user, group or managed identity, but below, I’m using Azure CLI, so have already logged into Azure using az login. Just use the method you’re most comfortable with. The following are some examples of RBAC roles you might wish to give your coding agent.

To allow the coding agent to experiment in a single group for resource deployment and management:

# Set environment variables
export AZURE_CLIENT_ID="12345678-1234-1234-1234-123456789012"
export AZURE_SUBSCRIPTION_ID="11111111-2222-3333-4444-555555555555"

# Create a resource group for the coding agent
az group create --name rg-coding-agent-sandbox --location eastus

# Contributor role to a single existing resource group
az role assignment create \
  --assignee $AZURE_CLIENT_ID \
  --role "Contributor" \
  --scope "/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/rg-coding-agent-sandbox"

For read-only monitoring of a subscription:

# Set environment variables
export AZURE_CLIENT_ID="12345678-1234-1234-1234-123456789012"
export AZURE_SUBSCRIPTION_ID="11111111-2222-3333-4444-555555555555"

# Reader role for subscription
az role assignment create \
  --assignee $AZURE_CLIENT_ID \
  --role "Reader" \
  --scope "/subscriptions/$AZURE_SUBSCRIPTION_ID"

For Key Vault access:

export AZURE_CLIENT_ID="12345678-1234-1234-1234-123456789012"
export AZURE_SUBSCRIPTION_ID="11111111-2222-3333-4444-555555555555"
export RESOURCE_GROUP_NAME="your-resource-group-name"
export KEY_VAULT_NAME="your-keyvault-name"

# Key Vault Secrets User
az role assignment create \
  --assignee $AZURE_CLIENT_ID \
  --role "Key Vault Secrets User" \
  --scope "/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.KeyVault/vaults/$KEY_VAULT_NAME"

I can’t stress this enough: Principle of least privilege. I’ve seen too many demos where people just hand out Contributor at the subscription level. That’s asking for trouble.

5. Add GitHub Repository Secrets

Your Copilot agents need three pieces of info to authenticate with Azure. Store these as GitHub repository environment level secrets—not environment variables in your code.

In your GitHub repository, go to Settings
Select Environments
Click on your copilot environment
Click Add environment secret
Add these repository secrets, one-by-one:

AZURE_CLIENT_ID: 12345678-1234-1234-1234-123456789012
AZURE_TENANT_ID: 87654321-4321-4321-4321-210987654321
AZURE_SUBSCRIPTION_ID: 11111111-2222-3333-4444-555555555555

If you’re not sure how to get your Azure Subscription ID, this page will tell you how.

These aren’t really “secrets” since they’re identifiers, but storing them as repository secrets keeps your config centralized and secure and it is better to treat them as secrets.

6. Create the GitHub Actions Workflow

Here’s where it all comes together. GitHub Copilot coding agents always look for a workflow called .github/workflows/copilot-setup-steps.yml in the GitHub repository that they are working on. This is actually a GitHub Actions workflow that provides all the necessary steps to set up the environment that the coding agent will work in. In our case, this also authenticates to Azure.

Create .github/workflows/copilot-setup-steps.yml:

name: Copilot Setup Steps

on:
  workflow_dispatch:

permissions:
  id-token: write
  contents: read

jobs:
  copilot-setup-steps:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    environment: copilot  # This must match your federated credential environment name

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Azure login
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      
      - name: Verify Azure access
        run: |
          echo "Successfully authenticated to Azure!"
          az account show

This workflow provides a solid foundation that your Copilot agents can build on for various Azure tasks that might need to perform while working on an issue. The key is the environment: copilot line—that’s what ties everything together with your federated credential.

Test It Out

Time to see if everything works. Let’s test the authentication flow:

Go to your GitHub repository
Click Actions → Copilot Setup Steps
Click Run workflow
Select the main branch
Click Run workflow

If everything’s configured correctly, you should see:

GitHub Copilot Coding Agent Run Copilot Setup Steps

When things go wrong (and they will):

Error: AADSTS7002138: No matching federated identity record found

Verify the environment name is exactly copilot (lowercase)
Check that your repository and organization names match exactly
Ensure the federated credential subject pattern is correct

Error: AADSTS70021: No matching federated identity record found for presented assertion

Verify your GitHub repository secrets are correct
Check that the app registration client ID matches
Ensure the federated credential is properly configured

Error: Authorization failed

Verify RBAC role assignments are correct
Check that permissions are assigned to the correct scope
Ensure the service principal has the necessary roles

Beyond the Basics

Configuring Coding Agents to use Azure MCP

To enable your coding agents to use an Azure MCP (Model Context Protocol) server, you’ll need to first add the Azure MCP in the Copilot settings for the repository. For more information on the Azure MCP server, see this page. The Azure MCP server actually makes calls to Azure CLI commands which is why it can only function if the coding agent environment is already connected to Azure.

Open the Settings tab of your GitHub repository.
Expand the Copilot section.
Select Coding agent.
In the MCP configuration section add:

 {
   "mcpServers": {
     "Azure": {
      "type": "local",
      "command": "npx",
      "args": [
        "-y",
        "@azure/mcp@latest",
        "server",
        "start"
       ],
      "tools": ["*"]
     }
   }
 }

Click Save MCP configuration.

GitHub Copilot Coding Agent Azure MCP Setup

A reminder: the Azure MCP server can only function if the coding agent environment is already connected to Azure. It uses the application registration provided in the GitHub repository secrets and will only have permissions to do what that identity can do. So make sure your permissions are set up correctly!

Testing with the Coding Agent

To really test out the setup, we need to have the coding agent perform a task that requires it to access Azure and use the Azure MCP server. If you don’t have the Azure MCP server configured, the coding agent won’t have the MCP tools to access to Azure, but technically can still call Azure CLI commands directly.

Open your GitHub repository.
Go to the Issues tab.
Click New issue.
Set the title to “Add Resource Group List” with a description that says:

Add a list of all the Azure Resource Groups in the subscription into a Markdown file called `resourcegroups.md`. Use the `#groups` tool if it is available, otherwise, run the `az` command.

Click the Assignee button and select Copilot
Assign the issue to the Copilot coding agent.
Click the Create button.

Create and assign issue to Copilot coding agent

After a few seconds Copilot will create a new branch and a draft pull request and assign the coding agent to get to work.
You can watch the progress by selecting the new pull request and clicking the View Session button.

Copilot coding agent completed Azure Resource Groups task

This is obviously a simplified example, and a very wasteful way to create this issue, but it demonstrates the potential for using GitHub Copilot coding agents to interact with Azure resources as part of a coding workflow.

Conclusion

Setting up GitHub Copilot coding agents with secure Azure access opens up options for coding agents to do more than just code. They can now interact with Azure resources, diagnose issues, and even automate infrastructure management tasks—all while adhering to security best practices.

The federated credential approach follows enterprise security best practices while actually being easier to set up than the old “secrets everywhere” approach.

What’s Next

Ready to expand this? Consider:

Limit coding agents to just reader over dev/test environments unless you have effective controls in place (cost, auditability)
Using copilot-instructions to tell coding agents when and why to use Azure MCP tools. See custom instructions. This should include specific scenarios where Azure MCP tools are beneficial, as well as any relevant context or constraints and requirements to delete deployed resources.

Remember: the goal isn’t to replace human oversight—it’s to allow your coding agents to be able to make better code descisions or provide better debugging assistance by having access to the Azure resources they need, when they need them.

And one last time:

Keeping Azure Bicep up-to-date the easy way with GitHub Copilot Agents

2025-06-16T00:00:00Z

Nobody likes doing janitorial work, but keeping your Infrastructure as Code (IaC) up-to-date is one of those critical maintenance tasks that often gets pushed to the bottom of your to-do list. Today, I want to show you how to use GitHub Copilot Coding Agent and an experimental feature in Visual Studio Code, Prompt Files to automate one of the most tedious but important maintenance tasks: updating your Azure Verified Modules (AVM) in Bicep to their latest versions.

Although I’m using the example of Bicep with Azure Verified Modules, the concepts and techniques discussed here can be applied to other IaC tools and module repositories as well. For example, updating ARM APIs, Terraform modules, or even DSC configurations can benefit from similar automation strategies.

Why Keep Your Infrastructure as Code Up-to-Date?

Before we dive into the solution, let’s talk about why this matters. When you’re building cloud solutions, your Infrastructure as Code isn’t just configuration—it’s the foundation of your entire system. Keeping it current means:

Security patches: New versions often include critical security fixes
New features: You get access to the latest Azure capabilities and improvements
Bug fixes: Issues you might not even know exist get resolved
Performance improvements: Better resource allocation and optimization
Compliance: Meeting organizational requirements for using current versions

The problem? Manually checking and updating dozens or hundreds of module references across multiple Bicep files is time-consuming and error-prone. So we often don’t do it as frequently as we should. And the worst reason of all? We forget about it until something breaks, and then we’re scrambling to catch up. Most engineering/platform teams are always overworked and asked to prioritize new features over maintenance tasks, which means that keeping your IaC up-to-date often falls by the wayside.

What are Azure Verified Modules?

If you’re not familiar with Azure Verified Modules (AVM), they’re Microsoft’s curated collection of Infrastructure as Code modules that follow consistent patterns and best practices. They wrap up common resources together along with Microsoft Well-Architected Best practices and simplify and speed up the process of defining your IaC.

Using AVM modules in your Bicep templates (or Terraform - yes, there are Terraform versions) means you’re building on a solid, well-maintained foundation. But like any dependency, you need to keep them current to get the full benefit.

For a quick primer into AVM, check out this video:

An Introduction to Azure Verified Modules (AVM)

A Bicep module using AVM might look something like this:

module aiSearchService 'br/public:avm/res/search/search-service:0.10.0' = {
  name: 'ai-search-service-deployment'
  scope: rg
  params: {
    name: aiSearchServiceName
    location: location
    sku: azureAiSearchSku
    // Other parameters and objects as needed
    publicNetworkAccess: azureNetworkIsolation ? 'Disabled' : 'Enabled'
    semanticSearch: 'standard'
    tags: tags
  }
}

In this example, the module br/public:avm/res/search/search-service:0.10.0 is an AVM module for deploying an Azure AI Search Service. The version 0.10.0 is specified, and over time, newer versions will be released with improvements and fixes.

The Manual Update Challenge

manually updating a Bicep file with lots of AVM resources is a real chore - but it must be done. Here’s what the typical process looks like:

Open each Bicep file that uses AVM modules
Identify which modules are being used and their current versions
Go to the AVM index page to find the latest version of each module
If it’s newer, update the module version in the Bicep file
Check the module documentation to see if any parameters have changed. An easy place for things to go wrong is if you update the module version but forget to update the parameters that may have changed in the new version
Update your parameter usage if needed
Test everything to make sure it still works - Automated tests are your best friend here, but simple linting and validation checks can help catch some issues early.

Multiply this by the number of Bicep files in your project, and it quickly becomes a significant time investment. This is exactly the kind of repetitive, rule-based task that AI can help us automate.

With the amount of work required it’s lucky if this is done more than once a year and often it is just done reactively when something breaks. This is not a good situation to be in, especially when you consider the security and compliance implications of running outdated modules.

Enter GitHub Copilot Agent Mode and Prompt Files

GitHub Copilot’s Agent mode allows you to create custom prompts that can perform complex, multi-step tasks. The experimental Prompt Files feature takes this a step further by letting you create reusable, parameterized prompts that you can apply across different files and projects.

Think of Prompt Files as templates for AI tasks. You define the steps, the tools the agent can use, and the parameters it needs, then you can invoke that prompt on any file or context where it makes sense.

The AVM Update Prompt File

Here’s the Prompt File I’ve created for updating Azure Verified Modules. This file should be saved with a .prompt.md extension in the /.github/prompts/ directory of your project. For example, you could name it update-avm-modules.prompt.md.

---
mode: 'agent'
description: 'Update the Azure Verified Module to the latest version for the Bicep infrastructure as code file.'
tools: ['changes', 'codebase', 'editFiles', 'extensions', 'fetch', 'githubRepo', 'openSimpleBrowser', 'problems', 'runTasks', 'search', 'searchResults', 'terminalLastCommand', 'terminalSelection', 'testFailure', 'usages', 'vscodeAPI']
---

Your goal is to update the Bicep file `${file}` to use the latest available versions of Azure Verified Modules (AVM).
You will need to perform these steps:

1. Get a list of all the Azure Verified Modules that are used in the specific `${file}` Bicep file and get the module names and their current versions.
2. Step through each module referenced in the Bicep file and find the latest version of the module. Do this by using the `fetch` tool to get the tags list from Microsoft Container Registry. E.g. for 'br/public:avm/res/compute/virtual-machine' fetch [https://mcr.microsoft.com/v2/bicep/avm/res/compute/virtual-machine/tags/list](https://mcr.microsoft.com/v2/bicep/avm/res/compute/virtual-machine/tags/list) and find the latest version tag.
3. If there is a newer version of the module available based on the tags list from Microsoft Container Registry than is currently used in the Bicep, use the `fetch` tool to get the documentation for the module from the Azure Verified Modules index page. E.g., for `br/public:avm/res/compute/virtual-machine` the docs are [https://github.com/Azure/bicep-registry-modules/tree/main/avm/res/compute/virtual-machine](https://github.com/Azure/bicep-registry-modules/tree/main/avm/res/compute/virtual-machine)
4. Display a list of the modules with their current versions and the latest available versions, along with the documentation links for each module to the user.
5. Compare the documentation for the module to the current usage in the Bicep file and identify any changes that need to be made to the module parameters or usage.
> [!IMPORTANT]
> If the changes to the module parameters are not compatible with the current usage, are new changes that would change the behaviour, are related to security or compliance, then these changes must be reviewed and approved before being applied. So, PAUSE and ask for user input before proceeding.
6. Update the Azure Verified Module version and the resource in the Bicep file to use the latest available version and apply any relevant changes based on the documentation and including guidance from the user if required, to the module parameters.
7. If there are no changes to the module, leave it as is and make no other changes.
8. Display a table summary of the modules detected, their current versions and whether they were updated or not, and the documentation links for each module.

## IMPORTANT

- Ensure that the Bicep file is valid after the changes and that it adheres to the latest standards for Azure Verified Modules and there are no linting errors.
- Do not try to find the latest version of an Azure Verified Module by any other mechanism than fetching the tags list from Microsoft Container Registry.
- The tags list returned from Microsoft Container Registry is an array of JSON strings, so is not in version order. You will need to parse the tags and find the latest version based on the semantic versioning scheme.

You can create prompt files in the /.github/prompts directory of your project and they’ll be available to all users accessing the repository, or you can also create prompts in a prompts directory in your own User Data Directory for personal use. The latter is useful for prompts that are specific to your workflow or environment.

Breaking Down the Prompt File

Let’s take a closer look at the key sections of this prompt file:

Front Matter

Mode: Set to agent to indicate that this prompt must be run in Agent mode. It can also be set to edit or ask. When you activate a prompt,chat will automatically switch to the appropriate mode - in this case agent.
Description: A brief description of what the prompt does.
Tools: An array of MCP tools that the agent can use to perform the task. Only agent mode supports tools, and the tools you specify here will be available to the agent when it runs the prompt.

On the Tools

There are several built-in tools including fetch for retrieving data from URLs, editFiles for modifying files, and others that help with retrieving information, running tasks, and interacting with the codebase. Many Visual Studio Code extensions also provide additional tools that can be used in prompts. And if you have custom tools defined in your MCP configuration, you can include those here as well.

Selecting the right tools for the task is crucial for the agent to be successful. If the prompt file needs a tool that the user doesn’t have installed, the agent will inform them that it can’t achieve the task that it needs to, but may try to work around it or simply stop and ask the user for guidance.

The Prompt Content

The prompt starts with a clear goal: to update the specified Bicep file to use the latest versions of Azure Verified Modules. It also specifies the file to act on using the ${file} placeholder, which will be replaced with the actual file that is selected in the Copilot context.

AVM Copilot Prompt File Update Chat Context

After that, it outlines a series of steps that the agent should follow to achieve this goal. Each step is clearly defined, and explicitly instructs the agent how to achieve each task, tools to use and documentation to reference.

Prompt engineering is an art and a science. The more specific and clear you can be about the steps the agent should take, the better the results will be. Being explicit rather than assuming the agent will “just know” what to do is key to getting the desired outcome. Providing explicit references/links to documentation, APIs, or other resources also ensures the agent will have the information it needs to make informed decisions.

It outlines a series of steps that the agent should follow, including:

Parse your Bicep files to find AVM module references
Check the Microsoft Container Registry for the latest versions
Compare current vs. latest versions using semantic versioning
Fetch documentation to understand any parameter changes
Update your files with the new versions and any necessary parameter adjustments

Finally, IMPORTANT guardrails and notes are provided at the end of the prompt to ensure the agent follows best practices and doesn’t make changes that could break your infrastructure.

Setting Up and Using Prompt Files

To use this prompt file, you’ll need to:

Pre-requisites

Visual Studio Code: Make sure you have the latest version of VS Code installed.
GitHub Copilot: Ensure you have GitHub Copilot enabled in your GitHub account.
Visual Studio Code GitHub Copilot Extension: Make sure you have the GitHub Copilot extension installed in Visual Studio Code.
Visual Studio Code GitHub Copilot Chat Extension: Ensure you have the GitHub Copilot Chat extension installed in Visual Studio Code.

GitHub Copilot Chat Extension

1. Enable the Experimental Feature

First, enable Prompt Files in Visual Studio Code:

Open VS Code settings (Ctrl/Cmd + ,)
Search for “prompt files”
Enable the “Copilot: Enable Prompt Files” setting (for either the User or Workspace settings, depending on your preference)

Enabling Copilot Prompt Files in Visual Studio Code

2. Create Your Prompt File

In Visual Studio Code, create a new file named update-avm-modules.prompt.md in the /.github/prompts/ directory of your project. If you don’t have this directory, you can create it.

You can also create this by:

Pressing Ctrl/Cmd + Shift + P to open the Command Palette
Typing “Chat: New Prompt File”
Select the prompts or the User Data Folder.
Enter the name update-avm-modules (without the .prompt.md and paste the content from the prompt file above.
Save the file.

3. Run the Prompt

To use the prompt:

Open the Bicep file you want to update
Open the Copilot Chat sidebar by clicking the Copilot icon in the Activity Bar or pressing Ctrl/Cmd + Shift + I

Before you run the prompt, make sure to enable all tools that will be used by this prompt file. You can do this by clicking on the gear icon in the Copilot Chat sidebar and ticking the tools. This will ensure that the agent has access to the fetch, editFiles, and other tools it needs to perform the update.

Enabling tools in GitHub Copilot Chat Agent mode

In the Copilot Chat sidebar, type / to invoke the prompt
Select your update-avm-modules prompt
Make sure to click the bicep file you want to update in the Chat context so that it is used as the ${file} parameter in the prompt.
Select the appropriate model to use. In my experience, GPT-4.1, Gemini 2.5 Pro and Claude Sonnet 4 all work well for this task. But I have found reasoning models like o4-mini aren’t as effective for this kind of task.
Start the agent by clicking the Send button or pressing Enter

The Process in Action

When you run this prompt on a Bicep file, here’s what happens behind the scenes:

Analysis: The agent scans your Bicep file and identifies all AVM module references
Version Check: For each module, it queries the Microsoft Container Registry to get the latest available version
Comparison: It compares your current version with the latest using semantic versioning rules
Documentation Review: If an update is needed, it fetches the module documentation to understand any breaking changes
Update: It updates your Bicep file with the new version and adjusts parameters if necessary
Validation: It ensures the updated file is valid and follows current best practices

The first time you run the prompt, the agent will ask you to confirm any tool calls it needs to make, such as fetching the tags list from the Microsoft Container Registry or updating the Bicep file. This is a safety measure to ensure you have control over what the agent is doing. You can approve them for the session, workspace or globally, depending on your preference.

The entire process that might take you 30-60 minutes manually happens in just a few minutes with the AI agent.

It is possible that the agent may decide that there are significant or non-trivial changes required to a resource, in which case it will pause and ask for your feedback and direction before proceeding. This is a good thing! It means the agent is being cautious and not making potentially breaking changes without your approval.
If you examine the prompt file, you’ll notice that it includes instructions to ensure that this occurs, especially for changes that could affect security or compliance. This is a critical part of the process to ensure that the agent doesn’t make changes that could have unintended consequences.

The Importance of Test Automation

Before you start using this kind of automated updating, make sure you have solid test automation in place. I can’t stress this enough—automated infrastructure updates are only as safe as your testing strategy.

You should have:

Linting to ensure your Bicep files follow best practices and standards
Bicep validation - perform Bicep What-If analysis to ensure the changes won’t break your deployment
Integration tests that deploy to a test environment
End-to-End tests that validate the entire system works as expected after the update
Rollback procedures (ideally roll forward) in case something goes wrong

Tools like Bicep’s built-in validation, Azure Resource Manager Template Test Toolkit, and Pester tests for infrastructure can help you build a comprehensive testing strategy.

For a complete video walkthrough of this process, check out my YouTube video showing the end to end process:

Keeping your Azure Bicep up-to-date the easy way with GitHub Copilot Agents

Beyond AVM Updates: The Bigger Picture

What excites me most about this approach isn’t just the AVM updates—it’s the pattern. You can create Prompt Files for any repetitive, rule-based task that involves:

Code analysis and transformation
Documentation updates
Security vulnerability scanning and fixing
Code style standardization
Dependency management

The key is identifying tasks that are:

Well-defined: Clear steps that you can articulate
Repetitive: You do them regularly across multiple files or projects
Rule-based: They follow consistent patterns and logic
Time-consuming: They take significant effort to do manually

Wrapping Up

GitHub Copilot’s Prompt Files feature is a great advancement that helps automate routine maintenance tasks and much more. By creating reusable prompts for complex processes or even using it to plan out one-off tasks helps us think through and document the process before we actually start coding - this in itself increases the chances of success and reduces the time spent on trial and error.

The future of development tooling is moving toward AI agents that can handle increasingly complex tasks. Prompt Files give us a way to encode our domain knowledge and processes into reusable AI workflows, making our entire team more productive.

Have you tried using GitHub Copilot Agent mode for infrastructure maintenance? I’d love to hear about your experiences and the creative ways you’re using Prompt Files in your projects.

Remember: the goal isn’t to replace human judgment, but to automate the tedious parts so we can focus on the creative and strategic aspects of building great cloud solutions. Less toil, more innovation!

Deploying Foundry VTT to Azure in 5 minutes

2025-05-26T00:00:00Z

What is Foundry VTT?

If you’re a tabletop RPG enthusiast like me, you might have heard of Foundry Virtual Table Top (VTT). It’s a self-hosted, modern application for playing tabletop roleplaying games online with your friends. Foundry VTT gives you full control over your TTRPG gaming experience - and is super extensible with a vibrant community creating modules, systems, and content to enhance your games.

Many people choose to just run Foundry VTT on a machine on their home network and make it accessible via port forwarding on their router, which all works perfectly. But if you’re playing with people spread over many different locations or just like to host things in the cloud, then this is for you - that’s where Azure comes in!

Why deploy Foundry VTT to Azure?

If you’re considering cloud hosting for Foundry VTT, and happen to have access to a Microsoft Azure subscription, deploying Foundry VTT to Azure has several advantages:

Always available - Your game server is always online, ready for your players to connect anytime.
No port forwarding - You keep your home network secure without needing to expose it to the internet.
Performance - Azure’s global infrastructure ensures low latency for players around the world.
Scalability - Need more power for a bigger campaign? Scale up easily.
Storage Reliability - Your game data is stored in Azure Files, which can be replicated and backed up for durability. This also means you can leverage features of Azure storage like snapshots and backups to protect your valuable campaign data.
Security - Azure provides built-in security features like managed identities and Key Vault for sensitive information.

Of course, you may not need all of these features, and be perfectly happy running Foundry VTT on your home network. But this blog post is for those who want to take advantage of the cloud to host their Foundry VTT server.

How to deploy the solution accelerator

So, to make deploying Foundry VTT to Azure as easy as possible, I’ve created a solution accelerator that automates the entire process. It uses the Azure Developer CLI (azd) and Azure Bicep. The Azure Developer CLI is a powerful tool that simplifies the process of provisioning and managing Azure resources as well as building and deploying applications (however, in this case we don’t need to build anything). I chose to use it for this solution because it makes it simple to “stamp” out environments as well as tearing them down when they’re not needed - perfect for gaming sessions that might not run continuously - although be careful not to delete your storage account if you want to keep your game data!

Let’s walk through the process:

Prerequisites

Before we start, you’ll need:

An Azure subscription (well, obviously!)
- If you don’t have one, you can sign up for a free Azure account which gives you some credits to get started.
A valid Foundry VTT license (purchase at foundryvtt.com)
Azure Developer CLI installed
Git installed

Deployment Steps

Open a terminal or command prompt.

Clone the Foundry VTT in Azure repository:

git clone https://github.com/PlagueHO/foundryvtt-azure.git
cd foundryvtt-azure

Log in to Azure using the Azure Developer CLI:
```
azd auth login
```
The next step is to initialize the Azure Developer CLI project by setting an environment name. This is a unique name that will be used to create the Azure resources. You can choose any name you like, but it should be unique across your Azure subscription. Run the following command, replacing <UniqueEnvironmentName> with your chosen name:
```
azd init -e <UniqueEnvironmentName>
```
Configure the Foundry VTT environment variables that allow the deployment to access your Foundry VTT account to get the application distribution and license. You can do this by running the following commands, replacing the placeholders with your actual Foundry VTT credentials:
```
azd env set FOUNDRY_USERNAME "<your-foundry-username>"
azd env set FOUNDRY_PASSWORD "<your-foundry-password>"
azd env set FOUNDRY_ADMIN_KEY "<your-foundry-admin-key>"
```
The FOUNDRY_USERNAME and FOUNDRY_PASSWORD are the credentials you use to log in to Foundry VTT website. These are used to retrieve your Foundry license and download the application distribution by the Foundry VTT Docker container. They are stored in an Azure Key Vault to protect them. The FOUNDRY_ADMIN_KEY is the admin password you’ll use to log into your Foundry VTT once it is deployed.
There are also a number of optional parameters you can also configure to control the deployment (this is just a few):
```
azd env set AZURE_DEPLOY_NETWORKING "true"
azd env set AZURE_STORAGE_CONFIGURATION "Premium_100GB"
azd env set AZURE_COMPUTE_SERVICE "Web App"
azd env set AZURE_APP_SERVICE_PLAN_SKUNAME "P0v3"
```
Start the deployment:
```
azd provision
```
You will then be asked to select the Azure Subscription and the Azure region to deploy the resources to.

The commands used to deploy the Foundry VTT solution accelerator

The Azure region should be the closest to you and your players for optimal latency.

And that’s it! In about 5 minutes (give or take a minute), you’ll have a fully deployed Foundry VTT instance running in Azure. At the end of the deployment, you’ll see the URL where your Foundry VTT server is accessible displayed in the console.

See it in action

Here’s a video walkthrough of the deployment process:

Deploying Foundry VTT to Azure in 5 minutes using Azure Developer CLI

If the embed doesn’t work, you can watch the video on YouTube

Configuration options

The solution accelerator does provide a number of configuration options you can use to customize your deployment. For example, you can change the SKU of the App Service Plan, the size of the Azure Storage account, whether to deploy a virtual network, and more. These can be set using the azd env set command as shown above.

Required Parameters

FOUNDRY_USERNAME - Your Foundry VTT username.
FOUNDRY_PASSWORD - Your Foundry VTT password
FOUNDRY_ADMIN_KEY - The admin key for Foundry VTT

Optional Parameters

AZURE_COMPUTE_SERVICE - Web App (default, recommended) or Container Instance
AZURE_DEPLOY_DDB_PROXY - true (default) or false to deploy a DDB Proxy. The DDB Proxy is second Web App that runs Mr. Primates DDB-Proxy container. For more information see DDB-Proxy.
AZURE_DEPLOY_NETWORKING - true (default) or false to deploy a virtual network
AZURE_STORAGE_CONFIGURATION - Premium_100GB (default) or Standard_100GB
AZURE_STORAGE_PUBLIC_ACCESS - false (default) to restrict public access to storage, or true to allow public access
AZURE_APP_SERVICE_PLAN_SKUNAME - App Service SKU (e.g., P0v3 is default)
AZURE_DEPLOY_DIAGNOSTICS - true or false (default) to deploy diagnostics

There are other optional parameters you can set to control the deployment. You can find the full list of parameters in the GitHub repository.

What gets deployed?

When you deploy the solution accelerator, these Azure resources are provisioned:

Azure Storage Account: Hosts your Foundry VTT data files, configurations, and worlds.
Azure Web App: Hosts the Foundry VTT application using the Felddy Docker container.
Azure Key Vault: Stores sensitive information like your Foundry VTT credentials securely.
Azure Virtual Network (optional): Provides a secure network for your resources.

The solution is designed with a separation of concerns - your data storage is independent from the compute resources. This means you can rebuild, upgrade, or modify your server without risking your valuable game data.

Advanced: Deploying with GitHub Actions

For those who want to automate deployments further, the solution accelerator can be integrated with GitHub Actions. This allows you to:

Version control your infrastructure configuration
Automate deployments when you push changes
Easily deploy to multiple environments (e.g., testing and production)

To set this up, check out the instructions in the GitHub repository. It provides a step-by-step guide on how to configure GitHub Actions to deploy your Foundry VTT instance automatically whenever you push changes to your repository.

What next?

I’m planning to add support to deploy the Foundry VTT and DDB-Proxy to Azure Container Apps in the future, which will allow you to run Foundry VTT in a serverless environment. In theory this should allow you to reduce hosting costs by enabling scale-to-zero for the Foundry VTT server when it’s not in use, and scale up automatically when players connect. However, this will need some testing and evaluation to ensure it works well with the Foundry VTT application.

If you encounter any issues, have any questions or feature requests, please create an issue in the GitHub repository.

Happy gaming in Azure!

Blogging with GitHub Copilot's Virtual Teammates

2025-05-20T00:00:00Z

Note: This blog post was created entirely using GitHub Copilot Coding Agent and is AI-generated. It’s published purely for demonstration and testing purposes to explore the capabilities of GitHub Copilot Coding Agent. This transparency aligns with Responsible AI practices.

Introduction to GitHub Copilot Coding Agent

GitHub has recently taken AI coding to a new level with the GitHub Copilot Coding Agent. Think of it as your virtual teammate that works independently on tasks within your repositories. Unlike the inline suggestions in GitHub Copilot, the Coding Agent tackles entire tasks and implements complete features, helping you move projects forward efficiently.

Capabilities That Will Change Your Workflow

GitHub Copilot Coding Agent extends beyond simple code suggestions with capabilities that include:

Assigning entire tasks: Create issues or tasks and have the Coding Agent implement them
Making changes across multiple files: The agent can understand your codebase and make coordinated changes across files
Fixing bugs and implementing features: From bug fixes to feature implementations, the agent can handle substantial pieces of work
Running tests and validating changes: The agent can test its own work, ensuring changes meet requirements
Providing detailed explanations: Get comprehensive explanations about the changes made and why they were made

What makes this particularly interesting is that the agent operates within the constraints of your GitHub repository’s structure and existing patterns, creating code that aligns with your project’s style and approach.

How to Get Started with Coding Agent

Getting started with GitHub Copilot Coding Agent is simple. According to the official documentation, you’ll need:

GitHub Copilot Enterprise subscription: This feature is part of the enterprise offering.
Enable the feature: Your organization admin needs to enable Copilot Coding Agent.
Assign a task: Create an issue, then assign it to Copilot.
Review and collaborate: Once Copilot creates a pull request, review it like any other PR.

The workflow integrates seamlessly into existing GitHub processes.

The Developer and Agent Partnership

Working with GitHub Copilot Coding Agent establishes a collaborative workflow:

Task definition: You define a task or issue with clear requirements.
Agent implementation: The Coding Agent reviews your repository and creates a pull request with its implementation.
Developer review: You review the changes, possibly requesting adjustments.
Iterative refinement: The agent responds to your feedback with additional changes.

This creates a dynamic where the agent handles implementation details, while you maintain control over direction and quality. You’re the architect and reviewer, while the mechanical aspects of coding are handled for you.

Steering the Agent and Handling Roadblocks

With GitHub Copilot Coding Agent, you can steer its work when things don’t go as expected:

Provide feedback: Comment directly on the PR to clarify requirements.
Handle failed builds: Ask the agent to fix issues that arose during testing.
Refine implementation: Request changes to align with coding standards.

For example, if a build fails, you might comment:

“The build is failing because we’re missing tests for the new authentication method. Could you add unit tests for this?”

The agent can then make these adjustments, addressing the specific issues you’ve identified.

The Meta Moment: AI Writing About AI

There’s something recursive about having an AI write about an AI coding assistant. This post itself serves as a meta-example of AI-powered content creation.

It raises questions about expertise and authorship:

Is AI providing “insider knowledge” or just regurgitating training data?
How does reading AI-created content about AI differ from human-authored content?
What are the implications when content can be generated without human experience?

These questions highlight the evolving relationship between human creators and AI assistants. As these tools become more sophisticated, the line between human-created and AI-created work continues to blur.

Conclusion: An Experiment in AI Capabilities

I want to emphasize that this blog post itself is an experiment in using GitHub Copilot Coding Agent. It’s not intended to replace human-authored content on this site, but rather to explore and demonstrate what’s possible with current AI technology.

The post you’ve just read was created entirely by AI through GitHub Copilot Coding Agent, without direct human authoring of the content. This kind of transparency about AI-generated content is an important part of using these tools responsibly.

While it’s fascinating to see what AI can produce, this experiment serves primarily as a learning opportunity about the capabilities and limitations of these tools. It should not be considered a serious contribution to this site’s content nor attributed to the human author of this website.

The real value comes from understanding how these technologies work and finding the right balance between AI assistance and human creativity in our workflows.

The Process: How This Post Was Created

This blog post was created by:

Opening an issue describing the requirements for a blog post about GitHub Copilot Coding Agent
Assigning the issue to GitHub Copilot
Letting GitHub Copilot Coding Agent analyze the repository structure and existing blog posts
Reviewing the AI-generated content and providing minimal guidance
Accepting the pull request with the new blog post

The entire process was handled by GitHub Copilot Coding Agent with minimal human intervention, demonstrating the capability of AI to understand repository structure, content requirements, and writing style guidelines.

Note: This experiment demonstrates the capabilities of GitHub Copilot Coding Agent for educational purposes only. Future content on this site will continue to be human-authored.

Video Demonstration

Check out this video showing me using GitHub Copilot Coding Agent to create this very blog post (yes, it’s inception!):

Using GitHub Copilot Coding Agent to create a blog post about GitHub Coding Agent! Inception!

If the embed doesn’t work, you can watch the video on YouTube.

Using AzureDefaultCredential with Semantic Kernel in Python

2025-05-17T00:00:00Z

Background

I am working on a project for simplifying the deployment of a zero-trust secure Azure AI Foundry environment. As part of this project I am creating some Python scripts to generate synthetic data for use as sample data.

Securing with Managed Identities

It is always best practice to use managed identities to authenticate to Azure services wherever possible, especially in production. I’m very familiar with using managed identities in C# and .NET, but I haven’t used them much in Python version of Semantic Kernel before. So, it’s time to learn how to do that.

To do this, I needed to use the DefaultAzureCredential class from the Azure Identity SDK to provide a token provider to the AzureChatCompletion class from the Semantic Kernel Python SDK.

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion

azure_openai_deployment = "<your_deployment_name>" # The name of your Azure OpenAI deployment
azure_openai_endpoint = "https://<your_endpoint>.openai.azure.com" # The endpoint for your Azure OpenAI Service

kernel = sk.Kernel() # Create the Semantic Kernel instance

# Create the Entra ID token provider using the DefaultAzureCredential
# https://learn.microsoft.com/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python
token_provider = get_bearer_token_provider(
    DefaultAzureCredential(),
    "https://cognitiveservices.azure.com/.default" # The scope for Azure OpenAI Service
)

# Attach an AzureChatCompletion service to the kernel using the token provider
service = AzureChatCompletion(
    deployment_name=azure_openai_deployment,
    endpoint=azure_openai_endpoint,
    ad_token_provider=token_provider, # Pass the token provider to the service
    service_id="azure_open_ai",
)

That’s it. It is fairly straight forward to get away from using API keys to authenticate to your Azure OpenAI Service endpoints. You can see the complete implementation in the Azure AI Foundry jumpstart data generator.

Conclusion

This is a fairly simple example and probably super obvious to many people, but I couldn’t find this clearly documented anywhere, and even my trusty GenAI tooling wasn’t getting it right. I suspect this is most likely because Semantic Kernel is undergoing rapid development.

Chances are I’ll run into this again in the future, so I thought it was worth documenting. But hopefully someone else finds it useful too.

Semantic Kernel Python SDK
Azure Identity SDK for Python
DefaultAzureCredential Class in Python
Python scripts to generate synthetic data that demonstrate this approach.

Assigning Roles for Azure Enterprise Apps using Bicep

2025-05-11T00:00:00Z

Welcome back

It’s been a while since I last posted, but with the amount of things going on in the world of AI, I thought it was time to get back into the swing of things. Today, I’m going to share a problem I faced recently and how I solved it with Bicep.

The problem

I am working on a project for simplifying the deployment of a zero-trust secure Azure AI Foundry environment, an Azure AI Foundry jumpstart if you will. This project deploys the resources using Bicep, but one of the requirements is to assign the reader role to the service principal for the Enterprise Application added Azure Machine Learning.

However, this is not as straightforward as it seems. The Azure Machine Learning Enterprise Application is a multi-tenant application, and the service principal object ID is different across all tenants. This means that you cannot hardcode the Object ID in your Bicep template and you can’t use the Application ID of the Azure Machine Learning Enterprise Application either, as the role assignement requires the Object ID of the service principal in your tenant.

I’m using the Azure Machine Learning Enterprise Application as an example, but this applies to any multi-tenant application in Entra ID that you want to assign roles to in your tenant. For more information on application and service principals objects in Entra ID, see this article.

The solution

The solution to this is to use the experiemental extensibility feature of Bicep that allows you to use the Microsoft Graph Extension to work with Entra ID resources, such as service principals, groups, and users. This feature is still in preview, but it works well for this use case.

Here is a list of the steps we’re going to follow:

Enable the extensibility feature of Bicep by adding a bicepconfig.json file to your project.
Add the Microsoft Graph Bicep extension to your Bicep template.
Create the service principal resource for the Azure Machine Learning Enterprise Application.
Create the role assignment for the service principal using the id property of the service principal resource.

And now for the fun part! Let’s get started with the code.

Step 1: Enable Bicep extensibility

To enable the extensibility feature of Bicep, you need to add a bicepconfig.json file to your Bicep project. In general this file should be in the same folder as your Bicep files. If you’re using Visual Studio Code you open the Command Palette ([CTRL/CMD]+[SHIFT]+P) and select Bicep: Create Bicep Configuration File to create the file. To the bicepconfig.json file, you need to add the following settings:

"extensibility": true to the experimentalFeaturesEnabled section to enable the extensibility feature.
"microsoftGraphV1": "br:mcr.microsoft.com/bicep/extensions/microsoftgraph/v1.0:0.2.0-preview" to the extensions section to use the Microsoft Graph Bicep extension.

{
  // See https://aka.ms/bicep/config for more information on Bicep configuration options
  // Press CTRL+SPACE at any location to see Intellisense suggestions
  "analyzers": {
    "core": {
      "rules": {
        "no-unused-params": {
          "level": "warning"
        }
      }
    }
  },
  "experimentalFeaturesEnabled": {
    "extensibility": true
  },
  "extensions": {
    "microsoftGraphV1": "br:mcr.microsoft.com/bicep/extensions/microsoftgraph/v1.0:0.2.0-preview"
  }
}

Once the extensibility feature is out of preview, setting the "extensibility": true will no longer be required.

Step 2: Add the Microsoft Graph Bicep extension

At the beginning of your Bicep template, you need to add the Microsoft Graph Bicep extension. This is done by adding the following line:

// Use the Microsoft Graph Bicep extension to work with Entra ID resources
extension microsoftGraphV1

Step 3: Reference the existing service principal resource

You can use the existing Microsoft.Graph/servicePrincipals@v1.0 resource that has the appId property set to the GUID of the Azure Machine Learning Enterprise Application.

// The Service Principal of the Azure Machine Learning service.
resource azureMachineLearningServicePrincipal 'Microsoft.Graph/servicePrincipals@v1.0' existing = {
  appId: '0736f41a-0425-4b46-bdb5-1563eff02385' // Azure Machine Learning service principal
}

If you don’t know the appId of the Enterprise Application you want to assign, you can find it in the Azure portal by going to Microsoft Entra ID > Enterprise applications and searching for Azure Machine Learning. The appId is the Application ID of the Enterprise Application. You might need to remove the Application type == Enterprise Applications filter to see the Azure Machine Learning Enterprise Application in the list.

The Azure Machine Learning enterprise application in Entra ID

Step 4: Create the role assignment

Once the azureMachineLearningServicePrincipal resource is available to your Bicep template, you can use the id property of the service principal to assign the reader role to the resource for the Enterprise Application. In this example, I’m using assigning the reader role to an existing Azure AI Search service, but you can assign it to any role to any resource, existing or being created in the template.

// The existing Azure AI Search service (can be a new or existing resource).
resource azureAiSearch 'Microsoft.Search/searchServices@2025-02-01-preview' existing = {
  name: 'my-azure-ai-search'
}

// The role assignment for the Azure AI Search service to grant reader role to Azure Machine Learning.
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(subscription().subscriptionId, azureAiSearch.id, 'acdd72a7-3385-48ef-bd42-f606fba81ae7' )
  scope: azureAiSearch
  properties: {
    principalType: 'ServicePrincipal'
    principalId: azureMachineLearningServicePrincipal.id
    roleDefinitionId: 'acdd72a7-3385-48ef-bd42-f606fba81ae7' // Role definition ID for Reader role
  }
}

The complete Bicep template

The complete Bicep template looks like this:

// Use the Microsoft Graph Bicep extension to work with Entra ID resources
extension microsoftGraphV1

// The Service Principal of the Azure Machine Learning service.
resource azureMachineLearningServicePrincipal 'Microsoft.Graph/servicePrincipals@v1.0' existing = {
  appId: '0736f41a-0425-4b46-bdb5-1563eff02385' // Azure Machine Learning service principal
}

// The existing Azure AI Search service (can be a new or existing resource).
resource azureAiSearch 'Microsoft.Search/searchServices@2025-02-01-preview' existing = {
  name: 'my-azure-ai-search'
}

// The role assignment for the Azure AI Search service to grant reader role to Azure Machine Learning.
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(subscription().subscriptionId, azureAiSearch.id, 'acdd72a7-3385-48ef-bd42-f606fba81ae7' )
  scope: azureAiSearch
  properties: {
    principalType: 'ServicePrincipal'
    principalId: azureMachineLearningServicePrincipal.id
    roleDefinitionId: 'acdd72a7-3385-48ef-bd42-f606fba81ae7' // Role definition ID for Reader role
  }
}

You can find the Github Gist containing the code above here.

To see the entire complete process in action, check out the Bicep for the Azure AI Foundry Jumpstart.

Conclusion

In this post, I showed you how to assign the reader role to the service principal for the Azure Machine Learning Enterprise Application using Bicep. This is a common scenario deploying services in Azure that are consumed by multi-tenant applications (such as Azure Machine Learning). Using experiemental features in production enviornments is generally not recommended, but hopefully the extensions feature of Bicep will be available in a stable release soon. So keep an eye out for that!

Protect your Environment from Malicious Pipeline Changes in Azure DevOps

2021-01-24T00:00:00Z

I’ve recently been looking into ways to increase control and governance of continuous delivery practices when Azure DevOps Pipelines when using multi-stage YAML pipelines. My reason for investigating this area is that there are certain gaps in control when you’re just relying on “pipeline as code”.

In this post I’ll demonstrate how a number of different features in Azure and Azure DevOps can be combined to provide a very high level of control and governance over your environments. This is especially important when working in industries that require adherence to specific compliance standards and controls (e.g. health and finance).

Once you’ve read through this post you should have a good understanding of the different features in Azure DevOps that can be combined to meet whatever controls you need.

This post is not going to be an in-depth look at Azure DevOps security and permissions. That would take far too long and is not the goal of this post. However, it is important to remember that if you don’t set the appropriate permissions on the entities (branches, branch policies, service connections, variable groups, pipelines etc.) then users will be able to bypass the controls you set up. Therefor it is necessary to take security and permissions into account when you’re planning your controls.

In this post I’ll be focusing on Pipeline Approvals and how they can be enabled in different ways when combined with Environments, Service Connections, Variable Groups and Azure Key Vault.

A Potential Gap in Control

The reason I decided to write this post is that it is not as straight to protect your secrets and environments when you’re implementing Pipeline as Code in Azure DevOps.

By way of example, consider you have a Git repository with a multi-stage Azure DevOps YAML pipeline defined in it:

trigger:
  branches:
    include:
    - 'main'
pr: none

stages:
  - stage: Build
    jobs:
      - template: templates/build.yml

  - stage: QA
    displayName: 'Quality Assurance'
    jobs:
      - deployment: deploy_qa
        displayName: 'Deploy to QA'
        pool:
          vmImage: 'Ubuntu-16.04'
        variables:
          - group: 'QA Secrets'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureResourceManagerTemplateDeployment@3
                  displayName: 'Deploy Azure Resources'
                  inputs:
                    azureResourceManagerConnection: 'Azure QA'
                    subscriptionId: '<redacted>'
                    resourceGroupName: 'dsr-qa-rg'
                    location: 'East US'
                    csmFile: '$(Pipeline.Workspace)/arm/azuredeploy.json'
                    overrideParameters: '-sqlServerName dsr-qa-sql -sqlDatabaseName dsrqadb -sqlAdministratorLoginUsername $(SQLAdministratorLoginUsername) -sqlAdministratorLoginPassword $(SQLAdministratorLoginPassword) -hostingPlanName "dsr-qa-asp" -webSiteName "dsrqaapp"'

  - stage: Production
    displayName: 'Release to Production'
    jobs:
      - deployment: deploy_production
        displayName: 'Deploy to Production'
        pool:
          vmImage: 'Ubuntu-16.04'
        variables:
          - group: 'PRODUCTION Secrets'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureResourceManagerTemplateDeployment@3
                  displayName: 'Deploy Azure Resources'
                  inputs:
                    azureResourceManagerConnection: 'Azure PRODUCTION'
                    subscriptionId: '<redacted>'
                    resourceGroupName: 'dsr-production-rg'
                    location: 'East US'
                    csmFile: '$(Pipeline.Workspace)/arm/azuredeploy.json'
                    overrideParameters: '-sqlServerName dsr-production-sql -sqlDatabaseName dsrproductiondb -sqlAdministratorLoginUsername $(SQLAdministratorLoginUsername) -sqlAdministratorLoginPassword $(SQLAdministratorLoginPassword) -hostingPlanName "dsr-production-asp" -webSiteName "dsrproductionapp"'

This definition is triggered to only run on ‘main’ branch and never from a pull request. The pipeline also references Variable Groups and Service Connections, which should be considered protected resources, especially for the Production environment.

We also have an Azure DevOps Pipeline called Environment Continuous Delivery that uses the YAML file:

Azure DevOps Pipeline ‘Environment Continuous Delivery’ linked to azure-pipelines.yml

The triggers are not being overridden:

The pipeline triggers are not being overridden

The fact that the pipeline triggers are not being overridden means that the triggers defined in the YAML will always be used.

Finally, we have also locked main branch to prevent pushing code directly to it without an approved pull request. There is also a branch policy enabled that runs a simple CI build:

The branch policy and lock prevents direct commits/pushes to main branch.

However, the branch policy specifics aren’t actually important here.

So, what is the problem?

The problem is that any user may create a new branch off main and add malicious (or accidental) code to the azure-pipelines.yml. For example, if I create a new branch called malicious-change with azure-pipelines.yml changed to:

trigger:
  branches:
    include:
    - 'main'
    - 'malicious-change'
pr: none

stages:
  - stage: Build
    jobs:
      - job: Malicious_Activities
        pool:
          vmImage: 'Ubuntu-16.04'
        continueOnError: true
        variables:
          - group: 'PRODUCTION Secrets'
        steps:
          - script: echo 'Send $(SQLAdministratorLoginUsername) to Pastebin or some external location'
          - task: AzurePowerShell@5
            displayName: 'Run malicious code in Azure Production envrionment'
            inputs:
              azureSubscription: 'Azure PRODUCTION'
              ScriptType: InlineScript
              Inline: '# Run some malicious code with access to Azure Production'
              azurePowerShellVersion: latestVersion

Create a new branch with malicious changes to pipeline definition.

If we then push that new malicious-change branch to Azure DevOps Git repo, then …

… the Azure DevOps pipeline Environment Continuous Delivery will automatically execute against this new branch with malicious/dangerous pipeline changes.

The pipeline runs and has access to all resources that this pipeline normally has.

Pipeline is able to run commands with access to Azure Production resources.

Now that we know where the gaps are in our controls we can look for potential solutions.

A Less than Ideal Solution

There is a “quick and dirty” solution to this issue, but it does move us away from true “pipeline as code”. To implement this we simply need to override the triggers section in the pipeline so that it is no longer controlled by the azure-pipelines.yml:

Overriding the triggers in the pipeline prevents the triggers section in azure-pipelines.yml from being used.

Although this solution is easy to implement it means that we’re not fully defining our pipelines with code. This means that someone with permissions to edit the pipeline would need to make any changes to branch or path filters, even when they are legitimate. Plus, there is a gap in the Azure DevOps UI which prevents us from overriding pull request triggers.

Alternatively, we could use Azure DevOps security to prevent creation of new branches by unapproved users but this will limit productivity and increases complexity, so I’m not even considering this a solution worth exploring.

So, let’s look at some better ways to protect our environments, secrets and service connections.

Increasing Controls the Right Way

I’m going to increase the level of control and governance over the pipelines by implementing the following changes:

Putting secrets into an Azure Key Vault and using a service connection with approvals & checks enabled on it. We’ll then create a variable group linked to the Key Vault.
Adding approvals & checks to deployment service connections and allowing them to only be used within approved pipelines.
Defining environments with approvals & checks and using them in your pipelines.

So, lets look at each of these in more detail.

Move Secrets into a Key Vault

The first task is to create an Azure Key Vault and add all the secrets that are used in a pipeline into an Azure Key Vault. In my case, I added SQL server login details as two secrets:

My production Azure Key Vault

In my case, I have two environments, QA and PRODUCTION. So, I created a resource group and a Key Vault for each. This is so that I can implement different levels of controls over QA to PRODUCTION.

As part of this process you should also use other techniques such as governance with Azure Policy, sending logs to Azure Log Analytics to harden and protect your Key Vaults. But this is beyond the scope of this post.

Next, I need to create a Service Connection to Azure Resource Manager to the resource group I created the Key Vault in:

Take note that I limited the connection to the Resource Group and didn’t grant permission to all pipelines.

I then need to edit the security for the Service Connection to grant access to it from specific pipelines:

What we then need to do is add approvals and checks to the service connection. This will cause these checks to be run any time a pipeline tries to use the service connection:

Adding Approvals and checks to a Service Connection.

There is one approval type (getting approval from a user or group) and several checks that can be enabled:

Adding approvals and checks on a Service Connection.

Depending on the level of control you’d like to implement on each environment, you might configure these checks differently. In my case, for QA I only used a branch control to only allow the connection to run against main branch.

The Azure Key Vault QA service connection can only be accessed when run within main branch.

Take care to format the Allowed branches using the full ref path.

By enabling verify branch protection it will ensure the service connection is only available if the branch protection for the branch is enabled. It should be ticked for QA and PRODUCTION.

For, PRODUCTION, I enabled both a branch control and Approvals from a security group:

The Azure Key Vault QA service connection can only be accessed when run within main branch and also requires approval from a group.

For the Approval gate I had a group defined called Production Environment Approvers. I could have used an Active Directory group here instead. Using a group is recommended rather than specifying individual users because only a single member of each group needs to approve. See this document for more information on setting approvers.

To enforce separation of duties, make sure approvers can not approve their own runs.

The final task is to create the Variable Groups linked to the Azure Key Vault, using our service connections:

Variable groups linked to Azure Key Vaults.

To keep this post short(er), I won’t describe the exact steps here. You can get more detail on the exact process on this page.

It is important to not allow access from all pipelines.

Because we unticked the allow access to all pipelines box, it will mean the owner (or someone with enough permissions) will be asked to approve the use of the variable group and Key Vault service connection the first time the pipeline is run:

Both the Azure Key Vault QA service connection and QA Secrets variable group need to be granted permission on the first run.

Subsequent runs of this pipeline won’t require permission.

The Variable Group gates and approvals only work if linking it to an Azure Key Vault - which is another good reason to use them.

Now we have a much higher level of governance over our pipeline secrets, so let’s move on to the next improvement we can make.

Add Approvals & Checks to Service Connections

The next method we’ll implement is to add approvals & checks to our PRODUCTION (and QA) service connections. This is just the same as I did in the previous section for the Azure Key Vault service connections:

PRODUCTION Service Connection approvals and checks.

We could implement similar approvals & checks to any service connection, not just to Azure. For example, we might do this for connections to Kubernetes clusters, Service Fabric clusters, SSH or Docker hosts, or any other service.

Next, we also want to limit the service connection to only be accessible to specific pipelines, just as we did for the Key Vault connections.

Grant permissions to pipelines we want to allow this Service Connection to be used in.

We now have individual controls over secrets and resources used within our continuous delivery pipelines.

Multiple Resource Approvals

If we have a pipeline with a stage that requires access to multiple service connections or environments protected with approvals we don’t need to approve them all individually:

We can approve all gates together or individually.

However, you can only approve when you’re a member of the Approvers that were specified in the Approval gate.

Environments with Approvals & Checks

The final improvement is to make use of the Azure DevOps Environments feature. This allows us to define an environment to target when using a deployment job of an Azure DevOps Multi-stage YAML pipeline. With the environment defined, we can assign approvals & checks to that, just like we did with the Service Connections and limit permissions to the environment to specific pipelines.

An environment can be used to define deployment targets for specific resources types such as Kubernetes namespaces and Virtual Machines. However, these are not required and you can still get a good deal of value from using environments without defining resources. See this blog post for more details.

In my case, I defined two environments, one for QA and one for PRODUCTION:

PRODUCTION and QA environments do not need to contain resources, but can still add value.

Just like before, I grant permissions to the environment for specific pipelines:

Limit environments to be used by specific pipelines.

I also define approvals & checks for the PRODUCTION environment just like before, but I also added an Exclusive Lock check that will prevent more than one pipeline deploying to the PRODUCTION environment at the same time. This isn’t strictly a governance control, but will reduce the risk of conflicting deployments occurring.

Prevent multiple deployments to this environment at the same time with the Exclusive Lock.

Finally, we need to update the azure-pipeline.yml to make use of the environment and the variable group:

Setting the environment to PRODUCTION in a deployment job.

trigger:
  branches:
    include:
    - 'main'
pr: none

stages:
  - stage: Build
    jobs:
      - template: templates/build.yml

  - stage: QA
    displayName: 'Quality Assurance'
    jobs:
      - deployment: deploy_qa
        displayName: 'Deploy to QA'
        pool:
          vmImage: 'Ubuntu-16.04'
        environment: 'QA'
        variables:
          - group: 'QA Secrets'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureResourceManagerTemplateDeployment@3
                  displayName: 'Deploy Azure Resources'
                  inputs:
                    azureResourceManagerConnection: 'Azure QA'
                    subscriptionId: '72ad9153-ecab-48c9-8a7a-d61f2390df78'
                    resourceGroupName: 'dsr-qa-rg'
                    location: 'East US'
                    csmFile: '$(Pipeline.Workspace)/arm/azuredeploy.json'
                    overrideParameters: '-sqlServerName dsr-qa-sql -sqlDatabaseName dsrqadb -sqlAdministratorLoginUsername $(SQLAdministratorLoginUsername) -sqlAdministratorLoginPassword $(SQLAdministratorLoginPassword) -hostingPlanName "dsr-qa-asp" -webSiteName "dsrqaapp"'

  - stage: Production
    displayName: 'Release to Production'
    jobs:
      - deployment: deploy_production
        displayName: 'Deploy to Production'
        pool:
          vmImage: 'Ubuntu-16.04'
        environment: 'PRODUCTION'
        variables:
          - group: 'PRODUCTION Secrets'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureResourceManagerTemplateDeployment@3
                  displayName: 'Deploy Azure Resources'
                  inputs:
                    azureResourceManagerConnection: 'Azure PRODUCTION'
                    subscriptionId: '72ad9153-ecab-48c9-8a7a-d61f2390df78'
                    resourceGroupName: 'dsr-production-rg'
                    location: 'East US'
                    csmFile: '$(Pipeline.Workspace)/arm/azuredeploy.json'
                    overrideParameters: '-sqlServerName dsr-production-sql -sqlDatabaseName dsrproductiondb -sqlAdministratorLoginUsername $(SQLAdministratorLoginUsername) -sqlAdministratorLoginPassword $(SQLAdministratorLoginPassword) -hostingPlanName "dsr-production-asp" -webSiteName "dsrproductionapp"'

We can now get now also get a single view of all deployments to an environment:

All environment deployments to PRODUCTION.

Because environments aren’t defined across projects, this is another reason to limit the number of Azure DevOps projects you’re creating. See my previous blog post on 12 Things you Should Know when Implementing Azure DevOps in your Organization.

Putting it all together

Now that we’ve completed all these additional checks and approvals, let’s see what happens when we attempt to get some malicious changes to run inside our Environment Deployment Pipeline:

After creating a new branch called Malicious_Activites off main with adjustments to azure-pipelines.yml the build fails.

As we can see from the screenshot above, the following things have happened:

The Environment Continuous Delivery pipeline was triggered automatically by our commit to the new Malicious_Activities branch. This was expected and is the same as before.
This time all our Branch control checks on the Service Connections that were maliciously trying to be accessed have caused the build to fail because this is not main branch.
The Approvals to access the service connections have been requested still, but because I created the commit that triggered this, I can’t approve them. This results in implementation of separation of duties control.

For a member of the Production Environment Approvers group it looks like this:

Approval allowed, even though the job has failed.

Even after the approving the job checks will still fail and the job won’t proceed. So, this means our PRODUCTION environment has been protected.

If we run the pipeline against main branch (either manually or via a commit via a Pull Request) then we will get the standard approvals:

QA checks passed automatically, and PRODUCTION Branch controls have passed. PRODUCTION approval is waiting.

A Quick Note About Approvals

By default approval notifications will be e-mailed to anyone who is in an Approval list. You can disable this by configuring your Notifications:

Enable/Disable Run stage waiting for approval notifications.

You can also choose to have notifications delivered to you in Microsoft Teams, if you use it. This is the best way to experience these features and you’re less likely to miss an important approval.

Wrapping Up

It is important to remember that all of these controls and methods are optional. If you don’t need this level of control and governance over your environments then you shouldn’t add the complexity that goes with it. That said, it is always good to know what you can do with the tools, even if you don’t need to use it.

I hope you found this (long, but hopefully not too long) post useful.

AKS Announcements Roll-up from Microsoft Ignite 2020

2020-09-23T00:00:00Z

There were a whole lot of announcements around Azure Kubernetes Service (AKS) at Ignite 2020. I thought I’d quickly sum them all up and provide links:

Brendan Burn’s post on AKS Updates

A great summary of recent investments in AKS from Kubernetes co-creator, Brendan Burns.

Preview: AKS now available on Azure Stack HCI

AKS on Azure Stack HCI enables customers to deploy and manage containerized apps at scale on Azure Stack HCI, just as they can run AKS within Azure.

Public Preview: AKS Stop/Start Cluster

Pause an AKS cluster and pick up where they left off later with a switch of a button, saving time and cost.

GA: Azure Policy add on for AKS

Azure Policy add on for AKS allows customers to audit and enforce policies to their Kubernetes resources.

Public Preview: Confidential computing nodes on Azure Kubernetes Service

Azure Kubernetes Service (AKS) supports adding DCsv2 confidential computing nodes on Intel SGX.

GA: AKS support for new Base image Ubuntu 18.04

You can now create Node Pools using Ubuntu 18.04.

GA: Mutate default storage class

You can now use a different storage class in place of the default storage class to better fit their workload needs.

Public preview: Kubernetes 1.19 support

AKS now supports Kubernetes release 1.19 in public preview. Kubernetes release 1.19 includes several new features and enhancements such as support for TLS 1.3, Ingress and seccomp feature GA, and others.

Public preview: RBAC for K8s auth

With this capability, you can now manage RBAC for AKS and its resources using Azure or native Kubernetes mechanisms. When enabled, Azure AD users will be validated exclusively by Azure RBAC while regular Kubernetes service accounts are exclusively validated by Kubernetes RBAC.

Public Preview: VSCode ext. diag+periscope

This Visual Studio Code extension enables developers to use AKS periscope and AKS diagnostics in their development workflow to quickly diagnose and troubleshoot their clusters.This Visual Studio Code extension enables developers to use AKS periscope and AKS diagnostics in their development workflow to quickly diagnose and troubleshoot their clusters.

Enhanced protection for containers

Enhanced protection for containers: As containers and specifically Kubernetes are becoming more widely used, the Azure Defender for Kubernetes offering has been extended to include Kubernetes-level policy management, hardening and enforcement with admission control to make sure that Kubernetes workloads are secured by default. In addition, container image scanning by Azure Defender for Container Registries will now support continuous scanning of container images to minimize the exploitability of running containers

Learn more about Microsoft Defender, Azure Defender and Azure Sentinel.

There may indeed been more, and I’ll update them as they come to hand. Hope this roll up helps.

Head over to https://myignite.microsoft.com and watch some of the AKS content to get even an even better view of the updates.

12 Things you Should Know when Implementing Azure DevOps in your Organization

2020-09-19T00:00:00Z

Azure DevOps is a really fantastic part of any DevOps tool chain. But when you’re first starting out with it in an organization, there are a few things you should know that will make it even better… and avoid making some doing some things you’ll later regret. These tips are most important if you’re implementing it across multiple teams or in a medium to large organization. Even if you’re implementing it in a small start-up, most of these tips will still help.

These tips are all based on my experience with implementing and using Azure DevOps, Visual Studio Online (VSO) and Visual Studio Team Services (VSTS). These are all things I wish I’d known earlier as they would have saved me time, made my life easier or kept me more secure. They are also just my opinion, so I encourage you to investigate further and decide what is best for you in your situation/environment.

This is by no means an exhaustive list either and they are in no particular order.

So, let’s get into it:

1. Projects: less is better

Less projects are better

Most things (work items, repositories, pipelines etc.) in Azure DevOps are organized into containers called Projects. It is tempting to try to break your work into lots of small projects (e.g. one for each library, or one per team, or one per department). This results in a lot of management overhead trying to keep everything organized and adds little value in most cases (there are exceptions). Implementing a project per team or a project per software component is usually wrong.

Recommendation: The less projects you have the better. Use Area Paths (covered next) to organize work in your project.

Documentation Reference: When to add another project

2. Area Paths: Organize work

Organizing Area Paths

Area Paths allow you to divide the work items and test plans into a hierarchy to make them easier to manage. Teams can be assigned to one or more area paths.

Area’s are easily moved around, so they are much better suited to arranging your work by software component/product and organizational hierarchies.

Recommendation: For your project, set up Area Paths and assign teams to them.

Documentation Reference: Define area paths for your project

3. Identity: Integrate with Azure AD

Connect AAD.

If you are using Azure AD as your primary identity source for your organization, then you should connect your Azure DevOps organization to Azure AD. This will allow your Azure AD identities to be used within Azure DevOps.

If you aren’t using Azure AD, but have Active Directory, consider setting up hybrid identity with Azure AD.

You should manage access to your Azure DevOps organization and to projects and other resources (e.g. Service Connections) using Azure AD Groups. I’d also strongly recommend reading the documentation on security groups and permissions for Azure DevOps as there are a lot of nuance to these and they deserve an entire post on their own.

Recommendation: Use Azure AD as the identity source for Azure DevOps. Create and manage users and security groups within Azure AD.

Documentation Reference: Connect organization to Azure Active Directory, Access with Active Directory Groups.

4. Git or TFVC?

Importing a TFVC repository as Git

This might be controversial, but it shouldn’t be. Unless you have a legacy TFVC repository that you need to keep around for historic/support reasons or some tools that only support TFVC that you can’t live without (or can’t replace) then you should be using Git as your version control system.

If you do have legacy TFVC repositories that you need to bring over, consider importing them as Git repositories.

Recommendation: Use Git. Make sure all your teams know how to use Git well.

Documentation Reference: When to add another project

5. Create a Sandbox Project

Create a Sandbox Project

You and your teams will often need a place to experiment and learn about Azure DevOps safely. A sandbox project is a great place to do this. You can create a sandbox project and give teams higher levels of permissions over it project to allow them to experiment with different settings (for example try out an alternate area path structure)

Don’t confuse a sandbox project with a project for building proof-of-concepts/experimental code: you should not use a Sandbox project for creating anything that could end up in production or anything that has any value. Content in sandbox projects often accidentally get deleted.

Recommendation: Create a Sandbox Project. Assign an image and description for the project to make it clear that it is a Sandbox and what it can be used for.

Documentation Reference: When to add another project

6. Install extensions… wisely

The Azure DevOps Extensions Marketplace

The Azure DevOps marketplace is filled with many great extensions that really enhance the value of Azure DevOps. It is well worth browsing through the extensions created by both Microsoft, Microsoft DevLabs and the hundreds of 3rd party ones to really experience the full power of Azure DevOps.

You should set up a formal process around validating, on-boarding and off-boarding extensions from your organization. It is all too easy to end up with “extension sprawl” that results in a management nightmare, especially if you have strict security or governance practices (you might be familiar with this if you’ve ever managed Jenkins within a large organization).

It is also tempting to install pipeline extensions for any minor task that you might want to execute in a CI/CD pipeline. But you should consider if the governance & management of a task is worth the time that might be saved using it, especially when a short Bash or PowerShell task might do just as well.

Recommendation: Install important extensions from marketplace. Formalize a process for validating, on-boarding and off-boarding extensions.

Documentation Reference: Azure DevOps marketplace, Install Azure DevOps Extension

7. Use Multi-stage YAML Pipelines

Do NOT use “Classic Editor” and create pipelines without YAML

Early on the evolution of Azure DevOps pipelines, all pipelines had to created using a visual designer. The structure of this visually designed pipeline was not stored in code, rather it was stored separately without proper version control. This is no longer the case.

You should always create new build pipelines using YAML and store them in your repository with your source code (pipeline as code). You can still use the assistant to help you design your YAML:

Click Show Assistant to edit your YAML.

The exception is release pipelines which don’t support YAML and being stored in version control.

Recommendation: Create all pipelines a multi-stage YAML pipelines.

Documentation Reference: Define pipelines using YAML syntax

9. Release Pipelines… in code?

Release Pipelines are awesome, but are they worth missing out on pipeline as code?

Release pipelines don’t support YAML. However, in many cases you don’t need release pipelines. Instead, you can use your multi-stage YAML build pipeline to release your software as well by adding a deployment job. This also aligns much more closely to GitHub, where there is no concept of a Release Pipeline and would make moving to GitHub Actions much easier should you want to.

As of writing this post, there are two key feature that are missing from YAML build pipelines: Gates and Deployment Group jobs. Also, the release pipeline visualization and dashboard widgets are quite useful, so you may prefer these over the build pipeline visualization. But in my opinion the visualization is not worth losing version control over your pipeline.

Recommendation: Use multi-stage YAML pipeline deployments if you don’t need Gates or Deployment Group Jobs. Use conditions to determine if a deployment job should be executed. Use approvals and checks on the environment to control deployment.

Documentation Reference: Deployment Job, Conditions, Approvals, Environments.

10. Deployment Group Agents?

Add a machine to a Deployment Group.

If the applications you are building and releasing need to be deployed to a physical or virtual machine (e.g. not to a Kubernetes cluster or managed service) that is not accessible by an Azure DevOps Hosted agent, then you can use a Deployment Group agent.

This is just the Azure DevOps Hosted agent installed onto the machine and registered with Azure DevOps as a Deployment Group agent in a Deployment Group. Deployment Group agents only require outbound connectivity to Azure DevOps services.

This is a good solution if you’re deploy to machines on-premises or on machines where inbound internet connectivity is blocked, but outbound internet is allowed.

Recommendation: If you have to deploy your application to a machine that is not accessible from Azure DevOps Microsoft Hosted Agents.

Documentation Reference: Deployment Group agent, Deployment Group

11. Automate. Automate

Get a list of Azure DevOps projects using Azure DevOps CLI

Just like most other Microsoft tools, you can automate them from the command line using either PowerShell, CMD or Bash (or even REST API). If you have to perform repetitive tasks in Azure DevOps, you might want to consider automating these processes.

This is also a good way to control certain processes and practices, such as creating Service Connections from code in a repository, or rolling secrets in a Library Variable Group.

You can also use these tools to interact with Azure DevOps from within Azure DevOps pipelines, leading to some interesting techniques such as release orchestrations (beyond the scope of this doc).

Recommendation: Use Azure DevOps CLI or the VSTeams PowerShell module (created by Donovan Brown) to automate Azure DevOps. Alternatively, use Azure DevOps REST API.

Documentation Reference: Azure DevOps CLI, VSTeam PowerShell module, Azure DevOps REST API.

12. Get Practical Experience

The best way to learn Azure DevOps is to get hands-on practical experience. Azure DevOps Labs provides free hands-on labs environments (via your own DevOps organization) and covers practically everything you could ever want to know. The Azure DevOps content on Microsoft Learn also has detailed walk throughs of the product and processes.

Making sure everyone in your organization has the skills/knowledge to work with Azure DevOps will help them be more successful and happy.

Recommendation: Do some of the hands-on labs and complete some Microsoft Learn learning pathways.

Documentation Reference: Azure DevOps Labs, Azure DevOps content on Microsoft Learn

Wrapping Up

There are definitely lots more recommendations and considerations I could suggest, especially security and DevOps best-practices but to keep this (reasonably) short, I’ll leave them for another post.

I hope you find this useful and it helps you avoid some of my mistakes.

Automate on-boarding Azure Log Analytics Container Monitoring of any Linux Docker Host using Azure Arc

2020-08-23T00:00:00Z

That title is a bit of a mouthful, but this post will show how easy it is to configure a Linux Docker host to be monitored by Azure Monitor.

Azure Monitor can be used to monitor machines that are running in Azure, in any cloud or on-premises. For a machine to be monitored by Azure Monitor, it needs to have the Microsoft Monitoring Agent (MMA) installed. The machine either needs to be able to connect to Azure directly or via a Log Analytics Gateway.

But to make things a lot easier, we’re going to set up the Docker host to allow it to be managed in Azure using Azure Arc. This will allow Azure Arc to install MMA for us. The Linux Docker host will appear in the Azure portal like other Azure resources:

Azure Arc managed machines running outside of Azure

We will also add the Container Monitoring solution to our Azure Monitor Log Analytics workspace. The Container Monitoring solution will set up the Log Analytics workspace to record telemetry data from your Linux Docker host and add a container monitoring dashboard.

Container Monitoring Solution dashboard in an Azure Monitor Log Analytics Workspace

To enable collection and sending of telemetry from all containers running on the Docker host, a microsoft/oms container is run on it. This Docker container will connect to the Azure Monitor Log Analytics workspace and send logs and performance counters from all Docker containers running on the host.

Once we have completed the configuration of the Docker Host, the following telemetry will be sent to your Log Analytics workspace:

Host diagnostics/logs.
Host performance metrics.
Diagnostics/logs from any Docker containers on the host.
Performance metrics from any Docker containers on the host.

The cost of sending this telemetry to your Azure Monitor Log Analytics workspace will depend on the volume of data ingested. You can control this by reducing the frequency with which performance counters are transmitted. By default this is sent every 60 seconds. You can configure this through the Log Analytics workspace.

Configuring performance counter frequency

What you need

These instructions assumes you have the following:

An Azure account - get a free account here.
A Resource Group to contain the machines you register with Azure Arc - instructions on how to create one.
A Log Analytics Workspace - instructions on how to create one. This is where all the diagnostic and metric data will be sent from the Docker hosts.
Azure Cloud Shell enabled - how to use Azure Cloud Shell.
SSH access to your Docker host to connect to Azure Arc.
Important: the Linux host you are going to install MMA on must have Python installed. If it is not installed you will receive an “Install failed with exit code 52 Installation failed due to missing dependencies.” error when you install the MMA Agent.

Connect Linux Host to Azure Arc

The first step is to connect our Linux host to Azure Arc so that we can use it to perform all the other steps directly from the Azure Portal. We are going to use a service principal for onboarding the machine as this will make it easier to automate.

We are going to run this Azure Arc Onboarding script generator PowerShell script in Azure Cloud Shell to create the Service Principal and generate the Linux Shell script for us. It can also generate a PowerShell script for onboarding Windows machines to Azure Arc.

Open Azure Cloud Shell and ensure you’re using PowerShell.

Download the script by running:

Invoke-WebRequest -Uri https://gist.githubusercontent.com/PlagueHO/64a2fd67489ea22b3ca09cd5bf3a0782/raw/Get-AzureArcOnboardingScript.ps1 -OutFile ~\Get-AzureArcOnboardingScript.ps1

Run the script by executing the following command and setting the TenantId, SubscriptionId, Location and ResourceGroup parameters:
```
./Get-AzureArcOnboardingScript.ps1 -TenantId '<TENANT ID>' -SubscriptionId '<SUBSCRIPTION ID>' -Location '<LOCATION>' -ResourceGroup '<RESOURCE GROUP>'
```
You will need to get your Tenant ID from the Azure Portal. The Subscription Id and Resource Group is the subscription and resource group respectively to register the machine in. The Location is the Azure region that the machine metadata will be stored.
Copy the script that was produced. We will execute it on any Linux machine we want to onboard.
SSH into the Linux Host and run (paste) the script:

In a real production environment you’d probably automate this process and you’d also need to protect the secrets in the script.
Once the installation is complete, the machine will appear in the Azure Portal in the resource group:

Now that the machine is onboarded into Azure Arc, we can use it to install Microsoft Monitoring Agent (MMA) and then run the microsoft/oms Docker container.

Further Improvements: we could easily have used something like PowerShell DSC or Ansible to apply the Azure Arc onboarding configuration to the machine, but this is beyond the scope of this post. In a fully mature practice, there would be no need for logging directly into the host at any point in this process.

Installing MMA with Azure Arc

At the time of writing this blog post, there wasn’t an Azure PowerShell module or AzCLI extension for Azure Arc. So automating this process right now will require the use of an ARM template:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "MachineName": {
            "type": "String"
        },
        "Location": {
            "type": "String"
        },
        "WorkspaceId": {
            "type": "String"
        },
        "WorkspaceKey": {
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.HybridCompute/machines/extensions",
            "apiVersion": "2019-12-12",
            "name": "[concat(parameters('MachineName'), '/OMSAgentForLinux')]",
            "location": "[parameters('Location')]",
            "properties": {
                "publisher": "Microsoft.EnterpriseCloud.Monitoring",
                "type": "OmsAgentForLinux",
                "autoUpgradeMinorVersion": true,
                "settings": {
                    "workspaceId": "[parameters('WorkspaceId')]"
                },
                "protectedSettings": {
                    "workspaceKey": "[parameters('WorkspaceKey')]"
                }
            }
        }
    ]
}

Important: Before you attempt this step, make sure the machine you are deploying MMA to has Python installed on it. If it is not installed you will receive an “Install failed with exit code 52 Installation failed due to missing dependencies.” error when you install the MMA Agent.

To apply the ARM Template in Azure Cloud Shell:

Run this command to download the ARM Template:

Invoke-WebRequest -Uri https://gist.githubusercontent.com/PlagueHO/74c5035543c454daf3d28f33ea91cde0/raw/AzureArcLinuxMonitoringExtensions.json -OutFile ~\AzureArcLinuxMonitoringExtensions.json

Apply the ARM Template to an Azure Arc machine by running this command (replacing the values in the strings):

New-AzResourceGroupDeployment `
  -ResourceGroupName '<NAME OF RESOURCE GROUP CONTAINING ARC MACHINES>' `
  -TemplateFile ~/AzureArcLinuxMonitoringExtensions.json `
  -TemplateParameterObject @{
    MachineName = '<NAME OF AZURE ARC MACHINE>'
    Location = '<LOCATION OF AZURE ARM MACHINE>'
    WorkspaceId = '<WORKSPACE ID OF LOG ANALYTICS WORKSPACE>'
    WorkspaceKey = '<WORKSPACE KEY OF LOG ANALYTICS WORKSPACE>'
  }

You can get the WorkspaceId and WorkspaceKey values by locating your Log Analytics Workspace in the Azure Portal and clicking Agents Management in the side bar.

Important: If you’re automating this, you’ll want to take care not to expose the Workspace Key.

You can navigate to the Azure Arc Machine resource in the Azure Portal and select the extension to see that it is “creating”. It will take a between 5 and 10 minutes before installation of the extension is completed.
Once installation has completed, you can navigate to your Azure Monitor Log Analytics Workspace, click Agents Management in the side bar and select Linux Agents. You should notice that the number of agents has increased:
Clicking Go to logs will show all Linux Machines that Azure Monitor Log Analytics has received a Heartbeat from:

Enable Container Telemetry

So far, so good. We’ve onboarded the machine to Azure Arc and enabled host logging to a Azure Monitor Log Analytics workspace. However, we’re only getting telemetry data from the host, not any of the containers. So the next thing we need to do is execute the following command on the host:

sudo docker run --privileged -d -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers -e WSID="<YOUR WORKSPACE ID>" -e KEY="<YOUR WORKSPACE KEY>" -h=`hostname` -p 127.0.0.1:25225:25225 --name="omsagent" --restart=always microsoft/oms:1.8.1-256

This will download and run the microsoft/oms container image on the host and configure it to send telemetry for all containers running on this host to your Azure Monitor Log Analytics workspace.

Important: If you are installing onto Ubuntu server, you can avoid problems in this stage by making sure you’ve installed Docker using the official Docker repository and instructions. I had used Snap on Ubuntu 18.05, which resulted in this error ‘Error response from daemon: error while creating mount source path ‘/var/lib/docker/containers’: mkdir /var/lib/docker: read-only file system.’ when running the script.

The way to automate the installation of this on the host is to again use an ARM Template, but this time use the Linux Custom Script extension to execute the above command. You can see the ARM Template here. This ARM template could easily be combined into the ARM template from the preceding stage, but I kept them separate for the purposes of showing the process.

Run this command to download the ARM Template:

Invoke-WebRequest -Uri https://gist.githubusercontent.com/PlagueHO/c3f09056cace496dded18da8bc1ed589/raw/AzureArcLinuxCustomScriptExtensions.json -OutFile ~\AzureArcLinuxCustomScriptExtensions.json

Apply the ARM Template to an Azure Arc machine by running this command (replacing the values in the strings with the same ones as before):

New-AzResourceGroupDeployment `
  -ResourceGroupName '<NAME OF RESOURCE GROUP CONTAINING ARC MACHINES>' `
  -TemplateFile ~/AzureArcLinuxCustomScriptExtensions.json `
  -TemplateParameterObject @{
    MachineName = '<NAME OF AZURE ARC MACHINE>'
    Location = '<LOCATION OF AZURE ARM MACHINE>'
    WorkspaceId = '<WORKSPACE ID OF LOG ANALYTICS WORKSPACE>'
    WorkspaceKey = '<WORKSPACE KEY OF LOG ANALYTICS WORKSPACE>'
  }

After a few minutes installation of the CustomScript extension should have completed and should show Succeeded in the Azure Portal.
If you SSH into the Linux Container host and run sudo docker ps you will see that the omsagent container is running:

The process is now complete and we’re getting telemetry from both the host and the containers running on it. We only needed to log into the host initially to onboard it into Azure Arc, but after that all other steps were performed by Azure. We could have performed the onboarding using automation as well and that would be the recommended pattern to use in a production environment.

Configure Performance Counters Sample Interval

The final (and optional) step is to configure sample interval that performance counters will be collected on each Linux host. To do this:

Open the Azure Portal.
Navigate to your Azure Monitor Log Analytics Workspace.
Click Advanced Settings:
Select Data, then Linux Performance Counters:
Configure the Sample Interval and click Save.

The updated counter sample interval will be updated in the Microsoft Monitoring Agent configuration on the host.

See It All In Action

Now that everything is all set up, let’s see what it looks like in Azure Monitor.

Open the Azure Portal.
Navigate to your Azure Monitor Log Analytics Workspace.
Click Workspace Summary in the side bar.
Click Container Monitoring Solution in the workspace overview:
You can now browse through the Container Monitoring Solution dashboard and see your hosts are being monitored as well as see performance information from your containers:

It really is fairly easy to get set up and once configured will give you much greater visibility over your entire estate, no matter where it is running.

Enable AKS Azure Active Directory integration with a Managed Identity from an ARM template

2020-08-08T00:00:00Z

When you’re deploying an Azure Kubernetes Service (AKS) cluster in Azure, it is common that you’ll want to integrate it into Azure Active Directory (AAD) to use it as an authentication provider.

The original (legacy) method for enabling this was to manually create a Service Principal and use that to grant your AKS cluster access to AAD. The problem with this approach was that you would need to manage this manually and as well as rolling worry about rolling secrets.

More recently an improved method of integrating your AKS cluster into AAD was announced: AKS-managed Azure Active Directory integration. This method allows your AKS cluster resource provider to take over the task of integrating to AAD for you. This simplifies things significantly.

You can easily do this integration by running PowerShell or Bash scripts, but if you’d prefer to use an ARM template, here is what you need to know.

You will need to have an object id of an Azure Active Directory group to use as your Cluster Admins.
```
$clusterAdminGroupObjectIds = (New-AzADGroup `
    -DisplayName "AksClusterAdmin" `
    -MailNickname "AksClusterAdmin").Id
```
This will return the object Id for the newly create group in the variable $clusterAdminGroupObjectIds. You will need to pass this variable into your ARM template.

You need to add an aadProfile block into the properties of your AKS cluster deployment definition:

For example:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "clusterAdminGroupObjectIds": {
            "defaultValue": [],
            "type": "array",
            "metadata": {
                "description": "Array of Azure AD Group object Ids to use for cluster administrators."
            }
        }
    },
    "resources": [
        {
            "name": "MyAksCluster",
            "type": "Microsoft.ContainerService/managedClusters",
            "apiVersion": "2020-04-01",
            "location": "eastus",
            "properties": {
                "kubernetesVersion": "1.18.4",
                "enableRBAC": true,
                "aadProfile": {
                    "managed": true,
                    "adminGroupObjectIds": "[parameters('clusterAdminGroupObjectIds')]",
                    "tenantId": "[subscription().tenantId]"
                }
                // Other cluster properties here...
            },
            "identity": {
                "type": "SystemAssigned"
            }
        }
    ]
}

When you deploy the ARM template (using whatever method you choose), you’ll need to pass the $clusterAdminGroupObjectIdsas a parameter. For example:

New-AzResourceGroupDeployment `
    -ResourceGroupName 'MyAksWithAad_RG `
    -TemplateFile 'ArmTemplateAksClusterWithManagedAadIntegration.json' `
    -TemplateParameterObject @{
        clusterAdminGroupObjectIds = @( $clusterAdminGroupObjectIds )
    }

That is all you really need to get AKS-managed AAD integration going with your AKS cluster.

For a fully formed ARM template for that will deploy an AKS cluster with AKS-managed AAD integration plus a whole lot more, check out this ARM template. It will deploy an AKS cluster including the following:

A Log Analytics workspace integrated into the cluster with Cluster Insights and Diagnostics.
A VNET for the cluster nodes.
An ACR integrated into the VNET with Private Link and Diagnostics into the Log Analytics Workspace.
Multiple node pools spanning availability zones:
- A system node pool including automatic node scaler.
- A Linux user node pool including automatic node scaler.
- A Windows node pool including automatic node scaler and taints.

Thanks for reading.

Refactoring PowerShell - Switch Statements

2019-10-05T00:00:00Z

Regardless of your experience within technology, the process of creating code usually starts out with a solution that is just enough to get the job done. The solution is then typically tweaked and improved continuously until it is either “production worthy” or “good enough for the requirements”. This process of improvement is called refactoring. Refactoring is a skill that all technical professionals should become proficient in, regardless if you are an IT Pro, a developer or just someone who needs to automate things.

There are many reasons to refactor code, including:

Add new features
Remove unused features
Improve performance
Increase readability, maintainability or test-ability
Improve design
Improve adherence to standards or best practices

Refactoring in Code Reviews

One of the jobs of a code reviewer is to suggest areas that could be refactored to improve some of the areas above. I’ve often found that I’ll suggest the same set of refactorings in my reviews. So rather than just putting the suggesting into the code review, I thought I’d start writing them down here in a series that I could refer contributors to as well as help anyone else who happens to come across these.

Unit Tests and Refactoring

Because refactoring requires changing code, how can we be sure that we’re not breaking functionality or introducing bugs? That is where unit testing comes in. With PowerShell, we’ll typically use the PowerShell Pester module to create unit tests that allow us to more safely refactor code. Unit testing is beyond the scope of this post.

Switch Statement Refactoring

One of the common patterns I’ve come across is PowerShell switch statements being used is to map from one values to another set of values. For example:

$colourName = 'green'

$colourValue = switch ($colourName) {
    'red' { 0xFF0000; break }
    'green' { 0x00FF00; break }
    'blue' { 0x0000FF; break }
    'white' { 0xFFFFFF; break }
    default { 0x0 }
}

return $colourValue

This converts a colour name (e.g. red, green, blue, white) to a colour value. If a colour name can not be matched then it returns 0x0 (black). Admittedly, this is a bit of an unlikely example, but it demonstrates the pattern.

This is by no means incorrect or “bad code”. It is completely valid and solves the problem perfectly well. But as you can see this requires a lot of code to perform a simple mapping.

Mapping using a Hash Table

An alternative to using a switch statement is to use a hash table:

$colourName = 'green'

$colourMap = @{
    red = 0xFF0000
    green = 0x00FF00
    blue = 0x0000FF
    white = 0xFFFFFF
}
$colourValue = $colourMap[$colourName]

return $colourValue

This can simplify the code slightly by removing the break statement and braces.

Note: The break statement is not strictly required in this example from a functional perspective, but including them increases overall performance of the switch statement.

You may have noticed that the hash table above does not quite match the behavior of the switch statement: the default mapping to 0x0 is not handled. So, in this case, we’d need to include additional code to handle this:

$colourName = 'green'

$colourMap = @{
    red = 0xFF0000
    green = 0x00FF00
    blue = 0x0000FF
    white = 0xFFFFFF
}
$colourValue = ($colourMap[$colourName], 0x0, 1 -ne $null)[0] # Null Coalescing

return $colourValue

To handle the default we’re using a quasi null coalescing operator. PowerShell doesn’t have a native null coalescing operator like many languages, but it does have a way of simulating it using the line:

$notNull = ($x, $y, 1 -ne $null)[0] # Null Coalescing in PowerShell

You could definitely argue that using a hash table mapping with a null coalescing operator does not make the code easier to read or maintain. But the purpose here is not to define which approach is best, rather to offer alternative patterns.

One other benefit of using a hash table for mapping is that it can be separated out into a separate psd1 file. This allows editing of the mapping table elements without having to edit the code itself:

$colourName = 'green'
$colourMap = Import-LocalizedData -FileName mapping.psd1
$colourValue = ($colourMap[$colourName], 0x0, 1 -ne $null)[0] # Null Coalescing
return $colourValue

The psd1 file containing the mapping data (mapping.psd1):

@{
    red = 0xFF0000
    green = 0x00FF00
    blue = 0x0000FF
    white = 0xFFFFFF
}

Reversing the Mapping

How do we use a similar pattern to reverse the mapping? For example, mapping a colour value back to a colour name:

$colourValue = 0x00FF00

$colourName = switch ($colourValue) {
    0xFF0000 { 'red'; break }
    0x00FF00 { 'green' ; break }
    0x0000FF { 'blue'; break }
    0xFFFFFF { 'white'; break }
    default { 'none' }
}

return $colourName

To implement the same functionality using a hash table also including the null coalescing operator you could use:

$colourValue = 0x00FF00

$colourMap = @{
    0xFF0000 = 'red'
    0x00FF00 = 'green'
    0x0000FF = 'blue'
    0xFFFFFF = 'white'
}
$colourName = ($colourMap[$colourValue], 0x0, 1 -ne $null)[0] # Null Coalescing

return $colourName

Using a Hash Table with Script Values

Switch blocks may contain more than just a single statement. For example:

$VerbosePreference = 'Continue'
$action = 'New'
$path = 'c:\somefile.txt'

$result = switch ($action) {
    'New' {
        Write-Verbose -Message 'Execute New-Item'
        New-Item -Path $path
        break
    }

    'Remove' {
        Write-Verbose -Message 'Execute Remove-Item'
        Remove-Item -Path $path
        break
    }

    'Get' {
        Write-Verbose -Message 'Execute Get-Item'
        Get-Item -Path $path
        break
    }

    Default {
        Write-Verbose -Message 'Invalid Action'
    }
}

return $result

If your switch blocks do more than just perform a mapping, you can assign script blocks to the hash table values instead:

$VerbosePreference = 'Continue'
$action = 'New'
$path = 'c:\somefile.txt'

$actions = @{
    'New' = {
        Write-Verbose -Message 'Execute New-Item'
        New-Item -Path $path
    }

    'Remove' = {
        Write-Verbose -Message 'Execute Remove-Item'
        Remove-Item -Path $path
    }

    'Get' = {
        Write-Verbose -Message 'Execute Get-Item'
        Get-Item -Path $path
    }
}

return ($actions[$action], {Write-Verbose -Message 'Invalid Action'}, 1 -ne $null)[0].Invoke()

Instead of containing a value in each hash table item, a script block is specified. The Invoke() method can then be called on the script block.

Enumerated Types

If you’re using PowerShell 5.0 or above (you hopefully are), you’re also able to use the enum keyword to define an enumerated type that can also be used to replace switch statements in some situations.

$colourName = 'green'

# Only needs to be declared once
enum colour {
    red = 0xFF0000 
    green = 0x00FF00
    blue = 0x0000FF
    white = 0xFFFFFF
}

return ([colour] $colourName).value__

The enumerated type only needs to be declared once.

But what do we need to do if we want to have a default returned if the value is invalid in the mapping? In that case we need to use the TryParse method of the enumerated type to try and parse the value and return a default value if it is invalid:

$colourName = 'grey'

# Only needs to be declared once
enum colour {
    red = 0xFF0000 
    green = 0x00FF00
    blue = 0x0000FF
    white = 0xFFFFFF
}

$colourValue = [colour]::white

if (-not [colour]::TryParse($colourName, $true, [ref] $colourValue)) {
    return 0x0
} else {
    return $colourValue.value__
}

However, we can’t assign scriptblocks to the values in an enumerated type - only constant values may be assigned. This means we can’t implement scenarios where we’d like to have the value contain more than one instruction. But this shouldn’t be too much of a problem, because if you do find yourself being limited by this, then you should probably be looking to use more advanced object oriented programming patterns such as polymorphism - which is well beyond the scope of this post. But if you’re interested to know more, review this article (not PowerShell specific).

Wrapping Up

This post is all about looking at different ways of writing the same code. It isn’t trying to say which way is better or worse. If you have a preference and it works for you, by all means, keep on using it. This is simply to provide alternative methods that may or may not make code more readable and maintainable.

Feel free to comment with alternative methods of refactoring switch statements.

Deploy Sonarqube to Azure App Service Linux Containers using an Azure DevOps Pipeline

2019-06-03T00:00:00Z

Update 2020-10-12: I have updated the 101-webapp-linux-sonarqube-azuresql Azure Resource Manager quick start template to default to 7.7-community edition and prevent deployment of versions that aren’t currently compatible with Azure App Service Web App Containers.

Update 2020-10-09: It was pointed out to me that the process in this post had stopped working. The container was not starting up correctly. Upon investigation I found that this was because newer versions of the Sonarqube container includes ElasticSearch which requires additional heap memory to be assigned. Therefore the latest versions of Sonarqube can’t be used with this process. I am working on a full resolution to this issue, but in the meantime ensure you’re only using Sonarqube 7.7-community edition. I have updated the ARM template to no longer default to latest for the sonarqubeImageVersion parameter. There is also an issue in GitHub against the ARM template.

Sonarqube is a web application that development teams typically use during the application development process to continuous validate the quality of the code.

This post is not specifically about Sonarqube and how it works. It is intended to show Developers & IT Pros how to deploy a service to Azure using contemporary infrastructure as code and DevOps patterns.

The Implementation

A Sonarqube installation is made up of a web application front end backed by database.

Sonarqube supports many different types of databases, but I chose to use Azure SQL Database. I decided to use Azure SQL Database for the following reasons:

It is a managed service, so I don’t have to worry about patching, securing and looking after SQL servers.
1. I can scale the database performance up and down easily with code. This allows me to balance my performance requirements with the cost to run the server or even dial performance right back at times when the service is not being used.
2. I can make use of the new Azure SQL Database serverless (spoiler alert: there are still SQL servers). This allows the SQL Database to be paused when not being accessed by the Sonarqube front end. It can be used to further reduce costs running Sonarqube by allowing me to delete the front end every night and only pay for the storage costs when developers aren’t developing code.

For the front end web application I decided to use the Azure Web App for Containers running a Linux container using the official Sonarqube Docker image. Because the Sonarqube web application is stateless it is a great target for being able to be delete and recreate from code. The benefits to using Azure Web App for Containers are:

Azure Web App for Containers is a managed service, so again, no patching, securing or taking care of servers.
I can scale the performance up and down and in and out from within my pipeline. This allows me to quickly and easily tune my performance/cost, even on a schedule.
I can delete and rebuild my front end web application by running the pipeline in under 3 minutes. So I can completely delete my front end and save money when it is not in use (e.g. when teams aren’t developing in the middle of the night).

Architectural Considerations

The Sonarqube web application, as it has been architected, is accessible from the public internet. This might not meet your security requirements, so you might wish to change the architecture in the following ways:

Putting an Azure Application Gateway (a layer 7 router) in front of the service.
Isolate the service in a Azure Virtual Network from the internet and make it only accessible to your development services. This may also require Azure ExpressRoute or other VPN technologies to be used.
We are using the SQL Server administrator account for the Sonarqube front end to connect to the backend. This is not advised for a production service - instead, a user account specifically for the use of Sonarqube should be created and the password stored in an Azure Key Vault.

These architectural changes are beyond the scope of this document though as I wanted to keep the services simple. But the pattern defined in this post will work equally well with these architectures.

Techniques

Before we get into the good stuff, it is important to understand why I chose to orchestrate the deployment of these services using an Azure Pipeline.

I could have quite easily built the infrastructure manually straight into the Azure Portal or using some Azure PowerShell automation or the Azure CLI, so why do it this way?

There are a number of reasons that I’ll list below, but this is the most mature way to deploy applications and services.

I wanted to define my services using infrastructure as code using an Azure Resource Manager template.
I wanted the infrastructure as code under version control using Azure Repos. I could have easily used GitHub here or one of a number of other Git repositories, but I’m using Azure Repos for simplicity.
I wanted to be able to orchestrate the deployment of the service using a CI/CD pipeline using Azure Pipelines so that the process was secure, repeatable and auditable. I also wanted to parameterize my pipeline so that I could configure the parameters of the service (such as size of the resources and web site name) outside of version control. This would also allow me to scale the services by tweaking the parameters and simply redeploying.
I wanted to use a YAML multi-stage pipeline so that the pipeline definition was stored in version control (a.k.a. pipeline as code). This also enabled me to break the process of deployment into two stages:
- Build - publish a copy of the Azure Resource Manager templates as an artifact.
- Deploy to Dev - deploy the resources to Azure using the artifact produced in the build.

Note: I’ve made my version of all these components public, so you can see how everything is built. You can find my Azure DevOps repository here and the Azure Pipeline definition here.

Step 1 - Create a project in Azure DevOps

First up we need to have an Azure DevOps organization. You can sign up for a completely free one that will everything you need by going here and clicking start free. I’m going to assume you have your DevOps organization all set up.

In your browser, log in to your Azure DevOps organization.
Click + Create project to create a new project.
Enter a Project Name and optionally a Description.
Select Public if you want to allow anyone to view your project (they can’t contribute or change it). Otherwise leave it as Private to make it only visible to you.
Click Create.

You’ve now got an Azure Repo (version control) as well as a place to create Azure Pipelines as well as a whole lot of other tools, such as Azure Boards, that we’re not going to be using for this project.

Step 2 - Add ARM Template Files to the Repo

Next, we need to initialize our repository and then add the Azure Resource Manager (ARM) template files and the Azure Pipeline definition (YAML) file. We’re going to be adding all the files to the repository directly in the browser, but if you’re comfortable using Git, then I’d suggest using that.

Select Repos > Files from the nav bar.
Make sure Add a README is ticked and click Initialize.
Click the ellipsis (…) next to the repo name and select Create a new folder.
Set the Folder name to infrastructure. The name matters because the pipeline definition expects to find the ARM template files in that folder.
Enter a checkin comment of “Added infrastructure folder”.
Click Create.
Once the folder has been created, we need to add two files to it:
- sonarqube.json - The ARM template representing the infrastructure to deploy.
- sonarqube.parameters.json - The ARM template default parameters.
Click here to download a copy of the sonarqube.json. You can see the content of this file here.
Click here to download a copy of the sonarqube.parameters.json. You can see the content of this file here.
Click the ellipsis (…) next to the infrastructure folder and select Upload file(s).
Click the Browse button and select the sonarqube.json and sonarqube.parameters.json files you downloaded.
Set the Comment to something like “Added ARM template”.
Ensure Branch name is set to master (it should be if you’re following along).
Click Commit.

We’ve now got the ARM Template in the repository and under version control. So we can track any changes to them.

Note: When we created the infrastructure folder through the Azure DevOps portal a file called _PlaceHolderFile.md was automatically created. This is created because Git doesn’t allow storing empty folders. You can safely delete this file from your repo if you want.

Step 3 - Create your Multi-stage Build Pipeline

Now that we’ve got a repository we can create our mulit-stage build pipeline. This build pipeline will package the infrastructure files and store them and then perform a deployment. The multi-stage build pipeline is defined in a file called azure-pipelines.yml that we’ll put into the root folde of our repository.

Click here to download a copy of the azure-pipelines.yml. You can see the content of this file here.
Click the ellipsis (…) button next to the repository name and select Upload file(s).
Click Browse and select the azure-pipelines.yml file you dowloaded.
Set the Comment to something like “Added Pipeline Defnition”.
Click Commit.
Click Set up build button.
Azure Pipelines will automatically detect the azure-pipelines.yml file in the root of our repository and configure our pipeline.
Click the Run button. The build will fail because we haven’t yet created the service connection called Sonarqube-Azure to allow our pipeline to deploy to Azure. We also still still need to configure the parameters for the pipeline.

Note: I’ll break down the contents of the azure-pipelines.yml at the end of this post so you get a feel for how a multi-stage build pipeline can be defined.

Step 4 - Create Service Connection to Azure

For Azure Pipelines to be able to deploy to Azure (or access other external services) it needs a service connection defined. In this step we’ll configure the service sonnection called Sonarqube-Azure that is referred to in the azure-pipelines.yml file. I won’t go into too much detail about what happens when we create a service connection as Azure Pipelines takes care of the details for you, but if you want to know more, read this page.

Important: This step assumes you have permissions to create service connections in the project and permissions to Azure to create a new Serivce Principal account with contributor permissions within the subscription. Many users won’t have this, so you might need to get a user with the enough permissions to the Azure subscription to do this step for you.

Click the Project settings button in your project.
Click Service connections under the Pipelines section.
Click New service connection.
Select Azure Resource Manager.
Make sure Service Principal Authentication is selected.
Enter Sonarqube-Azure for the Connection name. This must be exact, otherwise it won’t match the value in the azure-pipelines.yml file.
Set Scope level to Subscription.
From the Subscription box, select your Azure Subscription.
Make sure the Resource group box is empty.
Click OK.
An authorization box will pop up requesting that you authenticate with the Azure subscription you want to deploy to.
Enter the account details of a user who has permissions to create a Service Principal with contributor access to the subscription selected above**.**

You now have a service connection to Azure that any build pipeline (including the one we created earlier) in this project can use to deploy services to Azure.

Note: You can restrict the use of this Service connection by changing the Roles on the Service connection. See this page for more information.

Step 5 - Configure Pipeline Parameters

The ARM template contains a number of parameters which allow us to configure some of the things about the Azure resources we’re going to deploy, such as the Location (data center) to deploy to, the size of the resources and the site name our Sonarqube service will be exposed on.

In the azure-pipelines.yml file we configure the parameters that are passed to the ARM template from pipeline variables. Note: There are additional ARM template parameters that are exposed (such as sqlDatabaseSkuSizeGB and sonarqubeImageVersion), but we’ll leave the configuration of those parameters as a seprate exercise.

The parameters that are exposed as pipeline variables are:

siteName - The name of the web site. This will result in the Sonarqube web site being hosted at [siteName].azurewebsites.net.
sqlServerAdministratorUsername - The administrator username that will be used to administer this SQL database and for the Sonarqube front end to connct using. Note: for a Production service we should actually create another account for Sonarqube to use.
sqlServerAdministratorPassword - The password that will be used by Sonarqube to connect to the database.
servicePlanCapacity - The number of App Service plan nodes to use to run this Sonarqube service. Recommend leaving it at 1 unless you’ve got really heavy load.
servicePlanPricingTier - This is the App Service plan pricing tier to use for the service. Suggest S1 for testing, but for systems requiring greater performance then S2, S3, P1V2, P2V2 or P3V2.
sqlDatabaseSkuName - this is the performance of the SQL Server. There are a number of different performance options here and what you chose will need to depend on your load.
location - this is the code for the data center to deploy to. I use WestUS2, but chose whatever datacenter you wish.

The great thing is, you can change these variables at any time and then run your pipeline again and your infrastructure will be changed (scaled up/down/in/out) accordingly - without losing data. Note: You can’t change location or siteName after first deployment however.

To create your variables:

Click Pipelines.
Click the SonarqubeInAzure pipeline.
Click the Edit button.
Click the menu button (vertical ellipsis) and select Variables.
Click the Add button and add the following parameters and values:
- siteName - The globally unique name for your site. This will deploy the service to [siteName].azurewebsites.net. If this does not result in a globally unique name an error will occur during deployment.
- sqlServerAdministratorUsername - Set to sonarqube.
- sqlServerAdministratorPassword - Set to a strong password consisting of at least 8 characters including upper and lower case, numbers and symbols. Make sure you click the lock symbol to let Azure DevOps know this is a password and to treat it accordignly.
- servicePlanCapacity - Set to 1 for now (you can always change and scale up later).
- servicePlanPricingTier - Set to S1 for now (you can always change and scale up later).
- sqlDatabaseSkuName - Set to GP_Gen5_2 for now (you can always change and scale up later). If you want to use the SQL Serverless database, use GP_S_Gen5_1, GP_S_Gen5_2 or GP_S_Gen5_4.
- location - set to WestUS2 or whatever the code is for your preferred data center.
You can also click the Settable at Queue time box against any of the parameters you want to be able to set when the job is manually queued.
Click the Save and Queue button and select Save.

We are now ready to deploy our service by triggering the pipeline.

Step 6 - Run the Pipeline

The most common way an Azure Pipeline is going to get triggered is by committing a change to the repository the build pipeline is linked to. But in this case we are just going to trigger a manual build:

Click Pipelines.
Click the SonarqubeInAzure pipeline.
Click the Run pipeline.
Set any of the variables we want to change (for example if we wanted to scale up our services).
Click Run.
You can then watch the build and deploy stages complete.

Your pipeline should have completed and your resources will be on thier way to being deployed to Azure. You can rerun this pipeline at any time with different variables to scale your services. You could even delete the front end app service completely and use this pipeline to redeploy the service again - saving lots of precious $$$.

Step 7 - Checkout your new Sonarqube Service

You can login to the Azure Portal to see the new resource group and resources that have been deployed.

Open the Azure portal and log in.
You will see a new resource group named [siteName]-rg.
Open the [siteName]-rg.
Select the Web App with the name [siteName].
Click the URL.
Your Sonarqube application will open after a few seconds.
Note: It may take a little while to load the first time depending on the performance you configured on your SQL database.
Log in to Sonarqube with the username admin and the password admin.
Remember to change this password immediately.

You are now ready to use Sonarqube in your build pipelines.

Step 8 - Scaling your Sonarqube Services

One of the purposes of this process was to enable the resources to be scaled easily and non-destructively. All we need to do is:

Click Pipelines.
Click the SonarqubeInAzure pipeline.
Click Run pipeline.
Adjust any variables to scale the service up, down, in, or out.
Click Run.
Watch the build and deploy stages complete.

Of course you could do a lot of the scaling with Azure Automation, which is a better idea in the long term than using your build pipeline to scale the services because you’ll end up with hundreds of deployment records over time.

A Closer look at the Multi-stage Build Pipeline YAML

At the time of writing this post, the Multi-stage Build Pipeline YAML was relatively new and still in a preview state. This means that it is not fully documented. So, I’ll break down the file and highlight the interesting pieces:

Trigger

This section ensures the pipeline will only be triggered on changes to the master branch.

Stages

This section contains the two stages: Build and Deploy. We could have as many stages as we like. For example: Build, Deploy Test, Deploy Prod.

Build Stage

This defines the steps to run in the build stage. It also requires the execution of the stage on an Azure DevOps agent in the vs2017-win2016 pool.

Build Stage Checkout Step

This step causes the repository to be checked out onto the Azure DevOps agent.

Build Stage Publish Artifacts Step

This step takes the infrastructure folder from the checked out repository and stores it as an artifact that will always be accessible as long as the build record is stored. The artifact will also be made available to the next stage (the Deploy Stage). The purpose of this step is to ensure we have an immutable artifact available that we could always use to redeploy this exact build.

Deploy Stage

The deploy stage takes the artifact produced in the build stage and deploys it. It runs on an Azure DevOps agent in the vs2017-win2016 pool.

It also specifies that this is a deployment to an environment called “dev”. This will cause the environment to show up in the environments section under pipelines in Azure DevOps.

The strategy and runOnce define that this deployment should only execute once each time the pipeline is triggered.

Deploy Stage Azure Resource Group Deployment Step

This deploy step takes the ARM template from the infrastructure artifact and deploys it to the Sonarqube-Azure Service connection. It overrides the parameters (using the overrideParameters) property using build variables (e.g. $(siteName), $(servicePlanCapacity)).

But what about Azure Blueprints?

One final thing to consider: this deployment could be a great use case for implementing with Azure Blueprints. I would strongly suggest taking a look at using your build pipeline to deploy an Azure Blueprint containing the ARM template above.

Thank you very much for reading this and I hope you found it interesting.

Installing the Resource

If you have installed WMF 5.0 you can just download this directly from the PowerShell Gallery by running this command:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2019\06\2019-06-03-deploy-sonarqube-to-azure-app-service-linux-containers-using-an-azure-devops-pipeline.md
Install-Module -Name Az -AllowClobber -Scope CurrentUser

You can use the Azure CLI as well if you prefer:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2019\06\2019-06-03-deploy-sonarqube-to-azure-app-service-linux-containers-using-an-azure-devops-pipeline.md
az login
az group create --name <resource-group-name> --location <location>
az deployment group create --resource-group <resource-group-name> --template-file infrastructure/sonarqube.json --parameters infrastructure/sonarqube.parameters.json

Azure Resource Manager Templates Hands-on Lab and #GlobalAzure 2019

2019-05-04T00:00:00Z

Recently I helped organize and present at the 2019 Global Azure Bootcamp in Auckland. The Global Azure Bootcamp is an huge event run by Azure communities throughout the world all on the same day every year. It is an opportunity for anyone with an interest in Azure to come and learn from experts and presenters and share their knowledge. If you’re new to Azure or even if you’re an expert it is well worth your time to attend these free events.

The Global Azure Bootcamp is also an awful lot of fun to be a part of and I got to meet some fantastic people!

https://twitter.com/i/status/1122002366041968640

We also got to contribute to the Global Azure Bootcamp Science lab, which was a really great way to learn Azure as well as contribute to the goal of finding potential exosolar planets (how cool is that?). A global dashboard was made available where all locations could compare their contributions. The Auckland Team did fantastically well, given the size of Auckland comparatively: We managed to get 8th on the team leaderboard:

Hands-On Workshop Material

As part of my session this year, I produced a Hands-on workshop and presentation showing attendees the basics of using Azure Resource Manager templates as well as some of the more advanced topics such as linked/nested templates and security.

The topics covered are:

I’ve made all of this material open and free for the community to use to run your own sessions or modify and improve.

You can find the material in GitHub here:

https://github.com/PlagueHO/Workshop-ARM-Templates

Thanks for reading and hope to see some of you at a future Global Azure Bootcamp!

Allow Integer Parameter to Accept Null in a PowerShell Function

2019-04-06T00:00:00Z

One of the great things about PowerShell being based on .NET is that we get access to the huge number of types built into the framework.

A problem I came across today was that I needed to have a function that took a mandatory integer parameter, but that parameter needed to allow Null. In .NET, there is a generic type System.Nullable<T> that allows other types to take on a null value.

Here’s how I solved it:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2019\04\2019-04-06-allow-integer-parameter-to-accept-null-in-a-powershell-function.md
function Set-AdapterVlan {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        $Adapter,

        [Parameter(Mandatory = $true)]
        [AllowNull()]
        [Nullable[System.Int32]]
        $VlanId
    )
    
    if ($null -eq $VlanId) {
        # If VlanId is null, clear the VLAN ID from the adapter
        $Adapter | Set-VMNetworkAdapterVlan -Untagged
    }
    else {
        # Otherwise, set the VLAN ID
        $Adapter | Set-VMNetworkAdapterVlan -VlanId $VlanId -Access
    }
}

This allows me to call the function above like this:

Set-AdapterVlan -Adapter $adapter -VlanId $null

Which will clear the VLAN ID from the virtual network adapter.

The magic is in the parameter definition:

[Parameter(Mandatory = $true)]
[AllowNull()]
[Nullable[System.Int32]]
$VlanId

The [AllowNull()] attribute allows the $VlanId parameter to accept a null even though it is mandatory, and the [Nullable[System.Int32]] allows $VlanId to be assigned a null value.

This isn’t something I use often, but I thought it was worth sharing!

Enable CORS Support in Cosmos DB using PowerShell

2018-12-26T00:00:00Z

Support for Cross-Origin Resource Sharing (CORS) was recently added to Cosmos DB. If you want to enable CORS on an existing Cosmos DB account or create a new Cosmos DB account with CORS enabled it is very easy to do with Azure Resource Manager (ARM) templates or the Azure Portal.

But what if you’re wanting to find out the state of the CORS setting on an account or set it using PowerShell? Well, look no further.

The Cosmos DB PowerShell module (version 3.0.0 and above) supports creating Cosmos DB accounts with CORS enabled as well as updating and removing the CORS headers setting on an existing account. You can also retrieve the CORS setting for an existing Cosmos DB account.

Installing the CosmosDB Module

The first thing you need to do is install the CosmosDB PowerShell module from the PowerShell Gallery by running this in a PowerShell console:

Install-Module -Name CosmosDB -MinimumVersion 3.0.0.0

This will also install the Az PowerShell modules Az.Accounts and Az.Resources modules if they are not installed on your machine. The *-CosmosDbAccount functions in the CosmosDB module are dependent on these modules.

The CosmosDB PowerShell module and the Az PowerShell modules are completely cross-platform and support Linux, MacOS and Windows. Running in either Windows PowerShell (Windows) or PowerShell Core (cross-platform) is supported.
Versions of the CosmosDB PowerShell module earlier than 3.0.0.0 use the older AzureRm/AzureRm.NetCore modules and do not support the CORS setting.

Authenticating to Azure with ‘Az’

Before using the CosmosDB PowerShell module accounts functions to work with CORS settings you’ll first need to authenticate to Azure using the Az PowerShell Modules. If you’re planning on automating this process you’ll want to authenticate to Azure using a Service Principal identity.

If you’re using this module in an Azure DevOps build/release pipeline the Azure PowerShell task will take care of the Service Principal authentication process for you:

But if you’re just doing a little bit of experimentation then you can just use an interactive authentication process.

To use the interactive authentication process just enter into your PowerShell console:

Connect-AzAccount

then follow the instructions.

Create a Cosmos DB Account with CORS enabled

Once you have authenticated to Azure, you can use the New-CosmosDbAccount function to create a new account:

New-CosmosDbAccount `
  -Name 'dsrcosmosdbtest' `
  -ResourceGroupName 'dsrcosmosdbtest-rgp' `
  -Location 'westus' `
  -AllowedOrigin 'https://www.fabrikam.com','https://www.contoso.com'

This will create a new Cosmos DB account with the name dsrcosmosdbtest in the resource group dsrcosmosdbtest-rgp in the West US location and with CORS allowed origins of https://www.fabrikam.com and https://www.contoso.com.

The New-CosmosDbAccount command assumes the resource group that is specified in the ResourceGroup parameter already exists and you have contributor access to it. If the resource group doesn’t exist then you can create it using the New-AzResourceGroup function or some other method.

It will take Azure a few minutes to create the new Cosmos DB account for you.

But if you want your PowerShell automation or script to be able to get on and do other tasks in the meantime, then add the -AsJob parameter to the New-CosmosDbAccountcall. This will cause the function to immediately return and provide you a Job object that you can use to periodically query the state of the Job. More information on using PowerShell Jobs can be found here.
Be aware, you won’t be able to use the Cosmos DB account until the Job is completed.

If you look in the Azure Portal, you will find the new Cosmos DB account with the CORS allowed origin values set as per your command:

Get the CORS Allowed Origins on a Cosmos DB Account

Getting the current CORS Allowed Origins value on an account is easy too. Just run the following PowerShell command:

(Get-CosmosDbAccount `
  -Name 'dsrcosmosdbtest' `
  -ResourceGroupName 'dsrcosmosdbtest-rgp').Properties.Cors.AllowedOrigins

This will return a string containing all the CORS Allowed Origins for the Cosmos DB account dsrcosmosdbtest.

You could easily split this string into an array variable by using:

$corsAllowedOrigins = (Get-CosmosDbAccount `
  -Name 'dsrcosmosdbtest' `
  -ResourceGroupName 'dsrcosmosdbtest-rgp').Properties.Cors.AllowedOrigins -split ','

Update the CORS Allowed Origins on an existing Cosmos DB Account

To set the CORS Allowed Origins on an existing account use the Set-CosmosDbAccount function:

Set-CosmosDbAccount `
  -Name 'dsrcosmosdbtest' `
  -ResourceGroupName 'dsrcosmosdbtest-rgp' `
  -AllowedOrigin 'http://www.mycompany.com'

This will take a few minutes to update. So you can use the -AsJob parameter to run this as a Job.

Remove the CORS Allowed Origins from an existing Cosmos DB Account

You can remove the CORS Allowed Origins setting by setting using the Set-CosmosDbAccount function but passing in an empty string to the AllowedOrigin parameter:

Set-CosmosDbAccount `
  -Name 'dsrcosmosdbtest' `
  -ResourceGroupName 'dsrcosmosdbtest-rgp' `
  -AllowedOrigin ''

This will take a few minutes to update as well. As always, you can use the -AsJob parameter to run this as a Job.

Final Words

Hopefully, you can see it is fairly simple to automate and work with the Cosmos DB CORS Allowed Origins setting using the PowerShell Cosmos DB module.

If you have any issues or queries or would like to contribute to the PowerShell Cosmos DB module, please head over to the GitHub repository.

Use Pester to Test Azure Resource Manager Templates for Best Practices

2018-10-13T00:00:00Z

Recently I came across the amazing Secure DevOps Kit for Azure (AzSK). This contains a really useful AzSK PowerShell Module that contains cmdlets for performing different types of security scanning on Azure Resources, Subscriptions, and Resource Manager Templates.

The feature of this module that I was most interested in for my current project was being able to scan ARM templates for best practice violations.

To install the module, open a PowerShell Window and run:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2018\10\2018-10-13-use-pester-to-test-azure-resource-manager-templates-for-best-practices.md
Install-Module -Name AzSK

Important: At the time of writing this post, the AzSK module has dependencies on the AzureRM.Profile and other AzureRM.* PowerShell modules. As of December 2018, the AzureRM.* PowerShell Modules are going to be renamed to Az.* (see this post). The AzureRM and Az modules cannot be installed side-by-side, so if you’ve installed the Az PowerShell modules on your system, then the installation of AzSK will fail because the AzureRM modules will also be installed and a conflict will occur.

The cmdlet we’re most interested in is the Get-AzSKARMTemplateSecurityStatus. It can be used to scan one or more ARM templates or entire folders of ARM templates for best practice violations:

This will scan the ARM templates and produce a CSV report in a folder Microsoft\AzSKLogs\ARMChecker within your $ENV:LOCALAPPDATA folder and open the folder in Explorer. This isn’t ideal for automation scenarios or using during Continuous Integration or Continuous Delivery pipelines. I’ve raised an issue with the AzSK team on GitHub to see if this can be improved.

In my case, I wanted to be able to use the PowerShell Pester Module, a PowerShell testing framework, to execute tests on the output and then use the NUnit output Pester generates to publish into a Continuous Integration pipeline. To do that, I needed to create a custom test script that would take the CSV report, count the failures of each level (High, Medium, or Low), and fail if any are counted in the specific level.

This is what the script looks like:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2018\10\2018-10-13-use-pester-to-test-azure-resource-manager-templates-for-best-practices.md
<#PSScriptInfo
.VERSION 1.0.0
.GUID bf41177f-4d1e-481a-a126-5f0c07dd9aae
.AUTHOR Daniel Scott-Raynsford
.COMPANYNAME
.COPYRIGHT (c) 2018 Daniel Scott-Raynsford. All rights reserved.
.TAGS AzSK, Pester, Test
.LICENSEURI https://gist.github.com/PlagueHO/1af35ee65a2276ca90b3a8a5b224a5d4
.PROJECTURI https://gist.github.com/PlagueHO/1af35ee65a2276ca90b3a8a5b224a5d4
.ICONURI
.EXTERNALMODULEDEPENDENCIES
.REQUIREDSCRIPTS
.EXTERNALSCRIPTDEPENDENCIES
.RELEASENOTES First version.
.PRIVATEDATA 2016-Datacenter,2016-Datacenter-Server-Core
#>

#requires -Modules @{ ModuleName="AzSK"; ModuleVersion="3.6.1" }
#requires -Modules @{ ModuleName="Pester"; ModuleVersion="4.3.0" }
<#
    .SYNOPSIS
    Pester test for validating ARM template meets best-practices

    .DESCRIPTION
    This Pester test will validate one or more ARM templates in the specified
    file path to validate that they meet the best practices.

    .PARAMETER TemplatePath
    The full path to the ARM template to check. This may be a path with
    wild cards to check multiple files.

    .PARAMETER Severity
    An array of severity values that will count as failed tests. Any violation
    found in the ARM template that matches a severity in this list will cause
    the Pester test to count as failed. Defaults to 'High' and 'Medium'.

    .PARAMETER SkipControlsFromFile
    The path to a controls file that can be used to suppress rules.
#>
[CmdletBinding()]
param (
    [Parameter(Mandatory = $true)]
    [System.String]
    $TemplatePath,

    [Parameter()]
    [System.String[]]
    $Severity = @('High', 'Medium'),

    [Parameter()]
    [System.String]
    $SkipControlsFromFile
)

Describe 'ARM template best practices' -Tag 'AzSK' {
    Context 'When AzSK module is installed and run on all files in the Templates folder' {
        $resultPath = Get-AzSKARMTemplateSecurityStatus `
            -ARMTemplatePath $TemplatePath `
            -Preview:$true `
            -DoNotOpenOutputFolder `
            -SkipControlsFromFile $SkipControlsFromFile `
            -Recurse
        $resultFile = (Get-ChildItem -Path $resultPath -Filter 'ARMCheckerResults_*.csv')[0].FullName

        It 'Should produce a valid CSV results file ' {
            $resultFile | Should -Not -BeNullOrEmpty
            Test-Path -Path $resultFile | Should -Be $true
            $script:resultsContent = Get-Content -Path $resultFile | ConvertFrom-Csv
        }

        $groupedResults = $script:resultsContent | Where-Object -Property Status -EQ 'Failed' | Group-Object -Property Severity

        $testCases = $Severity.Foreach({ @{ Severity = $_ } })

        It 'Should have 0 failed Severity:<Severity> results' -TestCases $testCases {
            param ([System.String] $Severity)
            $failedCount = $groupedResults.Where({ $_.Name -eq $Severity })[0].Count
            $failedCount | Should -Be 0
        }
    }
}

You can download the script from GitHub Gist directly or get it from the PowerShell Gallery by running:

Install-Script -Name AzSKARMTemplateSecurityStatus.Test

To use it, you will need to install Pester 4.3.0 and AzSK 3.6.1 modules:

Install-Module -Name Pester -MinimumVersion 4.3.0
Install-Module -Name AzSK -MinimumVersion 3.6.1

Once that is done, you can use Invoke-Pester and pass in the TemplatePath and Severity parameters to the test script:

Invoke-Pester -Script @{ Path = 'd:\Invoke-AzSKARMTemplateSecurityStatusPesterTest.ps1'; Parameters = @{ TemplatePath = 'D:\101-webapp-basic-windows\azuredeploy.json' }}

This will execute the Pester tests in the file above on the specified ARM template. The tests will fail when there are any best practice violations with the specified Severity or above. If you didn’t pass in a Severity, then it will default to failing on Medium and High.

If you use the OutputFile and OutputFormat parameters, Pester can output an NUnit format file that most Continuous Integration tools will happily accept and use to display the output of the tests.

If you installed the script from the PowerShell Gallery, you can also run the tests like this:

AzSKARMTemplateSecurityStatus.Test -TemplatePath D:\101-webapp-basic-windows\azuredeploy.json

Finally, if you’re using Azure DevOps, you can also get this function as part of the Secure DevOps Kit (AzSK) CICD Extensions for Azure in the Azure DevOps Marketplace.

Whichever way you choose to consume AzSK, it is a great module and well worth including in your CI/CD pipelines to ensure your ARM templates meet best practices.

Converting a PowerShell Project to use Azure DevOps Pipelines

2018-09-25T00:00:00Z

I wrote an article on converting my Cosmos DB PowerShell module to use a CI process in Azure DevOps over on the awesome PowerShell Magazine. Check it out here.

List Global Assembly Cache using PowerShell

2018-09-10T00:00:00Z

The list of assemblies stored in the Global Assembly Cache (GAC) can be found in the registry under the HKEY_CLASSES_ROOT\Installer\Assemblies\Global key.

If you want to get a list of the assemblies registered in the GAC using PowerShell you can use this snippet:

New-PSDrive -Name HKCR -PSProvider 'Microsoft.PowerShell.Core\Registry' -Root HKEY_CLASSES_ROOT
Get-ItemProperty -Path 'HKCR:\Installer\Assemblies\Global' | Get-Member -MemberType NoteProperty

The first line registers a new drive called HKCR in PowerShell that maps to the HKEY_CLASSES_ROOT in the registry. This is required because, by default only the HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE registry hives are registered as drives in PowerShell.

The second line just gets the list of registry properties in the HKEY_CLASSES_ROOT\Installer\Assemblies\Global key and displays them.

Install Windows Admin Center (WAC) using DSC

2018-07-06T00:00:00Z

Windows Admin Center (WAC) is a locally deployed, browser-based app for managing servers, clusters, hyper-converged infrastructure, and Windows 10 PCs. It was previously known as Project Honolulu.

WAC really shines when being used to manage headless Windows Servers (e.g., Windows Server Core). The benefits of deploying Windows Server Core are huge, but it can be a bit daunting to system administrators that have only used the Windows GUI experience to manage servers.

It is pretty easy to install WAC, but if you want to install it with PowerShell DSC, then here is a configuration for you to use:

configuration WindowsAdminCenter {
    param (
        [System.String]
        $WacProductId = '{7019BE31-3389-46FB-A077-B813D53C1266}',
        
        [System.String]
        $WacDownloadPath = 'https://download.microsoft.com/download/1/0/5/1059800B-F375-451C-B37E-758FFC7C8C8B/WindowsAdminCenter1809.5.msi',

        [System.Int16]
        $Port = 6516,
        
        [System.String]
        $Thumbprint
    )

    Import-DscResource -ModuleName PSDscResources
    
    if ([System.String]::IsNullOrEmpty($Thumbprint)) {
        $wacInstallArguments = "/qn /l*v c:\windows\temp\windowsadmincenter.msiinstall.log SME_PORT=$Port SSL_CERTIFICATE_OPTION=generate"
    } else {
        $wacInstallArguments = "/qn /l*v c:\windows\temp\windowsadmincenter.msiinstall.log SME_PORT=$Port SME_THUMBPRINT=$Thumbprint"
    }
    
    Node localhost {
        MsiPackage InstallWindowsAdminCenter {
            ProductId = $WacProductId
            Path      = $WacDownloadPath
            Arguments = $wacInstallArguments
            Ensure    = 'Present'
        }
    }
}

The configuration is parameterized and supports specifying the port for WAC to listen on and using either a self-signed certificate or a local machine certificate by specifying a thumbprint.

To apply the DSC using a self-signed certificate and on the default port of 6516, run the following in an Administrator PowerShell console:

Invoke-WebRequest -Uri 'https://gist.githubusercontent.com/PlagueHO/e8120e1cc01b447d084322eb2ad14c95/raw/2aff9e1a8d94cdb6f8a7409874a3bdbfcf234f8e/WindowsAdminCenterDscConfiguration.ps1' -OutFile 'WindowsAdminCenterDscConfiguration.ps1'
Install-Module -Name PSDscResources
. .\WindowsAdminCenterDscConfiguration.ps1
WindowsAdminCenter
Start-DscConfiguration -Path .\WindowsAdminCenter\ -ComputerName localhost -Wait -Verbose

You can run this on a Windows Server Core machine by logging in and typing powershell to start a PowerShell console, then entering the commands above.

To apply the DSC configuration specifying a certificate with a thumbprint from the local machine store and on Port 4000, run these commands instead:

Invoke-WebRequest -Uri 'https://gist.githubusercontent.com/PlagueHO/e8120e1cc01b447d084322eb2ad14c95/raw/2aff9e1a8d94cdb6f8a7409874a3bdbfcf234f8e/WindowsAdminCenterDscConfiguration.ps1' -OutFile 'WindowsAdminCenterDscConfiguration.ps1'
Install-Module -Name PSDscResources
. .\WindowsAdminCenterDscConfiguration.ps1
WindowsAdminCenter -Port 4000 -Thumbprint 'fddfec2150b2a1c0d1166debffdbed1d55798485'
Start-DscConfiguration -Path .\WindowsAdminCenter\ -ComputerName localhost -Wait -Verbose

This DSC configuration can also be used on Virtual Machines deployed to Azure, using either the Azure DSC Extension Handler or an Azure Automation DSC Pull Server.

Easy as all that. Now you can use the awesome WAC GUI and still run headless while also taking advantage of the benefits that DSC brings.

Get the ForceChangePassword Office 365 User Setting with PowerShell

2018-06-22T00:00:00Z

Recently I was asked by a friend if I knew of a way to get the value of the setting that forces a user to change their password when they next log in to Office 365. The friend wanted to get this value for all users using PowerShell.

Changing this setting is fairly straight forward either in the Office 365 portal or using the Set-MsolUserPassword cmdlet in the MSOnline PowerShell module:

However, retrieving the current value of the setting isn’t possible using Get-MsolUser cmdlet—the attribute does not appear in the returned object:

Instead, we need to use the Get-AzureADUser cmdlet in the AzureAD PowerShell Module to query the Azure Active Directory for the Office 365 tenant.

If you don’t have the AzureAD module installed, use Install-Module cmdlet to install it from the PowerShell Gallery:

Install-Module -Name AzureAD

Then connect to AzureAD using the Connect-AzureAD cmdlet. Once connected, you can run the following command to get the user object and show only the appropriate property (ForceChangePasswordNextLogin of the PasswordProfile object):

Connect-AzureAD
(Get-AzureADUser -SearchString 'williammurderface@contoso.onmicrosoft.com').PasswordProfile.ForceChangePasswordNextLogin

If you wanted to get a list of all users with the ForceChangePasswordNextLogin property set to true, you could use:

Get-AzureADUser | Where-Object -FilterScript { $_.PasswordProfile.ForceChangePasswordNextLogin }

This is all fairly straight forward once you figure out which object in Azure AD contains the information required.

Disable TLS 1.0, TLS 1.1 and 3DES in Azure API Management using an ARM Template

2018-04-07T00:00:00Z

Recently, I’ve been putting together a continuous delivery pipeline (using VSTS) for our Azure API Management service using Azure Resource Manager (ARM) templates. One of the things I needed to be able to do to secure this service properly is to disable TLS 1.0, TLS 1.1 and 3DES. This is pretty easy to do in the portal:

However, if you only allow changes to be made via your continuous delivery pipeline (a good thing by the way), then you have to change the ARM template.

Side note: Disabling TLS 1.0, TLS 1.1 and 3DES is pretty important for keeping your system secure. But if you have an Azure Application Gateway in front of your API Management service, then you’ll also need to configure the Azure Application Gateway to disable TLS 1.0 and TLS 1.1. This is done in a slightly different way, but can also be done in an ARM Template (post a comment if you’re not sure how to do this and I’ll write another post).

I found the documentation for the API Management service resource here. This shows it can be done by setting the customProperties object in the ARM Template. But the documentation isn’t completely clear.

But after a little bit of trial and error I managed to figure it out and get it working. What you need to do is add the following customProperties to the properties of the API Management service resource:

"customProperties": {
  "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Ciphers.TripleDes168": "false",
  "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls11": "false",
  "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls10": "false"
}

This is what the complete ARM template looks like:

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "publisherEmail": {
            "type": "string",
            "minLength": 1,
            "metadata": {
                "description": "The email address of the owner of the service"
            }
        },
        "publisherName": {
            "type": "string",
            "minLength": 1,
            "metadata": {
                "description": "The name of the owner of the service"
            }
        },
        "sku": {
            "type": "string",
            "allowedValues": [
                "Developer",
                "Standard",
                "Premium"
            ],
            "defaultValue": "Developer",
            "metadata": {
                "description": "The pricing tier of this API Management service"
            }
        },
        "skuCount": {
            "type": "string",
            "allowedValues": [
                "1",
                "2"
            ],
            "defaultValue": "1",
            "metadata": {
                "description": "The instance size of this API Management service."
            }
        }
    },
    "variables": {
        "apiManagementServiceName": "[concat('apiservice', uniqueString(resourceGroup().id))]"
    },
    "resources": [
        {
            "apiVersion": "2017-03-01",
            "name": "[variables('apiManagementServiceName')]",
            "type": "Microsoft.ApiManagement/service",
            "location": "West US",
            "tags": {},
            "sku": {
                "name": "[parameters('sku')]",
                "capacity": "[parameters('skuCount')]"
            },
            "properties": {
                "publisherEmail": "[parameters('publisherEmail')]",
                "publisherName": "[parameters('publisherName')]",
                "customProperties": {
                    "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Ciphers.TripleDes168": "false",
                    "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls11": "false",
                    "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls10": "false"
                }
            }
        }
    ]
}

the template above is based off the Azure Quickstart Template for API Management.

Hopefully you find this if you’re looking for an example of how to do this and it saves you some time.

Managing Users and Permissions in Cosmos DB with PowerShell

2018-01-14T00:00:00Z

If you’re just getting started with Cosmos DB, you might not have come across users and permissions in a Cosmos DB database. However, there are certain use cases where managing users and permissions are necessary. For example, if you’re wanting to be able to limit access to a particular resource (e.g. a collection, document, stored procedure) by user.

The most common usage scenario for users and permissions is if you’re implementing a Resource Token Broker type pattern, allowing client applications to directly access the Cosmos DB database.

The Cosmos DB implementation of users and permissions only provides authorization - it does not provide authentication. It would be up to your own implementation to manage the authentication. In most cases you’d use something like Azure Active Directory to provide an authentication layer.

But if you go hunting through the Azure Management Portal Cosmos DB data explorer (or Azure Storage Explorer) you won’t find any way to configure or even view users and permissions.

To manage users and permissions you need to use the Cosmos DB API directly or one of the SDKs.

But to make Cosmos DB users and permissions easier to manage from PowerShell, I created the Cosmos DB PowerShell module. This is an open source project hosted on GitHub. The Cosmos DB module allows you to manage much more than just users and permissions, but for this post I just wanted to start with these.

Requirements

This module works on PowerShell 5.x and PowerShell Core 6.0.0. It probably works on PowerShell 3 and 4, but I don’t have any more machines running this version to test on.

The Cosmos DB module does not have any dependencies, except if you call the New-CosmosDbContext function with the ResourceGroup parameter specified as this will use the AzureRM PowerShell modules to read the Master Key for the connection directly from your Cosmos DB account. So I’d recommend installing the Azure PowerShell modules or if you’re using PowerShell 6.0, install the AzureRM.NetCore modules.

Installing the Module

The best way to install the Cosmos DB PowerShell module is from the PowerShell Gallery. To install it for only your user account execute this PowerShell command:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2018\01\2018-01-14-managing-users-permissions-in-cosmosdb-with-powershell.md
Install-Module -Name CosmosDB -Scope CurrentUser

Or to install it for all users on the machine (requires administrator permissions):

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2018\01\2018-01-14-managing-users-permissions-in-cosmosdb-with-powershell.md
Install-Module -Name CosmosDB

Context Variable

Update 2018-03-06
As of Cosmos DB module v2.0.1, the connection parameter has been renamed to context and the New-CosmosDbConnection function has been renamed New-CosmosDbContext. This was to be more inline with naming adopted by the Azure PowerShell project. The old connection parameters and New-CosmosDbConnection function is still available as an alias, so older scripts won’t break. But these should be changed to use the new naming if possible as I plan to deprecate the connection version at some point in the future.
This post was updated to specify the new naming, but screenshots still show the Connection aliases.

Before you get down to the process of working with Cosmos DB resources, you’ll need to create a context variable containing the information required to connect. This requires the following information:

The Cosmos DB Account name
The Cosmos DB Database name
The Master Key for the account (you can have the Cosmos DB PowerShell module get this directly from your Azure account if you wish).

To create the connection variable we just use the New-CosmosDbContext:

$account = 'MyCosmosDBAccount'
$database = 'MyDatabase'
$key = ConvertTo-SecureString -String 'this is your master key, get it from the Azure portal' -AsPlainText -Force
$context = New-CosmosDbContext -Account $account -Database $database -Key $key

If you do not wish to specify your master key, you can have the New-CosmosDbContext function pull your master key from the Azure Management Portal directly:

Add-AzureRmAccount
$account = 'MyCosmosDBAccount'
$database = 'MyDatabase'
$resourceGroup = 'MyCosmosDBResourceGroup'
$context = New-CosmosDbContext -Account $account -Database $database -ResourceGroup $resourceGroup

Note: This requires the AzureRM.Profile and AzureRM.Resources module on Windows PowerShell 5.x or AzureRM.Profile.NetCore and AzureRM.Resources.NetCore on PowerShell Core 6.0.0.

Managing Users

To add a user to the Cosmos DB Database use the New-CosmosDbUser function:

New-CosmosDbUser -Context $context -Id 'daniel'

To get a list of users in the database:

Get-CosmosDbUser -Context $context

To get a specific user:

Get-CosmosDbUser -Context $context -Id 'daniel'

To remove a user (this will also remove all permissions assigned to the user):

Remove-CosmosDbUser -Context $context -Id 'daniel'

Managing Permissions

Permissions in Cosmos DB are granted to a user for a specific resource. For example, you could grant a user access to just a single document, an entire collection or to a stored procedure.

To grant a permission you need to provide four pieces of information:

The Id of the user to grant the permission to.
An Id for the permission to create. This is just string to uniquely identify the permission.
The permission mode to the permission: All or Read.
The Id of the resource to grant access to. This can be generated from one of the Get-CosmosDb*ResourcePath functions in the CosmosDB PowerShell module.

In the following example, we’ll grant the user daniel all access to the TestCollection:

$userId = 'TestUserId'
$resourcePath = Get-CosmosDbCollectionResourcePath -Database 'TestDatabase' -Id 'TestCollection'
New-CosmosDbPermission -Context $context -Id 'AccessTestCollection' -UserId $userId -PermissionMode All -Resource $resourcePath

Once a permission has been granted, you can use the Get-CosmosDbPermission function to retrieve the permission and with it the Resource Token that can be used to access the resource for a limited amount of time (between 10 minutes and 5 hours).

Note: as you have the Master Key already, using the Resource Token isn’t required.

For example, to retrieve all permissions for the user with Id daniel and a resource token expiration of 600 seconds:

Get-CosmosDbPermission -Context $context -UserId 'daniel' -TokenExpiry '600' | 
    Format-List *

You can as expected delete a permission by using the Remove-CosmosDbPermission function:

Remove-CosmosDbPermission -Context $context -UserId 'daniel' -Id 'AccessTestCollection'

Final Thoughts

So this is pretty much all there is to managing users and permissions using the Cosmos DB PowerShell module. This module can also be used to manage the following Cosmos DB resources:

Attachments
Collections
Databases
Documents
Offers
Stored procedures
Triggers
User Defined Functions

You can find additional documentation and examples of how to manage these resources over in the Cosmos DB PowerShell module readme file on GitHub.

Hopefully this will help you in any Cosmos DB automation tasks you might need to implement.

Configure Azure SQL Server Automatic Tuning with PowerShell

2017-12-25T00:00:00Z

One thing I’ve found with configuring Azure services using automation (e.g. Azure PowerShell Modules, Azure Resource Manager template) is that the automation features are a little bit behind the feature set. For example, the Azure PowerShell modules may not yet implement settings for new or preview features. This can be a an issue if you’re strictly deploying everything via code (e.g. infrastructure as code). But if you run into a problem like this, all is not lost. So read on for an example of how to solve this issue.

Azure REST APIs

One of the great things about Azure is that everything is configurable by making direct requests to the Azure REST APIs, even if it is not available in ARM templates or Azure PowerShell.

Depending on the feature/configuration you can sometimes use the Set-AzureRmResource cmdlets to make calls to the REST APIs. But this cmdlet is limited to using an HTTP method of POST. So if you need to use PATCH, you’ll need to find an alternate way to make the call.

So, what you need then is to use the Invoke-RestMethod cmdlet to create a custom call to the REST API. This is the process I needed to use to configure the Azure SQL Server Automatic Tuning settings and what I’ll show in my script below.

The Script

The following script can be executed in PowerShell (of course) and requires a number of parameters to be passed to it:

SubscriptionId - the subscription Id of the Azure subscription that contains the Azure SQL Server.
ResourceGroupName - The name of the resource group containing SQL Server or database.
ServerName - The name of the Azure SQL Server to set the automatic tuning options on.
DatabaseName - The name of the Azure SQL Database to set the automatic tuning options on. If you pass this parameter then the automatic tuning settings are applied to the Azure SQL Database, not the server.
Mode - This defines where the settings for the automatic tuning are obtained from. Inherit is only valid if the DatabaseName is specified.
CreateIndex - Enable automatic tuning for creating an index.
DropIndex - Enable automatic tuning for dropping an index.
ForceLastGoodPlan - Enable automatic tuning for forcing last good plan.

Requirements: You need to have the installed the AzureRM.Profile PowerShell module (part of the AzureRM PowerShell Modules) to use this script. The script also requires you to have logged into your Azure Subscription using Add-AzureRmAccount (as a user or Service Principal).

#Requires -Modules 'AzureRM.Profile' 

<#
.SYNOPSIS
Configure Azure SQL Autotuning on an Azure SQL server
or database.

.DESCRIPTION
This function will retrieve a current access token from
the Azure RM PowerShell context and use it to make a direct
request to the Azure management portal endpoint for the
SQL Server or database. It will configure the Autotuning
parameters for the server.

Requires AzureRM PowerShell Modules 5.1.1 or above*.
* May work on lower versions but untested

.PARAMETER SubscriptionId
The Azure subscription Id of the subscription containing
SQL Server or database.

.PARAMETER ResourceGroupName
The name of the resource grou containing SQL Server or
database.

.PARAMETER ServerName
The name of the Azure SQL Server to set the autotuning
options on.

.PARAMETER DatabaseName
The name of the Azure SQL Database to set the autotuning
options on.

.PARAMETER Mode
This defines where the settings for the Autotuning are
obtained from.

Inherit is only valid if the DatabaseName is specified.

.PARAMETER CreateIndex
Enable autotuning for creating an index.

.PARAMETER DropIndex
Enable autotuning for dropping an index.

.PARAMETER ForceLastGoodPlan
Enable autotuning for forcing last good plan.
#>
param (
    [Parameter(Mandatory = $true)]
    [System.String]
    $SubscriptionId,

    [Parameter(Mandatory = $true)]
    [System.String]
    $ResourceGroupName,

    [Parameter(Mandatory = $true)]
    [System.String]
    $ServerName,

    [Parameter()]
    [System.String]
    $DatabaseName,

    [Parameter()]
    [ValidateSet('Auto', 'Custom', 'Inherit')]
    [System.String]
    $Mode = 'Auto',

    [Parameter()]
    [ValidateSet('On', 'Off', 'Default')]
    [System.String]
    $CreateIndex = 'Default',

    [Parameter()]
    [ValidateSet('On', 'Off', 'Default')]
    [System.String]
    $DropIndex = 'Default',

    [Parameter()]
    [ValidateSet('On', 'Off', 'Default')]
    [System.String]
    $ForceLastGoodPlan = 'Default'
)

# Get an access token from the Auzre RM PowerShell token cache for accessing the Azure Management Portal
$context = Get-AzureRmContext
$cache = $context.TokenCache

if (-not $cache)
{
    # Use an older method of accessing the Token Cache (for old versions of AzureRM.Profile)
    $cache = [Microsoft.IdentityModel.Clients.ActiveDirectory.TokenCache]::DefaultShared
}

$cacheItems = $cache.ReadItems()
$cacheItem = $cacheItems |
    Where-Object -FilterScript { $_.TenantId -eq $context.Tenant.TenantId } |
    Select-Object -First 1

if (-not $cacheItem)
{
    Throw ('A current access token could not be found for the tenant Id {0}.' -f $context.Tenant.TenantId)
}

$accessToken = $cacheItem.AccessToken
    
# Generate the Body of the request
$body = @{
    properties = @{
        desiredState = $Mode
        options      = @{
            createIndex       = @{
                desiredState = $CreateIndex
            }
            dropIndex         = @{
                desiredState = $DropIndex
            }
            forceLastGoodPlan = @{
                desiredState = $ForceLastGoodPlan
            }
        }
    }
}

# Generate the URI to the endpoint
$uri = ('https://management.azure.com/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.Sql/servers/{2}' -f $SubscriptionId, $ResourceGroupName, $ServerName )

if ($PSBoundParameters.ContainsKey('DatabaseName'))
{
    $uri = ('{0}/databases/{1}' -f $uri, $DatabaseName)
}
else
{
    if ($Mode -eq 'Inherit')
    {
        Throw 'Inherit mode is only valid for a SQL database. Either use a different not or specify a database name.'
    }
}

$uri = ('{0}/automaticTuning/current?api-version=2017-03-01-preview' -f $uri)

$bodyText = ConvertTo-Json -InputObject $body -Depth 10

$headers = @{
    'Authorization' = ('Bearer {0}' -f $accessToken)
    'Cache-Control' = 'no-cache'
}

$invokeRestMethodParameters = @{
    Uri         = $Uri
    Method      = 'PATCH'
    Headers     = $headers
    ContentType = 'application/json'
    Body        = $bodyText
}

return Invoke-RestMethod @invokeRestMethodParameters

Example Usage

To apply custom automatic tuning to an Azure SQL Server:

.\\Set-AzureRMSqlServerAutotuning.ps1 -SubscriptionId '<Subscription Id>' -ResourceGroupName '<Resource Group name>' -ServerName '<Azure SQL server name>' -Mode Custom -CreateIndex On -DropIndex On -ForceLastGoodPlan Off

To apply custom automatic tuning to an Azure SQL Database:

.\\Set-AzureRMSqlServerAutotuning.ps1 -SubscriptionId '<Subscription Id>' -ResourceGroupName '<Resource Group name>' -ServerName '<Azure SQL server name>' -DatabaseName '<Azure SQL database name>' -Mode Custom -CreateIndex On -DropIndex On -ForceLastGoodPlan Off

Conclusion

I’ve not yet encountered something in Azure that I can’t configure via the Azure REST APIs. This is because the Azure Management Portal uses the same APIs - so if it is available in the portal then you can do it via the Azure REST APIs. The biggest challenge is determining the body, header and methods available if the APIs are not yet documented.

If the API you need is not documented then you can raise a question in the Microsoft Azure Forums or on Stack Overflow. Failing that you can use the developer tools in your browser of choice to watch the API calls being made to the portal - I’ve had to resort to this many times, but documenting that process is something I’ll save for another day.

Create a Scheduled Task with unlimited Execution Time Limit in PowerShell

2017-12-16T00:00:00Z

When creating a scheduled task in PowerShell you may wish to set the Execution Time Limit of the task to be unlimited (no time limit).

This will prevent the task from being terminated if it is still running after a specific period of time.

Creating scheduled tasks using PowerShell is pretty easy using the *-ScheduledTask* cmdlets in Windows Server 2012 and above.

However, after working on this issue in the xScheduledTask DSC resource in the Microsoft DSC Resource Kit I found that there are some differences in how to do this between Windows Server 2012 R2 and Windows Server 2016.

So in this post I’m going to show how to create a scheduled task with no Execution Time Limit that will work on both Windows Server 2012 R2 (and Windows 8/8.1) and Windows Server 2016 (and Windows 10).

I’ll also show the method that works only on Windows Server 2016.

All Versions of Windows Server

To create a scheduled task with unlimited Execution Time Limit on Windows Server 2012 R2 and Windows Server 2016.

$trigger = New-ScheduledTaskTrigger -Once -At '13:00:00'
$action = New-ScheduledTaskAction -Execute 'powershell.exe'
$settingsSet = New-ScheduledTaskSettingsSet
# Set the Execution Time Limit to unlimited on all versions of Windows Server
$settingsSet.ExecutionTimeLimit = 'PT0S'
$task = New-ScheduledTask -Trigger $trigger -Action $action -Settings $settingsSet
Register-ScheduledTask -TaskName 'MyTask' -InputObject $task

This should also work on Windows Server 2012, but I have not confirmed this. It will NOT work on Windows Server 2008 R2. It should also work on Windows 8/8.1/10.

Windows Server 2016 Only

To create a scheduled task with unlimited Execution Time Limit on Windows Server 2016 only:

$trigger = New-ScheduledTaskTrigger -Once -At '13:00:00'
$action = New-ScheduledTaskAction -Execute 'powershell.exe'
# Set the Execution Time Limit to unlimited on Windows Server 2016
$settingsSet = New-ScheduledTaskSettingsSet -ExecutionTimeLimit '00:00:00'
$task = New-ScheduledTask -Trigger $trigger -Action $action -Settings $settingsSet
Register-ScheduledTask -TaskName 'MyTask' -InputObject $task

This method is a more elegant approach and arguably how the Scheduled Task cmdlets are intended to be used. But you would only use this method if your task does not need to created on an operating system earlier than Windows Server 2016/Windows 10.

So, hopefully this will help anyone else out there who has struggled with this.

Auto Formatting PowerShell in Visual Studio Code

2017-11-17T00:00:00Z

One of the features I’m most fond of in Visual Studio Code is the Format Document feature that is built into Visual Studio Code.

If you’re writing PowerShell scripts or modules then you should be using Visual Studio Code. You should only be using PowerShell ISE if you don’t have the ability to install Visual Studio Code.

The Format Document feature can be used in many different document types in Visual Studio Code to correct the layout based on your user settings or the workspace settings for the project you’re working on.

This enables me to configure Visual Studio Code to auto format PowerShell code the way I like it for my own projects, but still adhere to the code formatting standards of any other projects I work on without having to remember what they are for each. This saves so much time and hassle.

Tip: If you’re contributing code to an Open Source project, the project maintainers may have included a .vscode\settings.json in the project folder. This may contain workspace specific formatting settings that you should apply before submitting code back to the project.

But even if you’ve don’t define and code formatting settings Visual Studio Code will still do a great job of formatting your PowerShell code. Having nicely formatted code really is not a requirement to being awesome at writing PowerShell, but it does make it easier for not so awesome PowerShell people to read, understand and potentially maintain your work.

Formatting a PowerShell Script

Here are the simple instructions for auto formatting a document in Visual Studio Code:

Download and install Visual Studio Code.
Once Visual Studio Code is installed, add the PowerShell extension:
In Visual Studio Code, open the folder of the project containing the file you want to format or open an individual PowerShell file (PS1, PSD1, PSM1).
Workspace formatting settings are only used if you’ve opened the folder rather than an individual file.
Press **SHIFT+**ALT+F (or press F1 and type Format and select Format Document).
The code is now all well formatted, so save the document.

It really is as easy as that.

Happy formatting.

Stop, Start or Restart all Web Apps in Azure using PowerShell

2017-10-07T00:00:00Z

Here is a short (and sometimes handy) single line of PowerShell code that can be used to restart all the Azure Web Apps in a subscription:

(Get-AzureRmWebApp).GetEnumerator() | Restart-AzureRmWebApp

Use this with care if you’re working with production systems because this _will_ restart these Web Apps without confirming first.

This would be a handy snippet to be able to run in the Azure Cloud Shell. It could also be adjusted to perform different actions on other types of resources.

To stop all Web Apps in a subscription use:

(Get-AzureRmWebApp).GetEnumerator() | Stop-AzureRmWebApp

To start them all again:

(Get-AzureRmWebApp).GetEnumerator() | Start-AzureRmWebApp

The key part of this command is the GetEnumerator() method because most Azure Cmdlets don’t return an array of individual objects into the pipeline like typical PowerShell cmdlets. Instead returning a System.Collections.Generic.List object, which requires a slight adjustment to the code. This procedure can be used for most Azure Cmdlets to allow the results to be iterated through.

Thanks for reading.

Install Nightly Build of Azure CLI 2.0 on Windows

2017-10-06T00:00:00Z

The Azure PowerShell cmdlets are really first class if you’re wanting to manage Azure with PowerShell. However, they don’t always support the very latest Azure components and features. For example, at the time of writing this there is no Azure PowerShell module for managing Azure Container Instances.

The solution to this is to install the Nightly Build of Azure CLI 2.0. However, on Windows it is not entirely clear the easiest way to do this. So, in this post I’ll provide a PowerShell script that will:

Install Python 3.x using Chocolatey
Use PIP (Python package manager) to install the latest nightly build packages
Update the Environment Path variable so that you can use Azure CLI 2.0.

If you have the stable build of Azure CLI 2.0 installed using the MSI then you’ll need to configure your Environment Path variable to find the Az command that you’d like to use by default. I personally removed the stable build of Azure CLI 2.0 to make it easier.

Performing the Install

Make sure you’ve got Chocolatey installed. If you aren’t sure what Chocolatey is, it is a package management system for Windows - not unlike Apt-Get or Yum for Linux. It is free and awesome. In this process we’ll use Chocolatey to install Python for us. If you haven’t got Chocolatey installed, see this page for instructions.

Next, download and run this PowerShell script in a PowerShell Administrator Console:

<#
    .SYNOPSIS
        Install Azure CLI 2.0 Nightly Build on Windows using Chocolatey and PowerShell
#>
if (-not (Get-Command -Name Choco -ErrorAction SilentlyContinue))
{
    Throw 'Chocolatey is not installed. Please install it. See https://chocolatey.org/install for instructions.'
}

Write-Host -Object 'Installing Python 3 with Chocolatey...'
& choco @('install','python3','-y')

Update-SessionEnvironment

$pyhtonScriptsPath = Join-Path -Path $ENV:APPDATA -ChildPath 'Python\Python36\Scripts'
$currentPath = [System.Environment]::GetEnvironmentVariable('Path',[System.EnvironmentVariableTarget]::User) -split ';'
if ($currentPath -notcontains $pyhtonScriptsPath)
{
    Write-Host -Object 'Adding Python Scripts to User Environment Path...'
    $newPath = @()
    $newPath += $currentPath
    $newPath += $pyhtonScriptsPath
    $newPathJoined = $newPath -join ';'
    [System.Environment]::SetEnvironmentVariable('Path',$newPathJoined,[System.EnvironmentVariableTarget]::User)
}

if (-not $currentPath.Contains($pyhtonScriptsPath))
{
    Write-Host -Object 'Adding Python Scripts to Current PowerShell session path...'
    $ENV:Path = "$($ENV:Path);$pyhtonScriptsPath"
}

Write-Host -Object 'Installing nightly build of Az CLI 2.0...'
& pip @('install','--no-cache-dir','--user','--upgrade','--pre','azure-cli','--extra-index-url','https://azureclinightly.blob.core.windows.net/packages')

Write-Host -Object 'Installation of nightly build of Az CLI 2.0 complete. Execute "az" to start.'

You could save the content of this script into a PS1 file and then execute it like this:

It will then download and install Python, then use PIP to install the current nightly build packages. After a few minutes the installation will complete:

You can then run:

az login

To get started.

If you’re a bit new to Azure CLI 2.0, then another great way is to use Azure CLI Interactive:

az login interactive

If you need to update to a newer nightly build, just run the script again and it will update your packages.

Easy as that! Now you can experiment with all the latest automation features in Azure without needing to wait for a new version of Azure CLI 2.0 or for latest Azure PowerShell cmdlets.

Edge Builds

If you want to install even more “bleeding edge” builds (built straight off the master branch on every merge to master) then you can make a small adjustment to the script above:

On line 34 change the URL of the feed from:

https://azureclinightly.blob.core.windows.net/packages

To:

https://azurecliprod.blob.core.windows.net/edge

Thanks for reading!

Get Azure API Management Git Credentials using PowerShell

2017-09-16T00:00:00Z

One of the many great features of Azure API Management is the fact that it has a built in Git repository for storing the current configuration as well as publishing new configurations.

This allows you to push updated Azure API Management configurations to this internal Git repository as a new branch and then Deploy the configuration to API Management.

The internal Git repository in Azure API Management is not intended to be used for a normal development workflow. You’ll still want to develop and store your Azure API management configuration in an external Git repository such as GitHub or TFS/VSTS and then copy configuration updates to the internal Git repository in Azure API Management using some sort of automated process (e.g. Continuous Integration/Continuous Delivery could be adopted for this).

The Internal Git Repository

To access the Internal Git Repository requires short lived (30 days maximum) Git credentials to be generated. This is fairly easy through the Azure API Management portal:

Unfortunately using the portal to get these credentials is a manual process and so would not be so good for an automated delivery process (e.g. CI/CD). You’d need to update these Git credentials in your CI/CD automation system every time they expired (every 30 days).

Get Git Credentials

A better approach to generating the Git Credentials is to use Azure PowerShell API Management cmdlets connected with a Service Principal to generate the Git credentials whenever you need them in your CI/CD pipeline.

This is not a completely straightforward process right now (which is unusual for the Azure PowerShell team), so I’ve created a simple PowerShell script that will take care of the nuts and bolts for you.

Requirements

To run this script you’ll need:

PowerShell 5 (WMF 5.0) or greater.
Azure PowerShell Modules installed (make sure you’ve got the latest versions - 4.0.3 at the time of writing this).

You’ll also need to supply the following parameters to the script:

The Azure Subscription Id of the subscription containing the API Management instance.
The name of the Resource Group where the API Management instance is installed to.
The service name of the API Management instance.

You can also optionally supply which of the two internal API Management keys, primary or secondary, to use to generate the credential and also the length of time that the Git credential will be valid for (up to 30 days).

Steps

Download the Script

Download the script Get-AzureRMApiManagementGitCredential.ps1 using the PowerShell command:

iwr https://gist.githubusercontent.com/PlagueHO/70ae184e1c8d22848ade6a7bc0f8255d/raw/a9ca51e690c04654dfcb934ccbc7ca9358c97f08/Get-AzureRMApiManagementGitCredential.ps1 -OutFile Get-AzureRMApiManagementGitCredential.ps1

Unblock the script using the PowerShell command:

Unblock-File -Path .\Get-AzureRMApiManagementGitCredential.ps1

Using the Script

Use the Login-AzureRMAccount cmdlet to authenticate to Azure. This would normally be done using a Service Principal if using an automated process, but could be done interactively when testing.
Execute the script providing the SubscriptionId, ResourceGroup and ServiceName parameters (and optionally the KeyType and ExpiryTimespan) using the following PowerShell command:

.\Get-AzureRMApiManagementGitCredential.ps1 `
  -SubscriptionId '605e2ba7-056b-4982-a48b-12ff1da3c038' `
  -ResourceGroup 'apimanagement-shrp-p-rgp' `
  -ServiceName 'ApiManagementdsrshrpp' `
  -KeyType 'primary' `
  -ExpiryTimespan '4:00:00'

The script will return an object containing the properties GitUsername and GitPassword that can be provided to Git when cloning the internal Git repository.

The GitPassword is not escaped so can not be directly used within a Git Clone URL without replacing any / or @ with %2F and %40 respectively.

In the example above I generated an internal Git Credential using the Primary Secret Key that will expire in 4 hours.

Typically you’d assign the output of this script to a variable and use the properties to generate the URL to pass into the Git Clone. For example:

Tips

When cloning the internal Git Repository you’ll need the clone URL of the repository. This is always the name of your Azure API Management instance followed by with scm.azure-api.net appended to it E.g. https://myapimanagementinstance.scm.azure-api.net
Once you’ve uploaded a new Git branch containing a new or updated Azure API Management configuration you’ll need to use the Publish-AzureRmApiManagementTenantGitConfiguration cmdlet to tell Azure API Management to publish the configuration contained in the branch. I have not detailed this process here, but if there is interest I can cover the entire end-to-end process.
The Primary and Secondary Secret Keys that are used to generate the internal Git Credential can be re-generated (rolled) individually if a Git credential is compromised. However, this will invalidate all Git Credentials generated using that Secret Key.

The Script

If you wish to review the script itself, here it is:

param
(
    [Parameter(Mandatory = $True)]
    [System.String]
    $SubscriptionId,

    [Parameter(Mandatory = $True)]
    [System.String]
    $ResourceGroup,

    [Parameter(Mandatory = $True)]
    [System.String]
    $ServiceName,

    [Parameter()]
    [ValidateSet('primary','secondary')]
    [System.String]
    $KeyType = 'primary',

    [Parameter()]
    [timespan]
    $ExpiryTimespan = (New-Timespan -Hours 2)
    
)

$context = New-AzureRmApiManagementContext -ResourceGroupName $ResourceGroup -ServiceName $ServiceName

// Correction thanks to @Shaun Titus
$expiry = (Get-Date).ToUniversalTime() + $ExpiryTimespan
$parameters = @{
    "keyType"= $KeyType
    "expiry"= ('{0:yyyy-MM-ddTHH:mm:ss.000Z}' -f $expiry)
}

$resourceId = '/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.ApiManagement/service/{2}/users/git' -f $SubscriptionId,$ResourceGroup,$ServiceName

$gitUsername = 'apim'
$gitPassword = (Invoke-AzureRmResourceAction -Action 'token' -ResourceId $resourceId -Parameters $parameters -ApiVersion '2016-10-10' -Force).Value

return @{
    GitUsername = $gitUsername
    GitPassword = $gitPassword
}

So, hopefully that will be enough information to get anyone else started on building a CI/CD pipeline for deploying Azure API Management configurations.

Sonatype Nexus Containers with Persistent Storage in Azure Container Instances

2017-08-06T00:00:00Z

On the back of yesterdays post on running Azure Container Instance containers with persistent storage, I thought I’d try a couple of other containers with my script.

I don’t actually plan on running any of these apps, I just wanted to test out the process and my scripts to identify any problems.

I tried:

Sonatype Nexus 2 - sonatype/nexus:oss
Sonatype Nexus 3 - sonatype/nexus3:latest
Jenkins - jenkins/jenkins

And here are the results of my tests:

Sonatype Nexus 2

Works perfectly and the container starts up quickly (under 10 seconds):

I passed the following parameters to the script:

.\Install-AzureContainerInstancePersistStorage.ps1 `
    -ServicePrincipalUsername 'ce6fca5e-a22d-44b2-a75a-f3b20fcd1b16' `
    -ServicePrincipalPassword (ConvertTo-SecureString -String 'JUJfenwe89hwNNF723ibw2YBybf238ybflA=' -AsPlainText -Force) `
    -TenancyId '8871b1ba-7d3d-45f3-8ee0-bb60c0e4733e' `
    -SubscriptionName 'Visual Studio Enterprise' `
    -AppCode 'nexus' `
    -UniqueCode 'mine' `
    -ContainerImage 'sonatype/nexus:oss' `
    -ContainerPort '8081' `
    -VolumeName 'nexus' `
    -MountPoint '/sonatype-work/' `
    -Verbose

Note: The Nexus 2 server is only accessible on the path /nexus/.

Sonatype Nexus 3

Works perfectly but after takes at least a minute to be accessible after the container starts. But this is normal behavior for Nexus 3.

I passed the following parameters to the script:

.\Install-AzureContainerInstancePersistStorage.ps1 `
    -ServicePrincipalUsername 'ce6fca5e-a22d-44b2-a75a-f3b20fcd1b16' `
    -ServicePrincipalPassword (ConvertTo-SecureString -String 'JUJfenwe89hwNNF723ibw2YBybf238ybflA=' -AsPlainText -Force) `
    -TenancyId '8871b1ba-7d3d-45f3-8ee0-bb60c0e4733e' `
    -SubscriptionName 'Visual Studio Enterprise' `
    -AppCode 'nexus3' `
    -UniqueCode 'mine' `
    -ContainerImage 'sonatype/nexus3:latest' `
    -ContainerPort '8081' `
    -VolumeName 'nexus3' `
    -MountPoint '/nexus-data/' `
    -Verbose

Jenkins

Unfortunately Jenkins does not work with a persistent storage volume from an Azure Share. It seems to be trying to set the timestamp of the file that will contain the InitialAdminPassword, which is failing:

I passed the following parameters to the script:

.\Install-AzureContainerInstancePersistStorage.ps1 `
    -ServicePrincipalUsername 'ce6fca5e-a22d-44b2-a75a-f3b20fcd1b16' `
    -ServicePrincipalPassword (ConvertTo-SecureString -String 'JUJfenwe89hwNNF723ibw2YBybf238ybflA=' -AsPlainText -Force) `
    -TenancyId '8871b1ba-7d3d-45f3-8ee0-bb60c0e4733e' `
    -SubscriptionName 'Visual Studio Enterprise' `
    -AppCode 'jenkinshome' `
    -UniqueCode 'dsr' `
    -ContainerImage 'jenkins/jenkins:lts' `
    -ContainerPort '8080' `
    -VolumeName 'jenkinshome' `
    -MountPoint '/var/jenkins_home/' `
    -Verbose

So, this is still a little bit hit and miss, but in general Azure Container Instances look like a very promising way to run different types of services in containers without a lot of overhead. With a bit of automation, this could turn out to be a cost effective way to quickly and easily run some common services.

Persistent Storage in Azure Container Instances

2017-08-05T00:00:00Z

Update 2018-04-26: At some point Microsoft made a change to the requirements of the ARM template creating the Azure Container Instance. It now requires the Ports to be specified within the container as well as we the container group. I have improved the ARM template to meet the current requirements.

Update 2017-08-06: I have improved the script so that it is idempotent (can be run more than once and will only create anything that is missing). The Azure Container Instance resource group can be deleted once you’ve finished with the container and then recreated again with this same script when you next need it. The storage will be preserved in the separate storage account resource group. The script can now be run with the -verbose parameter and will produce much better progress information.

Azure Container Instances (ACI) is a new resource type in Azure that allows you to quickly and easily create containers without the complexity or overhead of Azure Service Fabric, Azure Container Services or provisioning a Windows Server 2016 VM.

It allows you to quickly create containers that are billed by the second from container images stored in Docker Hub or your own Azure Container Registry (ACR). Even though this feature is still in preview, it is very easy to get up and running with it.

But this post isn’t about creating basic container instances, it is about running container instances where some of the storage must persist. This is a basic function of a container host, but if you don’t have access to the host storage then things get more difficult. That said, Azure Container Instances do support mounting Azure File Shares into the container as volumes. It is fairly easy to do, but requires quite a number of steps.

There is some provided documentation for persisting storage in a container instance, but it is quite a manual process and the example ARM templates are currently broken: there are some typos and missing properties. So this post aims to make the whole thing a lot simpler and automatable.

So in this post, I’m going to share a PowerShell function and Azure Resource Manager (ARM) template that will allow you to easily provision an Azure Container Instance with an Azure File Share mounted. The process defaults to installing a GoCD Server container (version 17.8.0 if you’re interested), but you could use it to install any other Linux Container that needs persistent storage. The script is parameterized so other containers and mount points can be specified - e.g. it should be fairly easy to use this for other servers like Sonatype Nexus or Jenkins Server.

Update 2017-08-06: I documented my findings trying out these other servers in my following blog post.

Requirements

To perform this process you will need the following:

PowerShell 5.0+ (PowerShell 4.0 may work, but I haven’t tested it).
The Azure PowerShell module installed.
Created an Application Service Principal - see below.

Azure Service Principal

Before you start this process you will need to have created an Application Service Principal in Azure that will be used to perform the deployment. Follow the instructions on this page to create an application and then get the Service Principal from it.

You will need to record these values as they will be provided to the script later on:

Application Id
Application Key
Tenant Id
Subscription Name

The Process

The process will perform the following tasks:

The Service Principal is used to login to Azure to perform the deployment.
An Azure Resource Group is created to contain a Azure Storage Account and Azure Key Vault.
An Azure Storage Account is created and an Azure File Share is created in it.
An Azure Key Vault is created to store the Storage Account Key and make it accessible to the Azure Container Instance.
The Service Principal is granted permission to the Azure Key Vault to read and write secrets.
The key to the Storage Account Key is added as a secret to the Azure Key Vault.
The parameters are set in an ARM Template parameter file.
An Azure Resource Group is created to contain the Azure Container Instance.

The Script

This is the content of the script:

[CmdletBinding()]
param
(
    [Parameter(Mandatory = $True)]
    [String] $ServicePrincipalUsername,

    [Parameter(Mandatory = $True)]
    [SecureString] $ServicePrincipalPassword,

    [Parameter(Mandatory = $True)]
    [String] $TenancyId,

    [Parameter(Mandatory = $True)]
    [String] $SubscriptionName,

    [String] $AppCode = 'gocd', # just a short code to identify this app

    [String] $UniqueCode = 'dsr', # a short unique code to ensure that resources are unique

    [String] $ContainerImage = 'gocd/gocd-server:v17.8.0', # the container image name and version to deploy

    [String] $ContainerPort = '8153', # The port to expose on the container

    [String] $VolumeName = 'gocd', # The name of the volume to mount

    [String] $MountPoint = '/godata/', # The mount point

    [Int] $CPU = 1, # The number of CPUs to assign to the instance

    [String] $MemoryInGB = '1.5' # The amount of memory to assign to the instance
)

$supportRGName = '{0}{1}rg' -f $UniqueCode, $AppCode
$storageAccountName = '{0}{1}storage' -f $UniqueCode, $AppCode
$storageShareName = '{0}{1}share' -f $UniqueCode, $AppCode
$keyvaultName = '{0}{1}akv' -f $UniqueCode, $AppCode
$keyvaultStorageSecretName = '{0}key' -f $storageAccountName
$aciRGName = '{0}{1}acirg' -f $UniqueCode, $AppCode
$aciName = '{0}{1}aci' -f $UniqueCode, $AppCode
$location = 'eastus'

# Login to Azure using Service Principal
Write-Verbose -Message ('Connecting to Azure Subscription "{0}" using Service Principal account "{1}"' -f $SubscriptionName, $ServicePrincipalUsername)
$servicePrincipalCredential = New-Object -TypeName 'System.Management.Automation.PSCredential' -ArgumentList ($ServicePrincipalUsername, $ServicePrincipalPassword)
$null = Add-AzureRmAccount -TenantId $TenancyId -SubscriptionName $SubscriptionName -ServicePrincipal -Credential $servicePrincipalCredential

# Create resource group for Key Vault and Storage Account
if (-not (Get-AzureRmResourceGroup -Name $supportRGName -ErrorAction SilentlyContinue))
{
    Write-Verbose -Message ('Creating Resource Group "{0}" for Storage Account and Key Vault' -f $supportRGName)
    $null = New-AzureRmResourceGroup -Name $supportRGName -Location $location
}

# Create Key Vault
if (-not (Get-AzureRmKeyVault -ResourceGroupName $supportRGName -VaultName $keyVaultName -ErrorAction SilentlyContinue))
{
    Write-Verbose -Message ('Creating Key Vault "{0}" in Resource Group "{1}"' -f $keyVaultName, $supportRGName)
    $null = New-AzureRmKeyVault -ResourceGroupName $supportRGName -VaultName $keyVaultName -Location $location -EnabledForTemplateDeployment -EnabledForDeployment
}
Write-Verbose -Message ('Setting Key Vault "{0}" access policy to enable Service Principal "{1}" to Get,List and Set secrets' -f $keyVaultName, $ServicePrincipalUsername)
$null = Set-AzureRmKeyVaultAccessPolicy -ResourceGroupName $supportRGName -VaultName $keyVaultName -ServicePrincipalName $ServicePrincipalUsername -PermissionsToSecrets get, list, set
Write-Verbose -Message ('Getting Key Vault "{0}" Id' -f $keyVaultName)
$keyvaultNameId = (Get-AzureRmKeyVault -Name $keyVaultName).ResourceId

# Create Storage Account
if (-not (Get-AzureRmStorageAccount -ResourceGroupName $supportRGName -Name $storageAccountName -ErrorAction SilentlyContinue))
{
    Write-Verbose -Message ('Creating Storage Account "{0}" in Resource Group "{1}"' -f $storageAccountName, $supportRGName)
    $null = New-AzureRmStorageAccount -ResourceGroupName $supportRGName -Name $storageAccountName -SkuName Standard_LRS -Location $location
}
Write-Verbose -Message ('Getting Storage Account "{0}" key' -f $storageAccountName)
$storageAccountKey = Get-AzureRmStorageAccountKey -ResourceGroupName $supportRGName -Name $storageAccountName
$storageConnectionString = 'DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1};' -f $storageAccountName, $storageAccountKey[0].value
$storageContext = New-AzureStorageContext -ConnectionString $storageConnectionString
if (-not (Get-AzureStorageShare -Name $storageShareName -Context $storageContext -ErrorAction SilentlyContinue))
{
    Write-Verbose -Message ('Creating Azure Storage Share "{0}" in Storage Account {1}' -f $storageShareName, $storageAccountName)
    $null = New-AzureStorageShare -Name $storageShareName -Context $storageContext
}

# Add the Storage Key to the Key Vault
Write-Verbose -Message ('Adding Storage Account "{0}" key to Key Vault "{1}"' -f $storageAccountName, $keyvaultName)
$null = Set-AzureKeyVaultSecret -VaultName $keyvaultName -Name $keyvaultStorageSecretName -SecretValue (ConvertTo-SecureString -String $storageAccountKey[0].value -AsPlainText -Force)

# Create Azure Container Intstance
if (-not (Get-AzureRmResourceGroup -Name $aciRGName -ErrorAction SilentlyContinue))
{
    Write-Verbose -Message ('Creating Resource Group "{0}" for Container Group' -f $aciRGName)
    $null = New-AzureRmResourceGroup -Name $aciRGName -Location $location
}

# Generate the azure deployment parameters
$azureDeployParametersPath = (Join-Path -Path $PSScriptRoot -ChildPath 'aci-azuredeploy.parameters.json')
$azureDeployPath = (Join-Path -Path $PSScriptRoot -ChildPath 'aci-azuredeploy.json')

$azureDeployParameters = ConvertFrom-Json -InputObject (Get-Content -Path $azureDeployParametersPath -Raw)
$azureDeployParameters.parameters.containername.value = $aciName
$azureDeployParameters.parameters.containerimage.value = $ContainerImage
$azureDeployParameters.parameters.cpu.value = $CPU
$azureDeployParameters.parameters.memoryingb.value = $MemoryInGB
$azureDeployParameters.parameters.containerport.value = $ContainerPort
$azureDeployParameters.parameters.sharename.value = $storageShareName
$azureDeployParameters.parameters.storageaccountname.value = $storageAccountName
$azureDeployParameters.parameters.storageaccountkey.reference.keyVault.id = $keyvaultNameId
$azureDeployParameters.parameters.storageaccountkey.reference.secretName = $keyvaultStorageSecretName
$azureDeployParameters.parameters.volumename.value = $VolumeName
$azureDeployParameters.parameters.mountpoint.value = $MountPoint
Set-Content -Path $azureDeployParametersPath -Value (ConvertTo-Json -InputObject $azureDeployParameters -Depth 6) -Force

$deploymentName = ((Get-ChildItem -Path $azureDeployPath).BaseName + '-' + ((Get-Date).ToUniversalTime()).ToString('MMdd-HHmm'))
Write-Verbose -Message ('Deploying Container Group "{0}" to Resource Group "{1}"' -f $aciName, $aciRGName)
$null = New-AzureRmResourceGroupDeployment -Name $deploymentName `
    -ResourceGroupName $aciRGName `
    -TemplateFile $azureDeployPath `
    -TemplateParameterFile $azureDeployParametersPath `
    -Force `
    -ErrorVariable errorMessages

# Get the container info and display it
$subscriptionId = (Get-AzureRmSubscription -SubscriptionName $SubscriptionName).Id
$resourceId = ('/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.ContainerInstance/containerGroups/{2}' -f $subscriptionId, $aciRGName, $aciName)
$containerState = 'Unknown'
while ($containerState -ne 'Running')
{
    Write-Verbose -Message 'Waiting for container to enter running state'
    $containerResource = Get-AzureRmResource -ResourceId $resourceId
    $containerState = $containerResource.Properties.state
    Start-Sleep -Seconds 2
}

Write-Verbose -Message ('Container is running on http://{0}:{1}' -f $containerResource.Properties.ipAddress.ip, $containerResource.Properties.ipAddress.ports.port)

The script requires a four parameters to be provided:

ServicePrincipalUsername - the Application Id obtained when creating the Service Principal.
ServicePrincipalPassword - the Application Key we got (or set) when creating the Service Principal.
TenancyId - The Tenancy Id we got during the Service Principal creation process.
SubscriptionName - the name of the subscription to install the ACI and other resources into.

There are also some other optional parameters that can be provided that allow the container image that is used, the TCP port the container listens on and mount point for the Auzre File Share. If you don’t provide these parameters will be used which will create a GoCD Server.

AppCode - A short code to identify this application. It gets added to the resource names and resource group names. Defaults to ‘gocd’.
UniqueCode - this string is just used to ensure that globally unique names for the resources can be created. Defaults to ‘zzz’.
ContainerImage - this is the name and version of the container image to be deployed to the ACI. Defaults to ‘gocd/gocd-server:v17.8.0’.
CPU - The number of cores to assign to the container instance. Defaults to 1.
MemoryInGB - The amount of memory (in GB) to assign to the container instance. Defaults to 1.5.
ContainerPort - The port that the container listens on. Go CD Server defaults to 8153.
VolumeName - this is a volume name that is used to represent the volume in the ARM template. It can really be set to anything. Defaults to ‘gocd’.
MountPoint - this is the folder in the Container that the Azure File Share is mounted to. Defaults to ‘/godata/’.

ARM Template Files

There are two other files that are required for this process:

ARM template - the ARM template file that will be used to install the ACI.
ARM template parameters - this file will be used to pass in the settings to the ARM Template.

ARM Template

This file is called aci-azuredeploy.json and should be downloaded to the same folder as the script above.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "containername": {
      "type": "string"
    },
    "containerimage": {
      "type": "string"
    },
    "cpu": {
      "type": "int"
    },
    "memoryingb": {
      "type": "string"
    },
    "containerport": {
      "type": "string"
    },
    "sharename": {
      "type": "string"
    },
    "storageaccountname": {
      "type": "string"
    },
    "storageaccountkey": {
      "type": "securestring"
    },
    "volumename": {
      "type": "string"
    },
    "mountpoint": {
      "type": "string"
    }
  },
  "resources": [{
    "name": "[parameters('containername')]",
    "type": "Microsoft.ContainerInstance/containerGroups",
    "apiVersion": "2018-04-01",
    "location": "[resourceGroup().location]",
    "properties": {
      "containers": [{
        "name": "[parameters('containername')]",
        "properties": {
          "image": "[parameters('containerimage')]",
          "ports": [{
            "port": "[parameters('containerport')]"
          }],
          "resources": {
            "requests": {
              "cpu": "[parameters('cpu')]",
              "memoryInGb": "[parameters('memoryingb')]"
            }
          },
          "volumeMounts": [{
            "name": "[parameters('volumename')]",
            "mountPath": "[parameters('mountpoint')]"
          }]
        }
      }],
      "osType": "Linux",
      "ipAddress": {
        "type": "Public",
        "ports": [{
          "protocol": "tcp",
          "port": "[parameters('containerport')]"
        }]
      },
      "volumes": [{
        "name": "[parameters('volumename')]",
        "azureFile": {
          "shareName": "[parameters('sharename')]",
          "storageAccountName": "[parameters('storageaccountname')]",
          "storageAccountKey": "[parameters('storageaccountkey')]"
        }
      }]
    }
  }]
}

ARM Template Parameters

This file is called aci-azuredeploy.parameters.json and should be downloaded to the same folder as the script above.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "containername": {
            "value": ""
        },
        "containerimage": {
            "value": ""
        },
        "cpu": {
            "value": 1
        },
        "memoryingb": {
            "value": "1.5"
        },
        "containerport": {
            "value": ""
        },
        "sharename": {
            "value": ""
        },
        "storageaccountname": {
            "value": ""
        },
        "storageaccountkey": {
            "reference": {
                "keyVault": {
                    "id": ""
                },
                "secretName": ""
            }
        },
        "volumename": {
            "value": ""
        },
        "mountpoint": {
            "value": ""
        }
    }
}

Steps

To use the script the following steps need to be followed:

Download the three files above (the script and the two ARM template files) and put them into the same folder:
Open a PowerShell window.
Change directory to the folder you place the files into by executing:
CD <folder location>

Execute the script like this (passing in the variables):

.\Install-AzureContainerInstancePersistStorage.ps1 `
    -ServicePrincipalUsername 'ce6fca5e-a22d-44b2-a75a-f3b20fcd1b16' `
    -ServicePrincipalPassword (ConvertTo-SecureString -String 'JUJfenwe89hwNNF723ibw2YBybf238ybflA=' -AsPlainText -Force) `
    -TenancyId '8871b1ba-7d3d-45f3-8ee0-bb60c0e4733e' `
    -SubscriptionName 'Visual Studio Enterprise' `
    -AppCode 'gocd' `
    -UniqueCode 'mine' `
    -ContainerImage 'gocd/gocd-server:v17.8.0' `
    -ContainerPort '8153' `
    -VolumeName 'gocd' `
    -MountPoint '/godata/' `
    -Verbose

The process will then begin and make take a few minutes to complete:Note: I’ve changed the keys to this service principal and deleted this storage account, so I using these Service Principal or Storage Account keys won’t work!
Once completed you will be able to log in to the Azure Portal and find the newly created Resource Groups:
Open the resource group *gocdacirg and then select the container group *gocdaci**😗*
The IP Address of the container is displayed. You can copy this and paste it into a browser window along with the port the container exposed. In the case of Go CD it is 8153:
The process is now completed.

The Azure Container Instance can now be deleted and recreated at will, to reduce cost or simply upgrade to a new version. The Azure File Share will persist the data stored by the container into the mounted volume:

Hopefully this process will help you implement persisted storage containers in Azure Container Instances more easily and quickly.

Thanks for reading!

Publish an Azure RM Web App using a Service Principal in PowerShell

2017-07-12T00:00:00Z

Introduction

Deploying an Azure Web App is almost stupidly simple. If I were to list the methods and tools I’d still be typing next week. The problem with many of these tools and process is that they do a whole lot of magic under the hood which makes the process difficult to manage in source control.

I’m a big believer that all code (including deployment code) should be in the application source repository so it can be run by any tool or release pipeline - including manually by development teams. This ensures that whatever deployment process is used, it is the same no matter who or what runs it - and we end up continuously testing the deployment code and process.

So I decided to go and find out how to deploy an Azure Web App using PowerShell using an Service Principal.

Where is Publish-AzureRMWebsiteProject?

If you look through the Azure PowerShell cmdlets you’ll find a service manager one called Publish-AzureWebsiteProject. This cmdlet looks like it should do the trick, but it isn’t suitable because it requires authentication by a user account instead of a service principal.

Only service principal accounts can be authenticated using automation. Therefore using Publish-AzureWebsiteProject would only work if a development team member was able to interactively login- which would prevent the same process being used for automation or our continuous delivery pipeline. The newer Azure Resource Manager cmdlets (*-AzureRM*) all support a login using a service principal, but the problem is that there is no Publish-AzureRMWebsiteProject cmdlet.

So, to work around this limitation I determined I had to use Web Deploy/MSDeploy. The purpose of this post is to share the PowerShell function/code and process I used to do this. This will work with and without Web App deployment slots.

In my case our teams put all deployment code into a PowerShell PSake task in the application source code repository to make it trivial for anyone to run the deployment. The continuous delivery pipeline was also able to call the exact same task to perform the deployment. There is no requirement to use PowerShell PSake - just a simple PowerShell script will do.

The Code

So, I’ll start by just pasting the function that does performs the task:

[CmdletBinding()]
param (
    [Parameter(Mandatory = $True)]
    [pscredential]
    $Credential,

    [Parameter(Mandatory = $True)]
    [System.String]
    $TenantId,

    [Parameter(Mandatory = $True)]
    [System.String]
    $SubscriptionId,

    [Parameter(Mandatory = $True)]
    [System.String]
    $WebAppPath,

    [Parameter(Mandatory = $True)]
    [System.String]
    $ResourceGroupName,

    [Parameter(Mandatory = $True)]
    [System.String]
    $WebAppServiceName,

    [Parameter()]
    [System.String]
    $SlotName,

    [Parameter()]
    [System.String]
    $MSDeployPath = "$env:ProgramFiles\IIS\Microsoft Web Deploy V3\msdeploy.exe"
)

if (-not (Test-Path -Path $MSDeployPath))
{
    Throw "MSDeploy.exe not found at '$MSDeployPath'. Please install MSDeploy or specify the path to MSDeploy.exe on this system."
}

# Connect to Azure using SP
$connectParameters = @{
    Credential     = $Credential
    TenantId       = $TenantId
    SubscriptionId = $SubscriptionId
}

Write-Verbose -Message 'Connecting to Azure.'

$null = Add-AzureRmAccount @connectParameters -ServicePrincipal
  
# If a slot name is passed ensure all cmdlets use it
if ([String]::IsNullOrEmpty($SlotName)) {
    $slotParameters = $null
} else {
    $slotParameters = @{ Slot = $SlotName }
}

# Get the Publishing profile from Azure
$publishProfilePath = Join-Path -Path $ENV:Temp -ChildPath 'publishprofile.xml'

Write-Verbose -Message 'Getting publishing profile for web app'
$null = Get-AzureRmWebAppSlotPublishingProfile `
    -OutputFile $publishProfilePath `
    -Format WebDeploy `
    -ResourceGroupName $ResourceGroupName `
    -Name $WebAppServiceName `
    @slotParameters

# Stop the web app slot to make sure deployment is possible.
Write-Verbose -Message 'Stopping web app.'

$null = Stop-AzureRmWebAppSlot `
    -ResourceGroupName $ResourceGroupName `
    -Name $WebAppServiceName `
    @slotParameters

# Deploy the web site
$source = "-source:contentPath=$WebAppPath"
$dest = "-dest:contentPath=d:\home\site\wwwroot\,publishSettings=$publishProfilePath"

Write-Verbose -Message 'Publising web app content.'
& $MSDeployPath @('-verb:sync', $source, $dest)

# Start the web app back up
Write-Verbose -Message 'Starting web app.'

$null = Start-AzureRmWebAppSlot `
    -ResourceGroupName $ResourceGroupName `
    -Name $WebAppServiceName `
    @slotParameters

Write-Verbose -Message 'Waiting for web app to start up...'

$webappSlot = Get-AzureRmWebAppSlot `
    -ResourceGroupName $ResourceGroupName `
    -Name $WebAppServiceName `
    @slotParameters

# Wait for the site to report that it has started (optional)
while ($webappSlot.state -ne 'Running')
{
    Start-Sleep -Seconds 1

    Write-Verbose -Message 'Waiting for web app to start up...'
    $webappSlot = Get-AzureRmWebAppSlot `
        -ResourceGroupName $ResourceGroupName `
        -Name $WebAppServiceName `
        @slotParameters
}

Write-Verbose -Message 'Web app deployment complete.'

Just save this file as Publish-AzureRMWebappProject.ps1 and you’re ready to start publishing (almost).

Before you can use this function you’ll need to get a few things sorted:

Create a Service Principal with a password to use to deploy the web app using the instructions on this page.
Make sure you have got the latest version of the Azure PowerShell Modules installed (I used v4.0.0). See this page for instructions.
Make sure you’ve got MSDeploy.exe installed on your computer - see this page for instructions. You can pass the path to MSDeploy.exe into the Publish-AzureRMWebappProject.ps1 using the MSDeployPath parameter.
Gather the following things (there are many ways of doing that - but I’ll leave it up to you to figure out what works for you):
1. the Subscription Id of the subscription you’ll be deploying to.
2. the Tenant Id of the Azure Active Directory containing your Service Principal.
3. the Application Id that was displayed to you when you created the Service Principal.
4. the Password you assigned when you created the Service Principal.

Once you have got all this information you can call the script above like this:

$SubscriptionId = '3a54931f-5351-4ec4-9cf8-518e03257eff' # Not real
$TenantId = 'eef4615a-8a57-4519-99ea-e2a8bad20f82' # Not real
$Password = 'MyP@ssword99' # Not real
$Username = 'a3716a34-ae63-4ab8-8fb7-1e5f15ec3975' # Not real
$passwordSecure = ConvertTo-SecureString -String $Password -AsPlainText -Force
$Credential = New-Object -TypeName PSCredential ($Username, $passwordSecure)

.\Publish-AzureRMWebappProject `
    -Credential $Credential `
    -SubscriptionId $SubscriptionId `
    -TenantId $TenantId `
    -WebAppPath 'C:\Users\Dan\Source\MyAwesomeWebApp\debug\netcoreapp1.1\publish' `
    -ResourceGroupName 'MyAwesomeWebApp' `
    -WebAppServiceName 'WebApp' `
    -SlotName 'offline' `
    -Verbose

You’ll need to make sure to replace the variables $SubscriptionId, $TenantId, $Password and $Username with the values for your Azure Subscription, Tenancy and Service Principal.

When everything is done correctly this is what happens when you run it (with -Verbose enabled):

In the case above I was installing to a deployment staging slot called offline, so the new version of my website wouldn’t have been visible in my production slot until I called the Swap-AzureRmWebAppSlot cmdlet to swap the offline slot with my production slot.

All in all, this is fairly robust and allows our development teams and our automation and continuous delivery pipeline to all use the exact same deployment code which reduces deployment failures.

If you’re interested in more details about the code/process, please feel free to ask questions.

Thanks for reading.

Change the Friendly Name of a Cert with PowerShell

2017-06-09T00:00:00Z

While working on adding a new feature in the certificate request DSC resource, I came across this handy little trick: You can change the Friendly Name of a certificate using PowerShell.

All you need to do is identify the certificate using Get-ChildItem and then assign the new FriendlyName to it.

(Get-ChildItem -Path Cert:\LocalMachine\My\97CB2928C7AC163A750BF16CF1D2CF1A3DDAAA8E).FriendlyName = 'New Cert Name'

Sometimes PowerShell still surprises me at how easy it can make things. I didn’t need to search help or the internet - just typed it in and it worked!

Using Azure Key Vault with PowerShell - Part 1

2017-04-17T00:00:00Z

Azure Key Vault is used to safeguard and manage cryptographic keys, certificates and secrets used by cloud applications and services (you can still consume these on-premise though). This allows other application, services or users in an Azure subscription to store and retrieve these cryptographic keys, certificates and secrets.

Once cryptographic keys, certificates and secrets have been stored in a Azure Key Vault access policies can be configured to provide access to them by other users or applications.

Azure Key Vault also stores all past versions of a cryptographic key, certificate or secret when they are updated. So this allows easily rolling back if anything breaks.

This post is going to show how:

Set up an Azure Key Vault using the PowerShell Azure Module.
Set administration access policies on the Azure Key Vault.
Grant other users or applications access to cryptographic keys, certificates or secrets.
Add, retrieve and remove a cryptographic key from the Azure Key Vault.
Add, retrieve and remove a secret from the Azure Key Vault.

Requirements

Before getting started there is a few things that will be needed:

An Azure account. I’m sure you’ve already got one, but if not create a free one here.
The Azure PowerShell module needs to be installed. Click here for instructions on how install it.

Install the Key Vault

The first task is to customize and install the Azure Key Vault using the following PowerShell script.

# The name of the Azure subscription to install the Key Vault into
$subscriptionName = 'MySubscription'

# The resource group that will contain the Key Vault to create to contain the Key Vault
$resourceGroupName = 'MyKeyVaultRG'

# The name of the Key Vault to install
$keyVaultName = 'MyKeyVault'

# The Azure data center to install the Key Vault to
$location = 'southcentralus'

# These are the Azure AD users that will have admin permissions to the Key Vault
$keyVaultAdminUsers = @('Joe Boggs','Jenny Biggs')

# Login to Azure
Login-AzureRMAccount

# Select the appropriate subscription
Select-AzureRmSubscription -SubscriptionName $subscriptionName

# Make the Key Vault provider is available
Register-AzureRmResourceProvider -ProviderNamespace Microsoft.KeyVault

# Create the Resource Group
New-AzureRmResourceGroup -Name $resourceGroupName -Location $location

# Create the Key Vault (enabling it for Disk Encryption, Deployment and Template Deployment)
New-AzureRmKeyVault -VaultName $keyVaultName -ResourceGroupName $resourceGroupName -Location $location `
    -EnabledForDiskEncryption -EnabledForDeployment -EnabledForTemplateDeployment

# Add the Administrator policies to the Key Vault
foreach ($keyVaultAdminUser in $keyVaultAdminUsers) {
    $UserObjectId = (Get-AzureRmADUser -SearchString $keyVaultAdminUser).Id
    Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ResourceGroupName $resourceGroupName -ObjectId $UserObjectId `
        -PermissionsToKeys all -PermissionsToSecrets all -PermissionsToCertificates all
}

But first, the variables in the PowerShell script need to be customized to suit. The variables in the PowerShell script that needs to be set are:

$subscriptionName - the name of the Azure subscription to install the Key Vault into.
$resourceGroupName - the name of the Resource Group to create to contain the Key Vault.
$keyVaultName - the name of the Key Vault to create.
$location - the Azure data center to install the Key Vault to (use Get-AzureRMLocation to get a list of available Azure data centers).
$keyVaultAdminUsers - an array of users that will be given administrator (full control over cryptographic keys, certificates and secrets). The user names specified must match the full name of users found in the Azure AD assigned to the Azure tenancy.

It will take about 30 seconds for the Azure Key Vault to be installed. It will then show up in the Azure Subscription:

Assigning Permissions

Once the Azure Key Vault is setup and an administrator or two have been assigned, other access policies will usually need to be assigned to users and/or application or service principal.

To create an access policy to allow a user to get and list cryptographic key****s, certificates and secrets if you know the User Principal Name:

Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ResourceGroupName $resourceGroupName `
  -UserPrincipalName 'Joe.Boggs@contoso.com' `
  -PermissionsToCertificates list,get `
  -PermissionsToKeys list,get `
  -PermissionsToSecrets list,get

Tthe above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

If you only have the full name of the user then you’ll need to look up the Object Id for the user in the Azure AD:

$userObjectId = (Get-AzureRmADUser -SearchString 'Joe Bloggs').Id
Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ResourceGroupName $resourceGroupName `
  -ObjectId $userObjectId `
  -PermissionsToCertificates list,get `
  -PermissionsToKeys list,get `
  -PermissionsToSecrets list,get

The above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

To create an access policy to allow a service principal or application to get and list cryptographic key****s if you know the Application Id (a GUID):

Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ResourceGroupName $resourceGroupName `
  -ServicePrincipalName 'e9b1bc3c-4769-4a98-9014-b315fd2adf53' `
  -PermissionsToCertificates list,get `
  -PermissionsToKeys list,get `
  -PermissionsToSecrets list,get

The above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

Changing the values of the PermissionsToKeys**,** PermissionsToCertificates and PermissionsToSecrets parameters in the cmdlets above allow different permissions to be set for each policy.

The available permissions for certificates, keys and secrets are:

[gallery ids=“5624,5625” type=“rectangular”]

An access policy can be removed from users or service principals using the Remove-AzureRmKeyVaultAccessPolicy cmdet:

Remove-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ResourceGroupName $resourceGroupName `
  -UserPrincipalName 'Joe.Boggs@contoso.com'

The above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

Working with Secrets

Secrets can be created, updated, retrieved and deleted by users or applications that have been assigned with the appropriate policy.

Creating/Updating Secrets

To create a new secret, use the Set-AzureKeyVaultSecret cmdlet:

Set-AzureKeyVaultSecret -VaultName $keyVaultName -Name 'MyAdminPassword' `
  -SecretValue (ConvertTo-SecureString -String 'P@ssword!1' -AsPlainText -Force)

The above code assumes you still have the variables set from the ‘Install the Key Vault’ section.

This will create a secret called MyAdminPassword with the value P@ssword!1 in the Azure Key Vault.

The secret can be updated to a new value using the same cmdlet:

Set-AzureKeyVaultSecret -VaultName $keyVaultName -Name 'MyAdminPassword' `
  -SecretValue (ConvertTo-SecureString -String 'Sup3rS3cr3tP4ss!' -AsPlainText -Force)

Additional parameters can also be assigned to each version of a secret to control how it can be used:

ContentType - the type of content the secret contains (e.g. ‘txt’)
NotBefore - the date that the secret is valid after.
Expires - the date the secret is valid until.
Disable - marks the secret as disabled.
Tag - assigns tags to the secret.

For example:

Set-AzureKeyVaultSecret -VaultName $keyVaultName -Name 'MyAdminPassword' `
  -SecretValue (ConvertTo-SecureString -String 'Sup3rS3cr3tP4ss!' -AsPlainText -Force) `
  -ContentType 'txt' `
  -NotBefore ((Get-Date).ToUniversalTime()) `
  -Expires ((Get-Date).AddYears(2).ToUniversalTime()) `
  -Disable:$false `
  -Tags @{ 'Risk' = 'High'; }

Retrieving Secrets

To retrieve the latest (current) version of a secret, use the Get-AzureKeyVaultSecret cmdlet:

$secretText = (Get-AzureKeyVaultSecret -VaultName $keyVaultName -Name 'MyAdminPassword').SecretValue

This will assign the stored secret to the variable $secretText as a SecureString. This can then be passed to any other cmdlets that require a SecureString.

To list all the versions of a secret, add the IncludeVersions parameter:

Get-AzureKeyVaultSecret -VaultName $keyVaultName -Name 'MyAdminPassword' -IncludeVersions

To retrieve a specific version of a secret, use the Get-AzureKeyVaultSecret cmdlet with the Version parameter specified:

$secretText = (Get-AzureKeyVaultSecret -VaultName $keyVaultName -Name 'MyAdminPassword' -Version '02218af0521749b084bb08bd13184efb')

Removing Secrets

Finally, to remove a secret use the Remove-AzureKeyVaultSecret cmdlet:

Remove-AzureKeyVaultSecret -VaultName $keyVaultName -Name 'MyAdminPassword' -Force

That pretty much covers managing and using secrets in Azure Key Vault using PowerShell**.**

Cryptographic keys and Certificates

In the next part of this series I’ll cover using Azure Key Vault to use and manage cryptographic keys and certificates. Thanks for sticking with me this far.

Downloading GitHub .GitIgnore templates with PowerShell

2017-02-24T00:00:00Z

This will be a relatively short post today to get be back into the blogging rhythm. Most of my time has been spent of late working on the DSC Resource Kit adding code coverage reporting and new xCertificate features.

So, today’s post shows how you can use some simple PowerShell code to pull down the list of .gitIgnore templates from GitHub and then retrieve the one I wanted. There are lots of different ways I could have done this, but I decided to use the GitHub REST API.

First up, lets get the list of available .gitIgnore templates:

$templateList = (Invoke-WebRequest -URI 'https://api.github.com/gitignore/templates' -UseBasicParsing).Content |
    ConvertFrom-JSON

This will get the list of .GitIgnore templates to an array variable called $templateList. I could then display the list to a user:

Now, all I need to do is to download the named .gitIgnore Template to a folder:

Invoke-WebRequest -URI 'https://api.github.com/gitignore/templates/VisualStudio' -UseBasicParsing |
  Select-Object -ExpandProperty Content |
  ConvertFrom-JSON |
  Select-Object -ExpandProperty Source |
  Out-File -FilePath .\.gitignore

This will download the VisualStudio .giIgnore template and save it with the filename .gitignore to the current folder.

I could have specified a different .gitIgnore template by changing the VisualStudio in the URL to another template that appears in the $templateList.

You might have noticed that I included the -UseBasicParsing parameter in the Invoke-WebRequest call. This is to ensure the cmdlet works on machines that don’t have Internet Explorer installed - e.g. Nano Server or Linux/OSX. I haven’t tried this on PowerShell running on Linux or OSX, but I can’t see any reason why it wouldn’t work on those OS’s.

The next steps for this code might be to get these included as some new cmdlets in Trevor Sullivan’s PSGitHub PowerShell Module. You can download his module from the PowerShell Gallery if you’re not familiar with it.

Thanks for reading.

Using PFX Files in PowerShell

2017-01-10T00:00:00Z

One of the things I’ve been working on lately is adding a new resource to the xCertificate DSC Resource module for exporting an certificate with (or without) the private key from the Windows Certificate Store as a .CER or .PFX file. The very insightful (and fellow DSC Resource maintainer) @JohanLjunggren has been giving some really great direction on this new resource.

One of these suggested features was to be able to identify if the certificate chain within a PFX file is different to the chain in the Windows Certificate Store. This is because a PFX file can contain not just a single certificate but the entire trust chain required by the certificate being exported.

Therefore what we would need to do is be able to step through the certificates in the PFX and examine each one. It turns out this is pretty simple using the .NET Class:

System.Security.Cryptography.X509Certificates.X509Certificate2Collection

So, to read the PFX in to a variable called $PFX all we need to do is this:

$PFXPath = 'd:\Cert.pfx'
$PFXPassword = 'pass'
$PFX = New-Object -TypeName 'System.Security.Cryptography.X509Certificates.X509Certificate2Collection'
$PFX.Import($PFXPath,$PFXPassword,[System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::PersistKeySet)

The $PFXPath variable is set to the path to the PFX file we’re going to read in. The $PFXPassword is a string (not SecureString) containing the password used to protect the PFX file when it was exported.

We now have all the certificates loaded into an array in the $PFX variable and work with them like any other array:

Now, that we have the #PFX array, we can identify the thumbprint of the certificate that was actually exported (as opposed to the certificates in the trust chain) by looking at the last array item:

$PFX[$PFX.Count-1] | fl *

I’m piping the output Format-List so we can see the entire x509 certificate details.

In the case of the DSC Resource we’ll compare the certificate thumbprint of the last certificate in the PFX with the thumbprint that of the certificate in the Windows Certificate Store that we’re wanting to export. If they’re different we will then perform another export using the Export-PFXCertificate cmdlet.

Protip: You can actually verify the certificate and the entire trust chain is valid and not expired by calling the verify method on the last certificate:

foreach ($Cert in $PFX) { "$($Cert.Subject) is valid: $($Cert.Verify())" }

In the case above, the certificate I exported was actually invalid (it had expired):

So we could easily use the Validate method to test the certificates validity before we import them into the Windows Certificate Store. But beware, the Validate method will check that the certificate chain is trusted. To be trusted the entire chain must have been imported into the Windows Certificate Store in the appropriate stores (e.g. Trusted Root CA/Intermedicate CA stores).

So, finally this gives us the code required to implement the xCertificateExport Resource in the DSC Resource Kit. We can now perform a comparison of the certificates a PFX file to ensure that they are the same as the certificates that have already been exported.

This information is not something that you might use every day, but hopefully it’s information that someone might find useful. So thank you for taking the time to read this.

Test Website SSL Certificates Continuously with PowerShell and Pester

2016-12-23T00:00:00Z

One of the most common problems that our teams deal with is ensuring that SSL certificates are working correctly. We’ve all had that urgent call in telling us that the web site is down or some key API or authentication function is offline - only to find out it was caused by an expired certificate.

An easy way of preventing this situation would have been to set up a task that continuously tests your SSL endpoints (internal and external web apps and sites, REST API’s etc.) and warns us if:

The certificate is about to expire (with x days).
The SSL endpoint is using safe SSL protocols (e.g. TLS 1.2).
The certificate is using SHA256.

This seemed like a good task for Pester (or Operation Validation Framework). So, after a bit of digging around I found this awesome blog post from Chris Duck showing how to retrieve the certificate and SSL protocol information from an SSL endpoint using PowerShell.

Chris’ post contained this PowerShell cmdlet:

<#
    .DESCRIPTION
    Outputs the SSL protocols that the client is able to successfully use to connect to a server.

    .PARAMETER ComputerName
    The name of the remote computer to connect to.

    .PARAMETER Port
    The remote port to connect to. The default is 443.

    .EXAMPLE
    Test-SslProtocol -ComputerName "www.google.com"

    ComputerName       : www.google.com
    Port               : 443
    KeyLength          : 2048
    SignatureAlgorithm : rsa-sha1
    Ssl2               : False
    Ssl3               : True
    Tls                : True
    Tls11              : True
    Tls12              : True

    .NOTES
    Copyright 2014 Chris Duck
    http://blog.whatsupduck.net

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
#>
function Test-SslProtocol {
    param(
        [Parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true,ValueFromPipeline=$true)]
        $ComputerName,

        [Parameter(ValueFromPipelineByPropertyName=$true)]
        [int]$Port = 443
    )
    begin {
        $ProtocolNames = [System.Security.Authentication.SslProtocols] |
            Get-Member -Static -MemberType Property |
            Where-Object -Filter { $_.Name -notin @("Default","None") } |
            Foreach-Object { $_.Name }
    }
    process {
        $ProtocolStatus = [Ordered]@{}
        $ProtocolStatus.Add("ComputerName", $ComputerName)
        $ProtocolStatus.Add("Port", $Port)
        $ProtocolStatus.Add("KeyLength", $null)
        $ProtocolStatus.Add("SignatureAlgorithm", $null)

        $ProtocolNames | %{
            $ProtocolName = $_
            $Socket = New-Object System.Net.Sockets.Socket( `
                [System.Net.Sockets.SocketType]::Stream,
                [System.Net.Sockets.ProtocolType]::Tcp)
            $Socket.Connect($ComputerName, $Port)
            try {
                $NetStream = New-Object System.Net.Sockets.NetworkStream($Socket, $true)
                $SslStream = New-Object System.Net.Security.SslStream($NetStream, $true)
                $SslStream.AuthenticateAsClient($ComputerName,  $null, $ProtocolName, $false )
                $RemoteCertificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]$SslStream.RemoteCertificate
                $ProtocolStatus["KeyLength"] = $RemoteCertificate.PublicKey.Key.KeySize
                $ProtocolStatus["SignatureAlgorithm"] = $RemoteCertificate.SignatureAlgorithm.FriendlyName
                $ProtocolStatus["Certificate"] = $RemoteCertificate
                $ProtocolStatus.Add($ProtocolName, $true)
            } catch  {
                $ProtocolStatus.Add($ProtocolName, $false)
            } finally {
                $SslStream.Close()
            }
        }
        [PSCustomObject]$ProtocolStatus
    }
} # function Test-SslProtocol

So that was the hard part done, all I needed was to add this function to some Pester tests.

Note: If you are running these tests on an operating system older than Windows 10 or Windows Server 2016 then you will need to install the PowerShell Pester module by running this command in an Administrator PowerShell console:
Install-Module -Name Pester

So after a little bit of tinkering I ended up with a set of tests that I combined into the same file as Chris’ function from earlier. I called the file SSL.tests.ps1. I used the file extension .tests.ps1 because that is the file extension Pester looks for when it runs.

The tests are located at the bottom of the file below the Test-SslProtocol function.

<#
    .DESCRIPTION
    Outputs the SSL protocols that the client is able to successfully use to connect to a server.

    .PARAMETER ComputerName
    The name of the remote computer to connect to.

    .PARAMETER Port
    The remote port to connect to. The default is 443.

    .EXAMPLE
    Test-SslProtocol -ComputerName "www.google.com"

    ComputerName       : www.google.com
    Port               : 443
    KeyLength          : 2048
    SignatureAlgorithm : rsa-sha1
    Ssl2               : False
    Ssl3               : True
    Tls                : True
    Tls11              : True
    Tls12              : True

    .NOTES
    Copyright 2014 Chris Duck
    http://blog.whatsupduck.net

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
#>
function Test-SslProtocol {
    param(
        [Parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true,ValueFromPipeline=$true)]
        $ComputerName,

        [Parameter(ValueFromPipelineByPropertyName=$true)]
        [int]$Port = 443
    )
    begin {
        $ProtocolNames = [System.Security.Authentication.SslProtocols] |
            Get-Member -Static -MemberType Property |
            Where-Object -Filter { $_.Name -notin @("Default","None") } |
            Foreach-Object { $_.Name }
    }
    process {
        $ProtocolStatus = [Ordered]@{}
        $ProtocolStatus.Add("ComputerName", $ComputerName)
        $ProtocolStatus.Add("Port", $Port)
        $ProtocolStatus.Add("KeyLength", $null)
        $ProtocolStatus.Add("SignatureAlgorithm", $null)

        $ProtocolNames | %{
            $ProtocolName = $_
            $Socket = New-Object System.Net.Sockets.Socket( `
                [System.Net.Sockets.SocketType]::Stream,
                [System.Net.Sockets.ProtocolType]::Tcp)
            $Socket.Connect($ComputerName, $Port)
            try {
                $NetStream = New-Object System.Net.Sockets.NetworkStream($Socket, $true)
                $SslStream = New-Object System.Net.Security.SslStream($NetStream, $true)
                $SslStream.AuthenticateAsClient($ComputerName,  $null, $ProtocolName, $false )
                $RemoteCertificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]$SslStream.RemoteCertificate
                $ProtocolStatus["KeyLength"] = $RemoteCertificate.PublicKey.Key.KeySize
                $ProtocolStatus["SignatureAlgorithm"] = $RemoteCertificate.SignatureAlgorithm.FriendlyName
                $ProtocolStatus["Certificate"] = $RemoteCertificate
                $ProtocolStatus.Add($ProtocolName, $true)
            } catch  {
                $ProtocolStatus.Add($ProtocolName, $false)
            } finally {
                $SslStream.Close()
            }
        }
        [PSCustomObject]$ProtocolStatus
    }
} # function Test-SslProtocol

# List of Web sites that we want to check the SSL on
$WebSitesToTest = @(
    'www.google.com'
    'www.bing.com'
    'www.yahoo.com'    
)

# Number of days out to warn about certificate expiration
$WarningThreshold = 14

Describe 'SSL endpoints' {
    foreach ($WebSite in $WebSitesToTest) {
        Context $WebSite {
            $script:SSLResult = Test-SslProtocol -ComputerName $WebSite -Port 443
            
            It 'Should have Signature Algorithm of sha256RSA' {
                $script:SSLResult.SignatureAlgorithm | Should -Be 'sha256RSA'
            }
            
            It 'Should support TLS1.2' {
                $script:SSLResult.TLS12 | Should -BeTrue
            }
            
            It "Should not going to expire in $WarningThreshold days" {
                ($script:SSLResult.Certificate.NotAfter -gt (Get-Date).AddDays($WarningThreshold))| Should -BeTrue
            }
        }
    }
}

So, now to test these SSL endpoints all I need to do is run in a PowerShell console with the current folder set to the folder containing my SSL.tests.ps1 file:

cd C:\SSLTests\
Invoke-Pester

This is the result:

This shows that all the SSL endpoint certificates being used by google.com, bing.com and yahoo.com are all valid SHA-256 certificates and aren’t going to expire in 14 days.

All I would then need to do is put this in a task to run every hour or so and perform some task when the tests fail:

cd c:\
if ((Invoke-Pester -PassThru).FailedCount -gt 0) {
  Write-Host "An SSL Endpoint test failed. Notify someone here!"
}

At this point you will still need to use some mechanism to notify someone when they fail. One method could be to write an event into the Windows Event Log and then use Microsoft Operations Management Suite (or SCOM) to monitor for this event and send an e-mail or other alert to the appropriate administrators.

For an example showing how to use OMS to monitor custom events created by failed Pester and OVF tests, see my previous article here.

Potential Improvements

There are a number of ways you could go about improving this process, which our teams have in fact implemented. If you’re considering implementing this process then you might want to also consider them:

Put the Test-SSLProtocol cmdlet into a PowerShell Module that you can share easily throughout your organization.
Put your tests into source control and have the task clone the tests directly from source control every time they are run - this allows tests to be stored centrally and can be change tracked.
Parameterize the tests so that you don’t have to hard code the endpoints to test in the script file. Parameters can be passed into Pester tests fairly easily.
Use something like Jenkins, SCOM or Splunk to run the tests continuously.
Run the tests in an Azure Automation account in the Cloud.

Really, the options for implementing this methodology are nearly limitless. You can engineer a solution that will work for you and your teams, using whatever tools are at your disposal.

At the end of the day, the goal here should be:

Reduce the risk that your internal or external applications or websites are using bad certificates.
Reduce the risk that an application or website will be deployed without a valid certificate (write infrastructure tests before you deploy your infrastructure - TDD for operations).
Reduce the risk you’ll get woken up in the middle of the night with an expired certificate.

So, in this holiday season, I hope this post helps you ensure your certificates won’t expire in the next two weeks and you won’t get called into fix a certificate problem when you should be lying on a beach in the sun (in the southern hemisphere anyway).

Have a good one!

Continuously Testing your Infrastructure with OVF and Microsoft Operations Management Suite

2016-10-22T00:00:00Z

Introduction

One of the cool new features in Windows Server 2016 is Operation Validation Framework. Operation Validation Framework (OVF) is an (open source) PowerShell module that contains:

A set of tools for executing validation of the operation of a system. It provides a way to organize and execute Pester tests which are written to validate operation (rather than limited feature tests)

One of the things I’ve been using OVF for is to continuously test parts of my infrastructure. Any time a failure occurs an error event is written to the Event Log. I then have Microsoft Operations Management Suite set up to monitor my Event Log for any errors and then alert me (by e-mail) if they occur.

Note: OVF tests are just Pester tests, so if you’ve ever written a Pester test then you’ll have no trouble at all with OVF. If you haven’t written a Pester test before, here is a great introduction. If you’re going to just learn one thing about PowerShell this year, I’d definitely suggest Pester.

The great thing about OVF is that it can test anything that PowerShell can get information about - which is practically anything. For some great examples showing how to use OVF to test your infrastructure, see this article or this article by the insightful Irwin Strachan.

In this article I’m going to show you how to set up a basic set of OVF tests to continuously test some DNS components on a server, write any failures to the Event Log and then configure Microsoft OMS to monitor the Event Log for OVF failures and alert you if and something breaks.

I am calling my tests ValidateDNS which is reflected in the files and Event Log Source for the events that are created, but you can call these tests what ever you like. I’m also going to create my tests and related files as a PowerShell Module with the same name (ValidateDNS). You don’t have to do this - you could make a much simpler process, but for me it made sense because I could then publish my set of tests to my own private PowerShell Gallery or to a Sonatype Nexus OSS server.

I am also going to assume you’ve got an Microsoft Operations Management Suite account all setup. If you don’t have an OMS account, you can get a free trial one here. You should also have the OMS Windows Agent installed onto the machine that will execute your OVF tests.

Let’s Do This!

Step 1 - Installing the OperationValidation Module

The OperationValidation PowerShell module comes in-box with Windows Server 2016 (and Windows 10 AE), but must be downloaded for earlier operating systems.

To download and install the OperationValidation PowerShell module on earlier operating systems enter the following cmdlet in an Administrator PowerShell console:

Install-Module -Name OperationValidation

This will download the module from the PowerShell Gallery.

Note: to download modules from the PowerShell Gallery you’ll either need WMF 5.0 or PowerShell PackageManagement installed. I strongly recommend the former option if possible.

Step 2 - Create OVF Test Files

The next step is to create the OVF tests in a file. OVF work best if they are contained in a specific folder structure in the PowerShell Modules folder.

Note: you can create the the OVF tests in a different location if that works for you, but it requires specifying the -testFilePath parameter when calling Invoke-OperationValidation.

Create a folder in your PowerShell Modules folder (c:\program files\WindowsPowerShell\Modules) with the name of your tests. I used ValidateDNS.
In the ValidateDNS folder create the following folder structure:
- Diagnostics\
  - Simple_\_
  - Comprehensive_\_

In the Simple folder create a file called ValidateDNS.Simple.Tests.ps1 with the contents:

Describe 'DNS' {
    It 'Should be running' {
        (Get-Service -Name DNS).Status | Should Be Running
    }
        
    $Forwarders = Get-DnsServerForwarder
        
    It 'First forwarder should be 8.8.8.8' {
        $Forwarders.IPAddress[0] | Should Be '8.8.8.8'
    }

    It 'Second forwarder should be 4.4.4.4' {
        $Forwarders.IPAddress[1] | Should Be '4.4.4.4'
    }

    It 'Should resolve microsoft.com' {
        { Resolve-DnsName -Server LocalHost -Name microsoft.com } | Should Not Throw
    }
}

Edit the tests and create any that are validate the things you want to test for.

The OVF tests above just check some basic settings of a DNS Server and so would normally be run on a Windows DNS Server. As noted above, you could write tests for almost anything, including validating things on other systems. I intentionally have setup one of the tests to fail for demonstration purposes (a gold star for anyone who can tell which test will fail).

In a future article I’ll cover how to test components on remote machines so you can use a single central node to perform all your OVF testing.

Step 3 - Create a Module for Running the Tests

Although we could just run the tests as is, the output will just end up in the console, which is not what we want here. We want any failed tests to be put into the Application Event Log**.**

Create a file called ValidateDNS.psm1 in the ValidateDNS folder created earlier.

Add the following code to this ValidateDNS.psm1 file:

function Invoke-ValidateDNS {
    [cmdletbinding()]
    param (
        [String] $EventSource = 'ValidateDNS',

        [Int32] $EventId = 10000,

        [ValidateSet('Simple','Comprehensive')]
        [String] $TestType = 'Simple'
    )

    # Edit these settings

    # Add the Event Source if it doesn't exist
    if (-not [system.diagnostics.eventlog]::SourceExists($EventSource)) {
        [system.diagnostics.EventLog]::CreateEventSource($EventSource, 'Application')
    } # if

    # Execute the tests
    $FailedTests = Invoke-OperationValidation -ModuleName ValidateDNS -TestType $TestType |
        Where-Object -Property Result -EQ -Value 'Failed'

    # Add the Failed tests to the Event Log
    foreach ($FailedTest in $FailedTests) {
        Write-EventLog `
            -LogName Application `
            -Source $EventSource `
            -EntryType Error `
            -Message $FailedTest.Name `
            -EventId $EventId `
            -Category 0
        $EventId++
    } # foreach
}

Save the ValidateDNS.psm1

The above file is a PowerShell Module will make available a single cmdlet called Invoke-ValidateDNS. We can now just run Invoke-ValidateDNS in a PowerShell console and the following tasks will be performed:

create a new Event Source for the Applications Event Log that we can use in OMS to identify any errors thrown by our tests.
Execute the OVF tests in ValidateDNS.Simple.Tests.ps1.
Add Error entries to the Applications Event Log for each failed test.

Step 4 - Schedule the Script

This step we will create a Scheduled Task to run the cmdlet we created in Step 3. You could use the Task Scheduler UI to do this, but this is a PowerShell blog after all, so here is a script you can run that will create the scheduled task:

$Cred = Get-Credential -Message 'User to run task as'
$Action = New-ScheduledTaskAction `
    –Execute 'PowerShell.exe' `
    -Argument '-WindowStyle Hidden -Command "Import-Module ValidateDNS; Invoke-ValidateDNS;"'
$Trigger = New-ScheduledTaskTrigger `
    -Once `
    -At (Get-Date -Hour 0 -Minute 0 -Second 0) `
    -RepetitionInterval (New-TimeSpan -Minutes 60) `
    -RepetitionDuration ([System.TimeSpan]::MaxValue)
$Task = New-ScheduledTask `
    -Description 'Validate DNS' `
    -Action $Action `
    -Trigger $Trigger
Register-ScheduledTask `
    -TaskName 'Validate DNS' `
    -InputObject $Task `
    -User $Cred.UserName `
    -Password $Cred.GetNetworkCredential().Password

You will be prompted for the account details to run the task under, so enter valid credentials for this machine that give the task the correct access to run the tests. E.g. if the tests need Local Administrator access to the machine to run correctly, then ensure the account assigned is a Local Administrator.

This will run the script every 60 minutes. You could adjust it easily to run more or less frequently if you want to. This is what the Task Scheduler UI will show:

Every time the tests run and a test failure occurs the Application Event Log will show:

Now that we have any test failures appearing in the Event Log, we can move onto Microsoft Operations Management Suite.

Step 5 - Create a Log Search and Alert

As noted earlier, I’m assuming you have already set up the computer running your OVF tests to your OMS account as a data source:

What we need to do now is create and save a new Log Search that will select our OVF test failures. To do this:

In OMS, click the Log Search button.
In the Search box enter (adjust the Source= if you used a different name for your tests in earlier tests):
(Type=Event) (EventLevelName=error) (Source=ValidateDNS)
You will now be shown all the events on all computers matching these criteria:From here you could further refine your search if you want, for example, I could have added additional filters on Computer or EventId. But for me this is all I needed.
Click Save to save the Log Search.
In the Name enter ‘Validate DNS Events’ and in the Category enter ‘OVF’:
You can actually enter whatever works for you here.
Click Save.
Click the Alert button to add a new Alert Rule.
Configure the Alert Rule as follows (customizing to suit you):
Click Save to save the Alert.

You’re now done!

The DNS Admins will now receive an e-mail whenever any of the DNS validation tests fail:

If you look down in the ParamaterXML section you can even see the test that failed. So the DNS Admins can dive straight to the root of the problem.

How cool is that? Now we can feel more confident that problems will be noticed by our technical teams when they happen rather than waiting for an end user to complain.

Of course the tests above are fairly basic. They are just meant as an example of what sort of things can be done. Some of our teams have put together far more comprehensive sets of tests that validate things like ADFS tokens and SSL certificate validity.

Final Thoughts

There are a few things worth pointing about the process above:

I chose to use OVF to execute the tests but I could have just as easily used plain old Pester. If it makes more sense to you to use Pester, go right ahead.
I used Microsoft OMS to centrally monitor the events, but I could just of easily used Microsoft System Center Operations Manager (SCOM). There are many other alternatives as well. I chose OMS though because of the slick UI and mobile apps. Use what works for you.
This guide is intended to show what sort of thing can be done. It isn’t intended to tell you what you must do or how you must do it. If there is something else that works better for you, then use it!

Although I’ve focused on the technology and methods here, if you take away one thing from this article I’d like it to be this:

Continuous Testing of your infrastructure is something that is really easy to implement and has so many benefits. It will allow you and your stakeholders to feel more confident that problems are not going unnoticed and allow them to sleep better. It will also ensure that when things do go wrong (and they always do) that the first people to notice are the people who can do something about it!

Happy infrastructure testing!

Install Docker on Windows Server 2016 using DSC

2016-10-15T00:00:00Z

Windows Server 2016 is now GA and it contains some pretty exciting stuff. Chief among them for me is support for containers by way of Docker. So, one of the first things I did was start installing Windows Server 2016 VM’s (Server Core and Nano Server naturally) and installing Docker on them so I could begin experimenting with Docker Swarms and other cool stuff.

Edit: If you’re looking for a DSC configuration for setting up Docker on a Windows 10 Anniversary Edition machine, see the Windows 10 AE section below.

At first I started using the standard manual instructions provided by Docker, but this doesn’t really suit any kind of automation or infrastructure as code methodology. This of course was a good job for PowerShell Desired State Configuration (DSC).

So, what I did was put together a basic DSC config that I could load into a DSC Pull Server and build out lots of Docker nodes quickly and easily. This worked really nicely for me to build out lots of Windows Server 2016 Container hosts in very short order:

If you don’t have a DSC Pull server or you just want a simple script that you can use to quickly configure a Windows Server 2016 (Core or Core with GUI only) then read on.

Note: This script and process is really just an example of how you can configure Docker Container hosts with DSC. In a real production environment you would probably want to use a DSC Pull Server.

Get it Done

Edit: After a suggestion from Michael Friis (@friism) I have uploaded the script to the PowerShell Gallery and provided a simplified method of installation. The steps could be simplified even further into a single line, but I’ve kept them separate to show the process.

Using PowerShell Gallery

On a Windows Server 2016 Server Core or Windows Server 2016 Server Core with GUI server:

Log on as a user with Local Administrator privileges.
Start an Administrator PowerShell console - if you’re using Server Core just enter PowerShell at the command prompt:
Install the Install-DockerOnWS2016UsingDSC.ps1 script from the PowerShell Gallery using this command:
```
Install-Script -Name Install-DockerOnWs2016UsingDSC
```
You may be asked to confirm installation of these modules, answer yes to any confirmations.
Run the Install-DockerOnWS2016UsingDSC.ps1 script using:
```
Install-DockerOnWs2016UsingDSC.ps1
```

The script will run and reboot the server once. Not long after the reboot the Docker service will start up and you can get working with containers:

You’re now ready to start working with Containers.

The Older Method (without PowerShell Gallery)

On a Windows Server 2016 Server Core or Windows Server 2016 Server Core with GUI server:

Log on as a user with Local Administrator privileges.
Start an Administrator PowerShell console - if you’re using Server Core just enter PowerShell at the command prompt:
Install the DSC Resources required for the DSC configuration by executing these commands:
```
Install-Module -Name xPSDesiredStateConfiguration
Install-Module -Name xPendingReboot
```
You may be asked to confirm installation of these modules, answer yes to any confirmations.

Download the Docker installation DSC script by executing this command:

Invoke-WebRequest -Uri 'https://gist.githubusercontent.com/PlagueHO/d9595cae1788f436b97bd4c90d50d72e/raw/1146baa2b1e0c8b3869004074b4c97bf71ce9c3c/Install-DockerOnWS2016ByDSC.ps1' -OutFile 'Install-DockerOnWS2016ByDSC.ps1'

Run the Docker installation DSC script by executing this command:
```
.\Install-DockerOnWS2016ByDSC.ps1
```

The script will run and reboot the server once. Not long after the reboot the Docker service will start up and you can get working with containers:

You’re now ready to start working with Containers.

What the Script Does

In case you’re interested in what the script actually contains, here are the components:

Configuration ContainerHostDsc - the DSC configuration that configures the node as a Docker Container host.
Configuration ConfigureLCM - the LCM meta configuration that sets Push Mode, allows the LCM to reboot the node if required and configures ApplyAndAutoCorrect mode.
ConfigData - a ConfigData object that contains the list of node names to apply this DSC Configuration to - in this case LocalHost.
ConfigureLCM - the call to the Configuration ConfigureLCM to compile the LCM meta configuration MOF file.
Set-DscLocalConfigurationManager - this applies the compiled LCM meta configuration MOF file to LocalHost to configure the LCM.
ContainerHostDsc - the call to the Configuration ContainerHostDsc to compile the DSC MOF file.
Start-DSCConfiguration - this command starts the LCM applying the DSC MOF file produces by the ContainerHostDsc.

The complete script can be found here. Feel free to use this code in anyway that makes sense to you.

What About Windows 10 AE?

If you’re looking for a DSC configuration that does the same thing for Windows 10 Anniversary edition, Ben Gelens (@bgelens) has written an awesome DSC config that will do the trick. Check it out here.

Happy containering!

Easily Create a Hyper-V Windows Server 2016 AD & Nano Server Lab

2016-10-04T00:00:00Z

Introduction

One of the PowerShell Modules I’ve been working on for the last year is called LabBuilder.The goal of this module is:

To automatically build a multiple machine Hyper-V Lab environment from an XML configuration file and other optional installation scripts.

What this essentially does is allow you to easily build Lab environments using a specification file. All you need to do is provide the Hyper-V environment and the Operating System disk ISO files that will be used to build the lab. This is great for getting a Lab environment spun up for testing or training purposes.

Note: Building a new Lab can take a little while, depending on the number of VM’s in the Lab as well as the number of different Operating Systems used. For example, a Lab with 10 VMs could take an hour or two to spin up, depending on your hardware.

The LabBuilder module comes with a set of sample Labs that you can build “as is” or modify for your own purpose. There are samples for simple one or two machine Labs as well as more complex scenarios such as failover clusters and two tier PKI environments. Plus, if you’re feeling adventurous you can easily create your own LabBuilder configurations from scratch or by modifying an existing LabBuilder configuration.

In this article I’ll show how to use a configuration sample that will build a lab containing the following servers:

1 x Windows Server 2016 RTM Domain Controller (with DNS)
1 x Windows Server 2016 RTM DHCP Server
1 x Windows Server 2016 RTM Certificate Authority Server
1 x Windows Server 2016 RTM Edge Node (Routing and Remote Access server)
8 x Windows Server 2016 RTM Nano Servers (not yet automatically Domain Joined - but I’m working on it).

This is a great environment for experimenting with both Windows Server 2016 as well as Nano Server.

So, lets get started.

Requirements

To follow along with this guide your Lab host (the machine that will host your Lab) will need to have the following:

Be running Windows Server 2012 R2, Windows Server 2016 or Windows 10

I strongly recommend using Windows 10 Anniversary Edition.

If you are using Windows Server 2012 R2 you will need to install WMF 5.0 or above. Although WMF 4.0 should work, I haven’t tested it.

Have enough RAM, Disk and CPU available for your Lab

Running a lot of VMs at once can be fairly taxing on your hardware. For most Sample Lab I’d recommend at least a quad core CPU, 16 GB RAM and a fast SSD with at least 10 GB per VM free (although for Nano Server VMs only 800MB is required).

The amount of disk used is minimized by using differencing disks, but Labs can still get pretty big.

Hyper-V Enabled

If you’re using Windows 10, see this guide.

If you’re using Windows Server 2012 R2 or Windows Server 2016, you probably already know how to do this, so I won’t cover this here.

Copies of any Windows install media that is used by the Lab

In our case this is just a copy of the Windows Server 2016 Evaluation ISO. You can download this ISO from here for free.

You can use non-evaluation ISOs instead if you have access to them, but at the time of writing this the Windows Server 2016 non-evaluation ISO wasn’t yet available on my MSDN subscription.

An Internet Connection

Most Labs use DSC to configure each VM once it has been provisioned, so the ability to download any required DSC Resources from the PowerShell Gallery is required. Some sample Labs also download MSI packages and other installers that will be deployed to the Lab Virtual Machines during installation - for example RSAT is often installed onto Windows 10 Lab machines automatically.

The Process

Step 1 - Install the Module

The first thing you’ll need to do is install the LabBuilder Module. Execute this PowerShell command at an Administrator PowerShell prompt:

Install-Module -Name LabBuilder

Note: If you have an older version of LabBuilder installed**, I’d recommend you update it to at least 0.8.3.1081 because this was the version I was using to write this guide.**

Step 2 - Create the ISOs and VHDs Folders

Most labs are built using Windows Install media contained in ISO files. These are converted to VHD files that are then used by one or more Labs. We need a location to store these files.

By default all sample Labs expect these folders to be D:\ISOs and D:\VHDs. If you don’t have a D: Drive on your computer, you’ll need to adjust the LabBuilder configuration file in Step 4.

Execute the following PowerShell commands at an Administrator PowerShell prompt:

New-Item -Path 'd:\ISOs' -ItemType Directory
New-Item -Path 'd:\VHDs' -ItemType Directory

Step 3 - Create a Folder to Contain the Lab

When building a Lab with LabBuilder it will create all VMs, VHDs and other related files in a single folder.

For all sample LabBuilder configurations, this folder defaults to a folder in C:\vm. For the sample Lab we’re building in this guide it will install the Lab into c:\vm\NANOTEST.COM. This can be changed by editing the configuration in Step 4.

Note: Make sure you have enough space on your chosen drive to store the Lab. 10GB per VM is a good rough guide to the amount of space required (although it usually works out as a lot less because of the use of differencing disks).

Execute the following PowerShell commands at an Administrator PowerShell prompt:

New-Item -Path 'c:\VM' -ItemType Directory

Step 4 - Customize the Sample Lab file

We’re going to build the Lab using the sample Lab found in the samples folder in the LabBuilder module folder. The sample we’re using is called Sample_WS2016_NanoDomain.xml. I’d suggest editing this file in an editor like Notepad++.

If you changed the paths in Step 2 or Step 3 then you’ll need to change the paths shown in this screenshot:

You may also change other items in the Settings section, but be aware that some changes (such as changing the domain name) will also need to be changed elsewhere in the file.

If you already have an External Switch configured in Hyper-V that you’d like to use for this Lab to communicate externally, then you should set the name of the switch here:

If you don’t already have an External Switch defined in Hyper-V then one called General Purpose External will be created for you. It will use the first Network Adapter (physical or team) that is not already assigned to an External Switch. You can control this behavior in the LabBuilder configuration file but it is beyond the scope of this guide.

Save the Sample_WS2016_NanoDomain.xml once you’ve finished changing it.

Step 5 - Copy the Windows Media ISOs

Now that the ISOs folder is ready, you will need to copy the Windows Install media ISO files into it. In this case we need to copy in the ISO for Windows Server 2016 (an evaluation copy can be downloaded from here).

The ISO file must be name:

14393.0.160715-1616.RS1_RELEASE_SERVER_EVAL_X64FRE_EN-US.ISO

If it is named anything else then you will either need to rename it or go back to Step 4 and adjust the sample Lab configuration file.

Step 6 - Build the Lab

We’re now ready to build the lab from the sample configuration.

Execute the following PowerShell commands at an Administrator PowerShell prompt:

$ConfigPath = Join-Path `
  -Path (Split-Path -Path (Get-Module -Name LabBuilder -ListAvailable).Path -Parent) `
  -ChildPath 'Samples\Sample_WS2016_NanoDomain.xml'
Install-Lab -ConfigPath $ConfigPath -Verbose

This will begin the task of building out your Lab. The commands just determine the location of your LabBuilder sample file and then call the Install-Lab cmdlet. I could have specified the path to the sample file manually, and you can if you prefer.

So sit back and grab a tea or coffee (or beer), because this will take a little while.

Note: The individual virtual machines are configured using PowerShell DSC after they are first started up. This means that it might actually take some time for things like domain joins and other post configuration tasks to complete. So if you find a Lab VM hasn’t yet joined the domain, it is most likely that the DSC configuration is still being applied.

Using the Lab

Once you’ve built the Lab, you can log into the VMs like any other Hyper-V VM. Just double click the Virtual Machine and enter your login details:

For the sample Lab the Domain Administrator account password is configured as P@ssword!1. This is set in the Lab Sample configuration and you can change it if you like.

Note: Nano Server is not designed to have an interactive GUI. You interact with Nano Server via PowerShell Remoting. You’ll want to have a basic knowledge of PowerShell and PowerShell Remoting before attempting to administer Nano Servers**.**

Shutting Down the Lab

Once the Lab has been completely built, you can shut it down with the Stop-Lab command. You need to pass the path to the Lab Configuration file to shut it down:

$ConfigPath = Join-Path `
  -Path (Split-Path -Path (Get-Module -Name LabBuilder -ListAvailable).Path -Parent) `
  -ChildPath 'Samples\Sample_WS2016_NanoDomain.xml'
Stop-Lab -ConfigPath $ConfigPath -Verbose

The Virtual Machines in the Lab will be shut down in an order defined in the Lab Configuration file. This will ensure that the VMs are shut down in the correct order (e.g. shut down the domain controllers last).

Starting the Lab Up

If you need to start up a previously created Lab, use the Start-Lab command. You will again need to provide the path to the Lab Configuration file of the Lab you want to shut down:

$ConfigPath = Join-Path `
  -Path (Split-Path -Path (Get-Module -Name LabBuilder -ListAvailable).Path -Parent) `
  -ChildPath 'Samples\Sample_WS2016_NanoDomain.xml'
Start-Lab -ConfigPath $ConfigPath -Verbose

The Virtual Machines in the Lab will be started up in an order defined in the Lab Configuration file. This will ensure that the VMs are started up in the correct order.

Uninstalling the Lab

If you want to completely remove a Lab, use the Uninstall-Lab command. You will again need to provide the path to the Lab Configuration file of the Lab you want to unisntall:

$ConfigPath = Join-Path `
  -Path (Split-Path -Path (Get-Module -Name LabBuilder -ListAvailable).Path -Parent) `
  -ChildPath 'Samples\Sample_WS2016_NanoDomain.xml'
Uninstall-Lab -ConfigPath $ConfigPath -Verbose

Note: You will be asked to confirm the removals.

Wrapping Up

This article has hopefully given you a basic understanding of how to use LabBuilder to stand up a Hyper-V Lab in relatively short order and without a lot of commands and clicks. This project is still in Beta and so there may be bugs as well as some incomplete features. If you want to raise an issue with this project (or even submit a PR), head on over to the GitHub repository.

Export a Base-64 x.509 Cert using PowerShell on Windows 7

2016-09-10T00:00:00Z

Exporting a Base-64 Encoded x.509 certificate using PowerShell is trivial if you have the Export-Certificate cmdlet available. However, many of the nodes I work with are Windows 7 which unfortunately doesn’t include these cmdlets. Therefore I needed an alternate method of exporting these Base-64 encoded x.509 certificates from these nodes.

So I came up with this little snippet of code:

$certificate = Get-ChildItem -Path Get-ChildItem -path Cert:\CurrentUser\My\D675AE3AE9F7B56348C17EE527F261CFCEA0FD13
$base64certificate = @"
-----BEGIN CERTIFICATE-----
$([Convert]::ToBase64String($certificate.Export('Cert'), [System.Base64FormattingOptions]::InsertLineBreaks)))
-----END CERTIFICATE-----
"@
Set-Content -Path "$ENV:USERPROFILE\Documents\MyCert.cer" -Value $base64certificate

Hope someone finds it useful.

Tips for HQRM DSC Resources

2016-08-14T00:00:00Z

I’ve spent a fair amount of time recently working on getting some of my DSC Resources (SystemLocaleDsc, WSManDsc, iSCSIDsc and FSRMDsc) accepted into the Microsoft DSC Community Resource Kit. Some are nearly there (SystemLocaleDsc and WSManDsc), whereas others have a way to go yet.

I’ve had one resource already accepted (xDFS) into the DSC Community Resource kit, but this was before the High Quality Resource Module (HQRM) guidelines became available. The HQRM guidelines are a set of standards that DSC modules must meet and maintain to be considered a High Quality Resource Module. Once they meet these requirements they may be eligible to have the ‘x’ moniker removed with ‘Dsc’ being added to the name.

More information: If you want to read a bit more about the HQRM standards, you can find the HQRM Guidelines here.

Any modules being submitted for inclusion into the DSC Community Resource kit will be expected to meet the HQRM standards. The process of acceptance requires three reviewers from the Microsoft DSC team to review the module.

I thought it might be helpful to anyone else who might want to submit a DSC Resource into the DSC Community Resource kit to get a list of issues the reviewers found with my submissions. This might allow you to fix up your modules before the review process - which will help the reviewers out (they hate having to be critical of your code as much as you do). This enables the submission process to go much faster as well.

More information: If you want to read more about the submission process, you can find the documentation here.

I’ll keep this post updated with any new issues the reviewers pick up. Feel free to ask for clarifications on the issues.

So here is my list of what I have done wrong (so far):

Missing Get-Help Documentation

Every function (public or private) within the DSC resource module must contain a standard help block containing at least a .SYNOPSIS and .PARAMETER block:

This will get rejected:

This is good:

Examples Missing Explanation

All examples in the Examples folder and the Readme.md must contain an explanation of what the example will do.

This is bad:

This is good:

Old or Incorrect Unit/Integration Test Headers

There is a standard method of unit and integration testing DSC Resources. Your DSC resources should use these methods where ever possible. Any tests should therefore be based on the latest unit test templates and integration test templates. You should therefore ensure your tests are based on the latest practices and contain the latest header.

This is probably the hardest thing to get right if you’re not paying close attention to the current DSC community best practices around testing. So feel free to ask me for help.

This is bad:

This is good:

Incorrect Capitalization of Local Variables

Local variables must start with a lower case letter. I needed to correct this on several occasions.

Note: this is for local variables. Parameter names should start with Uppercase.

This is bad:

This is good:

Spaces around = in Localization Files

In any localization files you should make sure there is a space on either side of the = sign. This greatly improves message readability.

This is bad:

This is good:

Missing code of Conduct in Readme.md

All modules that are part of the DSC Resource Kit must contain this message in the Readme.md:

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

This is bad:

This is good:

Missing Localization file indent

All strings in localization files should be indented.

This is bad:

This is good:

Final Words

There were some other issues raised which I will also document, however I am still in discussion with the DSC team over the best methods to use to solve the issues (specifically the use of InModuleScope in unit tests).

The main thing you can do to help speed this process up and reduce the load on the reviewers however is to implement all the best practices and guidelines listed.

I hope this helps someone out there.

Failed to Start Docker Service on Windows 10 AE

2016-08-04T00:00:00Z

So, pretty much the first thing I did when the Windows 10 Anniversary Edition was installed onto my primary development machine was to installer the Windows Container Service and Docker on it.

I used the Windows Containers on Windows 10 Quick start guide to perform the installation. This is the same method I’d been using on my secondary development machine (running Insider Preview builds) since it was first available in build 14372.

Note: The Windows Containers on Windows 10 Quick Start guide doesn’t mention the Anniversary Edition specifically, but the method still works.

Unfortunately though, this time it didn’t work. When I attempted to start the Docker Service I received the error:

start-service : Failed to start service ‘Docker Engine (docker)’.

So, after a bit of digging around I found the following error in the Windows Event Log in the Application logs:

Basically what this was telling me was that the Docker Daemon couldn’t create the new virtual network adapter that it needed - because it already existed. So a quick run of Get-NetAdapter and I found that the docker adapter “vEthernet (HNS Internal)” already existed:

So what I needed to do was uninstall this adapter so that the Docker Service could recreate it. I’m not actually aware of a command line method of doing (except for using DevCon) so I had to resort to using Device Manager:

You’ll need to use the output of the Get-NetAdapter to find he right adapter uninstall. Once it has been uninstalled you should be able to start the service again:

This time the service should start successfully. A quick call to docker ps shows that the container service is indeed working. So now I can get onto the process pulling down the base container images.

Hopefully if anyone else runs into this problem in Windows 10 AE this will help them resolve it.

Allow PowerShell to Traverse a Secure Proxy

2016-06-24T00:00:00Z

One of the first things I like to do when setting up my development machine in a new environment is to update PowerShell help with the update-help cmdlet. After that, I will then go and download a slew of modules from the PowerShell Gallery.

However, recently I needed to set up my development machine on an environment that is behind an internet proxy that requires authentication. This meant that a lot of PowerShell cmdlets can’t be used because they don’t have support for traversing a proxy - or at least, not one that requires authentication. Take the aforementioned update-help and install-module cmdlets - I just couldn’t do with out these.

So I set about trying to find a way around this. So after lots of googling and trial and error (and also getting my Active Directory account locked out on more than one occasion) I came up with a solution.

Basically it requires using the NETSH command to configure the proxy settings and then configure the web client with my proxy credentials (which were AD integrated):

$ProxyAddress = 'http://myproxy.contoso.com'
$ProxyCredentials = Get-Credential
$null = & netsh @('winhttp','set','proxy',$ProxyAddress)
$webclient=New-Object System.Net.WebClient
$webclient.Proxy.Credentials = $ProxyCredentials

The code I needed to traverse the proxy could then be executed. Once it has completed the task using the proxy I would then reset it back to the default state (using settings from internet explorer):

$null = & netsh @('winhttp','import','proxy','source=ie')

After using this a bit I thought it would be great to turn it into a function that I could just call, passing a script block that I wanted to be able to traverse the proxy with. So I came up with this:

function Use-Proxy {
    param (
        [Parameter(Mandatory)]
        [ScriptBlock] $ScriptBlock,

        [String] $ProxyAddress = 'http://myproxy.contoso.com',

        [PSCredential] $ProxyCredentials
    )

    # If proxy creendials weren't passed in then see if global proxy creds
    # are set. Global Proxy credentials can save some typing by setting
    # them in the PS session.
    if (-not ($PSBoundParameters.ContainsKey('ProxyCredentials')))
    {
        if ($Global:ProxyCredentials)
        {
            $ProxyCredentials = $Global:ProxyCredentials
        }
        else
        {
            $ProxyCredentials = Get-Credential -Message "Enter proxy credentials"
        } # if
    } # if

    # Configure the proxy
    $null = & netsh @('winhttp','set','proxy',$ProxyAddress)
    try
    {
        $webclient=New-Object System.Net.WebClient
        $webclient.Proxy.Credentials = $ProxyCredentials
        # Call the actual code
        Invoke-Command -ScriptBlock $ScriptBlock
    }
    catch
    {
        Throw $_
    }
    finally
    {
        # Reset the proxy
        $null = & netsh @('winhttp','import','proxy','source=ie')
    } # try
} # Function Use-Proxy

To use this script, simply save it as a PS1 file (e.g. Use-Proxy.ps1), customizing the default proxy URL if you like and then dot source the file. Once it has been dot sourced it can be called, optionally passing the URL of the proxy server and credentials to use to authenticate to it:

. c:\Scripts\Use-Proxy.ps1
Use-Proxy {
  Update-Help
  Install-Module -Name Pester
  Install-Module -Name PSScriptAnalyzer
}

If you don’t pass any credentials, you will be prompted to enter them. I also added some code into this function so that you can specify a global variable containing the credentials to use to traverse the proxy. This can save on lots of typing, but might be frowned upon by your security team.

Finally, I also added the proxy reset code into the finally block of a try…catch to ensure that if the code in the script block throws an error the proxy will be reset. In my case I also loaded this function into a PowerShell module that can be distributed to other team members.

Happy proxy traversing!

VisualStudio and ISE Steroids Vertical Guides

2016-05-20T00:00:00Z

To make it easier for reviewers and other programmers to read PowerShell code it is recommended that lines of PowerShell code don’t exceed 100 characters. If you run past this limit you should usually split the line using a backtick (`). I also find that this limit prompts me to rethink my code logic if the line gets too long. Besides, no one wants to have to scroll several pages horizontally to be able to read your whole line of code.

However, one of the most common issues I run into when reviewing (or writing my own) PowerShell DSC resources is going over the 100 character limit. For your own projects you’re most welcome to use any styles that you want (although I’d still recommend this one for the reasons above). However, if you’re going to commit this code to the PowerShell DSC community resources you’ll need to ensure your code meets this requirement, otherwise reviewers will likely ask you change it.

But there is an easy way to help identify this problem if you’re using Visual Studio Code or PowerShell ISE Steroids: Setup a vertical guide/ruler at 100 characters.

A nice easy to see vertical ruler/guide tells us when our lines get too long… oh dear this line is too long.

To Set up a Vertical Ruler in Visual Studio Code

Select User Settings (you can use Workspace Settings if you want) from Preferences in the File menu:
Your user settings.json file will load on the right, with the Default Settings on the left:
Add the following line to your settings.json file in between the braces:
“editor.rulers”: [100]
Save the file and restart Visual Studio Code.

To Set up a Vertical Ruler in ISE Steroids

Select Show Secondary Toolbar from the View menu.
Click the Vertical Guides button and select Global Guide…
Enter 100 in the input box and press enter.
The vertical guide is added - better fix that long line!

That is all there is to using vertical guides. Having them setup if you’re planning on committing code to a community project might save you some extra commits and help the community reviewers merge your code faster.

Happy coding!

DSC Resource Kit - Anniversary Release

2016-05-19T00:00:00Z

The latest DSC Resource Kit (all your favorite DSC Resources in one handy pack) is available now. It is one mighty release with all sorts of awesomeness included! I strongly recommend picking it up if you’re doing DSC automation, as it has something for everyone.

Happy automating!

Bulk Updating Nano Servers using PowerShell and CIM

2016-05-14T00:00:00Z

Introduction

Feel free to skip this introduction and jump down to the Updating via CIM section to see the actual update process.

Yesterday, I decided to connect my Windows Server 2016 TP5 Active Directory lab to the new Azure Server Management Tools. This was mainly to look at the experience with managing Nano Server using these new Azure-based tools.

The interface is very slick, as you’d expect from anything managed within the new Azure Portal.

I did run into a minor glitch when installing updates, but the team over at Azure (major thanks @BrendanPower) had the problem identified and solved within 24 hours (and it was rolled out while I was writing this post). Also remember, this is an Azure Preview Feature updating a Windows Server 2016 Technical Preview 5 operating system, so you’d expect a few glitches. I did a lot of experimenting with these new features, and this was the only thing I ran into. I’m still extremely impressed with the responsiveness of the team over there in Redmond.

In fact, while I was writing this very post, I was contacted to let me know the fix had gone into production, and it worked perfectly:

Anyway, this post isn’t about how to configure an Azure Server Management Tools Gateway (it is extremely easy, but if anyone would be interested in a video, let me know and I’ll make one). It is about updating Nano Servers using CIM and, by extension, updating lots of servers in one go.

Updating via CIM

In Windows Nano Server TP4, Microsoft included CIM cmdlets. This enabled us to use the root/Microsoft/Windows/WindowsUpdate CIM Namespace to use Windows Update to install updates on a Nano Server. After a bit of searching around, I found this blog post covering the process.

However, I have lots of Nano Servers, and updating them one at a time would be a real pain. So I decided to write a short PowerShell snippet to update all of them at once. This snippet should actually work with any Windows Server 2016 TP4 (or greater) version.

function Get-AvailableUpdates {
    [CmdletBinding()]
    param (
        [PSCredential] $Credential = (Get-Credential -Message 'Credentials to use to update Servers'),

        [Parameter(Mandatory)]
        [ValidateNotNullOrEmpty()]
        [String[]] $Servers,

        [Switch] $Install,
        
        [Switch] $Restart
    )
    $servers | ForEach-Object {
        $session = New-CimSession `
            -ComputerName $_ `
            -Credential $credential
        $instance = New-CimInstance `
            -Namespace root/Microsoft/Windows/WindowsUpdate `
            -ClassName MSFT_WUOperationsSession `
            -CimSession $session
        # Due to a bug in CIM on Nano Server (TP4 and TP5), an error is returned when
        # there are no available updates.
        # We use ErrorAction SilentlyContinue to ignore this (DON'T do this in a production script!!!!)
        $scanResults = @($Instance | Invoke-CimMethod `
            -MethodName ScanForUpdates `
            -Arguments @{SearchCriteria="IsInstalled=0";OnlineScan=$true} `
            -CimSession $session `
            -ErrorAction SilentlyContinue)
        if ($scanResults) {
            "$_ has $($scanResults.Count) updates to be installed:"
            if ($Install) {
                $installResult = $Instance | Invoke-CimMethod `
                    -MethodName ApplyApplicableUpdates `
                    -CimSession $Session
                if ($installResult.ReturnValue -eq 0) {
                    'Updates were installed successfully:'
                    $scanResults.Updates
                    if ($Restart) {
                        "Restarting $_"
                        Invoke-Command `
                            -ComputerName $_ `
                            -Credential $credential `
                            -ScriptBlock { Restart-Computer }
                    } else {
                        'You may need to reboot this server for update installation to complete.'
                    }
                } else {
                    'An error occurred installing updates:'
                    $installResult
                }
            } else {
                'Set -Install flag to install updates'
                $scanResults.Updates
            }
        } else {
            "$_ has no updates to be installed."
        }
        Remove-CimSession `
            -CimSession $session
    }
}

This snippet contains a simple function that takes three parameters:

Credential - This is the credentials to use to connect to the Nano Server and update it. If not passed, you will be presented with a login dialog asking for them. It is assumed that all servers being updated use the same credentials.
Servers - This is the array of server names to apply updates to.
Install - Setting this optional switch will cause the updates to be installed. If it is not set, you will just be told for each server the list of updates that are required.
Restart - This optional switch will cause the servers to automatically restart after updates are installed (even if they don’t technically need a restart). Only set this switch if the Install switch is set.

For example, calling the function with:

Get-AvailableUpdates `
    -Servers 'SA-NANO1','SA-NANO2','SA-NANO3','SA-NANO4','SA-NANO5','SA-NANO6','SA-NANO7','SA-NANO8'

Would show the update status of the eight listed Nano Servers and show any updates available to be installed:

However, adding an -Install parameter to the call:

Get-AvailableUpdates `
    -Servers 'SA-NANO1','SA-NANO2','SA-NANO3','SA-NANO4','SA-NANO5','SA-NANO6','SA-NANO7','SA-NANO8' `
    -Install

Will cause all available updates to be installed onto the listed Nano Servers:

You may still need to restart these servers after the updates are installed. If you run the function again without restarting the servers first, you will be told the updates still need to be installed. If you want the servers to automatically be restarted, add a -Restart parameter:

Get-AvailableUpdates `
    -Servers 'SA-NANO1','SA-NANO2','SA-NANO3','SA-NANO4','SA-NANO5','SA-NANO6','SA-NANO7','SA-NANO8' `
    -Install `
    -Restart

Like this:

This is even more useful when you consider that you can update standard (Core and GUI) Windows Server 2016 installations like this as well. This might also work on earlier versions of Windows Server (2012, 2012 R2) as well, but I don’t have time to try this out.

Edit: 21 May 2016 - I tested this on Windows Server 2012 R2, but it does not work because the required CIM classes are not available on that version of the Operating System.

Note: I really haven’t put much work into error checking or reporting on this process, so if you run into any errors (servers not online, bad credentials, etc.), then they might not be handled elegantly. This code is really an example of what can be easily done.

This function would be a definite candidate for a PowerShell Workflow—allowing complete parallelization of the process.

I hope this is useful, and have a great weekend!

Using a Windows Virtual NAT with a Hyper-V Lab

2016-05-11T00:00:00Z

One of the new features introduced into Windows in build 10586 and above was the new NAT Virtual Switch. This feature was primarily introduced to ease the introduction of the Windows Containers in the upcoming release of Windows Server 2016.

In more recent builds of Windows (build 14295 and above) the NAT Virtual Switch has been removed in favor of a new Virtual NAT Device that exists separate from the Hyper-V Virtual Switch.

This new Virtual NAT Device is more inline with Microsoft’s Software Defined Networking approach. It also allows us to create multiple Hyper-V Lab environments where each Lab is completely isolated from any others but still be connected to the Internet by way of the Virtual NAT Device.

Previously, to give all the machines in a Lab internet access we would have had to use:

An External Switch - Connect all machines to an External Virtual Switch that was connected to the internet via one of the Hyper-V Host’s network adapters.
A Guest NAT - Install a NAT onto one of the Guest Virtual Machines in the Lab. For example, install Windows Server 2012 R2 with the Remote Access role and configure a NAT. This would still require at least this node in the Lab to be connected to the internet via an External Virtual Switch.

Each of these approaches had some drawbacks:

Each Lab was not completely isolated from the other labs.
An entire guest might need to be provisioned to provide internet access to the other machines in the Lab.

But using the Virtual NAT device allows us to configure Labs with complete network isolation but still being connected to the internet without the use of a guest NAT.

So, to configure a pair of Labs like in the diagram above all we need is to execute a few PowerShell Cmdlets.

Note: Make sure your Hyper-V host is at least build 14295 (Windows 10 build 14295 or Windows Server 2016 TP5). Otherwise these cmdlets will fail.

If you want some more detail on setting up a Virtual NAT, see Set up a NAT Network.

Configure Hyper-V Lab with NAT

To configure a Hyper-V Lab with NAT, perform the following steps, executing any PowerShell cmdlets in an Administrator PowerShell console.

Create a Hyper-V Internal Virtual Switch on your Host: [sourcecode language=“powershell”] New-VMSwitch -Name Lab1 -SwitchType Internal [/sourcecode] This will also create a Virtual Network Adapter connected to the host.
Assign the gateway IP address of the NAT to the Virtual Network Adapter: [sourcecode language=“powershell”] # Get the MAC Address of the VM Adapter bound to the virtual switch $MacAddress = (Get-VMNetworkAdapter -ManagementOS -SwitchName Lab1).MacAddress # Use the MAC Address of the Virtual Adapter to look up the Adapter in the Net Adapter list $Adapter = Get-NetAdapter | Where-Object { (($_.MacAddress -replace ‘-’,‘’) -eq $MacAddress) } New-NetIPAddress –IPAddress 192.168.140.1 -PrefixLength 24 -InterfaceIndex $Adapter.ifIndex [/sourcecode]
Create the Virtual NAT device: [sourcecode language=“powershell”] New-NetNat –Name Lab1NAT –InternalIPInterfaceAddressPrefix 192.168.140.0/24 [/sourcecode]
Configure the network settings on each guest virtual network adapter assigned to the virtual switch in the 192.168.140.0/24 subnet and configure the default gateway to be 192.168.140.1.

That’s it - all machines in the Lab should have access to the internet and be completely isolated as well. Naturally I have updated the LabBuilder system to support this new functionality as well.

I hope this was useful and happy NATing.

cDFS is dead, long live xDFS

2016-05-11T00:00:00Z

The xDFS DSC resource module has been officially released to the PowerShell Gallery thanks to the awesome review efforts of the Microsoft PowerShell Team. The cDFS DSC Resource has now been unlisted from the PowerShell Gallery. So now is the time to update any DSC configuration scripts to use xDFS.

Important: There were some minor changes to xDFS when it was converted from cDFS. For information on what you’ll need to change to convert to xDFS see my earlier post.

cDFS moving to the PowerShell Team

2016-05-06T00:00:00Z

Just a Friday afternoon heads up - if you’re using the cDFS DSC Resource I created to manage Windows Server Distributed File System (Replication and Namespaces), it has now been accepted into the PowerShell Community resources and will be under the control of the PowerShell Team.

This means that the GitHub source code repository will be moving over to the PowerShell organization in the next few days. This also means that any future releases of this resource module won’t be provided by me as cDFS, but will be released by the PowerShell team as xDFS.

So I recommend that when this happens you switch over to using the xDFS resource. I will put another post up here when the change over officially occurs. The first official release version under the new xDFS name will be 3.0.0.x. I won’t make any further changes or bug fixes to the cDFS resources.

It is also worth noting that as part of this move some minor changes were made to the DSC Resource modules. These are breaking changes and you will most likely need to update any DSC Configurations depending on this, but you would have to do this anyway because of the name change.

The changes are:

Resource xDFSRepGroup renamed to xDFSReplicationGroup
Resource xDFSRepGroupConnection renamed to xDFSReplicationGroupConnection
Resource xDFSRepGroupFolder renamed to xDFSReplicationGroupFolder
Resource xDFSRepGroupMembership renamed to xDFSReplicationGroupMembership
xDFSReplicationGroupConnection:
- Changed DisableConnection parameter to EnsureEnabled.
- Changed DisableRDC parameter to EnsureRDCEnabled.

These changes should only require minor changes to your configuration scripts to implement.

Thanks for reading and have a great Friday~

Nano Server Packages in Windows Server 2016 TP5

2016-04-28T00:00:00Z

Happy TP5 day!

Here is just a quick snapshot of the package list for Nano Server in Windows Server 2016 TP5:

Right off the bat I notice a few new ones are included:

BootFromWim
SecureStartup
ShieldedVM

I can guess at the purpose of these packages, but will be interested to learn a bit more about them. That is all I have time for as I now need to go and update the LabBuilder sample projects to use this new ISO.

Install a VMWare ESXi 6.0 Hypervisor in a Hyper-V VM

2016-04-22T00:00:00Z

Update 19th February 2018:

This article has had a lot more attention than I ever expected! This has brought to light several issues with the process as well as changes made to Hyper-V and ESXi that break the process. You could wade through all 256 comments and assemble the corrections yourself, but user @Gambit has helpfully done this for me! So I’ve included the summary here:

@Tom Watson: Adding a line to /etc/vmware/config with vmx.allowNested = "TRUE". Since all VMs will be running nested (whole point of article), this is a must if you want them to start! The alternative is adding this line to every VMX file for each guest. Tom’s solution was much more elegant.
@burnett437: Configure ESXi Management vSwitch to “accept” “promiscuous mode”. Apparently, this was a known issue even in nested ESXi 5.5. While the host network will operate for a while without a problem, the “promiscuous mode” policy will eventually be tripped, and you won’t be able to talk to the host at random times. Now I think this has to do with the nature of nested virtual switches (a VMWare vSwitch inside of a Hyper-V Virtual Switch). When this happens, I found you could “down/up” the management vnicX to get it back, but just set the setting to not worry about it.
@me: Network threat detection doesn’t like nested virtual switches either. My Symantec Endpoint Client would occasionally block traffic coming from my local NIC (vSphere Client/vSphere Converter) to this nested Host (Windows Firewall did NOT seem to care), but SEP occasionally tripped because of all “suspicious” addressing to the nested host.
@me: Use vSphere converter for guest images coming from other VMWare products….duh. ESXi doesn’t like split VHDKs; the guest won’t boot. This is a beginner mistake, but this article is for Hyper-V admins that may not know the subtle nuances between VMWare products.
@RichMD: I like the idea of hardware pass-through on the NIC. I may tinker with this because it may resolve several issues, not just the ESXi v6.5 NIC blacklist issue. Theoretically, passing the physical NIC directly to the nested ESXi Host could/should resolve the “promiscuous mode,” “Network Threat Protection,” and even the “Half Duplex Legacy Adapter Requirement” (very slow network performance) problems.

The Original Article Starts Here

Recently I’ve been playing around with the new Hyper-V Nested Virtualization feature within Windows 10 (build 10565 and greater) and Windows Server 2016. It is pretty cool to be able to create virtualized lab environments containing Hyper-V clusters. But what if we want a lab that contains VMWare ESXi Hypervisors running on a Hyper-V host? I couldn’t find the process documented anywhere and couldn’t even confirm if it should be possible. But after asking a lot of annoying questions—thanks Adam Burns—Googling, and hair-pulling, I managed to get it going:

So this seems like a good topic for a blog post.

What You’ll Need

You are going to need a few things to get this working:

A Hyper-V host running on Windows 10 (build 10565 or greater) or Windows Server 2016 TP4.
Enable-NestedVM.ps1 - A PowerShell script for enabling Nested Virtualization in a Hyper-V VM. Click here to get the file from the Microsoft team on GitHub.
A VMWare account - just sign up for one here if you don’t already have one.
VMWare PowerShell CLI installed - I used the 6.3 release 1 that I downloaded from here.
ESXi-Customizer-PS.ps1 - A PowerShell script for injecting network drivers into an ESXi 5.x/6.x ISO. I downloaded it from here.

I suggest you download all of the above items to a working folder—I called mine D:\ESX-In-Hyper-V, so these instructions will reflect that, but you can call your folder whatever you like.

You should end up with a folder containing these files:

And before you ask: No, you don’t need a VMWare ESXi 6.0 ISO—this will get downloaded and produced for us.

The Process

Part 1 - Prepare an ESXi 6.0 ISO with Network Drivers

The biggest problem I ran into when trying to install ESXi onto Hyper-V was that the ESXi kernel doesn’t come with drivers for the Microsoft Virtual Network Adapter or the Microsoft Legacy Network Adapter (emulates a DECchip 21140). So you’ll need to inject these drivers into the VMWare ESXi 6.0 ISO. Luckily, there is a script available and the appropriate drivers DECchip 21140 (called “net-tulip” for some reason) that makes this process a breeze:

Install VMWare PowerCLI.
Open a PowerShell console.

Enter the following commands:

CD D:\ESX-In-Hyper-V\
.\ESXi-Customizer-PS-v2.4.ps1 -v60 -vft -load net-tulip

After a few minutes, the VMWare ESXi 6.0 ISO will be downloaded, and the “net-tulip” drivers merged with it:

The ISO will now be available in the D:\ESX-In-Hyper-V folder:

Part 2 - Create the Hyper-V VM

In Hyper-V Manager, create a new Virtual Machine:
Click Next.
Select Generation 1 and click Next.
Set the Startup Memory to at least 4096MB.
Uncheck Use Dynamic Memory for this Virtual Machine:
Click Next.
Don’t bother to Configure Networking on the next step—just click Next.
Select Create a new virtual hard disk and set the Size to 10GB (this is just going to be the boot disk for the ESXi Hypervisor):
Click Next.
Select Install an operating system from a bootable CD/DVD-ROM.
Select Image file (.iso) and browse to the ISO created in Part 1:
Click Next, then click Finish to create the Virtual Machine:
Right-click the new Virtual Machine and select Settings.
Select the Processor node and increase the Number of Virtual Processors to at least 2:
Select the existing Network Adapter node and click Remove:
Select the Add Hardware node and select Legacy Network Adapter:
Click Add:
Select a Virtual Switch to connect the ESXi Host to.
Click OK.

The rest of the process involves enabling nested virtualization, booting the ESXi VM, and configuring the ESXi boot options. If you’d like me to continue with those steps, let me know!

Get an Array of Localized Hyper-V Integration Service Names

2016-04-19T00:00:00Z

Today’s PowerShell snippet is used to get a list of Localized captions for the available Integration Services available on a Hyper-V host. I needed this because LabBuilder allows the individual Integration Services to be enabled or disabled per Lab Virtual Machine.

It does this using the Integration Service names configured in the configuration XML file. The problem of course is localization - something I often overlook. If you need to enable/disable an Integration Service on a VM, you need to know the name of it. The name of course is a localized string, so you need to know what the possible values are on the current machine culture.

So, after a lot of digging around in the WMI/CIM I managed to locate the various classes I need and converted them into a simple function:

function GetIntegrationServiceNames {
    [CmdLetBinding()]
    param
    (
    )
    $Captions = @()
    $Classes = @(
        'Msvm_VssComponentSettingData'
        'Msvm_ShutdownComponentSettingData'
        'Msvm_TimeSyncComponentSettingData'
        'Msvm_HeartbeatComponentSettingData'
        'Msvm_GuestServiceInterfaceComponentSettingData'
        'Msvm_KvpExchangeComponentSettingData'
    )
    foreach ($Class in $Classes)
    {
        $Captions += (Get-CimInstance `
            -Class $Class `
            -Namespace Root\Virtualization\V2 `
            -Property Caption | Select-Object -First 1).Caption
    } # foreach
    # This Integration Service is registered in CIM but are not exposed in Hyper-V
    # 'Msvm_RdvComponentSettingData'
    return $Captions
} # GetIntegrationServiceNames

GetIntegrationServiceNames

The output of the function looks like this for English US:

VSS
Shutdown
Time Synchronization
Heartbeat
Guest Service Interface
Key-Value Pair Exchange

Hopefully someone will find it handy.

Install Jenkins using DSC – Part 2

2016-04-18T00:00:00Z

In my previous post, I showed how to create a PowerShell script that would install a Jenkins CI Master server onto a Windows Server Core installation. The obvious next step for such a script was to convert it into a DSC configuration file.

In this post, I’m assuming WMF 5.0 is installed onto the server that will be converted into a Jenkins Master. You could manage this without WMF 5.0, but you’d need to manually install the DSC Resource modules that the configuration will use.

Once again, the full DSC Configuration script can be found at the end of the post.

Requirements

You’ll need:

A physical or virtual machine running Windows Server 2012 R2 Core (or Full)—it should be a completely clean install with WMF 5.0 installed on it.
An administrator login to the server.
An internet connection to the server.

Resource Modules

This DSC Configuration requires the use of three DSC Resources:

cChoco - This community resource is used to install Chocolatey and Jenkins.
xNetworking - This resource is used to configure the networking on the server if required.
PSDesiredStateConfiguration - This resource comes with PowerShell by default and is used to provide the Script resource.

The easiest way to install these resource modules is by executing these commands on the Jenkins server:

# Make sure the DSC Resource modules are downloaded
Install-Module -Name cChoco -Force
Install-Module -Name xNetworking -Force

However, if you’re using a Pull server or compiling the DSC MOF on a development machine (rather than the Jenkins node), you would need to use other methods of ensuring the modules are available.

The Configuration Components

The DSC Configuration needs to do the following things:

Configure Networking (optional)
Install .NET 3.5 Framework
Install Chocolatey
Install JDK 8
Install Jenkins
Configure Jenkins Port (optional)

Configure Networking

I like to use the xNetworking DSC resource to configure the IPv4 and IPv6 settings on the network adapter to have a static configuration. However, you won’t need to do this if you’re using DHCP or manual configuration. Note, in my case, my adapter was called “Ethernet”.

xIPAddress IPv4_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv4'
    IPAddress      = '192.168.128.20'
    SubnetMask     = '24'
}
xDefaultGatewayAddress IPv4G_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv4'
    Address        = '192.168.128.19'
}
xDnsServerAddress IPv4D_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv4'
    Address        = '192.168.128.10'
}
xIPAddress IPv6_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv6'
    IPAddress      = 'fd53:ccc5:895a:bc00::14'
    SubnetMask     = '64'
}
xDefaultGatewayAddress IPv6G_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv6'
    Address        = 'fd53:ccc5:895a:bc00::13'
}
xDnsServerAddress IPv6D_1 {
    InterfaceAlias = 'Ethernet'
    AddressFamily  = 'IPv6'
    Address        = 'fd53:ccc5:895a:bc00::a'
}

Install .NET 3.5 Framework

Jenkins requires the .NET 3.5 Framework, so I’m going to use the WindowsFeature DSC Resource to install it:

WindowsFeature NetFrameworkCore {
    Ensure = "Present"
    Name   = "NET-Framework-Core"
}

Install Chocolatey

Next up, I’m going to use the cChocoInstaller resource in the cChoco resource module (available on PowerShell Gallery here) to install the Chocolatey package manager:

cChocoInstaller installChoco {
    InstallDir = "c:\choco"
    DependsOn  = "[WindowsFeature]NetFrameworkCore"
}

Install JDK 8 and Jenkins

The cChocoPackageInstaller resource module is then used to install JDK 8 and Jenkins:

# Install JDK8
cChocoPackageInstaller installJdk8 {
    Name      = "jdk8"
    DependsOn = "[cChocoInstaller]installChoco"
}

# Install Jenkins
cChocoPackageInstaller installJenkins {
    Name      = "Jenkins"
    DependsOn = "[cChocoInstaller]installChoco"
}

Configure Jenkins Port

The last step of the configuration is optional. By default, Jenkins is configured to listen on port 8080; however, I want to change it to 80. So this next part uses the Script resource to change the --httpPort setting in the Jenkins.xml file. I use Regex to do this:

Script SetJenkinsPort {
    SetScript = {
        Write-Verbose -Verbose "Setting Jenkins Port to $Using:JenkinsPort"
        $Config = Get-Content `
            -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
        $NewConfig = $Config `
            -replace '--httpPort=[0-9]*\s',"--httpPort=$Using:JenkinsPort "
        Set-Content `
            -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml" `
            -Value $NewConfig `
            -Force
        Write-Verbose -Verbose "Restarting Jenkins"
        Restart-Service `
            -Name Jenkins
    }
    GetScript = {
        $Config = Get-Content `
            -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
        $Matches = @([regex]::matches($Config, "--httpPort=([0-9]*)\s", 'IgnoreCase'))
        $CurrentPort = $Matches.Groups[1].Value
        Return @{
            'JenkinsPort' = $CurrentPort
        }
    }
    TestScript = { 
        $Config = Get-Content `
            -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
        $Matches = @([regex]::matches($Config, "--httpPort=([0-9]*)\s", 'IgnoreCase'))
        $CurrentPort = $Matches.Groups[1].Value
        
        If ($Using:JenkinsPort -ne $CurrentPort) {
            # Jenkins port must be changed
            Return $False
        }
        # Jenkins is already on correct port
        Return $True
    }
    DependsOn = "[cChocoPackageInstaller]installJenkins"
}

Create the MOF

The final thing to do is download the cChoco and xNetworking DSC Resources, create the MOF, and then ask the LCM to apply it:

$ConfigData = @{
    AllNodes = 
    @(
        @{
            NodeName = "LocalHost"
        }
    )
}

JENKINS_CI -JenkinsPort 80 -ConfigurationData $ConfigData

Start-DscConfiguration -Path .\JENKINS_CI -Wait -Verbose

The Complete DSC Configuration

Here is the complete DSC Configuration file. You just need to copy it to the server and run it. It will compile the configuration into a MOF and tell the LCM to apply it. Just remember to ensure required DSC Resource modules are installed.

Configuration JENKINS_CI {
    param (
        $JenkinsPort = 8080
    )
    Import-DscResource -ModuleName 'PSDesiredStateConfiguration'
    Import-DscResource -ModuleName 'cChoco'
    Import-DscResource -ModuleName 'xNetworking'

    Node $AllNodes.NodeName {
        # Configure networking (optional)
        xIPAddress IPv4_1 {
            InterfaceAlias = 'Ethernet'
            AddressFamily  = 'IPv4'
            IPAddress      = '192.168.128.20'
            SubnetMask     = '24'
        }
        xDefaultGatewayAddress IPv4G_1 {
            InterfaceAlias = 'Ethernet'
            AddressFamily  = 'IPv4'
            Address        = '192.168.128.19'
        }
        xDnsServerAddress IPv4D_1 {
            InterfaceAlias = 'Ethernet'
            AddressFamily  = 'IPv4'
            Address        = '192.168.128.10'
        }
        xIPAddress IPv6_1 {
            InterfaceAlias = 'Ethernet'
            AddressFamily  = 'IPv6'
            IPAddress      = 'fd53:ccc5:895a:bc00::14'
            SubnetMask     = '64'
        }
        xDefaultGatewayAddress IPv6G_1 {
            InterfaceAlias = 'Ethernet'
            AddressFamily  = 'IPv6'
            Address        = 'fd53:ccc5:895a:bc00::13'
        }
        xDnsServerAddress IPv6D_1 {
            InterfaceAlias = 'Ethernet'
            AddressFamily  = 'IPv6'
            Address        = 'fd53:ccc5:895a:bc00::a'
        }
        
        # Install .NET 3.5
        WindowsFeature NetFrameworkCore {
            Ensure = "Present"
            Name   = "NET-Framework-Core"
        }

        # Install Chocolatey
        cChocoInstaller installChoco {
            InstallDir = "c:\choco"
            DependsOn  = "[WindowsFeature]NetFrameworkCore"
        }

        # Install JDK8
        cChocoPackageInstaller installJdk8 {
            Name      = "jdk8"
            DependsOn = "[cChocoInstaller]installChoco"
        }

        # Install Jenkins
        cChocoPackageInstaller installJenkins {
            Name      = "Jenkins"
            DependsOn = "[cChocoInstaller]installChoco"
        }

        # Set the Jenkins Port
        Script SetJenkinsPort {
            SetScript = {
                Write-Verbose -Verbose "Setting Jenkins Port to $Using:JenkinsPort"
                $Config = Get-Content `
                    -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
                $NewConfig = $Config `
                    -replace '--httpPort=[0-9]*\s',"--httpPort=$Using:JenkinsPort "
                Set-Content `
                    -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml" `
                    -Value $NewConfig `
                    -Force
                Write-Verbose -Verbose "Restarting Jenkins"
                Restart-Service `
                    -Name Jenkins
            }
            GetScript = {
                $Config = Get-Content `
                    -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
                $Matches = @([regex]::matches($Config, "--httpPort=([0-9]*)\s", 'IgnoreCase'))
                $CurrentPort = $Matches.Groups[1].Value
                Return @{
                    'JenkinsPort' = $CurrentPort
                }
            }
            TestScript = { 
                $Config = Get-Content `
                    -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
                $Matches = @([regex]::matches($Config, "--httpPort=([0-9]*)\s", 'IgnoreCase'))
                $CurrentPort = $Matches.Groups[1].Value
                
                If ($Using:JenkinsPort -ne $CurrentPort) {
                    # Jenkins port must be changed
                    Return $False
                }
                # Jenkins is already on correct port
                Return $True
            }
            DependsOn = "[cChocoPackageInstaller]installJenkins"
        }
    }
}

$ConfigData = @{
    AllNodes = 
    @(
        @{
            NodeName = "LocalHost"
        }
    )
}

JENKINS_CI -JenkinsPort 80 -ConfigurationData $ConfigData

Start-DscConfiguration -Path .\JENKINS_CI -Wait -Verbose

Within five to ten minutes, the Jenkins server will be configured and ready to go.

Install Jenkins on Windows Server Core - Part 1

2016-04-14T00:00:00Z

I’ll admit it—I love Windows Server Core and I use it whenever possible. I think everyone should try and do the same. However, I know not everyone is a PowerShell expert or has any desire to be one.

So for this blog post, I’m going to show how I created a simple script that will install Jenkins CI on a Windows Server Core system to be a Jenkins Master server. Feel free to just skip down to the end and use the completed script if you want to. I did this on a Windows Server 2012 R2 Core system, but this would probably work on Windows Server 2016 TP4 (the currently available version). You could, of course, use this on a Windows Server Full system as well.

Note: Installing a Windows Server Core as a Jenkins Slave server is a similar process, but there is no need to install the Jenkins Server software or service. I won’t cover the process of installing a Windows Jenkins Slave in this post.

This is part one of a two-part post. In part two, I’ll convert the process over to a DSC configuration that can be applied to one or more nodes to make the process even easier and ensure your Jenkins servers maintain their state.

Requirements

You’ll need:

A physical or virtual machine running Windows Server 2012 R2 Core (or Full)—it should be a completely clean install with nothing already installed.
An administrator login to the server.
An internet connection to the server.

The Script

The first thing I like to do is get all the variables into one place so I can easily see what options I might want to set. In this case, the only thing I care about is setting a static IP Address for the server and also the port Jenkins will be assigned to:

# Configure the settings to use to set up this Jenkins Executor
$Port = 80
$IPAddress = '192.168.1.96'
$SubnetPrefixLength = 24
$DNSServers = @('192.168.1.1')
$DefaultGateway = '192.168.1.1'

The next thing I need to do is ensure the .NET Framework v3.5 is installed (required by Jenkins on Windows):

# Install .NET Framework 3.5
Install-WindowsFeature -Name NET-Framework-Core

For this installation, I’m actually going to let the Chocolatey package manager do most of the heavy lifting of actually downloading and installing the Jenkins bits. So I need to install Chocolatey:

# Install Chocolatey
iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))

Next up, I use Chocolatey to install both the Oracle JDK 8 and the Jenkins bits. These will be downloaded off the internet, so it may take a little while depending on your connection. The -y parameter forces the install to occur without prompting:

# Install JDK 8
choco install jdk8 -y

# Install Jenkins using Chocolatey
choco install Jenkins -y

What I’m going to do next is configure the port Jenkins should run on. This is done by changing the --httpPort setting in the C:\Program Files (x86)\Jenkins\Jenkins.xml file. I’ll use a simple RegEx to do this. Also, because the Jenkins Service is already running at this point, I’ll need to restart it before the changed setting will be read:

# Set the port Jenkins uses
$Config = Get-Content `
  -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
$NewConfig = $Config `
  -replace '--httpPort=[0-9]*\s',"--httpPort=$Port "
Set-Content `
  -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml" `
  -Value $NewConfig `
  -Force
Restart-Service `
  -Name Jenkins

The Chocolatey Jenkins package automatically configures a firewall rule named “Jenkins” that allows inbound traffic to the java.exe application. This means that external machines will be able to connect to this Jenkins server. You may want to change this by removing the “Jenkins” firewall rule and replacing it with something more specific to your needs; however, I didn’t do this in my script.

The final section is optional—it just configures the network connection on the machine to use a static IP address. You could omit this section completely if you were using DHCP or some other method of configuring the network connection:

# Set a static IP Address - optional
New-NetIPAddress `
  -IPAddress $IPAddress `
  -InterfaceAlias Ethernet `
  -DefaultGateway $DefaultGateway `
  -AddressFamily IPv4 `
  -PrefixLength $SubnetPrefixLength
Set-DnsClientServerAddress `
  -InterfaceAlias Ethernet `
  -Addresses $DNSServers

That’s all there is to it.

The Complete Script

Here is the complete script. You can just fire up PowerShell on the Core server and copy/paste this directly into the PowerShell console, or use some other method of running it:

# Configure the settings to use to set up this Jenkins Executor
$Port = 80
$IPAddress = '192.168.1.96'
$SubnetPrefixLength = 24
$DNSServers = @('192.168.1.1')
$DefaultGateway = '192.168.1.1'

# Install .NET Framework 3.5
Install-WindowsFeature -Name NET-Framework-Core

# Install Chocolatey
iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))

# Install JDK 8
choco install jdk8 -y

# Install Jenkins using Chocolatey
choco install Jenkins -y

# Set the port Jenkins uses
$Config = Get-Content `
  -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
$NewConfig = $Config `
  -replace '--httpPort=[0-9]*\s',"--httpPort=$Port "
Set-Content `
  -Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml" `
  -Value $NewConfig `
  -Force
Restart-Service `
  -Name Jenkins

# Set a static IP Address - optional
New-NetIPAddress `
  -IPAddress $IPAddress `
  -InterfaceAlias Ethernet `
  -DefaultGateway $DefaultGateway `
  -AddressFamily IPv4 `
  -PrefixLength $SubnetPrefixLength
Set-DnsClientServerAddress `
  -InterfaceAlias Ethernet `
  -Addresses $DNSServers

Tomorrow I’ll improve on this process by converting it into a DSC configuration, which will ensure the Jenkins Server maintains its state and makes provisioning them even easier.

Thanks for reading.

Which Physical Network Adapters are bound to Virtual Switches?

2016-04-08T00:00:00Z

Today’s post has quite a long title for what is going to be a fairly short post. While making some improvements to LabBuilder, I had a need to find out which physical network adapters on a host are bound to Hyper-V Virtual Switches. This is because a single physical adapter can only be bound to a single External Virtual Switch.

So I wrote a few lines of PowerShell that would do the trick:

$MacAddress = `
  (Get-VMNetworkAdapter `
    -ManagementOS `
    -Name (Get-VMSwitch | ? { 
      $_.SwitchType -eq 'External'
    }).Name).MacAddress

Get-NetAdapter -Physical | ? {
  ($_.MacAddress -replace '-','') -in $MacAddress
}

The first piece gets a list of MAC addresses for all Virtual Network Adapters that are configured for use by the host OS (managementOS) on External Switches.

The second piece then gets the list of Physical network adapters that match the MAC addresses from the first line. I had to use a -Replace to get rid of the dashes in the Physical network adapter MAC address so that I could compare it with the MAC Address in the Virtual Network Adapters. It would be nice if the MAC address format was standard across all modules, but it is a pretty minor complaint.

So as you can see, PowerShell makes this unbelievably easy. This piece of code allows me to ensure that when LabBuilder is creating a new External Switch it doesn’t use a physical adapter that has already been used.

Nano Server - CIM or WMI?

2016-04-05T00:00:00Z

Just a quick Nano server tip for this morning. As of Windows Server 2016 TP4, Nano Server only contains the CIM cmdlets. It does not contain the WMI cmdlets:

This is a good thing as it means that we can still perform some tasks that are not covered by the PowerShell modules that are provided with Nano Server.

Configuring iSCSI and iSNS with DSC

2016-04-04T00:00:00Z

Several months back I created a DSC Resource for configuring iSCSI Server Targets (including Virtual Disks) as well as iSCSI Initiators using Desired State Configuration (DSC). I created this for several reasons:

I needed a way for LabBuilder to automatically build Scale-Out File Servers (with CSVs).
I needed something to use as an example in my Creating Professional DSC Resources series.
No one else had already created one.

This weekend I decided to add iSNS Server support to the resource—for both the ciSCSIServerTarget and ciSCSIInitiator resources. So with that feature added, I thought it might be a good opportunity for me to write a quick blog post on how to use these DSC Resources.

Installing the Resource

You can find the new ciSCSI resource in the PowerShell Gallery.

For those of you using Windows Management Framework 5.0 (or have the PowerShellGet module installed), you can just use the command:

Install-Module -Name ciSCSI

If you don’t have Windows Management Framework 5.0 (and don’t have the PowerShellGet module installed), you will need to download and install the resource from the GitHub Repository.

Using the Resource

If you’d rather just jump right into the resource documentation and examples, you can find it here. Otherwise, read on and I’ll cover this resource to configure both an iSCSI Server Target and an iSCSI Initiator. I’ll also show how to register iSCSI Server Targets and Initiators with an iSNS Server.

Important: Although the ciSCSI DSC Resource will work on Windows Management Framework 4.0, these examples require the use of the WaitForAny DSC Resource, which is only available in Windows Management Framework 5.0. This resource is used to ensure that the iSCSI Server Target has been created before trying to connect any iSCSI Initiators to it. The resource could be omitted, but errors will be reported by the LCM on the iSCSI Initiator computers if the iSCSI Server Target is not available before the iSCSI Initiator DSC MOF is applied.

The Example Environment

In this example, the DSC Configurations that are being created will refer to the following servers:

FS1.CONTOSO.COM - This is the file server that will contain the iSCSI Virtual Disks and iSCSI Server Target.
CLUS1.CONTOSO.COM, CLUS2.CONTOSO.COM, CLUS3.CONTOSO.COM - These are the Windows Server 2012 R2 (or Windows Server 2016) Cluster Server nodes that will be connecting to the iSCSI Server Target.
ISNS1.CONTOSO.COM - This is a server with the iSNS Server Windows Feature installed on it. The iSNS default domain has been configured on this server already.

The DSC configurations that will be created will create four 128GB dynamic iSCSI Virtual Disks on the D:\ drive of FS1.CONTOSO.COM. An iSCSI Server Target called FS1-Server-Target will be created, and the four iSCSI Virtual Disks will be attached to it.

Configuring the iSCSI Server Target

A DSC configuration that creates an iSCSI Server Target requires the following steps to be performed in the DSC Resource:

Install the iSCSI Target Server Windows Feature (FS-iSCSITarget-Server).
Initialize any physical disks that will be used to store the iSCSI Virtual Disks (optional).
Create the iSCSI Virtual Disks that will be used by the iSCSI Server Target.
Create the iSCSI Server Target and optionally register it with an iSNS Server.

Here is the DSC Configuration:

Configuration ISCSI_SERVER_TARGET {
    Import-DscResource -ModuleName 'PSDesiredStateConfiguration'
    Import-DscResource -ModuleName xStorage
    Import-DscResource -ModuleName ciSCSI

    Node 'FS1' {
        # Step 1 - Install the iSCSI Target Server Windows Feature
        WindowsFeature ISCSITargetServerInstall {
            Ensure = "Present"
            Name = "FS-iSCSITarget-Server"
        }

        # Step 2 - Initialize any physical disks that will be used to store the iSCSI Virtual Disks
        xWaitforDisk Disk2 {
            DiskNumber = 1
            RetryIntervalSec = 60
            RetryCount = 60
        }
        xDisk DVolume {
            DiskNumber = 1
            DriveLetter = 'D'
            DependsOn = "[xWaitforDisk]Disk2"
        }
        File VirtualDisksFolder {
            Ensure = 'Present'
            DestinationPath = 'D:\iSCSIVirtualDisks'
            Type = 'Directory'
            DependsOn = '[xDisk]DVolume'
        }

        # Step 3 - Create the iSCSI Virtual Disks
        ciSCSIVirtualDisk VDisk1 {
            Ensure = 'Present'
            Path = 'D:\iSCSIVirtualDisks\VDisk1.vhdx'
            DiskType = 'Dynamic'
            SizeBytes = 128GB
            DependsOn = "[File]VirtualDisksFolder"
        }
        ciSCSIVirtualDisk VDisk2 {
            Ensure = 'Present'
            Path = 'D:\iSCSIVirtualDisks\VDisk2.vhdx'
            DiskType = 'Dynamic'
            SizeBytes = 128GB
            DependsOn = "[File]VirtualDisksFolder"
        }
        ciSCSIVirtualDisk VDisk3 {
            Ensure = 'Present'
            Path = 'D:\iSCSIVirtualDisks\VDisk3.vhdx'
            DiskType = 'Dynamic'
            SizeBytes = 128GB
            DependsOn = "[File]VirtualDisksFolder"
        }
        ciSCSIVirtualDisk VDisk4 {
            Ensure = 'Present'
            Path = 'D:\iSCSIVirtualDisks\VDisk4.vhdx'
            DiskType = 'Dynamic'
            SizeBytes = 128GB
            DependsOn = "[File]VirtualDisksFolder"
        }

        # Step 4 - Create the iSCSI Server Target and optionally register it with an iSNS Server
        ciSCSIServerTarget iSCSIServerTarget {
            Ensure = 'Present'
            TargetName = 'FS1-Server-Target'
            InitiatorIds = @(
                'Iqn:iqn.1991-05.com.microsoft:CLUS1.CONTOSO.COM',
                'Iqn:iqn.1991-05.com.microsoft:CLUS2.CONTOSO.COM',
                'Iqn:iqn.1991-05.com.microsoft:CLUS3.CONTOSO.COM'
            )
            Paths = @(
                'D:\iSCSIVirtualDisks\VDisk1.vhdx',
                'D:\iSCSIVirtualDisks\VDisk2.vhdx',
                'D:\iSCSIVirtualDisks\VDisk3.vhdx',
                'D:\iSCSIVirtualDisks\VDisk4.vhdx'
            )
            iSNSServer = 'isns1.contoso.com'
        }
    }
}

Important: Note that the TargetName is set to ‘FS1-Server-Target’, which will automatically configure the Target IQN to ‘iqn.1991-05.com.microsoft:FS1-FS1-Server-Target-Target’. This is because the Microsoft iSCSI Server Target cmdlets automatically name the Server Target for you using the following format:

iqn.1991-05.com.microsoft:$($ComputerName)-$($ServerTarget)-Target

This is very important to remember because the iSCSI Initiators use this string to identify the Server Target to connect to.

Configuring the iSCSI Initiator

A DSC configuration for each of the iSCSI Initiators that will connect to the iSCSI Server Target requires the following steps to be performed in the DSC Resource:

Start the Microsoft iSCSI Initiator Service service (MSiSCSI).
Use the WaitForAny WMF 5.0 DSC Resource to wait for the iSCSI Server Target to be created (optional).
Connect the iSCSI Initiator to the iSCSI Server Target and optionally register it with an iSNS Server.

Here is the DSC Configuration for CLUS1.CONTOSO.COM (the configuration for the other nodes would be similar except with different InitiatorPortalAddress values):

Configuration ISCSI_INITIATOR {
    Import-DscResource -ModuleName PSDesiredStateConfiguration
    Import-DscResource -ModuleName xPSDesiredStateConfiguration
    Import-DscResource -ModuleName ciSCSI

    Node 'CLUS1' {
        # Step 1 - Start the Microsoft iSCSI Initiator Service service (MSiSCSI)
        Service iSCSIService {
            Name = 'MSiSCSI'
            StartupType = 'Automatic'
            State = 'Running'
        }

        # Step 2 - Use the WaitForAny WMF 5.0 DSC Resource to wait for the iSCSI Server Target to be created (optional).
        WaitForAny WaitForiSCSIServerTarget {
            ResourceName = "[ciSCSIServerTarget]ClusterServerTarget"
            NodeName = 'FS1'
            RetryIntervalSec = 30
            RetryCount = 30
            DependsOn = "[Service]iSCSIService"
        }

        # Step 3 - Connect the iSCSI Initiator to the iSCSI Server Target and optionally register it with an iSNS Server
        ciSCSIInitiator iSCSIInitiator {
            Ensure = 'Present'
            NodeAddress = 'iqn.1991-05.com.microsoft:fs1-fs1-server-target-target'
            TargetPortalAddress = '192.168.129.10'
            InitiatorPortalAddress = '192.168.129.24'
            IsPersistent = $true
            iSNSServer = 'isns1.contoso.com'
            DependsOn = "[WaitForAny]WaitForiSCSIServerTarget"
        } # End of ciSCSITarget Resource
    }
}

Important: We need to make sure the NodeAddress is set to the the Target IQN from the iSCSI Server Target - in this case ‘iqn.1991-05.com.microsoft:FS1-FS1-Server-Target-Target’.

It is also recommended that you use IP Addresses for the TargetPortalAddress and InitiatorPortalAddress parameters rather than server names, as this will force the iSCSI traffic to use the appropriate network adapter.

iSNS Server Configuration

There are a few things to keep in mind when you have your iSCSI DSC Configurations registering with an iSNS Server:

The Default Domain on the iSNS Server should have been created.
If the iSNS Server is not available or contactable by the iSCSI Server Target or Initiator when the DSC Configuration is applied the DSC configuration will not throw an error, but the iSNS Server Address will not be set. However, next time the DSC configuration is applied by the LCM it will try again (and again the next time etc).

Using iSNS Server is completely optional and is mostly used in larger environments with more than twenty iSCSI Server Targets and where the Initiators will be connected to the iSCSI Server Targets manually or where DSC can’t be used on the iSCSI Server Targets.

That is all there is to using this resource to configure a Windows Server 2012 iSCSI SAN using DSC.

Note: I have submitted this DSC Resource to be included in the Microsoft Community DSC Resources project. If it is accepted then the name of the DSC Resource will change from ciSCSI to iSCSI. The resource hasn’t yet been reviewed and I’m not aware of an ETA for it. The old ‘c’ and ‘x’ nomenclature used by DSC Resources is being phased out.

Configure iSNS Server in an iSCSI Initiator with PowerShell

2016-03-27T00:00:00Z

This will just be a very quick post today.

Configuring an iSCSI Initiator to use an iSNS Server is fairly easy using the iSCSI configuration utility:

But lets face it, that’s no fun and it really doesn’t fit well when when we have to configure 10’s or 100’s of initiators or we’re using Server Core. Not to mention that this is something we’d really want to do with Desired State Configuration. Besides, this is a PowerShell blog.

It is easy enough to configure iSCSI Targets to register with an iSNS Server:

Set-WmiInstance -Namespace root\wmi -Class WT_iSNSServer –Arguments @{ServerName="ISNSSERVER.CONTOSO.COM"}

But unfortunately I couldn’t find any documentation on how to do this on an iSCSI Initiator. But after a little bit of digging around WMI I found the appropriate class:

MSiSCSIInitiator_iSNSServerClass

So, to add the iSNS Server to the iSCSI Initiator:

Set-WmiInstance `
  -Namespace root\wmi `
  -Class MSiSCSIInitiator_iSNSServerClass `
  -Arguments @{iSNSServerAddress="ISNSSERVER.CONTOSO.COM"}

Notice that the WMI Class argument name for the setting the iSNS Server name in an iSCSI Initiator is different (iSNSServerAddress) compared to setting it for an iSCSI Target (ServerName).

To list the currently set iSNS Servers:

Get-WmiObject `
  -Class MSiSCSIInitiator_iSNSServerClass `
  -Namespace root\wmi

And if you need to remove an iSNS Server from the iSCSI Initiator:

Get-WmiObject `
  -Class MSiSCSIInitiator_iSNSServerClass `
  -Namespace root\wmi `
  -Filter "iSNSServerAddress='ISNSSERVER.CONTOSO.COM'" | Remove-WmiObject -Verbose

Pretty easy right?

In a few weeks I plan to integrate iSNS registration with my iSCSI DSC Resources (also available on PowerShell Gallery) so that the whole process of registering iSCSI Initiators and Targets with iSNS is just a little bit easier.

Thanks for reading.

Creating a Cloneable Class in WMF 5.0

2016-03-11T00:00:00Z

Introduction

You might have noticed that instances of certain types of classes are created, a method called Clone is available that will create a (shallow) copy of the object. A PowerShell Hashtable object is a classic example of a class that supports the Clone method:

$Car = [Hashtable]::New()
$Car.Add('Make','Lamborghini')
$Car.Add('Model','Aventador')
$NewCar = $Car.Clone()

This is nothing new to developers, but for most Ops people it might be something they’re not that familiar with. But if you are an Ops person who is implementing more advanced PowerShell modules or scripts in WMF 5.0 that require custom classes, then this might be something you need to do.

Note: you can do this in WMF 3.0 and 4.0 but it requires reflection and a lot more code, so isn’t something I’m going to cover here.

For this post, I’m assuming you have a basic knowledge of how to create classes in WMF 5.0. If you aren’t familiar with creating classes, take a look at this series for a very good primer.

If you just go and create a new class in PowerShell and try to call the clone method, an error with be thrown:

This is because by default a new class that is defined in PowerShell is based off the System.Object class which does not implement the ICloneable interface. The ICloneable interface is what gives us the Clone method on an object. So we need to tell PowerShell that we want our new class to implement the ICloneable interface.

Note: You don’t really need to know what an interface is to use it, but if you do want a better understanding of it, this is a good place to start. Details on the ICloneable Interface can be found here.

Implementing an Interface

Creating a class that implements the ICloneable interface just requires that we add the name of the interface to implement after a colon following the class name:

class Car:ICloneable {
  [String] $Make
  [String] $Model
}

However, if we try to define this class as is we’ll get an error:

The problem is that we’ve told PowerShell that the Car class implements ICloneable and should therefore have a Clone method, but we haven’t actually created the Clone method in the class.

To do this we need to add the method to our class definition:

class Car:ICloneable {
  [Object] Clone () {
      $NewCar = [Car]::New()
      foreach ($Property in ($this | Get-Member -MemberType Property))
      {
          $NewCar.$($Property.Name) = $this.$($Property.Name)
      } # foreach
      return $NewCar
  } # Clone
  [String] $Make
  [String] $Model
}

The above code first creates a new Car object, then gets a list of the properties on the existing ($This) object and uses a foreach loop to copy the content of each property to the new Car object ($NewCar). The $NewCar object is returned to the calling code.

This performs a shallow copy of the object. If you want to perform a deep copy then you’ll need to implement code based on the child objects within the existing object._

You’ve now created an class that implements an interface. The .NET framework provides hundreds (if not thousands) of interfaces that you potentially could implement on your PowerShell classes. Of course, you could have cloned the Car object without implementing the ICloneable interface at all, but this post is intended to be a general introduction to implementing interfaces in WMF 5.0 as well as the ICloneable interface.

Thanks for reading.

PowerShell V5 New Feature: Protect/Unprotect-CmsMessage

2016-03-10T00:00:00Z

This interesting article gives some background details on some of the problems I ran into after upgrading my DSC dev machine to WMF 5.0 10586. This is because in WMF5.0 the DSC credential encryption mechanism was converted to use Protect/Unprotect-CMSMessage. It clears up a lot of things for me and is a worthwhile read if you’re using DSC credential encryption on WMF5.0.

Building a Enum that Supports Bit Fields in PowerShell

2016-03-09T00:00:00Z

I found this post extremely useful - definitely worth a read.
Building a Enum that Supports Bit Fields in PowerShell

Windows Management Framework 5.0 (WMF) RTM re-published

2016-02-25T00:00:00Z

After a bit of a false start the WMF 5.0 installer package has been republished. This released fixes the PSModulePath issue that popped up on the first release of this package.

The changes to this package are:

Changes in the republished packages

The KB numbers of these packages (KB3134758, KB3134759, and KB3134760) are different than previously released WMF 5.0 RTM packages (KB3094174, KB3094175, and KB3094176).
These packages have fixes only for the PSModulePath issue compared to the previously released WMF 5.0 RTM packages.

Note: if you’re install the previous (broken) WMF 5.0 RTM package you’ll have to uninstall it before installing this new one.

Uninstall previously released WMF 5.0 RTM packages

You must uninstall previously released WMF 5.0 RTM (KB3094174, KB3094175, and KB3094176) packages.

Creating a Chocolatey Package in AppVeyor CI

2016-02-21T00:00:00Z

Introduction

Recently I had a need to have an application published in Chocolatey. If you’re not familiar with Chocolatey, it is:

a Machine Package Manager, somewhat like apt-get, but built with Windows in mind.

The application I needed to package was actually a Microsoft tool called DevCon.exe and is available freely from Microsoft as part of the Windows Driver Kit (WDK). The WDK is a massive 2.5GB download, and I needed this one tiny (80kb) executable to be automatically installed and used as part of some Integration tests for another project using AppVeyor CI. Forcing AppVeyor CI to download and install a 2.5GB WDK just so some integration tests could be run was unreasonable and unworkable.

If you are interested: the reason I needed DevCon.exe was to be able to automatically create Microsoft Loopback Adapters for use in testing some functions of the xNetworking DSC resource. I did this by creating a PowerShell LoopbackAdapter Module that automatically downloads and installs this Chocolatey package.

The Package

If you fancy skipping the details of the process and just want to jump in and have a look at the project, you can find it on GitHub.

The package is fairly simple. It just contains a few files:

AppVeyor.yml - this is the AppVeyor CI definition file that tells AppVeyor how to build this project.
Readme.md - some details about this project.
devcon.portable\DevCon32.exe - a 32-bit version of the DevCon.exe copied straight out of the WDK.
devcon.portable\DevCon64.exe - a 64-bit version of the DevCon.exe copied straight out of the WDK.
devcon.portable\devcon.portable.nuspec - this is the nuspec package manifest file I created for this package.

The really important files here are AppVeyor.yml and devcon.portable.nuspec. So I’ll cover those in a bit more detail.

Compiling DevCon from Source

The source code for the DevCon application is actually available Microsoft’s own GitHub repository. So in my original version of this project I was actually using AppVeyor to clone this source code and then compile the application using MSBuild as part of the packaging process. This worked really well, but the problem was that the purpose of Chocolatey is to actually install applications that can be validated as being from a trusted source.

What this means is that if I can’t prove that the DevCon32.exe and DevCon64.exe files that are in the Chocolatey package are the ones provided by Microsoft then Chocolatey (will rightly) reject the package.

So if AppVeyor CI compiles the DevCon every time the package is built then the bits won’t match those in the WDK. So unfortunately I had to disable this step in the process and include the DevCon32.exe and DevCon64.exe files copied straight out of the WDK. That way the team at Chocolatey can use the hashes of the files to ensure that they are the same ones in the WDK.

I have however left the compile step in the AppVeyor CI build even though the compiled executable files are not being used anymore.

Devcon.Portable.Nuspec

This is the Chocolatey manifest file for the package. It is really just an XML file that contains the details of your package. To create it I just made a copy of the template manifest file, named it devcon.portable.nuspec and filled in the details.

Alternately, if you have Chocolatey installed, you can run (from PowerShell or CMD):

choco new devcon.portable

This will create a new folder called devcon.portable with a file devcon.portable.nuspec in it. You can then just customize the devcon.portable.nuspec with the details of the package. I also deleted the tools folder that was created as I had no need for that.

In my case I ended up with this:

<?xml version="1.0" encoding="utf-8"?>
<!-- Do not remove this test for UTF-8: if “Ω” doesn’t appear as greek uppercase omega letter enclosed in quotation marks, you should use an editor that supports UTF-8, not this one. -->
<package xmlns="http://schemas.microsoft.com/packaging/2015/06/nuspec.xsd">
  <metadata>
    <id>devcon.portable</id>
    <title>DevCon (Portable)</title>
    <version>1.0</version>
    <authors>Microsoft</authors>
    <owners>Microsoft</owners>
    <summary>Device Console (DevCon) Tool (Portable)</summary>
    <description>
[DevCon](http://msdn.microsoft.com/en-us/library/windows/hardware/ff544707) is a command-line tool that displays detailed information about devices, and lets you search for and manipulate devices from the command line. DevCon enables, disables, installs, configures, and removes devices on the local computer and displays detailed information about devices on local and remote computers. DevCon is included in the WDK.

This pacakge includes both an x86 and an x64 version of the DevCon application:
* DevCon32.exe
* DevCon64.exe

Please use the version applicable to your Operating System.
    </description>
    <projectUrl>https://msdn.microsoft.com/en-us/windows/hardware/hh852365</projectUrl>
    <packageSourceUrl>https://github.com/PlagueHO/devcon-choco-package</packageSourceUrl>
    <projectSourceUrl>https://github.com/Microsoft/Windows-driver-samples/tree/master/setup/devcon</projectSourceUrl>
    <tags>DevCon DeviceConsole WDK WindowsDriverKit</tags>
    <copyright>Copyright (c) 2015 Microsoft</copyright>
    <licenseUrl>https://github.com/Microsoft/Windows-driver-samples/blob/master/LICENSE</licenseUrl>
    <requireLicenseAcceptance>true</requireLicenseAcceptance>
  </metadata>
</package>

Why “Portable”

Most Chocolatey packages are designed to automatically download installers from the application creator themselves and install the app silently using the installer. But in our case this is unworkable (2.5GB download). So we need to include the application executables in the package itself and we need them to end up in our %PATH% so they can be found.

To enable this, Chocolatey has implemented a feature where packages that contain the .portable extension will automatically have the contained executables into a special folder called the Chocolatey Tools Folder. This folder is always in the system path.

For more information on .portable packages, see this page.

AppVeyor.YML

The AppVeyor.yml file looks like this:

#---------------------------------#
#      environment configuration  #
#---------------------------------#
os: WMF 5

version: 10.0.10586.{build}

configuration: Release

platform: Any CPU

install:
   - git clone https://github.com/Microsoft/Windows-driver-samples.git

#---------------------------------#
#      build configuration        #
#---------------------------------#

build_script:
   - cmd: msbuild "Windows-driver-samples\setup\devcon\devcon.vcxproj" /p:Configuration=Release;Platform=Win32 /t:Clean;Build /verbosity:normal /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"
   - cmd: msbuild "Windows-driver-samples\setup\devcon\devcon.vcxproj" /p:Configuration=Release;Platform=x64 /t:Clean;Build /verbosity:normal /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"

#---------------------------------#
#      test configuration         #
#---------------------------------#

#---------------------------------#
#      deployment configuration   #
#---------------------------------#

deploy_script:
  - ps: |
      # Create Chocolately Package

      # Because the bits need to be signed by MSFT, we are no longer using
      # the compiled version, but instead we'll use the one from the WDK.
      # The only problem is that the WDK versions aren't signed by MSFT either.
      
      # Copy-Item -Path .\Windows-driver-samples\setup\devcon\Release\devcon.exe `
      #  -Destination .\devcon.portable\Devcon32.exe
      # Copy-Item -Path .\Windows-driver-samples\setup\devcon\x64\Release\devcon.exe `
      #  -Destination .\devcon.portable\Devcon64.exe

      Set-Location -Path .\devcon.portable\
      (Get-Content '.\devcon.portable.nuspec' -Raw).Replace("<version>1.0</version>", "<version>$($env:APPVEYOR_BUILD_VERSION)</version>") | Out-File '.\devcon.portable.nuspec'
      cpack
      Push-AppveyorArtifact ".\devcon.portable.$($ENV:APPVEYOR_BUILD_VERSION).nupkg"

The key part of this file is in the deploy_script section:

Set-Location -Path .\devcon.portable\
(Get-Content '.\devcon.portable.nuspec' -Raw).Replace("<version>1.0</version>", "<version>$($env:APPVEYOR_BUILD_VERSION)</version>") | Out-File '.\devcon.portable.nuspec'
cpack
Push-AppveyorArtifact ".\devcon.portable.$($ENV:APPVEYOR_BUILD_VERSION).nupkg"

The first line changes to the folder containing the devcon.portable.nuspec and devcon*.exe files.

The second line sets the version number in the devcon.portable.nuspec package manifest to match the current AppVeyor version number.

The third line creates the package using the cpack tool.

The last line pushes the completed package to AppVeyor CI as an artifact that we can then download and submit to Chocolatey:

It is probably possible to have AppVeyor CI automatically submit the package to Chocolatey on my behalf, but I didn’t want that in this case. But if you’re planning on doing that, you’ll want to ensure you use the AppVeyor Encrypt Data tool to encrypt any Chocolatey credentials that your AppVeyor.yml file might use - otherwise your Chocolatey credentials are available for the world to see and abuse. This would be very bad indeed!

The Old Build

I have left in the original commands that pull the DevCon repository from GitHub and then compile it using MSBuild. The commands relating to this can be found in the build_script and install sections of the AppVeyor.yml file.

Final Thoughts

Hopefully this post might help you package up some of your own tools to use distribute with Chocolatey for easy installation or for use with CI services such as AppVeyor.

If you want some more information on using Chocolatey with PowerShell, check out this blog post.

Creating Professional DSC Resources – Part 7

2016-01-25T00:00:00Z

The purpose of this series of articles is to try and document a few of the lessons I learned while releasing new DSC resources as well as contributing to the existing Microsoft Community DSC resources. These articles are not intended to tell you how to write DSC resources from a programming perspective, but to give you some ideas on what might be expected of a DSC resource you’re releasing to the public. For example, unit and integration tests (don’t worry if you aren’t familiar with those terms).

These articles are also not intended to tell you what you must do to release your resource, but more document what will help your resource be easier to use and extend by other people. Some of these these things are obvious for people who have come from the development community, but may be quite new to operations people.

If you missed any previous articles you can find them here:

Recap

In the last couple of articles I covered the importance of automated testing with unit testing in particular. I had covered creating new unit tests using the unit test templates that are now available here. I also covered how to complete the Pester Test Initialization and the Get-TargetResource, Set-TargetResource and Test-TargetResource function areas of the unit test.

Integration Testing

Integration testing is a great way of catching many errors that can’t be easily picked up by Unit testing. It effectively tests your DSC Resource by actually using it in a DSC configuration file and applying it to a computer and checking the results. So this is as close to real-life testing as you can get.

Integration testing of a PowerShell DSC resource should be performed after unit testing. When a PowerShell DSC Resource is integration tested the following process occurs:

A DSC configuration file using the DSC resource to be integration tested is compiled into a MOF.
The MOF file is applied to the test machine.
The parameters current DSC Configuration of this DSC Resource on the test machine is obtained.
The parameters of the current DSC Configuration are compared with what was set in the DSC configuration file in step 1.

Just like unit testing we use Pester to test the above steps and ensure that errors don’t occur and the output is as expected.

Sometimes Integration Tests are not Possible

Integration testing is not always possible on a resource. Some resources may rely on external servers being available or they might be destructive to the machine performing the tests.

For example, integration tests could not be implemented for the MSFT_xIPAddress resource in the xNetworking DSC Resource module because it would have caused the network to disconnect during testing which would have resulted in a failure of the AppVeyor CI machine running the tests.

But, if there is a reasonable way of implementing integration tests for a resource in a non-destructive manor, then I’d strongly recommend it - especially as it is usually really easy.

Don’t Be Destructive

Unlike unit testing, integration testing actually changes configuration on the machine performing the tests. If you’re using a continuous integration service like AppVeyor to perform your tests then this isn’t such a problem as the test machine is “destroyed” after your tests are run.

However, many people also run any integration tests on their local machines before committing code, therefore, your integration tests should always leave the machine in the state that it was before running them. This means that any changes that will be made applying the integration tests should be undone at the completion of your integration tests script.

Integration Test Files

Integration tests for a DSC resource actually consist of two different files:

*.config.ps1 - The DSC Configuration file that will use the DSC Resource being tested.
*.Integration.Tests.ps1 - The Integration Test script file containing the Pester tests.

These files should be stored in the Tests\Integration folder in the DSC Resource module:

You must also ensure that the names of these files exactly matches the name of the resource itself. For example, if your DSC Resource is called BMD_MyResource then these files must be called:

BMD_MyResource.config.ps1
BMD_MyResource.Integration.Tests.ps1

Creating a New Integration Test

Luckily, a good amount of the work in implementing integration tests is already done for you. Like unit tests, templates for the two integration files are available in the DscResources repository in GitHub:

You need to copy the integration test template files and rename them to match your DSC Resource.

The easiest way to do this is to clone the repository containing the test template files and copy the integration_template.ps1 and integration_config_template.ps1 files to your Tests/Integration folder:

git clone https://github.com/PowerShell/DSCResources.git
Copy-Item .\DSCResources\Tests.Template\integration_config_template.ps1 .\ciSCSI\Tests\Integration\BMD_ciSCSIVirtualDisk.config.ps1
Copy-Item .\DSCResources\Tests.Template\integration_template.ps1 .\ciSCSI\Tests\Integration\BMD_ciSCSIVirtualDisk.Integration.Tests.ps1

You’ll now have two new integration test files that you can open in your PowerShell editor of choice.

Modifying the Config File

The first file I usually edit is the \config.ps1 file:

Next, you’ll want to change any <ResourceName> occurrences in this file to the name of your resource. I also like to remove the #TODO bits at the same time so I know what I’ve completed:

configuration 'BMD_ciSCSIVirtualDisk_config' {
    Import-DscResource -Name 'BMD_ciSCSIVirtualDisk'
    node localhost {
       BMD_ciSCSIVirtualDisk Integration_Test {
            # TODO: Fill Configuration Code Here
       }
    }
}

Next, we need to configure the config file with the parameters we want to use as tests of the resource.

The best way of doing this is actually to create a hash table object at the beginning of the file with the parameters that we’re going to set. This is so that we can use this hash table object in the other integration file (*.Integration.Tests.ps1) when we’re comparing the values that are expected to be set.

$VirtualDisk = @{
    Path            = Join-Path -Path $ENV:Temp -ChildPath 'TestiSCSIVirtualDisk.vhdx'
    Ensure          = 'Present'
    DiskType        = 'Dynamic'
    Size            = 100MB
    Description     = 'Integration Test iSCSI Virtual Disk'
}

Configuration BMD_ciSCSIVirtualDisk_Config {
    Import-DscResource -Name BMD_ciSCSIVirtualDisk_Config
    node localhost {
        BMD_ciSCSIVirtualDis Ikntegration_Test {
            Path            = $VirtualDisk.Path
            Ensure          = $VirtualDisk.Ensure
            DiskType        = $VirtualDisk.DiskType
            SizeBytes       = $VirtualDisk.Size
            Description     = $VirtualDisk.Description
        }
    }
}

As you can see in the example above, I create a $VirtualDisk hash table that contains all the parameters and values that will be used to test this DSC Resource. The $VirtualDisk object is then also accessible in the *.Integration.Tests.ps1 file.

Modifying the Integration Tests File

Now that the integration tests config file has been completed it is time to move on to the integration test script (*.Integration.Tests.ps1) itself, so open it in your editor of choice:

Next, customize the TODO area in the header with the your DSC Resource Name and DSC Module Name:

Feel free to remove the TODO comments if you want (I always do).

Initialization Code

After customizing the header we need to add any code that might be required to set this machine up to actually perform these integration tests. The first thing I like to do is add code to check that these integration tests can actually be performed on this machine. In my example resource, the iSCSI Virtual Disk resource will require the iSCSI Target Server feature to be installed, which also means the OS must be a Server OS. So, first thing in the try/catch block I add these checks:

# Ensure that the tests can be performed on this computer
$ProductType = (Get-CimInstance Win32_OperatingSystem).ProductType
Describe 'Environment' {
    Context 'Operating System' {
        It 'Should be a Server OS' {
            $ProductType | Should Be 3
        }
    }
}
if ($ProductType -ne 3)
{
    Break
}

$Installed = (Get-WindowsFeature -Name FS-iSCSITarget-Server).Installed
Describe 'Environment' {
    Context 'Windows Features' {
        It 'Should have the iSCSI Target Feature Installed' {
            $Installed | Should Be $true
        }
    }   
}
if ($Installed -eq $false)
{
    Break
}

This will cause the try/catch block to be exited straight away if these tests can’t actually be performed on this machine.

The cleanup code in the finally block will still be called if we exit with a break command._

After this you might also need to add code to configure anything that these integration tests might depend on. For example, if you were implementing integration tests for testing an iSCSI Server Target, you’d need to make sure that there was an iSCSI Virtual Disk available to use, so you’d need to create one at this point. However, in the integration tests for the iSCSI Virtual Disk resource I don’t need anything else.

Testing the Resource Was Applied

Next, we need to add the tests that check that after the DSC Configuration has been applied to the machine that the changes have actually been made and that the parameters match those set by the Configuration:

To do this, we complete this section:

In this case, I’ve changed it to:

It 'Should have set the resource and all the parameters should match' {
  # Get the Rule details
  $virtualDiskNew = Get-iSCSIVirtualDisk -Path $VirtualDisk.Path
  $VirtualDisk.Path               | Should Be $virtualDiskNew.Path
  $VirtualDisk.DiskType           | Should Be $virtualDiskNew.DiskType
  $VirtualDisk.Size               | Should Be $virtualDiskNew.Size
  $VirtualDisk.Description        | Should Be $virtualDiskNew.Description
}

What this code does is gets the iSCSI Virtual Disk that is at the path specified in the $VirtualDisk.path into a variable $VirtualDiskNew.

The parameters in $VirtualDiskNew are then matched to ensure they are the same as those in the $VirtualDisk hash table object that was created in the DSC Configuration script (*.config.ps1).

Cleaning Up

It is important that after the tests have been run that any changes that were made to the testing computer are reverted. So, after the end of the last test I add any clean up code. In my case, I want to remove the iSCSI Virtual Disk that was created:

# Clean up
Remove-iSCSIVirtualDisk `
  -Path $VirtualDisk.Path
Remove-Item `
  -Path $VirtualDisk.Path `
  -Force

The above code just removes the iSCSI Virtual Disk and then also makes sure that the VHD file was also deleted. This is also very important because if the clean up does not occur and the tests are run again on the same computer they may fail.

And We’re Done

Now, that may all seem like quite a bit of work, but it becomes second nature after creating a few of them. They will also save you far more time in addressing future issues with the resource every time you make a simple change to the MOF (but forget to change the resource code). These tests will give users and other maintainers much more confidence in your resources as well.

This series actually ended up being a bit longer than I intended, but hopefully you’ve stuck with it and it has helped in some small way. If you’ve got this far and you’re wanting to know what to do next, why not head over to the PowerShell DSCResources GitHub repository and see if you could help out on some resources. You could start off adding some small but useful parameter to an existing resource, fixing a bug or contribute an entire new resource to an existing module. There are numerous issues that need to be addressed on these resources, many of which are requests for new features or resources.

If you have an idea for a new resource in an existing module, raise an issue in the DSC Resource Module repository and offer to create the new resource. You may find that someone is already working on one, but if not, then this is a great opportunity to get started. It is quite a rewarding feeling the first time one of your contributions gets published in the official community DSC Resources!

So, thanks again for reading.

Speed up iSCSI PowerShell cmdlets

2016-01-02T00:00:00Z

I’ve been spending a bit of time lately completing a set of DSC Resources for configuring iSCSI Targets and iSCSI Initiators. One thing I’ve noticed is that these iSCSI cmdlets are extremely slow:

21 seconds is just too slow!

Now, if I was just calling this cmdlet every now and then it really wouldn’t matter so much - as long as it works. However, because this cmdlet is going to be called every few minutes when used in a DSC Resource it is unacceptable.

I’ve seen this sort of issue before. It is often caused by the cmdlet looking for things on the network that don’t exist. This means that a TCP timeout might occur before the cmdlet will complete. So I thought a way of possibly eliminating this is to specify the computer the cmdlet should work against - in the case of a DSC Resource it is always LocalHost.

So I gave this a shot:

49 milliseconds is very acceptable.

That is a 42,000% speed increase, which is definitely acceptable.

To confirm that the cmdlet is still working as expected:

Yes, all the iSCSI Virtual Disks are there.

Important Note: You must set the ComputerName to the exact test "LocalHost" and not the actual computer name of the local machine. If you use the local computer name, the cmdlet will still be slow:

21 seconds again, no good.

All of the iSCSI Target cmdlets in the iSCSI Target module seem to suffer from this problem, so adding the -ComputerName LocalHost parameter to them should speed things up across the board. Obviously, this is only going to work if you’re actually manipulating iSCSI Targets on the LocalHost - if you’re trying to configure a remote computer then you’ll need to set the remote computer name.

Hope this one shaves a number of seconds off some scripts out there.

Also, Happy 2016!

Remove an iSCSI Target Portal with PowerShell

2015-12-26T00:00:00Z

I ran into a small problem with removing iSCSI Target Portals using PowerShell the other day and thought it might be worth documenting.

Pretend you have an iSCSI Target Portal configured with a Target Portal Address of 192.168.129.24:

You might therefore expect that you could remove this Target Portal with the command:

Remove-IscsiTargetPortal -TargetPortalAddress 192.168.129.24

Unfortunately this won’t work:

And neither does this:

What you actually have to do is specify both the Target Portal Address and the Initiator Portal Address when deleting an iSCSI Target Portal:

Remove-IscsiTargetPortal -TargetPortalAddress 192.168.129.24 -InitiatorPortalAddress 192.168.129.30

Over and out.

WMF 5.0 Download Removed from Download Center

2015-12-24T00:00:00Z

Bit of a bump in the WMF 5.0 road today: Microsoft has removed the WMF 5.0 RTM download from the download center because of a significant bug:

We recently released Windows Management Framework (WMF) 5.0 RTM delivering many requested improvements and fixes, via the Microsoft Download Center as announced in a previous blog post. However, we have discovered a bug which resets the PowerShell module environment during installation. As this issue can have a serious impact on our customers, we are taking the action to stop delivery of WMF 5.0 RTM, and have removed the packages from the Download Center. Additionally, we will be unpublishing Azure DSC Extension Handler versions 2.11 and 2.12 as they automatically install WMF 5.0 RTM.

So if you’re planning on rolling WMF5.0 out to production, best hold off for the moment. See the original blog post here.

Creating Professional DSC Resources – Part 6

2015-12-23T00:00:00Z

If you missed any previous articles you can find them here:

Recap

In the last couple of articles I covered the importance of automated testing and covered unit testing in particular (I’ll get to integration testing later). I had covered creating new unit tests using the unit test templates that are available here (although they will probably move here). I also covered how to complete the Pester Test Initialization and the Get-TargetResource and Set-TargetResource function areas of the unit test.

Unit Testing Completion

The final task in completing the unit tests is to complete the **Set-TargetResource ** in tests and also optionally tests for any other supporting functions your DSC Resource may have required.

In these unit tests I am using a DSC Resource for creating iSCSI Virtual Disks to illustrate the process. You don’t need to know anything about iSCSI Virtual Disks to understand these articles or resources, but if you’re interested to know the cmdlets I’m using for these, see this page. I’m using the *_iSCSIVirtualDisk cmdlets in this DSC Resource.

Function Test-TargetResource

This area will contain the actual Pester tests that test the Test-TargetResource function. These are fairly similar to the Set-TargetResource except we will be checking these two things:

The output of the Test-TargetFunction is correct. E.g. it returns false if changes are required, which will cause Set-TargetFunction to be called.
The expected Mocks are called by the Function.

This area may contain a large number of tests depending on the complexity of your DSC Resource. In most cases, you should expect there to create the tests from the following list, but often you will need even more for 100% code coverage:

Does the function return false when the resource being configured does exist and should, but one of the configured parameters does not match the current values? This test is usually repeated for each parameter in the DSC Resource.
Does the function return true when the resource being configured does exist and should, and all the configured parameters match the current values?
Does the function return false when the resource being configured does not exist but should?
Does the function return false when the resource being configured does exist but should not?
Does the function return true when the resource being configured does not exist and should not?

The bottom four of these tests are very similar. So I’ll only show examples of the top two contexts here.

Context ‘Virtual Disk exists and should but has a different …’

In this scenario we Mock the Get-iSCSIVirtualDisk cmdlet to return the object we defined in the Pester Test Initialization section. This is the behavior we’d expect if the resource being configured does exist:

Context 'Virtual Disk exists and should but has a different Description' {
    
    Mock Get-iSCSIVirtualDisk -MockWith { return @($MockVirtualDisk) }

    It 'should return false' {
        { 
            $Splat = $TestVirtualDisk.Clone()
            $Splat.Description = 'Different'
            Test-TargetResource @Splat | Should Be $False
        } | Should Not Throw
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
    }
}

This context will perform two tests:

Should return false - The Test-TargetResource should return false because we are changing the Description parameter so that the resource will require changes (e.g. Set-TargetResource should be called).
Should call the expected mocks - The Test-TargetResource should call the mocked cmdlets the expected number of times. In all contexts in this function this will always be just once.

The purpose of cloning the $TestVirtualDisk object is so we can modify the properties to simulate a property difference without modifying the $TestVirtualDisk object.

You should expect to repeat this context for each parameter that might be different.

Context ‘Virtual Disk does not exist but should’

In this scenario we Mock the Get-iSCSIVirtualDisk cmdlet to return nothing. This is the behavior we’d expect if the resource being configured does not exist:

Context 'Virtual Disk does not exist but should' {
    
    Mock Get-iSCSIVirtualDisk

    It 'should return false' {
        $Splat = $TestVirtualDisk.Clone()
        Test-TargetResource @Splat | Should Be $False
        
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
    }
}

As you can see, there is not too much different with these tests and you shouldn’t have any problems figuring out the remaining ones. Just remember, the goal is always to get 100% code coverage.

Unit Testing Supporting Functions

It is quite common that you might have implemented some supporting functions in your DSC Resource. These supporting functions are usually called by your standard *-TargetResource functions. If that is the case there are two important things you should do:

Write unit tests that cover all code paths in your supporting functions.
Add mocks to your *-TargetResource unit tests that prevent any constructive/destructive cmdlets that exist in your supporting functions from being called.

The first item is fairly self explanatory. For example, I often implement a get-* function in my DSC Resources which is used to pull the actual objects that will be used by the *-TargetResource functions (e.g. Get-VirtualDisk):

Function Get-VirtualDisk {
    param
    (
        [parameter(Mandatory = $true)]
        [System.String]
        $Path
    )
    try
    {
        $VirtualDisk = Get-iSCSIVirtualDisk `
            -Path $Path `
            -ErrorAction Stop
    }
    catch [Microsoft.Iscsi.Target.Commands.IscsiCmdException]
    {
        $VirtualDisk = $null
    }
    catch
    {
        Throw $_
    }
    Return $VirtualDisk
}

To unit test this function I’d write unit tests that tested the following contexts:

Context ‘Virtual Disk does not exist’
Context ‘Virtual Disk does exist’

For example:

Describe "$($Global:DSCResourceName)\Get-VirtualDisk" {

    Context 'Virtual Disk does not exist' {
        
        Mock Get-iSCSIVirtualDisk

        It 'should return null' {
            $Splat = $TestVirtualDisk.Clone()
            $Result = Get-VirtualDisk -Path $Splat.Path 
            $Result | Should Be $null             
        }
        It 'should call expected Mocks' {
            Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
        }
    }

    Context 'Virtual Disk does exist' {
        
        Mock Get-iSCSIVirtualDisk -MockWith { return @($MockVirtualDisk) }

        It 'should return expected parameters' {
            $Splat = $TestVirtualDisk.Clone()
            $Result = Get-VirtualDisk -Path $Splat.Path 
            $Result.Path                    | Should Be $MockVirtualDisk.Path
            $Result.DiskType                | Should Be $MockVirtualDisk.DiskType
            $Result.Size                    | Should Be $MockVirtualDisk.Size
            $Result.Description             | Should Be $MockVirtualDisk.Description
            $Result.ParentPath              | Should Be $MockVirtualDisk.ParentPath
        }
        It 'should call expected Mocks' {
            Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
        }
    }
}

As you can see, there isn’t much to it.

Earn a Chocolate Fish: If you look at the above supporting function and unit tests carefully, you’ll notice that I haven’t got 100% code coverage on it!

To get 100% code coverage I would have had to implement a unit test that covered the situation where the Get-iSCSIVirtualDisk function threw an exception that wasn’t a [Microsoft.Iscsi.Target.Commands.IscsiCmdException] exception.

In case you’re wondering, the Get-iSCSIVirtualDisk function throws a [Microsoft.Iscsi.Target.Commands.IscsiCmdException] when the cmdlet is called with the path parameter set to a path that does not contain a valid iSCSI Virtual Hard Disk file.

Unit Testing Exceptions

When creating unit tests you’ll often need to test a scenario where the function that is being tested is expected to throw an exception. If you read the Pester documentation, you’d might write a test for an exception like this:

Context 'Virtual Disk exists and should but has a different ParentPath' {
    
    Mock Get-iSCSIVirtualDisk -MockWith { return @($MockVirtualDisk) }

    It 'should throw an exception' {
        { Test-TargetResource @Splat } | Should Throw
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
    }
}

This would of course will work. It will ensure that the code throws an exception in this situation. The problem is we aren’t really sure if it is the exception that we expected it to throw. It could have been thrown by some other part of our code.

So to improve on this we need to do things:

Customize the exception that is thrown.
Change the unit test so that it checks for the customized exception.

Customize the Exception

To create a custom exception we need to create a new exception object containing our custom error message. The exception object is then used to create a custom Error Record:

$errorId = 'iSCSIVirtualDiskRequiresRecreateError'
$errorCategory = [System.Management.Automation.ErrorCategory]::InvalidArgument
$errorMessage = $($LocalizedData.iSCSIVirtualDiskRequiresRecreateError) -f $Path
$exception = New-Object -TypeName System.InvalidOperationException `
    -ArgumentList $errorMessage
$errorRecord = New-Object -TypeName System.Management.Automation.ErrorRecord `
    -ArgumentList $exception, $errorId, $errorCategory, $null

$PSCmdlet.ThrowTerminatingError($errorRecord)

In the above code, you just need to customize the $errorId and $errorMessage variables. The $errorId should just contain a simply string identifier for this particular type of error, but the $errorMessage can contain a full description of the error, including related parameters.

Once you’ve created the $errorRecord object you can call the ThrowTerminatingError method of the $PSCmdLet object, passing the $errorRecord object as the parameter.

Important: the $PSCmdLet object is only available in Functions that include the [CmdletBinding()] function attribute. So ensure your *-TargetResource and supporting functions include this attribute if you want to be able to access this object.

Test for the Customized Exception

To test for the custom exception object we need to create an identical object in the unit test and test for it:

Context 'Virtual Disk exists and should but has a different ParentPath' {
    
    Mock Get-iSCSIVirtualDisk -MockWith { return @($MockVirtualDisk) }

    It 'should throw an iSCSIVirtualDiskRequiresRecreateError exception' {
        $Splat = $TestVirtualDisk.Clone()
        $Splat.ParentPath = 'c:\NewParent.vhdx'

        $errorId = 'iSCSIVirtualDiskRequiresRecreateError'
        $errorCategory = [System.Management.Automation.ErrorCategory]::InvalidArgument
        $errorMessage = $($LocalizedData.iSCSIVirtualDiskRequiresRecreateError) -f $Splat.Path
        $exception = New-Object -TypeName System.InvalidOperationException `
            -ArgumentList $errorMessage
        $errorRecord = New-Object -TypeName System.Management.Automation.ErrorRecord `
            -ArgumentList $exception, $errorId, $errorCategory, $null
            
        { Test-TargetResource @Splat } | Should Throw $errorRecord
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
    }
}

The above code creates an identical exception object to the one produced by the exception in our DSC Resource code. The exception object can then be passed to the should throw cmdlet. If a different exception is thrown by the code then the test will fail - it will only pass if the exception object is exactly the same.

Important: Make sure both the $errorId and $errorMessage variables are exactly the same as what would be produced by the code when your unit test calls it. This includes ensuring that if your $errorMessage contains any parameters that the unit test $errorMessage contains the same parameter values.

That about completes creating unit tests. After you’ve implemented a few unit tests you’ll no doubt come up with your own method of implementing them, but hopefully this has given you a place to start.

Up Next - Integration Tests

In the next article, I’ll cover the integration tests. There are often the most difficult to implement, but if you can take the time to implement them then your DSC Resources are guaranteed to be extremely robust and bugs are far less likely to slip through.

Further parts in this series:

Creating Professional DSC Resources - Part 7

Creating Professional DSC Resources - Part 5

2015-12-20T00:00:00Z

These articles are also not intended to tell you what you must do to release your resource, but more to document what will help your resource be easier to use and extend by other people. Some of these things are obvious for people who have come from the development community, but may be quite new to operations people.

If you missed any previous articles, you can find them here:

Recap

Yesterday I talked about the importance of automated testing and covered unit testing in particular (I’ll get to integration testing later). I had covered creating new unit tests using the unit test templates that are available here (although they will probably move here). I also covered how to complete the Pester Test Initialization and the Function Get-TargetResource areas of the unit test.

Unit Testing Continued

The next task in completing the unit tests is to complete the Set-TargetResource area in the unit test.

In these unit tests, I am using a DSC Resource for creating iSCSI Virtual Disks to illustrate the process. You don’t need to know anything about iSCSI Virtual Disks to understand these articles or resources, but if you’re interested to know the cmdlets I’m using for these, see this page. I’m using the *_iSCSIVirtualDisk cmdlets in this DSC Resource.

Function Set-TargetResource

This area will contain the actual Pester tests that test the Set-TargetResource function. Unlike the Get-TargetResource, this area may contain a large number of tests depending on the complexity of your DSC Resource. In most cases, you should expect to create the tests from the following list, but often you will need even more for 100% code coverage:

Does each parameter get set/updated correctly when the resource being configured does exist and should? This test is usually repeated for each parameter in the DSC Resource.
Does it work when the resource being configured does not exist but should?
Does it work when the resource being configured does exist but should not?
Does it work when the resource being configured does not exist and should not?

Context ‘Virtual Disk exists and should but has a different …’

In this scenario, we Mock the Get-iSCSIVirtualDisk cmdlet to return the object we defined in the Pester Test Initialization section. This is the behavior we’d expect if the resource being configured does exist.

We are also going to Mock the Set-iSCSIVirtualDisk, New-iSCSIVirtualDisk, and Remove-iSCSIVirtualDisk. This is so we can ensure the expected cmdlets are called as well as prevent the real cmdlets from being run:

Context 'Virtual Disk exists and should but has a different Description' {
    Mock Get-iSCSIVirtualDisk -MockWith { return @($MockVirtualDisk) }
    Mock New-iSCSIVirtualDisk
    Mock Set-iSCSIVirtualDisk
    Mock Remove-iSCSIVirtualDisk

    It 'should not throw error' {
        { 
            $Splat = $TestVirtualDisk.Clone()
            $Splat.Description = 'Different'
            Set-TargetResource @Splat
        } | Should Not Throw
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
        Assert-MockCalled -commandName New-iSCSIVirtualDisk -Exactly 0
        Assert-MockCalled -commandName Set-iSCSIVirtualDisk -Exactly 1
        Assert-MockCalled -commandName Remove-iSCSIVirtualDisk -Exactly 0
    }
}

This context will perform two tests:

Should not throw error - The Set-TargetResource should not throw an error when called in this context.
Should call the expected mocks - The Set-TargetResource should call the mocked cmdlets the expected number of times.

The purpose of cloning the $TestVirtualDisk object is so we can modify the properties to simulate a property difference without modifying the $TestVirtualDisk object.

It is also important to ensure that we are not only checking that the expected cmdlets are called, but also that the other cmdlets in this function are not called. This is why we are checking the New-iSCSIVirtualDisk and Remove-iSCSIVirtualDisk are being called zero times.

You should expect to repeat this context for each parameter that might be updated.

Note: It is possible that updating some parameters may not be possible because of limitations in the underlying cmdlets. In this case I like to throw an exception so that the user is made aware that they are configuring a scenarios that can not be performed. In that case the test would be to ensure the correct exception occurs. I’ll cover testing exceptions in a later article.

Context ‘Virtual Disk does not exist but should’

In this scenario we Mock the Get-iSCSIVirtualDisk cmdlet to return nothing. This is the behavior we’d expect if the resource being configured does not exist.

Context 'Virtual Disk does not exist but should' {
    Mock Get-iSCSIVirtualDisk -MockWith { return @() }
    Mock Get-iSCSIVirtualDisk
    Mock New-iSCSIVirtualDisk
    Mock Set-iSCSIVirtualDisk
    Mock Remove-iSCSIVirtualDisk

    It 'should not throw error' {
        { 
            $Splat = $TestVirtualDisk.Clone()
            Set-TargetResource @Splat
        } | Should Not Throw
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
        Assert-MockCalled -commandName New-iSCSIVirtualDisk -Exactly 1
        Assert-MockCalled -commandName Set-iSCSIVirtualDisk -Exactly 0
        Assert-MockCalled -commandName Remove-iSCSIVirtualDisk -Exactly 0
    }
}

The context tests are very similar to all the other tests so I won’t go into detail on them here. It is important to note that the expected Mocks will be different.

Context ‘Virtual Disk exists but should not’

Context 'Virtual Disk exists but should not' {
    
    Mock Get-iSCSIVirtualDisk -MockWith { return @($MockVirtualDisk) }
    Mock New-iSCSIVirtualDisk
    Mock Set-iSCSIVirtualDisk
    Mock Remove-iSCSIVirtualDisk

    It 'should not throw error' {
        { 
            $Splat = $TestVirtualDisk.Clone()
            $Splat.Ensure = 'Absent'
            Set-TargetResource @Splat
        } | Should Not Throw
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
        Assert-MockCalled -commandName New-iSCSIVirtualDisk -Exactly 0
        Assert-MockCalled -commandName Set-iSCSIVirtualDisk -Exactly 0
        Assert-MockCalled -commandName Remove-iSCSIVirtualDisk -Exactly 1
    }
}

The context tests are very similar to all the other tests so I won’t go into detail on them here. It is important to note that the expected Mocks will be different.

Context ‘Virtual Disk does not exist and should not’

Context 'Virtual Disk does not exist and should not' {
    
    Mock Get-iSCSIVirtualDisk
    Mock New-iSCSIVirtualDisk
    Mock Set-iSCSIVirtualDisk
    Mock Remove-iSCSIVirtualDisk
        
    It 'should not throw error' {
        { 
            $Splat = $TestVirtualDisk.Clone()
            $Splat.Ensure = 'Absent'
            Set-TargetResource @Splat
        } | Should Not Throw
    }
    It 'should call expected Mocks' {
        Assert-MockCalled -commandName Get-iSCSIVirtualDisk -Exactly 1
        Assert-MockCalled -commandName New-iSCSIVirtualDisk -Exactly 0
        Assert-MockCalled -commandName Set-iSCSIVirtualDisk -Exactly 0
        Assert-MockCalled -commandName Remove-iSCSIVirtualDisk -Exactly 0
    }
}

The context tests are very similar to all the other tests so I won’t go into detail on them here. It is important to note that the expected Mocks will be different.

Unit Tests to be Continued

In the next article, I’ll cover the unit tests for Get-TargetResource as well as unit testing any additional functions. Thanks again for reading, and I hope it is useful.

Further parts in this series:

Markdown Preview in Visual Studio Code

2015-12-19T00:00:00Z

I used to keep around a copy of Markdown Pad to enable me to preview any markdown files I created. However, after an update to a Awesomium SDK Markdown Pad stopped working. I reported the issue but it hasn’t been resolved yet.

But after Irwin Strachan mentioned using Visual Studio Code as a Markdown editor, I wondered if there was an extension that allowed it to Preview the Markdown as well - because that was really all that was missing for me to do away with Markdown Pad. So I went and did a search for Markdown extensions in VS Code:

Want to install extensions in VS Code? Press F1 and type ‘ext install’.

I found a few, but after playing around with them a bit I noticed a small button in the latest version of Visual Studio Code at the top right corner when a markdown file is open:

The Preview button!

I wonder what that might do?

That is a great looking markdown preview.

Oh, now look at that! Just what I needed. I don’t know which version of Visual Studio Code this was added in, but it is fantastic!

You can also use the Visual Studio Code split window function to edit and Preview at the same time:

Changing the markdown automatically updates the preview - nice!

While we’re on the topic of markdown, Irwin also pointed me at this awesome markdown tutorial - it makes learning markdown fun!

Now, Visual Studio Code is not on par with ISE Steroids when it comes to editing PowerShell files, but when you’re dealing with lots of different types of files (including PS files) it can be a real time saver. I highly recommend downloading a copy of Visual Studio Code, installing the PowerShell Extension and giving it a try.

Windows Management Framework (WMF) 5.0 RTM Available

2015-12-18T00:00:00Z

I don’t know about you, but I am excited about the release to manufacture of Windows Management Framework (WMF) 5.0. It has been a long time coming and I’m hoping this release will resolve the issues I’ve been having with DSC configurations built on Windows 10 build 10586 (see this article).

You can download the WMF 5.0 installer for here.

Creating Professional DSC Resources - Part 4

2015-12-18T00:00:00Z

The purpose of this series of articles is to try and document a few of the lessons I learned while releasing new DSC resources as well as contributing to the existing Microsoft Community DSC resources. These articles are not intended to tell you how to write DSC resources from a programming perspective but to give you some ideas on what might be expected of a DSC resource you’re releasing to the public. For example, unit and integration tests (don’t worry if you aren’t familiar with those terms).

These articles are also not intended to tell you what you must do to release your resource but more to document what will help your resource be easier to use and extend by other people. Some of these things are obvious for people who have come from the development community but may be quite new to operations people.

If you missed any previous articles, you can find them here:

Automated Testing

Automated testing is something that is familiar to most developers, but for operations people, it is usually a new concept. However, it is one of the most important things you can add to your DSC Resources—and most DSC resource projects won’t even accept your code contributions if they don’t contain automated tests.

So, what are automated tests? Well, they are just PowerShell scripts that you create and run that will check your DSC Resource is working correctly. Usually, automated tests are run on your DSC Resources every time you commit your code—and they’ll tell you if anything has gone wrong. I could spend the next day listing reasons why automated testing is extremely important, but that is not the purpose of this post.

PowerShell contains a great automated test framework called Pester that allows you to describe your tests using special PowerShell functions.

If you aren’t familiar with Pester and automated testing, you should get familiar with it before reading any further. This series is a fantastic place to start. Even if you’re familiar with Pester, it is a good read.

An example of a Pester test on a DSC Resource:

Describe 'MSFT_xFirewall\Get-TargetResource' {
    Context 'Absent should return correctly' {
        Mock Get-NetFirewallRule

        It "Should return absent on firewall rule $($FirewallRule.Name)" {
            $result = Get-TargetResource -Name 'FirewallRule'
            $result.Name | Should Be 'FirewallRule'
            $result.Ensure | Should Be 'Absent'
        }
    }

    Context 'Present should return correctly' {
        $result = Get-TargetResource -Name $FirewallRule.Name

        # Looping these tests
        foreach ($parameter in $ParameterList) {
            $ParameterSource = (Invoke-Expression -Command "`$($($parameter.source))")
            $ParameterNew = (Invoke-Expression -Command "`$result.$($parameter.name)")
            It "should have the correct $($parameter.Name) on firewall rule $($FirewallRule.Name)" {
                $ParameterSource | Should Be $ParameterNew
            }
        }
    }
}

The above test is a unit test of the xFirewall resource in the xNetworking module. Don’t worry if you don’t completely understand this yet—that is the purpose of this article—although you should understand the basic structure of the Pester test. If you don’t, you’ll definitely want to go and review this series.

Types of Automated Tests

There are two types of automated tests you should create for your DSC Resources:

Unit Tests - These test that each function in your DSC Resource works correctly in isolation. This means that if you call the function with a set of parameters, you get the expected output.
Integration Tests - These tests ensure that your DSC Resource works in a real environment—e.g., works correctly when they are actually integrated into a DSC Configuration file.

For every DSC Resource in your DSC Resource Module, you should ensure that there is one unit test file and one integration test file (although usually for integration tests, a support file is also needed, but we’ll cover this later).

Test Folders

You should place all tests inside a Tests folder in the root of your DSC Module folder:

Unit tests should be placed in a Unit folder within Tests, and … I’m sure you get where I’m going here.

In the next article, I’ll cover unit tests for Set-TargetResource and Test-TargetResource, as well as unit testing any additional functions. Thanks again for reading, and I hope it is useful.

Further parts in this series:

Creating Professional DSC Resources - Part 3

2015-12-16T00:00:00Z

If you missed any previous articles you can find them here:

Coding Style

Everyone has there own preferences of how they like their code to look. Just take a look at all the PowerShell repositories on GitHub and you’d see a lot of different coding styles.

When I refer to coding style, this covers a many different things, for example:

Variable name format
Tab format (spaces or tabs?)
Function name format
Maximum line length
Comments
Curly braces allowed at the end of a line
Allow more than one blank line in a row

If you’re writing code that only you will look at and maintain, it doesn’t much matter what your coding style is - as long as you can understand it 6 months later. However, if you would like other people to use and possibly contribute and maintain your code or if you are contributing to community project, you’ll want to adopt a standard coding style.

The best reason for adopting a standard coding style is to ensure consistency in your code - especially across DSC resources within the same module. This makes it much easier for people to become familiar with your coding style and therefore easier for them to read and understand.

If you don’t have a particular coding style you have adopted, Microsoft has released a simple style guidelines document for DSC Resources that you can adopt quite easily.

Tip: Using the Microsoft style guidelines document is a requirement for contributing code to the Microsoft Community DSC Resources. So if you’re planning on contributing code or even entire resource modules to this project, I’d recommend you adopt the Microsoft style guidelines.

Message Localization

Adding support for language localization to your DSC Resources is trivial, but it is often over looked. Supporting data localization is much easier to add when you are creating your DSC Resource rather than adding it later on. This doesn’t mean you have to provide the language files of course, just provide support so that someone else could contribute them should they want to.

To ensure language support, create a new file in the same folder as your DSC Resource with the same name as the DSC Resource but with a psd1 extension. In that file create a Data section named LocalizedData containing the messages for your default culture. For example:

data LocalizedData
{
    # culture="en-US"
    ConvertFrom-StringData -StringData @'
GettingiSCSIVirtualDiskMessage=Getting iSCSI Virtual Disk "{0}".
iSCSIVirtualDiskExistsMessage=iSCSI Virtual Disk "{0}" exists.
iSCSIVirtualDiskDoesNotExistMessage=iSCSI Virtual Disk "{0}" does not exist.
SettingiSCSIVirtualDiskMessage=Setting iSCSI Virtual Disk "{0}".
EnsureiSCSIVirtualDiskExistsMessage=Ensuring iSCSI Virtual Disk "{0}" exists.
EnsureiSCSIVirtualDiskDoesNotExistMessage=Ensuring iSCSI Virtual Disk "{0}" does not exist.
iSCSIVirtualDiskCreatedMessage=iSCSI Virtual Disk "{0}" has been created.
iSCSIVirtualDiskUpdatedMessage=iSCSI Virtual Disk "{0}" has been updated.
iSCSIVirtualDiskRemovedMessage=iSCSI Virtual Disk "{0}" has been removed.
TestingiSCSIVirtualDiskMessage=Testing iSCSI Virtual Disk "{0}".
iSCSIVirtualDiskParameterNeedsUpdateMessage=iSCSI Virtual Disk "{0}" {1} is different. Change required.
iSCSIVirtualDiskDoesNotExistAndShouldNotMessage=iSCSI Virtual Disk "{0}" does not exist and should not. Change not required.
iSCSIVirtualDiskRequiresRecreateError=iSCSI Virtual Disk "{0}" needs to be deleted and recreated. Please perform this manually.
'@
}

Each line contains a localized message - in this case for the culture en-us. You could of course use a different default culture if you wanted to.

At the beginning of your DSC Resource you would the following command to ensure the appropriate localization strings are imported:

Import-LocalizedData -BindingVariable LocalizedData -Filename BMD_cMyNewResource.psd1

Alternately, if you want to support Message Localization but don’t want to have to supply your default messages in a separate file, you can place the LocalizedData section for your default culture at the top of your DSC Resource and exclude the Import-LocalizedData command.

Using Localization Messages

Once you’ve got the messages in, using them is extremely easy:

Write-Verbose -Message ($LocalizedData.iSCSIVirtualDiskExistsMessage -f $Path)

You can of course consume the messages anyway you like, but all of your localized messages are just properties of the LocalizedData object.

Other Localization Files

You put LocalizedData for other languages in separate PowerShell files in sub-folders named after the culture the file contains. For example, you might create a sub-folder called en-uk and place a file called BMD_cMyNewResource**.psd1** containing the en-uk messages.

Example DSC Configuration Files

Another element of any easy to use DSC Resource are example DSC Configuration files. These can usually be found in the Examples folder in the root of the DSC Module:

There should usually be a number of different DSC Configuration files in this folder, showing common scenarios for using your DSC Resources:

The file name of any example files should be prefixed with Sample or Example so that they can be easily identified and differentiated from types of DSC Module files. The summary of the purpose of the configuration should also be included in the file name. This is fairly obvious I realize, but I have seen public DSC Resources named Example_1, Example_2, Example_3 etc - which reduces usability of the examples.

Each example should contain a DSC Configuration in a form that it could be used without any modification to actually test the resource. This might include installing prerequisite Windows Features or starting services etc. This allows a potential user to test drive the resource without investing a whole lot of work trying to figure out how to use it.

For example:

configuration Sample_ciSCSIInitiator
{
    Param
    (
         [String] $NodeName = 'LocalHost'
    )

    Import-DscResource -Module ciSCSI

    Node $NodeName
    {
        Service iSCSIService 
        { 
            Name = 'MSiSCSI'
            StartupType = 'Automatic'
            State = 'Running'  
        }

        ciSCSITargetPortal iSCSITargetPortal
        {
            Ensure = 'Present'
            TargetPortalAddress = '192.168.128.10' 
            InitiatorPortalAddress = '192.168.128.20'
            DependsOn = "[WindowsFeature]iSCSIService" 
        } # End of ciSCSITargetPortal Resource

        ciSCSITarget iSCSITarget
        {
            Ensure = 'Present'
            NodeAddress = 'iqn.1991-05.com.microsoft:fileserver01-cluster-target'
            TargetPortalAddress = '192.168.128.10'
            InitiatorPortalAddress = '192.168.128.20' 
            IsPersistent = $true 
            DependsOn = "[ciSCSITargetPortal]iSCSITargetPortal" 
        } # End of ciSCSITarget Resource
    } # End of Node
} # End of Configuration

The above DSC Resource example will ensure the MSiSCSI service is running before configuring an iSCSI Initiator.

Tip: Make the name of the configuration the same as the sample configuration file (without the extension of course).

It is also a great idea to copy the content of any example DSC Configuration files into the Examples section of the Readme.md of your DSC Resource Module along with a brief description of what the configuration will produce. You’ll find that all Microsoft Community DSC Resources do this.

In The Next Article

I intended on covering creating unit and integration tests in this article, but as that is by far the most involved part of creating a community DSC resource I’ve decided I’ll dedicate an entire part to each. So, rest assured the next ones will contain this very important component of any public DSC Resource. Thank you for reading this far and I hope you’re finding it useful.

Further parts in this series:

Creating Professional DSC Resources - Part 2

2015-12-14T00:00:00Z

If you missed any previous articles you can find them here:

Creating Professional DSC Resources - Part 1

Before You Start Coding

So, you have an idea or need for a set of super new DSC Resources. Before you write a single line of code you should first go and take a look at the documentation provided by the DSC Community. These guys (many of them from Microsoft) have been doing this stuff for a while and they’ve come up with a set of best practices and instructions on how to get started.

The above GitHub repository should be your first port of call and for DSC creation and it is worth keeping an eye on this repository by watching it:

This will cause you to be notified whenever any changes to this repository are made (which isn’t that often). So if the best practices are updated you’ll be kept in the loop!

This repository also contains some template files you can use to create a new DSC Resource module.

Creating the Resource Module

Sure, there is no reason why you can’t just jump straight in and knock out a PSD1 file and some PSM1/MOF files and be done with it. But creating a resource that other people can easily use, usually requires a few other files.

First, you need to decide on the name for the DSC Resource module folder. This is usually simple enough, but if you think your module may contain more than one resource it is worth naming it with a more generic name. For example, if you were creating a DSC Resource module that will contain resources for configuring iSCSI Targets and iSCSI Initiators you might name your folder ciSCSI.

Tip: Your resource folder should begin with a lower case c. This indicates it is a Community DSC resource. This is not a requirement, but it tells people that the resource was created by the community (in this case you). DSC Resource modules starting with an x indicate that this is a Microsoft Community DSC resource and maintained by the PowerShell team as well as the community, but are not built in to DSC.

Once you’ve created the folder to store your new DSC Resource module, you should make a copy of all the files in the GitHub repository folder found here to the root of your new DSC Resource folder:

The easiest way to get a copy of these files is to use Git to clone the DSCResource repository on your computer and then copy the files from the DSCResource.Template folder to your new DSC module folder:

The DSCResource.Template Files

At the time of writing this the DSCResource.Template folder only contains two files:

Readme.md - tells people how to use your DSC Resource as well as containing usage examples and Version information. You should fill this in as soon as possible and keep it up-to-date everytime you change your DSC Resource.
AppVeyor.yml - this is a file that configures the AppVeyor Continuous Integration (CI) for your DSC Resource Module. I will cover AppVeyor CI later in the series. At the moment don’t worry about this file, just copy it to your module folder and leave it alone.

Markdown

The readme.md file (like most other files with an .md file extension) is text file in the Markdown format. Because this file is being made available up on GitHub you can use GitHub Flavored Markdown. Markdown is really easy to create straight in your editor of choice and you’ll find it in use all over GitHub, so it is a good idea to get somewhat familiar with it (it should only take you about 2 minutes to get a handle on it).

If you want an example of how your Readme.md file might be constructed, have a look at this example. Of course you are completely free to format it any way you like, but keeping to a standard makes it easy for users to know what they can expect.

In The Next Article

As I’m trying to keep these parts short I’ll break for today and continue tomorrow. In the next part I intend to cover code guidelines and examples, with unit and integration testing to follow. I hope you have found this interesting!

Further parts in this series:

Creating Professional DSC Resources -Part 1

2015-12-14T00:00:00Z

Introduction

Writing desired state configuration (DSC) resources can be a little bit tricky at first if you’ve not come from a programming background. But once you’ve written one or two, you’ll quickly find yourself churning them out like Disney releases Star Wars branded rubbish. That is how it went for me.

However, I quickly found that there is a big difference between writing a simple resource for your own consumption and releasing a community DSC resource that may be used and abused by tens, hundreds or even thousands of people. After I decided to release my first resource (this one) to the public, I quickly found that it really wasn’t measuring up to what was being released by Microsoft and other folk in the community. So I spent about 6 months releasing and re-releasing my resources as I got to know the best practices.

So that gets us to the purpose of this post: to try and document the lessons I learned over the last year creating my own DSC resources and contributing to the Microsoft Community DSC resources. Hopefully by doing this it might save someone else a bit of time and avoid the joys or re-writing your resources.

Tip: The best practices for creating DSC resources change quite often as better ways of doing things are discovered it is useful to keep and eye on some of the more active DSC resources (like this one) in the Microsoft Community DSC resources. Even since I started contributing the processes and best practices have changed quite a lot.

Where to Start

If have no idea what Desired State Configuration (DSC) is, then this is a good place to start.

If you’ve not created a custom DSC resource before, then you should first get to know the basic process before reading this guide. There are many really great tutorials on creating DSC resources.

If you have written a resource or two or are familiar with the process, but you want to know how to go about releasing your custom DSC resource to the community then this post might interest you. So read on!

Does it Already Exist?

Before even getting started writing a custom DSC Resource it is a good idea to see if it already exists. It might not be included in the standard DS resources or in the Microsoft Community DSC Resources, but someone else may have already created one that will work for you. So why re-invent the wheel?

The best places to look for DSC resources is to search on PowerShell Gallery (you’ll find the Microsoft Community DSC Resources here as well). If you can’t find anything that looks like it’ll work for you, then you could also do a search on GitHub.

If you have WMF 5.0 installed (or have installed the PowerShell Package Modules installed) then you can search the PowerShell Gallery for DSC resources using PowerShell cmdlets:

Find-Module -Tag DSC

This will only find modules tagged with DSC. Some resources may not be tagged with this so they will not appear in this search. Therefore you may need to tweak the search.

If you find a resource you want to examine, you can install it using:

Install-Module -Name cFSRM

Or if you’re not sure you want to trust it you can use the save-module cmdlet to save a copy to a folder so you can open and examine the resource code:

Save-Module -Name cFSRM -Path c:\temp

If you find the DSC resource module else where you’ll need to download and install it manually - the same way install any other module.

Modifying an Existing Resource

If you find something that is close to what you need, but is missing some parameter or functionality, rather than starting a whole new resource you could ask the maintainer if they would be able to add the feature for you. Or even better, you could ask the maintainer if they’d object to you making the change and submitting it back to them via a Pull Request. Even if they’re not interested including your change, at least you can modify their resource adding the functionality you need - which will save you a lot of time**.** But in my experience, most maintainers are grateful for the assistance and welcome new features (as long as they go along with the resources overall design).

In fact, I’ve learned more from helping out on the community resources than from writing my own. If you have some spare time and want to help the community out, I’d strongly recommend helping out on the Microsoft Community DSC Resources. Just head over there, take a look at a resource you’re interested in then have a look at the issues for the resource and you’re bound to find a request or two for additional features. If you don’t find a request for a new feature but you see that one is missing that you’d like to see implemented, raise an issue yourself and offer to create it.

At the time I wrote this post there were some big holes in the functionality of most of the existing DSC community resources. The xDNSServer resource is a prime example.

Tip: one of the best ways to learn how to write professional quality resources is to look at some of the Microsoft Community DSC Resources. I’d strongly recommend the xNetworking resource (no, not because I contributed to it a bit) because it is very active and when best practices are updated this tends to be one of the first resources to get updated.

Source Control

So you’ve decided you’re going to write a new DSC resource (or modify an existing one). First thing you’ll need to be a little bit familiar with is Source Control. This used to be mainly the realm of developers, but it should now be a key tool in the Operations toolbox. I’m going to assume that you are a little bit familiar with Git and GitHub for this series.

If you’re not familiar with using Git and GitHub, this is the article you should read. It is written by the DSC Community and covers the basics, all from a DSC perspective. Even if you are familiar with Git and GitHub but haven’t used it for DSC resources, this document is worth a read.

Here is a summary of the Source Control things you should already know/have done:

You should have a GitHub account (it’s free). This is the most likely place you’ll be storing your DSC Resources, and you’ll find the other community ones here too.
You should have downloaded and installed the Git client, GitHub Desktop client or be using a tool like Microsoft Code that has a Git client built in.
Ideally you should be familiar with what Forks, Branches and a Pull Requests are. There are some pretty amazing guides out there that will tell you everything you need to know in about 10 minutes.
Decided if you’re going to create a new resource or fork someone elses and modify it.

Gitter: Asking Questions

Something I didn’t find out till many months after I started to contribute was that there is a Gitter chat channel where you can chat directly with some of the many of the people who contribute to the Microsoft Community DSC Resources. It is a great place to lurk and just see what is going on. If you want to contribute to a resource but you’re not sure what, if you ask, I’m sure you’ll receive lots of suggestions as to what needs some work. You’ll learn a lot just watching the discussion and if you need some help it’s a great place to ask.

Tip: If you have a GitHub account, you can sign in to Gitter with it.

What Next?

So, that is pretty the introduction over with. In the next article in this series I’m going to cover what you should do when you’re going to start a DSC Resource from scratch - one that you intend on releasing to the community. The final part of the series I’ll cover adding features to an existing Microsoft Community DSC Resource.

Hopefully this is of interest to a few out there and it doesn’t scare off any potential DSC contributors. It really is a great feeling to contribute back to the community.

Further parts in this series:

File Server Resource Manager DSC Resource

2015-12-10T00:00:00Z

With Our Powers Combined, We Are Captain FSRM

I’ve been recently working on combining my three File Server Resource Manager DSC Resources into a single module. This made more sense and will make it a lot easier to maintain.

Integration Testing

At the same time as combining these resources, I also added integration tests (based on the ones that were added recently to the Microsoft xNetworking resource). This identified a number of bugs that had previously been overlooked. If you’re in the DSC resource writing game (or just starting), I strongly recommend adding integration tests early on—it’ll definitely save you a lot of grief and just make life more enjoyable for you and everyone who uses your resource. I intend on writing an introduction article to unit and integration testing DSC resources over the next month—so keep an eye out for it if you’re interested.

The new File Server Resource Manager (cFSRM) resource replaces these old resources which have now been deprecated:

So, if you’re using any of the above resources, you should update your DSC Configuration files to use the new cFSRM one. The resources are completely compatible with the old ones, so you should just need to update the Import-DSCModule cmdlet in any configuration files.

Installing the Resource

You can find the new cFSRM resource on Microsoft Script Center or on the PowerShell Gallery.

For those of you using Windows Management Framework 5.0 (or have the PowerShellGet module installed), you can just use the command:

Install-Module -Name cFSRM

Using the Resource

Rather than go into detail on using this resource, you can find the full documentation and usage examples here.

If you need some additional guidance or other specific examples, please feel free to let me know and I’ll do my best to help you out.

Feedback

If you’re interested in contributing to this resource, providing feedback, raising issues, or requesting features, please feel free (anything is appreciated). You’ll find the resource GitHub repository here where you can fork, issue pull requests, and raise issues/feature requests.

Get the BIOS GUID of a Hyper-V VM

2015-11-29T00:00:00Z

I’ve just spent the last few hours looking into how I can get the BIOS GUID from a Hyper-V VM from inside the Host OS. I needed this so I could use it to pre-stage devices in Windows Deployment Services. I could have used the MAC address of course, but I decided I wanted to use the BIOS GUID instead.

So after a fair bit of hunting all I could turn up was an older VBS script. I decided this wasn’t ideal and so went about investigating how I might do this in PowerShell (this is a PowerShell blog mainly after all). Well after a few minutes I came up with this (rather long) command:

$VMName = 'My VM'
(Get-CimInstance -Namespace Root\Virtualization\V2 -ClassName Msvm_VirtualSystemSettingData -Filter "ElementName = '$VMName'").BiosGUID

It uses WMI/CIM, but does seem to work nicely (don’t forget to set the name of the VM):

Good night!

Nano Server TP4

2015-11-20T00:00:00Z

Just a quick one for Friday. After downloading the new Windows Server 2016 TP4 ISO, I quickly fired up my New-NanoServerVHD script to see how it went. Unfortunately, I ran straight into a bug in the Convert-WindowsImage script. The bug in this script only occurs when the WIM file being converted only contains a single image—which as of TP4 includes the NanoServer.wim.

If you try and run the New-NanoServerVHD script using the unfixed version of the Convert-WindowsImage script and TP4, you’ll run into the following error message:

ERROR : The variable cannot be validated because the value $null is not a valid value for the Edition variable

So, after reporting the error to the original script creator, I went ahead and fixed the problem myself and uploaded a working version to GitHub (until it has been fixed in the official version). You can download my fixed version from here.

Installing Nano Server TP4

After fixing the bug in the Convert-WindowsImage.ps1 file, here are some updated instructions on using this script to quickly create a new Nano Server TP4 VHD or VHDx.

Create a New Nano Server VHD

It is fairly straightforward to install and use:

Create a Working Folder on your computer. In this example, I used C:\Nano.
Download the _New-NanoServerVHD.ps1_ to the Working Folder.
Download the _Convert-WindowsImage.ps1_ (download here) to the Working Folder.
Download the Windows Server 2016 Technical Preview ISO (download here) to the Working Folder.
Open an Administrative PowerShell window.
Change directory to the Working Folder (cd C:\Nano).
Execute the following command (customizing the parameters to your needs):

.\New-NanoServerVHD.ps1 `
    -ServerISO 'C:\Nano\10586.0.151029-1700.TH2_RELEASE_SERVER_OEMRET_X64FRE_EN-US.ISO' `
    -DestVHD C:\Nano\NanoServer01.vhdx `
    -VHDFormat VHDX `
    -ComputerName NANOTEST01 `
    -AdministratorPassword 'P@ssword!1' `
    -Packages 'Containers','OEM-Drivers','Guest','IIS','DNS' `
    -IPAddress '192.168.1.65'

Available Packages in TP4

There are a bunch of new packages that are now available in TP4 for integrating into your Nano Server builds. I’m not quite sure of the exact purpose of some of them, but I’ve listed them here:

Compute: Hyper-V Server
OEM-Drivers: Standard OEM Drivers
Storage: Storage Server
FailoverCluster: Failover Cluster Server
ReverseForwarders: ReverseForwarders to allow some older App Servers to run
Guest: Hyper-V Guest Tools
Containers: Support for Hyper-V and Windows containers
Defender: Windows Defender
DCB: Unsure
DNS: DNS Server
DSC: PowerShell Desired State Configuration Support
IIS: Internet Information Server (Web Server)
NPDS: Unsure
SCVMM: System Center VMM
SCVMM-Compute: System Center VMM Compute

Over and out.

Windows Server 2016 TP4 is available

2015-11-19T00:00:00Z

After yesterday’s exciting release of Visual Studio Code with PowerShell support, today we get Windows Server 2016 Technical Preview 4.

I’m diving straight into the Nano Server to see what has changed and if Containers is working.

PowerShell Language Support in Visual Studio Code

2015-11-18T00:00:00Z

I feel like my birthday has come early: Microsoft has added PowerShell language support in Visual Studio Code:

I mainly use PowerShell ISE with ISE Steroids for most of my coding, but I find when I’m needing to switch between working on multiple files of different types Visual Studio Code is so much faster and slicker. Visual Studio 2015 with PoSH tools would be the ideal way to go, but the PoSH extension is quite slow with any files larger than 20KB. So I’m really excited with what this can do.

Windows 10 Build 10586 - PowerShell Problems

2015-11-15T00:00:00Z

PowerShell Direct Broken

Unfortunately my Sunday afternoon of planned study has been slightly derailed, as last night I just upgraded my primary work machine to Windows 10 Build 10586. Everything went fine with the upgrade and seemed to be working just perfectly. However, when I started to get to work with my Hyper-V lab machines today I ran into a significant bug with PowerShell on this build:

PowerShell Direct no longer connects to any of my VMs. It pauses for approximately 30 seconds and then reports a rather mysterious error:

An error has occurred which Windows PowerShell cannot handle. A remote session might have ended.

But it was working yesterday!

Passing credentials that are correct or incorrect for the VM have no effect - the error message is always the same.

Fortunately connecting via plain old PowerShell Remoting still works fine, but I have many scripts that are dependent on using PowerShell Direct that are no longer functioning - including my LabBuilder project, which I use every day to get my Hyper-V lab up and running for my studies.

I’ve logged this issue in Windows Connect right here. So if you’re being affected by this issue please go and vote it up to see if it can’t be resolved quickly. This was a fantastic feature that was only added recently and it is sad to see it being broken so soon!

Encrypting Credentials in DSC MOF Files

Anyone who has put properly encrypted credentials in DSC configuration files knows that they need to be encrypted using a certificate that was issued to the Node that will be reading the configuration. This is usually fairly straight forward, but some care needs to be taken when generating the certificate for the Node to use.

I have an automated process designed for my Lab where any new VM’s that get built are automatically issued with a self-signed certificate that will be used to encrypt the DSC config files. This certificate is automatically downloaded to the host (via PowerShell Direct of course) and then used to encrypt the DSC config files for that node. All completely seamless and automatic. Until build 10586, when these certificates are no longer able to be used to encrypt the MOF file. Instead I get this error:

ConvertTo-MOFInstance : System.ArgumentException error processing property ‘Password’ OF TYPE ‘MSFT_Credential’: Certificate ‘8E474886A6AA72859BDC3C2FBEEFAAD7E089A5DD’ cannot be used for encryption. Encryption certificates
must contain the Data Encipherment or Key Encipherment key usage, and include the Document Encryption Enhanced Key Usage (1.3.6.1.4.1.311.80.1).

Ok, so this isn’t the end of the world and it is pretty clear what has changed here. After looking at my existing self-signed certificates they didn’t include the EKU (Enhanced Key Usage) of Document Encrpytion.

My previous certificates - now useless because the Document Encryption EKU is missing.

It seems this is now required to encrypt credentials in MOF Files. I guess this makes sense and I’m sure not that many people are going to run into the problem. But in case you do, you’ll need to reissue these certificates including the following EKU:

Document Encryption (1.3.6.1.4.1.311.80.1)

You also need to ensure the Key Usage contains either Data Encipherment or Key Encipherment.

Finally, I have one strong recommendation related to the topic of encrypting DSC credentials: Don’t use the built in PowerShell cmdlet New-SelfSignedCertificate to create self-signed certificates for this purpose. It creates certificates that are not compatible for other reasons (I won’t go into detail but you can look the issue up I’m sure). Instead I strongly recommend you use this script on MSDN Script Center.

Decryption Failed

Update 2015-12-18: Installing Windows Management Framework (WMF) 5.0 RTM on the DSC node resolves the Decryption Failed error described below. So if you’re experiencing this issue, install this update on any DSC nodes experiencing this problem.

Edit: Karl in his comment on this post mentioned a problem he was having where the DSC node was failing to decrypt any credentials provided in DSC MOF files created on the build 10586. He was receiving a Dercyption Failed error when the MOF was being applied to the node:

I hadn’t noticed this issue because I hadn’t been working on DSC for a week, but when I tried to apply a rebuilt MOF file I experienced the same issue.

It was also reported that the password property format of the MSFT_Credential object in the MOF seems to have changed in one of the recent releases from:

instance of MSFT_Credential as $MSFT_Credential2ref
{
Password = “…Base64Password…”;
UserName = “LABBUILDER.COM\\Administrator”;
};

To:

instance of MSFT_Credential as $MSFT_Credential2ref
{
Password = “-----BEGIN CMS-----\n…base64Password…\n-----END CMS-----”;
UserName = “LABBUILDER.COM\\Administrator”;
};

After a full day of investigating this issue, I can confirm it has been caused by a change the Microsoft has made in the PSDesiredStateConfigration module supplied with this build (specifically the Get-EncryptedPassword function, in case you’re interested). This issue has been reported to Microsoft on PowerShell Connect here. Please go an upvote it if you’re having this problem (even if you’re not). Karl has posted this issue on Stack Overflow.

In the mean time I have posted a work around (roll back to a previous version of the PSDesiredStateConfiguration module on the Stack Overflow page) - so if you want to work around this problem, go and take a look.

DSC is Practically Broken in 10586

Update: After finishing this last post I’ve run into some critical problems with DSC on build 10586. Specifically, when I build a DSC configuration on this machine and include the PSDesiredStateConfiguration resource (by way of Import-DSCResource cmdlet) the MOF file that is created references a 1.0 version of the module - which doesn’t exist:

Version 1.0 isn’t on the machine!

Applying the MOF file to any node immediately throws an error because of course this module doesn’t exist (1.1 is the earliest version of this module that are on any of the nodes).

However, if I force the module version to 1.1 in the Import-DSCResource cmdlet then the MOF file that is created has the correct module version and can be applied to the node without any issue:

Forcing the Module Version.

But of course going around all my config files and forcing the module version to 1.1 is a very unsatisfactory solution. Also, I’m not sure if it is just the PSDesiredStateConfiguration resource that has this problem or all modules. I haven’t had the time to investigate this further yet.

If you are suffering from any of these issues in build 10586, please let me know.

Thanks for reading!

Pester as an Operation Validation Framework

2015-11-13T00:00:00Z

In this latest video on Channel 9, Jeffrey Snover (the grand wizard of PowerShell) is suggesting what might be on the horizon in Windows Server 2016. In it, he says they’re looking at using Pester (or a form of it) to allow you to create Operational Validation tests for your servers and environment, so that after any environmental changes are made, the environment is validated automatically. This sounds like a fantastic idea to me and such an obvious fit for Pester! After doing a bit of digging around, it seems like this idea has been around for a while—see this post here for an example of how it can be used in practice.

Of course, there does feel like there is a little bit of an overlap here with DSC, but I’m sure the implementation will play well with DSC. All of these new ideas and technologies (Nano, Containers, DSC, Operational Pester tests, etc.) are just more tools in the “Infrastructure as Code” tool belt. So I’m very happy!

I suggest watching the whole video (found here) as it is really interesting, but if you want to just jump to the bit about Pester, it starts at about 11:48. I’m really eager to see where Microsoft is going with this stuff in Windows Server 2016. Roll on TP4!

WFAS Firewall Rules Group vs. DisplayGroup

2015-11-10T00:00:00Z

Recently I’ve been helping resolve a couple of issues with the behavior of the xFirewall resource in the xNetworking DSC Module. One of these was trying to implement both the Group and DisplayGroup parameters when creating a new Firewall Rule.

A Firewall Rule Group/Display Group.

The obvious assumption is that the Group and DisplayGroup parameters behave in a similar fashion to the Name and DisplayName parameters. Unfortunately, this is not the case and it took me quite a lot of digging to figure out how they actually work.

Why is this Important?

This became an issue for me while trying to determine how to best implement these parameters in the xFirewall resource. It became clear that it is not actually possible to set the DisplayGroup parameter using any of the *-netfirewallrule cmdlets. So adding this parameter to the resource caused a lot of problems and was also causing quite a bit of confusion, especially as the relationship between Group and DisplayGroup is not clearly documented anywhere I could find.

You can’t even set the DisplayGroup parameter via NETSH or in the WFAS (Windows Firewall with Advanced Security) UI. In fact, the WFAS UI only shows the DisplayGroup and is labeled as Group—the actual Group is hidden completely:

WFAS UI showing the DisplayGroup (labeled as Group)

The Relationship

So, onto the actual reason for this post: What is the relationship between Group and DisplayGroup and how does DisplayGroup get set? I’m hoping this will help someone else who is trying to understand this undocumented behavior.

There are two possible values that the DisplayGroup can have and these can never be set directly. They are:

If Group does not start with an @ then the DisplayGroup will be the same as Group.
If Group starts with an @ then the DisplayGroup will be a value pulled from a DLL or Windows Universal App resource. This creates the rule as a predefined rule.

For example, here is the DisplayGroup being pulled from a DLL:

And here is one being pulled from a Universal App Resource:

So, what is the take away from all this?

The DisplayGroup can’t be set manually by you and will never be different to the Group unless the Group starts with an @.

A predefined rule will prevent most settings of the rule from being changed in the WFAS UI, although they still can be changed in PowerShell.

A predefined rule can’t be changed via the WFAS UI.

And one final piece of info: The Group (and by extension the DisplayGroup) can’t be changed for an existing rule. You can only change it by deleting and recreating the rule!

How to tell if a PowerShell variable has not been Declared

2015-11-10T00:00:00Z

In most situations in PowerShell, I am really only interested if a variable has a value or not (e.g. not null). Checking for this is easy:

if ($myvariable -eq $null) {
    Write-Host -Message '$MyVariable is null'
} else {
    Write-Host -Message '$MyVariable has a non-null and non-blank value'
}

But what if I want to know if a variable is not declared at all? The method of doing that is not so obvious and being PowerShell there are many ways of doing it. By far the clearest I think is to use the Test-Path cmdlet using the Variable provider:

if (Test-Path -Path Variable:\MyVariable) {
    Write-Host -Message '$MyVariable is declared'
} else {
    Write-Host -Message '$MyVariable is not declared'
}

If there is a cleaner or officially recommended way of doing this I’d be most keen to hear about it.

File Server Resource Manager (FSRM) Classifications DSC Resource

2015-11-04T00:00:00Z

Introduction

I’ve been spending a bit of time lately working on some issues and improvements on the xNetworking DSC Resource so haven’t been spending as much time working on the series of File Server Resource Manager (FSRM) DSC Modules as I’d like. That said, I have managed to complete another module. This one is used for configuring Classification Properties and Property Values, Classification Configuration and Classification Rules.

If you missed any of the previous FSRM DSC Modules:

Resources

This module contains the following resources:

cFSRMClassification- configures FSRM Classification settings. cFSRMClassificationProperty- configures FSRM Classification Property Definitions. cFSRMClassificationPropertyValue- configures FSRM Classification Property Definition Values. This resource only needs to be used if the Description of a Classification Property Definition Value must be set. cFSRMClassificationRule- configures FSRM Classification Rules.

The purpose of the resources should be fairly self explanatory, as long as you have a basic understanding of how FSRM Classifications are used.

Installing the Resource

If you have installed WMF 5.0 you can just download this directly from the PowerShell Gallery by running this command:

Install-Module -Name cFSRMClassifications

Otherwise you’ll need to download this from the Microsoft Script Center here and unzip it into your PowerShell modules path.

Using the Resource

As per the last post on these resources, rather than go into detail on using this resource, I thought I’d try and keep it short and just provide a link to the documentation. This covers the parameters available in the resources as well as some usage examples.

If you need some additional guidance or other specific examples, please feel free to let me know and I’ll do my best to help you out.

Hopefully this resource finds some use out there, but either way it has been extremely helpful to me really imprint the underlying FSRM features and usage into my own mind.

Feedback

If you’re interested in contributing to this resource, providing feedback or raising issues or requesting features, please feel free (anything is appreciated). You’ll find the resource GitHub repository here where you can fork, issue pull requests and raise issues/feature requests.

NanoServer Container Base Image - It does Exist...Somewhere!

2015-10-29T00:00:00Z

A really interesting video from Microsoft was just released with Mark Russinovich (CTO of Azure if you don’t already know) demonstrating Windows Server Containers. What is really interesting about this demo is that he is demonstrating containers using a Windows NanoServer Base Image:

Nano Server Containers Base Image - it does exist.

If you’ve read any of my previous posts here and here you’ll know I spent quite some time looking at this and trying to get it going with TP3. I deduced it was not possible yet without the Windows NanoServer Base Image for containers - which had not been provided by Microsoft.

Other eagle-eyed viewers will also note that he appears to be running a Nano Server container on a Full Server container host, which I didn’t actually think was possible. From what I originally understood about containers, you could only instantiate a container using a base container image matching the version of the OS the container host used. For example, you cannot instantiate a Server Core container on a NanoServer container host—I confirmed this was the case in TP3. But perhaps I misunderstood, or perhaps containers can be instantiated on “up” version container hosts but not “down” version.

Actually, on further examination he is remoting into a different server that is acting as a Container Host (10.205.158.127). So I can’t assume that this remote host is a Full Server—it could well be a NanoServer. So the above paragraph isn’t relevant.

I also notice that he demos Hyper-V Containers, which as far as I am aware aren’t working on TP3. So this would indicate a more recent build than TP3.

So perhaps we’ll see this image being made available in the Windows Server 2016 TP4 release?

File Server Resource Manager (FSRM) File Screen DSC Resource

2015-10-26T00:00:00Z

Introduction

Continuing on with implementing File Server Resource Manager (FSRM) DSC Modules, I’ve added a new module for configuring File Screens, File Screen Templates and File Screen Exceptions. If you missed it the previous module for configuring quotas can be found here.

Resources

This module contains the following resources:

cFSRMFileScreen - configures FSRM File Screen. cFSRMFileScreenAction - configures FSRM File Screen Actions for File Screens. cFSRMFileScreenTemplate - configures FSRM File Screen Templates. cFSRMFileScreenTemplateAction - configures FSRM File Screen Template Actions for File Screen Templates. cFSRMFileScreenExclusion - configures FSRM File Screen Exclusions.

The purpose of the resources should be fairly self explanatory, as long as you have a basic understanding of how FSRM File Screens are used.

Installing the Resource

If you have installed WMF 5.0 you can just download this directly from the PowerShell Gallery by running this command:

Install-Module -Name cFSRMFileScreens

Otherwise you’ll need to download this from the Microsoft Script Center here and unzip it into your PowerShell modules path.

Using the Resource

If you need some additional guidance or other specific examples, please feel free to let me know and I’ll do my best to help you out.

Hopefully this resource finds some use out there, but either way it has been extremely helpful to me really imprint the underlying FSRM features and usage into my own mind.

Feedback

File Server Resource Manager (FSRM) Quotas DSC Resource

2015-10-23T00:00:00Z

Introduction

After implementing (but not yet completing) the my DFS Replication Groups resource last week, I had an epiphany about another resource that I had begun writing some time ago but had run into problems with. The epiphany allowed me to resolve the issues holding up completion of this resource as well as dig more deeply into the FSRM for my studies.

Resources

Initially I was going to create all of the FSRM Resources (File Groups, File Classifications etc) in a single module, but I quickly realized that this wasn’t ideal as the number of modules to support this was actually quite large. Therefore I’ve decided to break this down into more manageable chunks. This is the first chunk. It contains the following resources:

cFSRMFileQuota - configures FSRM Quotas.
cFSRMFileQuotaAction - configures FSRM Quota Actions for Quotas.
cFSRMFileQuotaTemplate - configures FSRM Quota Templates.
cFSRMFileQuotaTemplateAction - configures FSRM Quota Template Actions for Quota Templates.
cFSRMAutoQuota - configures FSRM Auto Quotas.

The purpose of the resources should be fairly self explanatory, as long as you have a basic understanding of how FSRM Quotas are used. If you aren’t familiar with FSRM Quotas, this is a good place to start - although why you’d be reading this if you’re not familiar with FSRM Quotas already is beyond me.

There are some other Quota management DSC Resources available online and they look very easy to use, but they don’t provide the complete set of functionality that these resources do because I tried to ensure that every Quota is available and as complete as possible. Which resources to use depends on your needs.

Installing the Resource

If you have installed WMF 5.0 you can just download this directly from the PowerShell Gallery by running this command:

Install-Module -Name cFSRMQuotas

Otherwise you’ll need to download this from the Microsoft Script Center here and unzip it into your PowerShell modules path.

Using the Resource

Rather than go into detail on using this resource in this post, I thought I’d try and keep it short and just provide a link to the documentation. This covers the parameters available in the resources as well as some usage examples.

If you need some additional guidance or other specific examples, please feel free to let me know and I’ll do my best to help you out.

Summary

Well, there is not much more to say about this. Hopefully someone finds it useful. I intend to add complete the other chunks of the FSRM Resources over the coming weeks when I have time.

Feedback

DSC Resource Kit Updates are Available

2015-10-23T00:00:00Z

The Microsoft DSC Resource Kit has been updated with a bunch of new stuff. You can check this out here.

If you’re using DSC, you’re no doubt familiar with the Microsoft DSC Resource Kit as it provides some of the most useful resources available outside of the base DSC Resources. You should really go and check it out to see what sort of thing you can be configuring with DSC straight out of the box.

I’m especially proud of this release as it contains some stuff I’ve be contributing to outside of my own resources. I’ve been helping out with adding some code to the xNetworking resource.

IPv6, DHCP and Get-NetIPInterface - DHCP State can be WRONG!

2015-10-17T00:00:00Z

Recently I’ve been attempting to help out with the awesome Microsoft Community DSC Resources by throwing in a bit of code here and there - especially into the xNetworking resource. I started contributing to them because I had a need for some specific features in these resources for some other projects I was working on.

Anyway, long story short I found myself investigating an odd little bug with the xIPAddress resource (it configures an IPv4 or IPv6 address on a Network adapter). The problem was that even though I had a network adapter with a statically assigned IPv6 address, the Get-NetIPInterface cmdlet always seemed to say that DHCP was enabled:

The IPv6 address is clearly statically assigned but it says DHCP is enabled!

I am not sure if this is a bug in Get-NetIPInterface that causes the DHCP property to be misreported for IPv6 interfaces or if using this property to determine DHCP status on an IPv6 address is not recommended.

Either way, I’m a bit stumped. I need an alternate and reliable way that can be used to detect the DHCP state of an IPv6 interface. I’ve looked at using the PrefixOrigin and/or SuffixOrigin properties of objects returned by Get-NetIPAddress but this feels a little bit untrustworthy to me.

Well, if anyone reads this and has any ideas I’d be very grateful to hear about it!

Edit: After a bit more investigation on this, it seems you can quite happily set the DHCP property on an IPv6 Interface using the Set-NetIPInterface cmdlet to whatever you like, regardless of whether or not a static IP address is assigned. So it seems that the DHCP property returned by the Get-NetIPInterface cmdlet for IPv6 addresses is meaningless. But I’d still love to know for sure.

Distributed File System DSC Resource Update

2015-10-11T00:00:00Z

After releasing the DFS DSC Resource Module yesterday, I had an idea of how to simplify it if you’re deploying a DFS folder that contains the same path content path for all members. I added a ContentPaths parameter (an array of strings) to the cDFSRepGroup resource so that if the folder exists in the same location on every member, you won’t need to use the cDFSRepGroupMembership resource to individually set the Content Path for each member.

For example:

configuration Sample_cDFSRepGroup_Simple {
    Import-DscResource -Module cDFS

    Node $NodeName {
        [PSCredential]$Credential = New-Object System.Management.Automation.PSCredential (
            "CONTOSO.COM\Administrator",
            (ConvertTo-SecureString "MyP@ssw0rd!1" -AsPlainText -Force)
        )

        # Install the Prerequisite features first
        # Requires Windows Server 2012 R2 Full install
        WindowsFeature RSATDFSMgmtConInstall {
            Ensure = "Present"
            Name   = "RSAT-DFS-Mgmt-Con"
        }

        # Configure the Replication Group
        cDFSRepGroup RGPublic {
            GroupName            = 'Public'
            Description          = 'Public files for use by all departments'
            Ensure               = 'Present'
            Members              = 'FileServer1','FileServer2'
            Folders              = 'Software','Misc'
            Topology             = 'Fullmesh'
            ContentPaths         = 'd:\public\software','d:\public\misc'
            PSDSCRunAsCredential = $Credential
            DependsOn            = "[WindowsFeature]RSATDFSMgmtConInstall"
        }
        # End of RGPublic Resource
    }
    # End of Node
}
# End of Configuration

The above example creates a DFS Replication Group called Public containing two folders, Software and Misc. The DFS Replication Group replicates to two members, FileServer1 and FileServer2. It is maintaining a Fullmesh connection topology.

The thing to note is that the ContentPaths array should have the elements in a matching order to the Folders parameter. So this:

Folders = 'Misc','Software'
ContentPaths = 'd:\public\software','d:\public\misc'

Would result in the Misc folder being set with the Content Path ‘d:\public\software’ and the Public folder being set with the Content Path ‘d:\public\misc’ - which is probably not ideal.

The Primary Member

Every Resource Group Folder needs a Primary Member set for initial replication to take place. If you use this automatic assigning of content paths the Primary Member will automatically be set to the computer listed first in the Members parameter. If you want to change this you’ll need to use the manual cDFSRepGroupMembership resource instead.

Partially Setting Content Paths

It is actually possible to only automatically configure some of the content paths in a DFS Replication Group by leaving the appropriate ContentPaths array entry blank. This would allow you to automatically configure some folders but leave other folders to be manually configured.

For example:

cDFSRepGroup RGPublic {
    GroupName            = 'Public'
    Description          = 'Public files for use by all departments'
    Ensure               = 'Present'
    Members              = 'FileServer1','FileServer2'
    Folders              = 'Software','Misc','Video'
    Topology             = 'Fullmesh'
    ContentPaths         = 'd:\public\software','','e:\video'
    PSDSCRunAsCredential = $Credential
    DependsOn            = "[WindowsFeature]RSATDFSMgmtConInstall"
}
# End of RGPublic Resource

This would create a Replication Group called Public, with three folders Software, Misc and Video. The Software and Video folders will be automatically configured with Content Paths but the Misc folder will be left unconfigured so that it can be configured manually.

Optional Use

Using the ContentPaths or Topology parameters is optional. You can still define the folder Content Paths manually using the cDFSRepGroupMembership resource and/or configure the connection topology manually using the cDFSRepGroupConnection resource if you want to.

Important: It is not recommended that you define a ContentPath for a folder in the cDFSRepGroup ContentPaths parameter if you are also setting it in a cDFSRepGroupMembership resource. The same applies to defining and automatic Topology and using the cDFSRepGroupConnection resource.

And again, in case you missed it, the post covering the original resource is here.

Windows Distributed File System DSC Resource

2015-10-10T00:00:00Z

Introduction

While studying for my MS 70.411 exam, I found that one way of getting a good understanding of a feature is to perform as many feature tasks as possible using PowerShell. One especially useful way of doing this for me was to implement a DSC resource for the feature. So, this week the feature was Distributed File System Replication Groups. I’ll refer to Distributed File Systems as DFS in future to save typing.

Note: I am going to implement DFS Namespaces as well, but that will be left to next week.

Update: After releasing this version I had an idea for some improvements to simplify this resource. See the details here.

Node vs. Active Directory

The first thing to note with implementing a DSC resource for Windows Distributed File System is that the resource is actually setting the Desired State of Active Directory elements rather than that of a Node. What that means is that when you use the PowerShell (or Management Console) to manage Windows DFS Replication Groups or DFS Namespaces you’re actually configuring items in the Active Directory database - you’re not changing anything on the actual node/computer you’re running the commands on.

This means that a DSC Resource for configuring Windows DFS could be run on any computer within the AD Domain. This is actually very handy as it turns out. At first though, I wasn’t sure DSC should be used for configuring elements that aren’t actually on a Node/Computer, but I couldn’t see why not, and then I remembered that there are other resources that do this (xActiveDirectory for example).

Server Core Not Supported

The first problem I ran into when implementing this DSC Resource is that you can’t install the DFS Replication (DFSR) PowerShell module onto a Windows Server Core installation. This is because the PowerShell DFSR module is only installed with the DFS Management Tools feature, which requires a Full Server install (or at least the Graphical Management Tools and Infrastructure feature).

This feature is required to enable the DFSR PowerShell Module.

This isn’t the end of the world, but it is annoying because all my file servers are Server Core. Therefore I’d need to run this resource on a node with a Full Server install that is also part of the AD Domain. So it is great that this resource can be run on any Full Server install (or even a Desktop with RSAT).

Setting AD Credentials

Because this resource calls PowerShell CmdLets that interact with the AD Database, AD credentials need to be supplied that can have the appropriate permissions. This means that the PSDSCRunAsCredential property must be set for each resource entry, which in turn means this Resource can only be used on nodes with Windows Management Framework 5.0 (WMF 5.0) or greater installed. If you’re not familiar with this property, see this link.

Installing the Resource

Because this resource requires WMF 5.0 you can just download this directly from the PowerShell Gallery by running this command:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\10\2015-10-10-windows-distributed-file-system-dsc-resource.md
Install-Module -Name cDFS

Using the Resource

The following example creates a DFS Replication Group called Public containing two members, FileServer1 and FileServer2. The Replication Group contains a single folder called Software. A description will be set on the Software folder and it will be set to exclude the directory Temp from replication.

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\10\2015-10-10-windows-distributed-file-system-dsc-resource.md
configuration Sample_cDFSRepGroup {
    Import-DscResource -Module cDFS

    Node $NodeName {
        [PSCredential]$Credential = New-Object System.Management.Automation.PSCredential (
            "CONTOSO.COM\Administrator",
            (ConvertTo-SecureString "MyP@ssw0rd!1" -AsPlainText -Force)
        )

        # Install the Prerequisite features first
        # Requires Windows Server 2012 R2 Full install
        WindowsFeature RSATDFSMgmtConInstall {
            Ensure = "Present"
            Name   = "RSAT-DFS-Mgmt-Con"
        }

        # Configure the Replication Group
        cDFSRepGroup RGPublic {
            GroupName            = 'Public'
            Description          = 'Public files for use by all departments'
            Ensure               = 'Present'
            Members              = 'FileServer1','FileServer2'
            Folders              = 'Software'
            PSDSCRunAsCredential = $Credential
            DependsOn            = "[WindowsFeature]RSATDFSMgmtConInstall"
        }

        cDFSRepGroupConnection RGPublicC1 {
            GroupName                = 'Public'
            Ensure                   = 'Present'
            SourceComputerName       = 'FileServer1'
            DestinationComputerName  = 'FileServer2'
            PSDSCRunAsCredential     = $Credential
        }

        cDFSRepGroupConnection RGPublicC2 {
            GroupName                = 'Public'
            Ensure                   = 'Present'
            SourceComputerName       = 'FileServer2'
            DestinationComputerName  = 'FileServer1'
            PSDSCRunAsCredential     = $Credential
        }

        cDFSRepGroupFolder RGSoftwareFolder {
            GroupName                = 'Public'
            FolderName               = 'Software'
            Description              = 'DFS Share for storing software installers'
            DirectoryNameToExclude   = 'Temp'
            PSDSCRunAsCredential     = $Credential
            DependsOn                = '[cDFSRepGroup]RGPublic'
        }

        cDFSRepGroupMembership RGPublicSoftwareFS1 {
            GroupName                = 'Public'
            FolderName               = 'Software'
            ComputerName             = 'FileServer1'
            ContentPath              = 'd:\Public\Software'
            PrimaryMember            = $true
            PSDSCRunAsCredential     = $Credential
            DependsOn                = '[cDFSRepGroupFolder]RGSoftwareFolder'
        }

        cDFSRepGroupMembership RGPublicSoftwareFS2 {
            GroupName                = 'Public'
            FolderName               = 'Software'
            ComputerName             = 'FileServer2'
            ContentPath              = 'e:\Data\Public\Software'
            PSDSCRunAsCredential     = $Credential
            DependsOn                = '[cDFSRepGroupFolder]RGPublicSoftwareFS1'
        }
    }
}

Example Breakdown

The resource usage hopefully is fairly straight forward and the Module itself contains documentation in the Readme.md (you can also see it here). But I’ll provide a quick breakdown of the resources just in case.

WindowsFeature RSATDFSMgmtConInstall

Install the Windows Feature that is required to use this DSC Resource. It installs the Windows DFSR/DFSN PowerShell Modules.

cDFSRepGroup

This resource creates, configures or removes a DFS Replication Group. You should specify both the Members and the Folders that are in this Replication Group. Both of these properties take an array of strings so you can specify more than one member (not much of a Distributed File System without that right?) and more than one folder. You of course also need to specify a DFS Replication Group Name.

This resource also contains an optional Topology parameter that defaults to Manual. If this parameter is set to Fullmesh then a Full Mesh connection topology will be configured automatically for this Replication Group, based on the members specified in the resource.

cDFSRepGroupConnection

This is an optional resource that allows the Replication Group Connections to be defined manually. I used the above example, so that it was obvious how they should be used. It allows a Replication Group Connection to be defined for a Replication Group between two members. A description can also be set on each connection. The connections can be disabled and also have RDC (Remote Differential Compression) disabled.

Note: this resource should only be used if the Topology parameter of the cDFSRepGroup resource is set to Manual (which is the default). If you set the Topology parameter to Fullmesh, a set of Replication Group Connections will automatically be created in a Full Mesh structure. The Hub and Spoke structure is not currently supported but may be in the future.

cDFSRepGroupFolder

This is an optional resource that can be used to configure specific properties of any of the folders in a DFS Replication Group. It is not used to create a folder within the Replication Group, that is the job of the cDFSRepGroup resource. This job of this resource is to configure the following properties of a Replication Group Folder:

Description
FilenameToExclude - if this is not specified the default value that DFS assigns is automatically used.
DirectoryNameToExclude - if this is not specified the default value that DFS assigns is automatically used.

cDFSRepGroupMembership

This resource is used to configure the actual content folders on each member of the Replication Group Folder. An instance of this resource should be used for each combination of member and folder in a Replication Group to set the Content Folder. It can also be used to set the following optional properties:

StagingPath - this can be used to override the default staging path. Usually this should be left to the default.
ReadOnly - this property can be used to make this content folder read only.
PrimaryMember - this property allows a Primary Member of the replication group to be set. At least one member of each Replication Group folder must set as the Primary Member otherwise initial replication will never take place.

Common Parameters

There are a couple of parameters that are common to each resource:

GroupName - this is the name of the Replication Group.
Domain - this is the name of the AD Domain this Replication Group is part of. If not specified then the AD Domain that the computer that is running the config is part of is used. Usually it should not be specified.

Summary

Well, there is not much more to say about this. Hopefully someone finds it useful. I intend to add DFS Namespace support over the next week or so, so if you’re needing that, keep an eye out.

Feedback

Programming Sucks

2015-10-07T00:00:00Z

Even if you’re not a programmer, this is one of the funniest things I’ve read in a long time (and, terrifyingly, not completely untrue):

Programming Sucks

Go and read it now.

Creating WS-Man Listeners with DSC

2015-10-05T00:00:00Z

Introduction

After my last post showing how to create an SSL/HTTPS listener using GPO, I thought this might be a good fit for a Desired State Configuration Resource. So after a rainy Saturday morning coding I had it working nicely.

You might ask “what is the point of adding HTTPS/SSL WS-Man Listeners when HTTP WS-Man Listeners are usually enabled by default”? Well, first off, it ensures you’re going to be connecting to the server you actually think you’re connecting to. This is pretty important and helps protect against DNS poisoning and man-in-the-middle attacks. It also means you don’t have to set the WS-Man client trusted hosts setting on your client machines to bypass host name checking for your servers:

No more of this!

HTTPS/SSL and Certificates

This DSC Resource essentially allows you to create, configure and remove HTTP and HTTPS/SSL WS-Man Listeners. That is pretty much it.

However, the most common use is going to be creating an HTTPS/SSL listener by automatically detecting the appropriate certificate to use. It uses the exact same method of doing this as described in this post, so I won’t go over it here. But essentially, all you need to do is provide the full name of the Issuing CA that will have issued the certificate to the computer. The certificate would normally have been created and assigned to each server using Certificate Autoenrollment using an Active Directory Certificate Services PKI and GPO, but this is not required - you could use any certificate enrollment method.

Installing the Resource

The first thing that needs to be done is installing the cWSMan Module. If you’re using WMF 5.0 you can get this directly from the PowerShell Gallery by running this command:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\10\2015-10-05-creating-ws-man-listeners-with-dsc.md
Install-Module -Name cWSMan -MinimumVersion 1.0.0.0

If you’re using WMF 4.0 then you’ll need to get this from the Microsoft Script Center. But of course, you’re using WMF 5.0 right?

Once it is installed you can integrate it into your DSC Scripts.

Using the Resource

The most likely thing you’re going to want to do is install an HTTPS/SSL Listener. To do that all you need to do is something like this:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\10\2015-10-05-creating-ws-man-listeners-with-dsc.md
configuration Sample_cWSManListener_HTTPS {
    Import-DscResource -Module cWSMan

    Node Server01 {
        cWSManListener HTTPS {
            Transport = 'HTTPS'
            Ensure    = 'Present'
            Issuer    = 'CN=CONTOSO.COM Issuing CA, DC=CONTOSO, DC=COM'
        }
        # End of cWSManListener Resource
    }
    # End of Node
}
# End of Configuration

This would install an HTTPS/SSL Listener onto the default port of 5986 using a certificate that was issued by CN=CONTOSO.COM Issuing CA, DC=CONTOSO, DC=COM. There really is nothing to it - it is actually more fiddly getting your PKI set up than doing this part.

You can also configure the port and address to bind the HTTPS/SSL Listener to by passing the port and address parameters as well:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\10\2015-10-05-creating-ws-man-listeners-with-dsc.md
configuration Sample_cWSManListener_HTTPS {
    Import-DscResource -Module cWSMan

    Node Server01 {
        cWSManListener HTTPS {
            Transport = 'HTTPS'
            Ensure    = 'Present'
            Issuer    = 'CN=CONTOSO.COM Issuing CA, DC=CONTOSO, DC=COM'
            Port      = 7000
            Address   = '192.168.1.55'
        }
        # End of cWSManListener Resource
    }
    # End of Node
}
# End of Configuration

If you don’t provide the port and address parameters they default to 5986 (or 5985 for HTTP listeners) and ‘*’ respectively.

You can also use this resource to remove an HTTP or HTTPS listener. For example you might want to remove the default HTTP listener so that it can’t be used once your HTTPS listener has been created. To do that:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\10\2015-10-05-creating-ws-man-listeners-with-dsc.md
configuration Remove_cWSManListener_HTTP {
    Import-DscResource -Module cWSMan

    Node Server01 {
        cWSManListener HTTP {
            Transport = 'HTTP'
            Ensure    = 'Absent'
        }
        # End of cWSManListener Resource
    }
    # End of Node
}
# End of Configuration

Feedback

NAP, DHCP and Windows 10 - Nope!

2015-09-29T00:00:00Z

I just spent a good hour trying to figure out why my Windows 10 clients were not getting assigned an IP Address from my DHCP servers once I enabled NAP integration on the scope. The reason, of course, is obvious: NAP was deprecated in Windows Server 2012 R2.

The NAP client is not available on Windows 10 computers. You can’t even see the Network Access Policy node when you edit a GPO using Windows 10 RSAT:

So if you’re wanting to configure your Windows 10 computers with DHCP and you’re using NAP, you’ll need to disable it or create a special scope without NAP enabled with a DHCP Scope Policy for your Windows 10 clients. As this technology has been deprecated, you’re probably better off removing NAP entirely. Pity I’m having to spend time studying it for my 70.411 exam.

Install an SSL WS-Management Listener with GPO

2015-09-27T00:00:00Z

Introduction

One of the things I like to do whenever I install a new server is to enable an HTTPS/SSL WS-Management Listener on it so that I can disable the more insecure HTTP WS-Management listener. For more information on WS-Management Listeners see this MSDN article.

There are many benefits to using a secure HTTPS/SSL WS-Management Listener:

Security - the communication channel between client and server is encrypted using SSL.
Authentication - the server is authenticated to the client so you can trust you’re talking to the server you think you’re talking to.

The downside to this is that you need a valid and trusted server authentication certificate on the server to enable this - but if you have a PKI then this is no big deal as you’ll probably have certificate autoenrollment enabled. If you don’t have a PKI, then you should look into installing one.

Installing these listeners manually is fairly straight forward and requires only a single command:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\09\2015-09-27-install-an-ssl-ws-management-listener-with-gpo.md
New-WSManInstance -ResourceURI winrm/config/Listener `
  -SelectorSet @{ Address='*'; Transport="HTTPS" } `
  -ValueSet @{ Hostname="SERVER01.CONTOSO.COM"; CertificateThumbprint="09 49 93 24 53 81 32 16 b7 44 8b 47 ca af 56 3a ef 9f 10 2d" }

All you need to do is enter the appropriate hostname and certificate thumbprint for a server authentication certificate that exists on the server. But who wants to do this manually, right?

Installing with GPO

The slightly tricky part of installing this automatically onto your servers with a GPO is detecting which certificate to use. The certificate must:

exist in the local computer personal certificate store.
have an Extended Key Usage of Server Authentication.
be issued by a CA trusted by any client connecting to the server.
The Subject must contain a Common Name that contains either the FQDN computer name or the flat computer name (e.g. CN=SERVER1.CONTOSO.COM or CN=SERVER1).

It is easy to ensure a certificate meets these criteria by using a GPO enabling certificate autoenrollment for computer certificates and that the Computer autoenrollment certificate template will create certificates meeting these requirements. See this page for some basic information on certificate autoenrollment. There are some much more detailed instructions on this around the net if you’re happy to search.

Once you’ve ensured all your computers have been issued such a certificate you would normally need to lookup the certificate on each computer and get the certificate thumbprint and execute the command I showed earlier. This would be a complete pain on any more than 10 computers, and probably pure insanity at a lot of facilities.

So, I put together the following PowerShell commands that could be used to automatically pull the certificate thumbprint for an appropriate certificate:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\09\2015-09-27-install-an-ssl-ws-management-listener-with-gpo.md
[String] $Issuer = 'CN=CONTOSO.COM Issuing CA, DC=CONTOSO, DC=COM'
[String] $HostName = [System.Net.Dns]::GetHostByName($ENV:computerName).Hostname
[String] $Thumbprint = (
    Get-ChildItem -Path Cert:\localmachine\my |
        Where-Object {
            ($_.Extensions.EnhancedKeyUsages.FriendlyName -contains 'Server Authentication') -and
            ($_.IssuerName.Name -eq $Issuer) -and
            ($HostName -in $_.DNSNameList.Unicode) -and
            ($_.Subject -eq "CN=$HostName")
        } |
        Select-Object -First 1
).Thumbprint

All you’d need to set was the Issuer to whatever the Distinguished Name of your issuing CA is - which should be the same for all computers. This simplifies things a lot because the same code could be run on any computer and should always return the correct thumbprint.

The next step was to put it into a script where you could just pass the Distinguished Name of the issuing CA as a parameter. I did this and also added some other optional parameters and uploaded the result to Microsoft Script Center. So you can download this script and put it into a GPO Startup Script:

Installing an HTTPS WS-Management Listener with GPO

The script is actually a little bit smarter than the above command. If a certificate with a subject can’t be found that matches the FQDN of the computer it will automatically look for one that just uses the flat computer name. You can control this behavior by setting the DNSNameType parameter.

There are some other optional parameters that control other the behavior of the script as well:

DNSNameType

The allowed DNS Name types that will be used to find a matching certificate. If set to FQDN then the script will only try to find a certificate with a subject matching the FQDN of the computer. If set to ComputerName it will only match on the computer name of the computer. By default this is set to Both which will try to match on FQDN first and then ComputerName if it can’t find one matching the FQDN.

MatchAlternate

The certificate found must also have an alternate subject name containing the DNS name found in the subject as well. This places additional restrictions on the certificate that is used, but is not usually required. This defaults to False.

Port

This parameter lets you specify an alternate port to install the HTTPS/SSL listener on. If you don’t specify it, it will use the default port of 5986.

Don’t Forget your Firewall

It is important to remember that by default this listener is installed onto port 5986, which is not usually open inbound. So you’ll want to add a firewall rule to ensure this port can be reached - another job for GPO. You could even add this setting into the GPO that installs the HTTPS/SSL listener.

Installing with DSC

In theory it should be possible to adapt this code to run in a DSC Script Resource. I haven’t tried this yet, but you can be assured that I will fairly soon. If there was some interest in this I could convert it into a proper DSC Resource (unless there was one already - I haven’t checked). If you are interested, let me know.

Links

If you want to make a copy of the repository, you’ll find it here:

https://github.com/PlagueHO/WSManGPOTools

Right, that is me out for another Sunday. Thanks for reading.

New PowerShell Icon in Windows 10 Build 10547

2015-09-20T00:00:00Z

I know this is a bit of a non event, but anyone notice the new PowerShell icon in Windows 10 insider build 10547?

I wonder if there are any other PowerShell changes in this build?

Installing a Two-Tier PKI using nothing but Desired State Configuration – Part 2

2015-09-13T00:00:00Z

Continuing on from yesterday, the goal of this series is show how it is possible to install a two-tier Active Directory Certificate Services environment using only Desired State Configuration. In Part 1, I covered the basic DSC setup and requirements, the AllNodes hash table and the first part of the Root CA configuration script.

Other Parts in this Series

Installing a Two-Tier PKI using nothing but Desired State Configuration - Part 1

Lets get going then!

Step 2: Installing the Subordinate CA

In this configuration we’ll need both Local Credentials for installing the Web Enrollment feature and Domain Credentials for joining the Sub CA to the domain and for registering the CA in AD:

# Assemble the Local Admin and Domain Admin Credentials
Node $AllNodes.NodeName {
    if ($Node.LocalAdminPassword) {
        [PSCredential]$LocalAdminCredential = New-Object System.Management.Automation.PSCredential (
            "Administrator",
            (ConvertTo-SecureString $Node.LocalAdminPassword -AsPlainText -Force)
        )
    }
    if ($Node.DomainAdminPassword) {
        [PSCredential]$DomainAdminCredential = New-Object System.Management.Automation.PSCredential (
            "$($Node.DomainName)\Administrator",
            (ConvertTo-SecureString $Node.DomainAdminPassword -AsPlainText -Force)
        )
    }
}

Just like the Root CA the ADCS Certificate Authority and the ADCS Web Enrollment features need to be installed. But I’m also going to install the Online Responder service as well - you of course don’t need to. I really should configure the CRLPublicationURLs node property as well to make use of this Online Responder, but I’m sure you can figure that part out.

# Install the RSAT PowerShell Module which is required by the xWaitForResource
WindowsFeature RSATADPowerShell {
    Ensure = "Present"
    Name   = "RSAT-AD-PowerShell"
}

# Install the CA Service
WindowsFeature ADCSCA {
    Name      = 'ADCS-Cert-Authority'
    Ensure    = 'Present'
    DependsOn = "[WindowsFeature]RSATADPowerShell"
}

# Install the Web Enrollment Service
WindowsFeature WebEnrollmentCA {
    Name      = 'ADCS-Web-Enrollment'
    Ensure    = 'Present'
    DependsOn = "[WindowsFeature]ADCSCA"
}

# Install the Online Responder Service
WindowsFeature OnlineResponderCA {
    Name      = 'ADCS-Online-Cert'
    Ensure    = 'Present'
    DependsOn = "[WindowsFeature]WebEnrollmentCA"
}

You might have noticed that we’re also installing the RSAT-AD-PowerShell. This is required by the xWaitForADDomain DSC resource. If you don’t install this feature the domain will never be detected and the DSC Script will progress no further (I found this out the hard way).

On the agenda next, this machine needs to be joined to the domain. It is important to check the domain is up before trying to join it. In my case I was also creating the DC’s (by DSC of course) at the same time as the CA’s so sometimes there was a long wait for the Domain to come up (which is why the large retry count):

# Wait for the Domain to be available so we can join it.
xWaitForADDomain DscDomainWait {
    DomainName          = $Node.DomainName
    DomainUserCredential = $DomainAdminCredential
    RetryCount          = 100
    RetryIntervalSec    = 10
    DependsOn           = "[WindowsFeature]OnlineResponderCA"
}

# Join this Server to the Domain so that it can be an Enterprise CA.
xComputer JoinDomain {
    Name     = $Node.NodeName
    DomainName = $Node.DomainName
    Credential = $DomainAdminCredential
    DependsOn = "[xWaitForADDomain]DscDomainWait"
}

The next step is to create a CAPolicy.inf file, but this file is slightly different from the one created on the Root CA. The process is the same though:

# Create the CAPolicy.inf file that sets basic parameters for certificate issuance for this CA.
File CAPolicy {
    Ensure = 'Present'
    DestinationPath = 'C:\\Windows\\CAPolicy.inf'
    Contents = "\[Version\]\`r\`n Signature= \`"$Windows NT$\`"\`r\`n\[Certsrv\_Server\]\`r\`n RenewalKeyLength=2048\`r\`n RenewalValidityPeriod=Years\`r\`n RenewalValidityPeriodUnits=10\`r\`n LoadDefaultTemplates=1\`r\`n AlternateSignatureAlgorithm=1\`r\`n"
    Type = 'File'
    DependsOn = '\[xComputer\]JoinDomain' }

Easy enough so far. What I did next was create a CertEnroll folder (c:\windows\System32\CertSrv\CertEnroll) where the Root CA certificate needed to be put. The Web Enrollment Service would have created this too but I can’t configure this service until later. So I’m going to create it manually:

# Make a CertEnroll folder to put the Root CA certificate into.
# The CA Web Enrollment server would also create this but we need it now.
File CertEnrollFolder {
    Ensure = 'Present'
    DestinationPath = 'C:\\Windows\\System32\\CertSrv\\CertEnroll'
    Type = 'Directory'
    DependsOn = '\[File\]CAPolicy'
}

Next up I wanted to download the Root CA Cert to this Sub CA. Strictly this isn’t required till later but I was basically emulating the steps in this document.

The important thing to remember here though is that we need to ensure the Root CA DSC has reached the point where the Root CA certificate and Certificate Revocation List (CRL) is produced and available to us. So this is where we use the new PowerShell DSC 5.0 WaitFor resource:

# Wait for the RootCA Web Enrollment to complete so we can grab the Root CA certificate # file.
WaitForAny RootCA {
    ResourceName = '\[xADCSWebEnrollment\]ConfigWebEnrollment'
    NodeName = $Node.RootCAName
    RetryIntervalSec = 30
    RetryCount = 30
    DependsOn = "\[File\]CertEnrollFolder"
}

# Download the Root CA certificate file.
xRemoteFile DownloadRootCACRTFile {
    DestinationPath = "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Node.RootCAName)\_$($Node.RootCACommonName).crt"
    Uri = "http://$($Node.RootCAName)/CertEnroll/$($Node.RootCAName)\_$($Node.RootCACommonName).crt"
    DependsOn = '\[WaitForAny\]RootCA'
}

# Download the Root CA certificate revocation list.
xRemoteFile DownloadRootCACRLFile {
    DestinationPath = "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Node.RootCACommonName).crl"
    Uri = "http://$($Node.RootCAName)/CertEnroll/$($Node.RootCACommonName).crl"
    DependsOn = '\[xRemoteFile\]DownloadRootCACRTFile'
}

Note: using HTTP to copy files between the Root CA and the Sub CA’s is not strictly recommended by Microsoft when installing a two-tier PKI because that means the Root CA system has to be connected to the network. Because the Root CA and Sub CA DSC scripts need the machines to directly interact there isn’t any way around this that I can see. But if you were using this in a production environment you could put the Root CA machine onto an isolated virtual network consisting of the Root CA and Sub CA machines only. It is not a perfect solution but it should be reasonable for most situations. The Root CA can still be taken off line and removed after the Sub CA’s have been created.

Following this the Root CA Certificate and CRL can be imported into the local machine root certificate store and also the Active Directory domain. This is done in a single script resource:

# Install the Root CA Certificate to the LocalMachine Root Store and DS
Script InstallRootCACert {
    PSDSCRunAsCredential = $DomainAdminCredential SetScript = {
        Write-Verbose "Registering the Root CA Certificate C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Using:Node.RootCAName)\_$($Using:Node.RootCACommonName).crt in DS..."
        "$($ENV:SystemRoot)\\system32\\certutil.exe" -f -dspublish "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Using:Node.RootCAName)\_$($Using:Node.RootCACommonName).crt" RootCA
        Write-Verbose "Registering the Root CA CRL C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Node.RootCACommonName).crl in DS..." "$($ENV:SystemRoot)\\system32\\certutil.exe" -f -dspublish "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Node.RootCACommonName).crl" "$($Using:Node.RootCAName)"
        Write-Verbose "Installing the Root CA Certificate C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Using:Node.RootCAName)\_$($Using:Node.RootCACommonName).crt..." "$($ENV:SystemRoot)\\system32\\certutil.exe" -addstore -f root "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Using:Node.RootCAName)\_$($Using:Node.RootCACommonName).crt"
        Write-Verbose "Installing the Root CA CRL C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Node.RootCACommonName).crl..." "$($ENV:SystemRoot)\\system32\\certutil.exe" -addstore -f root "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Node.RootCACommonName).crl"
    }
    GetScript = {
        Return @{
            Installed = ((Get-ChildItem -Path Cert:\\LocalMachine\\Root | Where-Object -FilterScript { ($\_.Subject -Like "CN=$($Using:Node.RootCACommonName),\*") -and ($\_.Issuer -Like "CN=$($Using:Node.RootCACommonName),\*") } ).Count -EQ 0)
        }
    }
    TestScript = {
        If ((Get-ChildItem -Path Cert:\\LocalMachine\\Root | Where-Object -FilterScript { ($\_.Subject -Like "CN=$($Using:Node.RootCACommonName),\*") -and ($\_.Issuer -Like "CN=$($Using:Node.RootCACommonName),\*") } ).Count -EQ 0) {
            Write-Verbose "Root CA Certificate Needs to be installed..."
            Return $False
        }
        Return $True
    }
    DependsOn = '\[xRemoteFile\]DownloadRootCACRTFile'
}

I’d actually prefer to break the above code into for separate resources and detect if each one has occurred (and I might do for a later version), but this configuration is extremely large as it is.

Notice here we also used another PowerShell DSC 5.0 feature, the PSDSCRunAsCredential parameter. This parameter is available in all DSC Resources and allows us to specify an alternate credential to run this DSC Resource as. By default a DSC Resource is run as NT AUTHORITY/SYSTEM, which is usually OK, but in this case some of the commands write certificates into DS and therefore need to be run under a Domain Admin account.

Onwards: It is now time to configure the AD CS Certificate Authority and Web Enrollment. Except this time the Certificate Authority configuration will produce a certificate request (REQ) that has to be issued by our Root CA. So what I did was ensure the REQ file is put into the CertEnroll folder - this should make it accessible by in the http:\\SA_SUBCA\CertEnroll\ web site.

# Configure the Sub CA which will create the Certificate REQ file that Root CA will use
# to issue a certificate for this Sub CA.
xADCSCertificationAuthority ConfigCA {
    Ensure                    = 'Present'
    Credential                = $DomainAdminCredential
    CAType                    = 'EnterpriseSubordinateCA'
    CACommonName              = $Node.CACommonName
    CADistinguishedNameSuffix = $Node.CADistinguishedNameSuffix
    OverwriteExistingCAinDS   = $True
    OutputCertRequestFile     = "c:\windows\system32\certsrv\certenroll\$($Node.NodeName).req"
    DependsOn                 = '[Script]InstallRootCACert'
}

# Configure the Web Enrollment Feature
xADCSWebEnrollment ConfigWebEnrollment {
    Ensure     = 'Present'
    Name       = 'ConfigWebEnrollment'
    Credential = $LocalAdminCredential
    DependsOn  = '[xADCSCertificationAuthority]ConfigCA'
}

Seems simple enough - except one small problem. By default IIS doesn’t include REQ files as supported mime types so the file can’t be downloaded. To get around this we need to add REQ as a supported mime type. Unfortunately there is no DSC resource to do this so it’s time to resort to the Script resource:

# Set the IIS Mime Type to allow the REQ request to be downloaded by the Root CA
Script SetREQMimeType {
    SetScript = {
        Add-WebConfigurationProperty -PSPath IIS:\ -Filter //staticContent -Name "." -Value @{fileExtension='.req';mimeType='application/pkcs10'}
    }
    GetScript = {
        return @{
            'MimeType' = ((Get-WebConfigurationProperty -Filter "//staticContent/mimeMap[@fileExtension='.req']" -PSPath IIS:\ -Name *).mimeType)
        }
    }
    TestScript = {
        if (-not (Get-WebConfigurationProperty -Filter "//staticContent/mimeMap[@fileExtension='.req']" -PSPath IIS:\ -Name *)) {
            # Mime type is not set
            return $False
        }
        # Mime Type is already set
        return $True
    }
    DependsOn = '[xADCSWebEnrollment]ConfigWebEnrollment'
}

Right, now an issuing certificate needs to be issued to this Sub CA by the Root CA using the REQ that has been created in the CertEnroll virtual folder on the Sub CA. To do this we need to go back to the Root CA DSC script and continue on with it.

Step 3: Issuing the Sub CA certificate on the Root CA

This is the second component of the Root CA DSC configuration. It is a bit more complicated than the first part because it may need to be run more than once - once for each Sub CA that is being created. Therefore the whole part is wrapped in foreach loop. This is also the purpose of the SubCAs array property of the AllNodes object. Each Sub CA that will be bought up should be in the list:

SubCAs=@('SA\_SUBCA1','SA\_SUBCA2','SA\_SUBCA3')

So now that we’ve got that covered we can start adding to the Root CA DSC Configuration. So here’s the start of that foreach loop I was talking about:

# Generate Issuing certificates for any SubCAs Foreach ($SubCA in $Node.SubCAs) {

The first thing to do is wait for the Sub CA to complete creation of the REQ file and download it. So once again we use the WaitForAny resource. Also note the use of the $SubCA variable that is defined by the foreach loop:

# Wait for SubCA to generate REQ
WaitForAny "WaitForSubCA_$SubCA" {
    ResourceName     = '[xADCSCertificationAuthority]ConfigCA'
    NodeName         = $SubCA
    RetryIntervalSec = 30
    RetryCount       = 30
    DependsOn        = '[Script]ADCSAdvConfig'
}

# Download the REQ from the SubCA 
xRemoteFile "DownloadSubCA\_$SubCA" {
    DestinationPath = "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$SubCA.req"
    Uri = "http://$SubCA/CertEnroll/$SubCA.req"
    DependsOn = "\[WaitForAny\]WaitForSubCA\_$SubCA"
}

To make things simple I just downloaded the REQ to the CertEnroll folder of this Root CA. Now, things got a little bit tough here. There is no DSC Resource or even PowerShell modules for issuing a certificate from the REQ. We have to fall back to using the DSC Script resource and the CertReq.exe and CertUtil.exe tools. This is a little bit fiddly and reminds me why I love PowerShell’s object based output. I won’t go into detail of what is going on here, but if you want me to expand on it let me know.

# Generate the Issuing Certificate from the REQ
Script "IssueCert\_$SubCA" {
    SetScript = {
        Write-Verbose "Submitting C:\\Windows\\System32\\CertSrv\\CertEnroll\\$Using:SubCA.req to $($Using:Node.CACommonName)"
        [String]$RequestResult = "$($ENV:SystemRoot)\\System32\\Certreq.exe" -Config ".\\$($Using:Node.CACommonName)" -Submit "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$Using:SubCA.req"
        $Matches = [Regex]::Match($RequestResult, 'RequestId:\\s(\[0-9\]\*)')
        If ($Matches.Groups.Count -lt 2) {
            Write-Verbose "Error getting Request ID from SubCA certificate submission."
            Throw "Error getting Request ID from SubCA certificate submission."
        }
        [int]$RequestId = $Matches.Groups\[1\].Value
        Write-Verbose "Issuing $RequestId in $($Using:Node.CACommonName)"
        [String]$SubmitResult = "$($ENV:SystemRoot)\\System32\\CertUtil.exe" -Resubmit $RequestId
        If ($SubmitResult -notlike 'Certificate issued.\*') {
            Write-Verbose "Unexpected result issuing SubCA request."
            Throw "Unexpected result issuing SubCA request."
        }
        Write-Verbose "Retrieving C:\\Windows\\System32\\CertSrv\\CertEnroll\\$Using:SubCA.req from $($Using:Node.CACommonName)"
        [String]$RetrieveResult = "$($ENV:SystemRoot)\\System32\\Certreq.exe" -Config ".\\$($Using:Node.CACommonName)" -Retrieve $RequestId "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$Using:SubCA.crt"
    }
    GetScript = {
        Return @{ 'Generated' = (Test-Path -Path "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$Using:SubCA.crt"); }
    }
    TestScript = {
        If (-not (Test-Path -Path "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$Using:SubCA.crt")) {
            # SubCA Cert is not yet created
            Return $False
        }
        # SubCA Cert has been created
        Return $True
    }
    DependsOn = "\[xRemoteFile\]DownloadSubCA\_$SubCA"
}

That is all we actually need to do in the loop on the Root CA. It is now up to each Sub CA to download the new Issuing Certificate and install it.

Step 4: Installing the Issuing Certificate on the Sub CA

Now that an Issuing Certificate is available to be downloaded from the Root CA for each Sub CA, the configuration script for each Sub CA can continue. But as always the script needs to use the WaitFor resource (really have to love this resource) to ensure that the certificate is available:

# Wait for the Root CA to have completed issuance of the certificate for this SubCA.
WaitForAny SubCACer {
    ResourceName = "\[Script\]IssueCert\_$($Node.NodeName)"
    NodeName = $Node.RootCAName
    RetryIntervalSec = 30
    RetryCount = 30
    DependsOn = "\[Script\]SetREQMimeType"
}

# Download the Certificate for this SubCA.
xRemoteFile DownloadSubCACERFile {
    DestinationPath = "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Node.NodeName).cer"
    Uri = "http://$($Node.RootCAName)/CertEnroll/$($Node.NodeName).cer"
    DependsOn = '\[WaitForAny\]SubCACer'
}

# Register the Sub CA Certificate with the Certification Authority
Script RegisterSubCA {
    PSDSCRunAsCredential = $DomainAdminCredential
    SetScript = {
        Write-Verbose "Registering the Sub CA Certificate with the Certification Authority C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Using:Node.NodeName)\_$($Using:Node.CACommonName).crt..."
        "$($ENV:SystemRoot)\\system32\\certutil.exe" -installCert "C:\\Windows\\System32\\CertSrv\\CertEnroll\\$($Using:Node.NodeName)\_$($Using:Node.CACommonName).crt"
    }
    GetScript = {
        Return @{ }
    }
    TestScript = {
        If (-not (Get-ChildItem 'HKLM:\\System\\CurrentControlSet\\Services\\CertSvc\\Configuration').GetValue('CACertHash')) {
            Write-Verbose "Sub CA Certificate needs to be registered with the Certification Authority..."
            Return $False
        }
        Return $True
    }
    DependsOn = '\[xRemoteFile\]DownloadSubCACERFile'
}

Note: It is always important to remember that when using the Script DSC Resource if you want to use any variables that are declared outside the resource you’ll need to prefix them with the Using: keyword. I have wasted many hours tracking down issues caused by missing this vital keyword!

Again, we’re running the above script resource using the PSDSCRunAsCredential parameter to run it using Domain Admin credentials so that the command can register the certificates into AD DS.

Once this is done the AIA and CDP extensions can be configured using the same method as we did for the Root CA. This will also start up the Certificate Service:

Script ADCSAdvConfig {
    SetScript = {
        if ($Using:Node.CADistinguishedNameSuffix) {
            "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSConfigDN "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)"
            "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSDomainDN "$($Using:Node.CADistinguishedNameSuffix)"
        }
        if ($Using:Node.CRLPublicationURLs) {
            "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CRLPublicationURLs $($Using:Node.CRLPublicationURLs)
        }
        if ($Using:Node.CACertPublicationURLs) {
            "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CACertPublicationURLs $($Using:Node.CACertPublicationURLs)
        }
        Restart-Service -Name CertSvc
        Add-Content -Path 'c:\windows\setup\scripts\certutil.log' -Value "Certificate Service Restarted ..."
    }
    GetScript = {
        return @{
            'DSConfigDN' = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN')
            'DSDomainDN' = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN')
            'CRLPublicationURLs' = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs')
            'CACertPublicationURLs' = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs')
        }
    }
    TestScript = {
        if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN') -ne "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
        if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN') -ne "$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
        if (($Using:Node.CRLPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs') -ne $Using:Node.CRLPublicationURLs)) { return $false }
        if (($Using:Node.CACertPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs') -ne $Using:Node.CACertPublicationURLs)) { return $false }
        return $true
    }
    DependsOn = '[Script]RegisterSubCA'
}

Step 5: Shut down the Root CA

Once all the Sub CAs have installed their certificates the Root CA can be shutdown. This is a nice way of identifying that everything has gone according to plan and all Sub CAs can now issue certificates. It also helps reduce the amount of time the Root CA is online. To do this, once again we use the WaitFor DSC Resource. If there is more than one Sub CA being installed then the Root CA script should wait for the last one to be complete.

WaitForAny "WaitForComplete_$SubCA" {
    ResourceName     = '[Script]InstallSubCACert'
    NodeName         = $SubCA
    RetryIntervalSec = 30
    RetryCount       = 30
    DependsOn        = "[Script]IssueCert_$SubCA"
}

Script ShutdownRootCA {
    SetScript = { Stop-Computer }
    GetScript = { return @{} }
    TestScript = {
        # SubCA Cert is not yet created
        return $false
    }
    DependsOn = "[WaitForAny]WaitForComplete_$SubCA"
}

At this point all the Sub CAs should be operational and the Root CA will have been shut down ready to be put away in a safe somewhere. There are still some minor tasks yet to complete such as configuring the Online Responder, generating and installing a Web Server certificate for the Web Enrollment Server etc. But seeing as this part is now getting extremely long I think I’ll leave them till Part 3 in the next few days. I hope this has been useful!

Additional Information

It is probably very useful to see the full complete DSC configuration files. These files change frequently as I optimize and test the process. As noted they are actually part of another project I’m working on - LabBuilder. They are currently available in my LabBuilder project repository on GitHub.

I will cover the LabBuilder project another day once I have completed testing and documentation on it.

Installing a Two-Tier PKI using nothing but Desired State Configuration - Part 1

2015-09-12T00:00:00Z

I am a firm believer in the concept of Infrastructure as Code. I do think technologies such as Chef and Windows PowerShell Desired State Configuration (DSC) will eventually replace ‘clickety-click’ administration in medium to large environments, and even some smaller sites. If you’re not familiar with these technologies or concepts I’d strongly recommend you take a look at the above links.

Note: This post is going to be quite long and it does assume you have a basic understanding of Desired State Configuration (DSC) and Windows Active Directory Certificate Services (AD CS). If you’re not comfortable creating basic DSC configuration files or have never installed AD CS then you might want to get familiar with doing this before jumping into this post.

Other Parts in this Series

Installing a Two-Tier PKI using nothing but Desired State Configuration - Part 2

The Goal

But it’s all well and good to say I think it is the future, but how about I put my money where my mouth is and actually use DSC to implement something much more complicated than a simple IIS web site. Sure, DSC can and should be used for straight forward infrastructure configuration, but what about something like a two-tier PKI with an offline Standalone Root CA as well as one or more Enterprise Subordinate CAs? As part of my LabBuilder project I was going to have to find out. Basically, I was going to try and implement this using nothing more than Desired State Configuration. If you’re interested in seeing how this is done, continue reading.

Why is this Complicated?

The difficulty with installing a two-tier PKI is that the Root CA and Issuing/Sub CA installation processes are interdependent. For example, to install the Sub CA an Issuing certificate must be issued by the Root CA, but this can only be done by the Sub CA issuing the request. The Request gets copied to the Root CA and issued and then the Issuing Certificate copied back to the Sub CA and installed. Therefore, there would need to be at least two DSC configurations, one on the Root CA and one on each Sub CA and they would be running at the same time, interacting with each other and waiting for various processed on each machine to complete before proceeding. This will become clearer later on in this post. To allow a DSC configuration to wait for a step to complete on another machine in another DSC configuration requires the WaitFor DSC resource that is only available in WMF 5.0.

Requirements

To be able to do this you’ll need several things:

A Hyper-V Host with the following Guest VMs:
1. A standalone clean Windows Server 2012 R2 Core server that will become the Standalone Root CA. In my system this computer is called SS_ROOTCA.
2. A Domain Controller that the Enterprise Issuing/Sub CA will become a part of. My domain was called LABBUILDER.COM.
3. One or more standalone clean Windows Server 2012 R2 Core servers that will become the Enterprise Issuing/Sub CAs. In my system I was only using a single Sub CA and it was named SA_SUBCA. But you could use multiple Sub CAs, the following scripts do support more than one Sub CA.
A computer to create the DSC configuration files on that has RSAT installed (the version appropriate to the operating system).
WMF 5.0 installed on all the above servers as well as the computer you’re creating the DSC configuration on.
The above servers need to be able to communicate with one another via networking (virtual or physical).

In a production environment it is recommended that the Root CA is kept offline and is never connected to a network. Therefore this DSC process wouldn’t actually work. It could be made to work by doing some tricky things like waiting for external storage to be connected and so on, but I’m not even going to go there for this post.

Server Core vs. Full

I used Windows Server Core installations for my PKI servers, but you could just as easily use Full installations if you wanted. However, I think given what I’m trying to achieve, using Server Core makes more sense - and besides, it’s simply the way to go when you’re talking about Infrastructure as Code. In fact, if AD CS was available on Server Nano I would be trying to use that instead. You can still use RSAT to work with these Server Core installations should you need to.

Resources

The DSC configuration files are going to require a few additional DSC resources. These DSC resources will need to be installed onto the PKI servers and the computer you’re using to compile the DSC Configurations into MOF files. The resources you’ll need are:

PSDesiredStateConfiguration - this is build in to the core DSC installation, so it doesn’t need to be downloaded.
xADCSDeployment - this is a community DSC resource required to perform post installation configuration of ADCS.
xPSDesiredStateConfiguration - we need this community DSC resource for the xRemoteFile DSC Resource.
xComputerManagement - this community DSC resource is required to join the Sub CA to the domain.

The easiest way to do this on PowerShell 5.0 is using the Find-Module and Install-Module cmdlets from the PowerShellGet module to download these from the PowerShell Gallery:

Find-Module xPSDesiredStateConfiguration, xADCSDeployment, xComputerManagement | Install-Module

AllNodes

To make things a little bit more generic I like to put all the variables that the DSC configuration files are going to require into an AllNodes hash table, with one for each server. Also, normally I’ll have a self-signed certificate generated on each server and copied down to the computer that is creating the configuration files so that the various credentials can be encrypted - see this post or this post for details on how this works. This is optional and you can use the PSDscAllowPlainTextPassword = $true option in the Node if you want to just send the credentials in the clear.

Important Note Regarding Credential Encryption in DSC

Don’t use the New-SelfSignedCertificate cmdlet to create a self-signed certificate to encrypt your credentials. It creates a certificate that will not work here (the private key is not accessible). Instead, use this script from the script center. You will waste a lot of time trying to figure out what is wrong.

AllNodes for Root CA

These are the Node parameters that contain the variables that you’ll want to configure for the Root CA. They are fairly self explanatory but they will be covered later on in the post.

AllNodes = @(
    @{
        NodeName = 'SS_ROOTCA'
        Thumbprint = 'CDD4EEAE6000AC7F40C3802C171E30148030C072'
        LocalAdminPassword = 'P@ssword!1'
        CACommonName = "LABBUILDER.COM Root CA"
        CADistinguishedNameSuffix = "DC=LABBUILDER,DC=COM"
        CRLPublicationURLs = "1:C:\Windows\system32\CertSrv\CertEnroll\%3%8%9.crl\n10:ldap:///CN=%7%8,CN=%2,CN=CDP,CN=Public Key Services,CN=Services,%6%10\n2:http://pki.labbuilder.com/CertEnroll/%3%8%9.crl"
        CACertPublicationURLs = "1:C:\Windows\system32\CertSrv\CertEnroll\%1_%3%4.crt\n2:ldap:///CN=%7,CN=AIA,CN=Public Key Services,CN=Services,%6%11\n2:http://pki.labbuilder.com/CertEnroll/%1_%3%4.crt"
        SubCAs = @('SA_SUBCA')
    }
)

AllNodes for Sub CA

And these are the parameters for each Subordinate CA. If you had more than one Sub CA then you could add additional nodes. The variables are fairly self explanatory but they will be covered later on in the post.

AllNodes = @(
    @{
        NodeName = 'SA_SUBCA'
        Thumbprint = '8F43288AD272F3103B6FB1428485EA3014C0BCFE'
        LocalAdminPassword = 'P@ssword!1'
        DomainName = "LABBUILDER.COM"
        DomainAdminPassword = "P@ssword!1"
        PSDscAllowDomainUser = $True
        CACommonName = "LABBUILDER.COM Issuing CA"
        CADistinguishedNameSuffix = "DC=LABBUILDER,DC=COM"
        CRLPublicationURLs = "65:C:\Windows\system32\CertSrv\CertEnroll\%3%8%9.crl\n79:ldap:///CN=%7%8,CN=%2,CN=CDP,CN=Public Key Services,CN=Services,%6%10\n6:http://pki.labbuilder.com/CertEnroll/%3%8%9.crl"
        CACertPublicationURLs = "1:C:\Windows\system32\CertSrv\CertEnroll\%1_%3%4.crt\n2:ldap:///CN=%7,CN=AIA,CN=Public Key Services,CN=Services,%6%11\n2:http://pki.labbuilder.com/CertEnroll/%1_%3%4.crt"
        RootCAName = "SS_ROOTCA"
        RootCACRTName = "SS_ROOTCA_LABBUILDER.COM Root CA.crt"
    }
)

Step 1: Installing the Root CA

First things first. We need to create a credential object that will be used to perform various steps in the process. This is a local credential as this is a standalone server.

Node $AllNodes.NodeName {
    # Assemble the Local Admin Credentials
    if ($Node.LocalAdminPassword) {
        [PSCredential]$LocalAdminCredential = New-Object System.Management.Automation.PSCredential (
            "Administrator",
            (ConvertTo-SecureString $Node.LocalAdminPassword -AsPlainText -Force)
        )
    }
}

Next up we’ll install the ADCS Certificate Authority and the ADCS Web Enrollment features. Normally on a standalone Root CA you wouldn’t bother installing the ADCS Web Enrollment feature, but in our case it is an easy way to have the CertEnroll website virtual folder created which we use to transfer the Root CA Cert and the Issuing CA Cert (later on).

# Install the ADCS Certificate Authority
WindowsFeature ADCSCA {
    Name   = 'ADCS-Cert-Authority'
    Ensure = 'Present'
}

# Install ADCS Web Enrollment - only required because it creates the CertEnroll virtual folder
# Which we use to pass certificates to the Issuing/Sub CAs
WindowsFeature ADCSWebEnrollment {
    Ensure    = 'Present'
    Name      = 'ADCS-Web-Enrollment'
    DependsOn = '[WindowsFeature]ADCSCA'
}

Next on the agenda is we create a CAPolicy.inf file - this file configures some basic parameters that will be used by the Root CA certificate and server:

# Create the CAPolicy.inf file which defines basic properties about the ROOT CA certificate
File CAPolicy {
    Ensure          = 'Present'
    DestinationPath = 'C:\Windows\CAPolicy.inf'
    Contents        = "[Version]`r`nSignature=`"$Windows NT$`"`r`n[Certsrv_Server]`r`nRenewalKeyLength=4096`r`nRenewalValidityPeriod=Years`r`nRenewalValidityPeriodUnits=20`r`nCRLDeltaPeriod=Days`r`nCRLDeltaPeriodUnits=0`r`n[CRLDistributionPoint]`r`n[AuthorityInformationAccess]`r`n"
    Type            = 'File'
    DependsOn       = '[WindowsFeature]ADCSWebEnrollment'
}

And now the ADCS Certificate Authority and the ADCS Web Enrollment features can be configured. Notice we are using some of the Nodes parameters here as well as the Local Administrator Credentials:

# Configure the CA as Standalone Root CA
xADCSCertificationAuthority ConfigCA {
    Ensure                   = 'Present'
    Credential               = $LocalAdminCredential
    CAType                   = 'StandaloneRootCA'
    CACommonName             = $Node.CACommonName
    CADistinguishedNameSuffix = $Node.CADistinguishedNameSuffix
    ValidityPeriod           = 'Years'
    ValidityPeriodUnits      = 20
    DependsOn                = '[File]CAPolicy'
}

# Configure the ADCS Web Enrollment
xADCSWebEnrollment ConfigWebEnrollment {
    Ensure    = 'Present'
    Name      = 'ConfigWebEnrollment'
    Credential = $LocalAdminCredential
    DependsOn = '[xADCSCertificationAuthority]ConfigCA'
}

Now, here is where things get interesting. We need to configure some of the more advanced properties of the CA such as the AIA and CDP extensions. The problem is that there is no DSC resource for doing this and there aren’t even any native PowerShell cmdlets either! So I had to resort to the DSC Script resource in combination with the CertUtil.exe tool and registry entries:

# Set the advanced CA properties
Script ADCSAdvConfig {
    SetScript = {
        if ($Using:Node.CADistinguishedNameSuffix) {
            & "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSConfigDN "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)"
            & "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSDomainDN "$($Using:Node.CADistinguishedNameSuffix)"
        }
        if ($Using:Node.CRLPublicationURLs) {
            & "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CRLPublicationURLs $($Using:Node.CRLPublicationURLs)
        }
        if ($Using:Node.CACertPublicationURLs) {
            & "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CACertPublicationURLs $($Using:Node.CACertPublicationURLs)
        }
        Restart-Service -Name CertSvc
        Add-Content -Path 'c:\windows\setup\scripts\certutil.log' -Value "Certificate Service Restarted ..."
    }
    GetScript = {
        return @{
            'DSConfigDN'          = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN')
            'DSDomainDN'          = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN')
            'CRLPublicationURLs'  = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs')
            'CACertPublicationURLs' = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs')
        }
    }
    TestScript = {
        if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN') -ne "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
        if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN') -ne "$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
        if (($Using:Node.CRLPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs') -ne $Using:Node.CRLPublicationURLs)) { return $false }
        if (($Using:Node.CACertPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs') -ne $Using:Node.CACertPublicationURLs)) { return $false }
        return $true
    }
    DependsOn = '[xADCSWebEnrollment]ConfigWebEnrollment'
}

The above section was actually detailed in my previous post here. With all that done the Root CA is installed and ready to go. But this DSC configuration script is not yet finished, but we can’t go any further until the Sub CA DSC has progressed. It is important to keep in mind that these DSC scripts are running at the same time on different machines and will interact with one another during this process.

This also seems like an appropriate time to take a break. The really interesting stuff is yet to come, but it is getting close to my bedtime and so I’ll continue this in part 2 tomorrow. This will cover the Sub CA DSC configuration and the final part of the Root CA DSC configuration. Hopefully someone out there has stuck with me till this point! 😃

Next Part

Installing a Two-Tier PKI using nothing but Desired State Configuration - Part 2

PowerShell DON'T of the Week - $Switch in a Switch { }

2015-09-08T00:00:00Z

I just spent the last hour bashing my head against my keyboard trying to figure out what I had done wrong in one of my scripts.

It turns out when you are inside a switch construct, the variable $Switch is redefined (presumably by the switch construct itself) as an empty variable of type System.Collections.IEnumerator. The value is set to $null. This won’t be a problem if you’re not using a variable with the name $Switch. Unfortunately, I was—because I was working with a set of Virtual Switches, so $Switch seemed like a fair choice of variable name.

I could go and research further into this and find out why this is, but I just don’t have time right now. If anyone else has looked into this, I’d be really interested to know why.

Get the IP Address of a VM Attached to an External Switch

2015-09-07T00:00:00Z

My LabBuilder project is coming along nicely and it is building a large lab environment in Hyper-V within a few minutes. However, a problem I ran into was that sometimes the host couldn’t connect (using New-PSSession or equivalent) to a Guest VM to copy files or invoke commands. This was because I was usually using the computer name to connect to the Guest VM—which won’t always work. Instead, I needed to use the IP address of the VM’s Virtual NIC that is attached to the External Switch.

So what I needed was a command that I could fire on the host that would tell me the IPv4 address of the external-facing NIC on a VM. So this is what I came up with:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\09\2015-09-07-get-the-ip-address-of-a-vm-attached-to-an-external-switch.md
$IPAddress = (Get-VMNetworkAdapter -VMName 'Server01').Where({
    $_.SwitchName -eq (Get-VMSwitch -SwitchType External).Name
}).IPAddresses.Where({ $_.Contains('.') })

This wouldn’t be required at all, and I wouldn’t need to have the Guest VM connected to the Host via an External switch, if PowerShell Direct was integrated into the New-PSSession cmdlet. But unfortunately it isn’t yet. If you’d like to see this happen too, please go and vote for this PowerShell Connect feedback item. If you haven’t heard of PowerShell Direct, see this post.

Convert a Domain Name to a Distinguished Name in PowerShell

2015-09-03T00:00:00Z

Here is a small PowerShell snippet to easily convert a domain name (e.g. corp.bmdlab.com) to a distinguished name (DC=corp,DC=bmdlab,DC=com):

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\09\2015-09-03-convert-a-domain-name-to-a-distinguished-name.md
[String]$Domain = 'corp.bmdlab.com'

# Create an empty string that the DN will be stored in
[String]$DN = ''

# Assemble the DN by splitting the DC and then looping to concatenate the new
$Domain.Split('.') | ForEach-Object { $DN = "DC=$($_),$DN" }

# An extra , will be left on the end of DN, so strip it off
$DN = $DN.Substring(0, $DN.Length - 1)

An even easier way would be to use the Replace method on a string object:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\content\blog\2015\09\2015-09-03-convert-a-domain-name-to-a-distinguished-name.md
[String]$Domain = 'corp.bmdlab.com'

# Assemble the DN by replacing
$DN = 'CN=' + $Domain.Replace('.', ',CN=')

That is all!

Advanced Certificate Services Configuration with DSC

2015-09-03T00:00:00Z

Recently I’ve been rebuilding my Hyper-V lab environment from scratch (as part of my MCSA/MCSE studying) and decided I would completely script the process using PowerShell only. My goal was to not require a single interactive session with any of the servers to set up the entire environment. This was a multi-site AD environment with several other member servers performing other duties including NPS & NAP, ADCS, WDS, WSUS, ADFS, ADRMS, IIS, DirectAccess, SQL, etc. I also wanted this to include a proper Multi-tier PKI environment with both a standalone Root CA and an Enterprise Subordinate CA (as is recommended by Microsoft).

The Problem

To configure an AD CS using DSC is very straightforward using the xADCSDeployment DSC Resource. However, anyone who has installed an enterprise PKI is probably familiar with the fact that AIA and CDP settings need to be configured for the CA so that any generated certificates can include these extensions. Unfortunately, the xADCSDeployment DSC Resource doesn’t support setting these Certificate Services options because the underlying AD CS PowerShell cmdlets don’t allow setting these options either.

The Solution

The solution to this problem is to use the DSC Script Resource to call the CertUtil.exe application to set these values. To do this, I set the SetScript resource parameter like this:

SetScript = {
    if ($Using:Node.CADistinguishedNameSuffix) {
        & "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSConfigDN "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)"
        & "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSDomainDN "$($Using:Node.CADistinguishedNameSuffix)"
    }
    if ($Using:Node.CRLPublicationURLs) {
        & "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CRLPublicationURLs $($Using:Node.CRLPublicationURLs)
    }
    if ($Using:Node.CACertPublicationURLs) {
        & "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CACertPublicationURLs $($Using:Node.CACertPublicationURLs)
    }
    Restart-Service -Name CertSvc
    Add-Content -Path 'c:\windows\setup\scripts\certutil.log' -Value "Certificate Service Restarted ..."
}

The above code expects the $Node object to contain several properties that it will use to set applicable settings in the CA server:

CADistinguishedNameSuffix - This is the Directory Services Distinguished Name (e.g., DC=bmdlab,DC=com). If left blank, then your CDP and AIA LDAP Addresses will be incorrect.
CRLPublicationURLs - The CRL Publication URLs in the same format as you would normally pass to the certutil.exe application.
CACertPublicationURLs - The CA Cert Publication URLs (AIA Extension) in the same format as you would normally pass to the certutil.exe application.

Note: To access the $Node object in the script resource, you’ll need to use the Using keyword; otherwise, the $Node object won’t be available in the external script scope.

Finally, after these values have been set, the CertSvc needs to be restarted.

What about TestScript?

You might look at the above code and wonder, “won’t the CertSvc be restarted every 30 minutes no matter what?” That is where the TestScript resource parameter comes in. We’ll use this to decide if the current values of the CA are different from what we want them to be. In this case, we dive straight to the registry rather than using the CertUtil.exe application.

TestScript = {
    if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN') -ne "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
    if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN') -ne "$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
    if (($Using:Node.CRLPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs') -ne $Using:Node.CRLPublicationURLs)) { return $false }
    if (($Using:Node.CACertPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs') -ne $Using:Node.CACertPublicationURLs)) { return $false }
    return $true
}

Once again, I need to ensure the Using scope is used with the Node variable. If any of the node properties don’t match the registry value, then false is returned, which triggers SetScript. If all of the values match, true is returned (meaning everything is the same), and SetScript isn’t fired.

Is That It?

Almost. Finally, I needed to implement the GetScript parameter of the resource. This was the easiest part:

GetScript = {
    return @{
        'DSConfigDN'          = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN')
        'DSDomainDN'          = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN')
        'CRLPublicationURLs'  = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs')
        'CACertPublicationURLs' = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs')
    }
}

I could also adjust the TestScript to call the GetScript and then use the returned hash table to compare with the Node values instead of comparing them directly with the registry values. But I didn’t.

Final Script

Here is what the final script looks like (I didn’t include everything in the DSC Configuration as there were lots of other resources for creating the machine):

Script ADCSAdvConfig {
    SetScript = {
        if ($Using:Node.CADistinguishedNameSuffix) {
            & "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSConfigDN "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)"
            & "$($ENV:SystemRoot)\system32\certutil.exe" -setreg CA\DSDomainDN "$($Using:Node.CADistinguishedNameSuffix)"
        }
        if ($Using:Node.CRLPublicationURLs) {
            & "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CRLPublicationURLs $($Using:Node.CRLPublicationURLs)
        }
        if ($Using:Node.CACertPublicationURLs) {
            & "$($ENV:SystemRoot)\System32\certutil.exe" -setreg CA\CACertPublicationURLs $($Using:Node.CACertPublicationURLs)
        }
        Restart-Service -Name CertSvc
        Add-Content -Path 'c:\windows\setup\scripts\certutil.log' -Value "Certificate Service Restarted ..."
    }
    GetScript = {
        return @{
            'DSConfigDN'          = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN')
            'DSDomainDN'          = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN')
            'CRLPublicationURLs'  = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs')
            'CACertPublicationURLs' = (Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs')
        }
    }
    TestScript = {
        if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSConfigDN') -ne "CN=Configuration,$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
        if (((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('DSDomainDN') -ne "$($Using:Node.CADistinguishedNameSuffix)")) { return $false }
        if (($Using:Node.CRLPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CRLPublicationURLs') -ne $Using:Node.CRLPublicationURLs)) { return $false }
        if (($Using:Node.CACertPublicationURLs) -and ((Get-ChildItem 'HKLM:\System\CurrentControlSet\Services\CertSvc\Configuration').GetValue('CACertPublicationURLs') -ne $Using:Node.CACertPublicationURLs)) { return $false }
        return $true
    }
    DependsOn = '[xADCSWebEnrollment]ConfigWebEnrollment'
}

Hopefully, someone will make sense of all this. It should also be useful in other similar situations where there are no relevant DSC resources.

WMF 5.0 Production Preview is Available

2015-09-01T00:00:00Z

Have a look here. Still not quite RTM, but close!

The Demise of SMB1 in the Windows Stack

2015-09-01T00:00:00Z

If you’re interested in SMB and the general progress of the protocol, I’d recommend this 30-minute video on Channel 9: The Demise of SMB1 in the Windows Stack. It motivated me to rid all my desktops and servers of SMB 1.0, the ancient and insecure protocol support.

To disable SMB 1.0 on a Windows Server:

Set-SmbServerConfiguration -EnableSMB1Protocol $false

You can uninstall SMB 1.0 on a Windows desktop:

Disable-WindowsOptionalFeature -FeatureName SMB1Protocol -Online

Warning: Don’t do this if you have any older devices or operating systems that can only use SMB 1.0.

Install all DSC Resources in a Repository

2015-08-28T00:00:00Z

Something quick for Friday morning! Are you feeling enthusiastic about DSC? Do you want to update or install every DSC resource in a repository onto your computer?

Try this (only on PowerShell 5.0 computers - what do you mean you’re not using PowerShell 5.0?!):

Find-DscResource |
    Select-Object -ExpandProperty ModuleName -Unique |
    ForEach-Object { Install-Module -Name $_ -Force }

This downloads and installs (or updates, if you have older versions) every DSC resource in all PowerShell repositories registered on your computer. It can take a while.

If you want to limit the action to a specific repository, do this:

$RepoName = 'PSGallery'

Find-DscResource -Repository $RepoName |
    Select-Object -ExpandProperty ModuleName -Unique |
    ForEach-Object { Install-Module -Name $_ -Repository $RepoName -Force }

Important: only install DSC resources from repositories you trust. Mark trusted repositories so you don’t have to confirm each module installation:

Set-PSRepository -Name 'PSGallery' -InstallationPolicy Trusted

Detach from a Docker Container Without Stopping It

2015-08-28T00:00:00Z

Saturday-morning Docker fun times (still only on Windows Server Core) – here’s something I found out that might be useful. It is documented in the official Docker docs but not in the Microsoft container docs.

Once you have attached to a Docker container via a CMD console, typing exit detaches from the container and stops it. That’s usually not what I want. To detach from the container without stopping it, press Ctrl + P followed by Ctrl + Q.

The container is still running after being detached.

This only applies to Docker containers that you connected to with docker attach or docker run. Windows Server Containers that you entered with Enter-PSSession can be exited with the standard Exit command.

Well, that’s enough for a Saturday morning!

Docker and Containers on Nano Server Continued

2015-08-27T00:00:00Z

This is a continuation of my investigation of how to get Containers and also possibly the Docker engine running on Windows Server Nano 2016 TP 3. The initial investigation into this can be found here: How to use Containers on Windows Nano Server.

This post is mainly documenting the process of manually creating containers on Windows Nano Server 2016 TP3 as well as some additional details about what I have managed to find out. The documentation on Windows Server Containers from Microsoft is relatively thin at the moment (not surprising - this is very much in technical preview) and so a lot of the information here is speculation on my part. Still, it might be useful to get an idea of how things eventually will work. But of course a good deal of it could change in the near future. This information is really to help me get my head around the concepts and how it will work, but it might be useful for others.

Step 1 - Create a Nano Server Virtual Machine

Anyone who has played around with Nano Server should already be very familiar with this step. The only thing to remember is that the following packages must be included in the VHDx:

Guest - All Nano Server VHDx files running as VM should have this package. If you’re installing Nano Server onto bare metal you won’t need this.
Compute - Includes the Nano Server Hyper-V components. Required because Containers use Hyper-V networking and are a form of Virtualization.
OEM-Drivers - Not strictly required but I tend to include it anyway.
Containers - This package provides the core of Windows Server Containers.

If you’re unfamiliar with creating a Nano Server VHDx, please see this post.

Step 2 - Configure the Container Host Networking

Any container that needs to be connected to a network (most of them usually) will need to connect to a Hyper-V Virtual Switch configured on this Container Host. There are two virtual switch types that can be configured for this purpose:

NAT - This seems to be a new switch type in Windows Server 2016 that causes performs some kind of NAT on the connected adapters.
DHCP - this is actually just a standard External switch with a connection to a physical network adapter on the Container Host.

The installation script normally performs one of the above depending on which option you select. However on Nano Server both of these processes fail:

NAT

Creating a NAT VM Switch on Nano Server actually works. But the command to create a NAT Network connection to the VM Switch fails because the NETNAT module is not available on Nano Server.

DHCP

Creating a DHCP/External VM Switch on Nano Server just fails with a cryptic error message. The same error occurs when creating a Private or Internal VM Switch, so I expect Hyper-V on Nano Server isn’t working so well (or at all). Not much point pursuing this method of networking.

Step 3 - Install a Base OS Image from a WIM File

Every container you create requires a Base OS Image. This Base OS Image contains all the operating system files and registry settings for the OS a container uses. Windows Server Containers expects to be provided with at least one Base OS Image in the form of a WIM file. You can’t create a container without one of these. At this point I am unsure if the WIM file that Windows Server Containers will use is a customized version of the WIM file provided with an OS or if it is standard.

During an installation of Windows Server Containers onto a Windows Server Core operating system, the process automatically downloads a WIM file that is used as the Base OS Image.

To install a Base OS Image from a WIM file on the Container Host, use:

Install-ContainerOSImage -WimPath CoreServer.wim -Verbose

This function does several things:

Creates a new folder in C:\ProgramData\Microsoft\Windows\Images whose name is the canonical name of the new Base OS Image.
Inside that folder a sub-folder called files is created and the image is expanded there.
Another sub-folder called hives is created which contains the default registry hives for the image.
Two metadata files are written – Metadata.json and Version.wcx.
Finally, the image is added to the list of container images available for new containers.

I have tried using Install.wim from the ISO, NanoServer.wim from the ISO, and the Core.wim downloaded via the Core-edition container install script. Note that Install.wim on the TP3 ISO still reports Windows Server 2012 R2 SERVERSTANDARDCORE (double-checked via the version number inside the image).

The Test-ContainerImage cmdlet can be used to identify “problems” with container images:

None of the container images report any problems which is nice to know.

Step 4 - Create a Container

This is obviously where things should start to get exciting! The next step is to create a shiny new container using one of our Base OS Images. However, if you try and create a new container at this point a cryptic error message will occur:

I don’t know what causes this, but if you reboot your Nano Server VM the error goes away and you should be able to successfully create the container:

Unfortunately only the Base OS image downloaded from Microsoft for Windows Server 2016 Core results in a valid container. It seems that certain customisations are required before an image can be containerised.

Step 5 - Start the Container

I’m not holding my breath here. This is what happens when the container is started:

Looking closely at the text of the error it would appear that there was a mismatch between the Container Host OS version and that of the Base OS version that the container was using. This is probably because the Container Host is a Nano Server and the Base OS that was downloaded was for a Core Server.

Next Steps

It would seem at this point we have to wait for Microsoft to provide a Base OS file for Nano Server and also fix the Virtual Switch issues with Nano Server before any further progress can be made experimenting with Containers on Nano Server.

However, it may still be possible to get the Docker Engine working under Nano Server and see if that offers any more information. So that will be what I’ll look into next.

Also, it is interesting to dig around into the files that are created when the new container was created:

When a container is created the container files are stored in the C:\ProgramData\Microsoft\Windows\Hyper-V\containers folder. Unfortunately the files all binary so we aren’t able to dig around in them to glean any other information.

Well, that is enough for today.

How to Use Containers on Windows Nano Server

2015-08-26T00:00:00Z

Edit: I wrote this article when examining containers on Windows Nano Server TP3 – which wasn’t in a working state. I have not yet had a chance to fully examine containers on Windows Nano Server TP4, but when I get a spare few hours I will no doubt deep-dive into it.

If you’re looking for instructions on installing and using containers on Windows Nano Server TP4, start here.

These instructions are more focused on setting up a container host on Windows Server Core TP4, but I have managed to get them working on Windows Nano Server TP4 just fine:

I do plan to document this process over the next week or so.

You’d be forgiven for believing that it was just a simple click of a button (or addition of a package) to get Docker containers working on a shiny new Windows Nano Server TP3 install. That’s what I thought too. But after careful examination of the available documentation I found that there isn’t much information on actually getting containers working on Nano Server. Sure, there is lots of information on running them on a full or core version of Windows Server TP3, but Nano is lacking. So, because I’m a bit obsessive, I decided I’d try adapting the standard installation process.

Edit: Initially I had a bit of success, but I’ve run into some rather stop dead issues that I haven’t been able to resolve (see later on in this post).

I have continued the investigation here with a much more in depth look at the issues.

tl;dr: Containers on Windows Server Nano 2016 TP3 does not work yet! The Base OS WIM file for Windows Server Nano is required, but has not been provided.

Problems with the Standard Containers Install Script

First up I grabbed a copy of this script from Microsoft which is what is used to install containers on a full Windows Server 2016 install. I took a look at it and identified the things that wouldn’t work on a Nano Server 2016 install. This is what I found:

The script can optionally configure a NAT switch - this requires the NetNat PS module which isn’t available on Nano.
The script will install a VM Switch - therefore the Compute package is required to be installed on the Nano Server (the Compute package contains the Hyper-V components).
The script can download various files from the internet (using the alias wget). Wget and Invoke-WebRequest are not available on Nano Server - so we’ll need to download the files to another machine and pre-copy them to the Nano Server.
The Expand-Archive is used to extract the NSSM executable, but this cmdlet is not available on Nano Server either - so we’ll need to extract the NSSM.exe on another machine and copy it to the server.

The Process of Installing a Container Host

The process of actually installing a Container Host in Windows Nano Server is as follows:

Create a Nano Server VHDx with the packages Guest, OEM-Drivers, Compute and Containers.
Create a new VM booting from the VHDx - this is our Container Host.
Upload a Base OS WIM file to the Container Host containing that will be used to create new containers.
Upload Docker.exe to c:\windows\system32\ on the Container Host.
Upload NSSM.exe to c:\windows\system32\ on the Container Host - this is used to create and run the Docker Service.
Run the installation script on the Container Host - this will install the networking components and configure the Docker service as well as create the container OS image.
Create a Container!

In theory the Container Host is now ready to go!

What is Required to build a Nano Server Container Host

A bit of experience with PowerShell is a good help here!

So, to create a Nano Server Container Host you’ll need a few things:

A machine that can run Generation 2 Hyper-V machines (Gen 1 will probably work but I’m using Gen 2) - this will host your Nano Server. This machine must also be running PowerShell 5.0 (I’m using some PS5.0 only cmdlets)!
A copy of the Windows Server 2016 TP 3 ISO from here.
A working folder (I used D:\Temp) where you’ll put all the scripts and other files etc.
The scripts (I’ll provide them all in a zip file), but they are:
1. New-ContainerHostNano.ps1 - this will do everything and is the only script you’ll run.
2. Install-ContainerHostNano.ps1 - this is the script that gets automatically run on the Container Host. It is a version of the Microsoft one from here that I have adjusted to work with Nano Server.
3. New-NanoServerVHD.ps1 - this is a script I wrote a while back to create Nano Server VHDx files (see this post for more details).
4. Convert-WindowsImage.ps1 - this script is required by New-NanoServerVHD.ps1 and is available on Microsoft Script Center.

How Can I use all This?

I haven’t really finished implementing or testing these scripts and I am encountering a problem creating the VM Switch on the Nano Server, but if you’re interested you can get a hold of the scripts in my GitHub repository.

To use them:

Create a working folder (I used d:\temp).
Download the four PS1 scripts from the GitHub repository to the working folder.
Download the Windows Server 2016 TP3 ISO from here and put it in the working folder.
Download the Base OS Container Image from here (3.5GB download) and put it in the working folder.
Edit the New-ContainerHostNano.ps1 file in the working folder and customize the variables at the top to suit your paths and such - fairly self explanatory.
In an Administrative PowerShell run the New-ContainerHostNano.ps1 file.

Please note: This is a work in progress. There are definitely some bugs in it:

An error is occurring when trying to create the VM Switch in DHCP mode or NAT mode.
If using NAT mode the NAT module isn’t included in Nano Server so although the VM switch gets created the NAT Network adapter can’t be created.
NSSM isn’t creating the Docker Service - which may just be an issue with running the PowerShell installation script remotely.

None of the above will stop containers being created though. The containers might not be able to communicate with the world via networking and the Docker management engine might not work, but in theory the containers should still work (at least that is my understanding).

The BIG Problem

Any container that you create requires a WIM file that contains the container base OS image that container will use. Microsoft has so far only provided a base WIM file for WIndows Server 2016 Core installations - they haven’t provided a container base OS Image for Windows Server 2016 Nano yet. You can download the Core one from here (3.5GB download).

If you try to use the NanoServer.WIM file from the Windows Server 2016 ISO as the container base OS image you can’t even create the container at all.

I did try putting the Core WIM file downloaded above onto the Nano Server. I could then create a container OK, but an error would occur starting it up:

Nope - can’t use the Core WIM with a Nano Server Container Host!

Update 2015-10-29: There is a new video available online from Microsoft of Mark Russinovich (Azure CTO) doing a container demonstration using a Nano Server. It clearly shows that the NanoServer Base Container Image does exist. So perhaps we’ll see this in the TP4 release.

The video can be seen here.

Feel free to let me know if you can solve any of these issues! Any help is appreciated. I’ll continue to work on this and post any additional results.

RSAT for Windows 10 is now Available!

2015-08-25T00:00:00Z

You can finally download Remote Server Administration tools for Windows 10 from here. Go and get it!

PowerShell Cmdlets Available in the Containers Module on a Nano Server

2015-08-23T00:00:00Z

Just a Monday-morning quickie:

Here’s a list of all the cmdlets available in the Containers PowerShell module on a Nano Server with the containers package installed:

Containers – the next big thing!

Function Install-ContainerOSImage                    1.0.0.0  Containers
Function Uninstall-ContainerOSImage                  1.0.0.0  Containers
Cmdlet   Add-ContainerNetworkAdapter                 1.0.0.0  Containers
Cmdlet   Connect-ContainerNetworkAdapter             1.0.0.0  Containers
Cmdlet   Disconnect-ContainerNetworkAdapter          1.0.0.0  Containers
Cmdlet   Export-ContainerImage                       1.0.0.0  Containers
Cmdlet   Get-Container                               1.0.0.0  Containers
Cmdlet   Get-ContainerHost                           1.0.0.0  Containers
Cmdlet   Get-ContainerImage                          1.0.0.0  Containers
Cmdlet   Get-ContainerNetworkAdapter                 1.0.0.0  Containers
Cmdlet   Import-ContainerImage                       1.0.0.0  Containers
Cmdlet   Move-ContainerImageRepository               1.0.0.0  Containers
Cmdlet   New-Container                               1.0.0.0  Containers
Cmdlet   New-ContainerImage                          1.0.0.0  Containers
Cmdlet   Remove-Container                            1.0.0.0  Containers
Cmdlet   Remove-ContainerImage                       1.0.0.0  Containers
Cmdlet   Remove-ContainerNetworkAdapter              1.0.0.0  Containers
Cmdlet   Set-ContainerNetworkAdapter                 1.0.0.0  Containers
Cmdlet   Start-Container                             1.0.0.0  Containers
Cmdlet   Stop-Container                              1.0.0.0  Containers
Cmdlet   Test-ContainerImage                         1.0.0.0  Containers

Comparing Objects using JSON in PowerShell for Pester Tests

2015-08-23T00:00:00Z

Recently I spent the good part of a weekend putting together Pester Tests (click here if you aren’t familiar with Pester) for my LabBuilder PowerShell module- a module to build a set of Virtual Machines based on an XML configuration file. In the module I have several cmdlets that take an XML configuration file (sample below) and return an array of hash tables as well as some hash table properties containing other arrays - basically a fairly complex object structure.

A Pester Test config file for the LabBuilder module

In the Pester Tests for these cmdlets I wanted to ensure the object that was returned exactly matched what I expected. So in the Pester Test I programmatically created an object that matched what the Pester Test should expect the output of the cmdlets would be:

# Sample of the expected object built inside the Pester test
$ExpectedSwitches = @(
    @{
        name     = 'General Purpose External'
        type     = 'External'
        vlan     = $null
        adapters = [System.Collections.Hashtable[]]@(
            @{ name = 'Cluster';    macaddress = '00155D010701' }
            @{ name = 'Management'; macaddress = '00155D010702' }
            @{ name = 'SMB';        macaddress = '00155D010703' }
            @{ name = 'LM';         macaddress = '00155D010704' }
        )
    },
    @{ name = 'Pester Test Private Vlan'; type = 'Private'; vlan = '2'; adapters = @() },
    @{ name = 'Pester Test Private';      type = 'Private'; vlan = $null; adapters = @() },
    @{ name = 'Pester Test Internal Vlan'; type = 'Internal'; vlan = '3'; adapters = @() },
    @{ name = 'Pester Test Internal';     type = 'Internal'; vlan = $null; adapters = @() }
)

What I needed to do was try and make sure the objects were the same. At first I tried to use the Compare-Object cmdlet - this actually wasn’t useful in this situation as it doesn’t do any sort of deep property comparison. What was needed was to serialize the objects and then perform a simple string comparison. The ConvertTo-JSON cmdlet seemed to be just what was needed. I also decided to use the [String]::Compare() method instead of using the PowerShell -eq operator because the -eq operator seems to have issues with Unicode strings.

The Pester test that I first tried was:

Context 'Valid configuration is passed' {

    $Switches = Get-LabSwitches -Config $Config

    It 'Returns Switches object that matches Expected object' {
        [string]::Compare(
            ($Switches        | ConvertTo-Json -Depth 4),
            ($ExpectedSwitches | ConvertTo-Json -Depth 4)
        ) | Should -Be 0
    }
}

This initially seemed to work, but if I changed any of the object properties below the root level (e.g. the adapter name property) the comparison still reported the objects were the same when they weren’t. After reading the documentation it states that the ConvertTo-JSON cmdlet provides a Depth property that defaults to 2 - which limits the depth that an object structure would be converted to. In my case the object was actually 4 levels deep. So I needed to add a Depth parameter to the ConvertTo-JSON calls:

[string]::Compare(
    ($Switches        | ConvertTo-Json -Depth 4),
    ($ExpectedSwitches | ConvertTo-Json -Depth 4)
) | Should -Be 0

This then did pretty much exactly what I wanted. However, I also needed the comparison to be case-insensitive, so I added a boolean parameter to the [String]::Compare static call:

# Case-insensitive comparison
[string]::Compare(
    ($Switches        | ConvertTo-Json -Depth 4),
    ($ExpectedSwitches | ConvertTo-Json -Depth 4),
    $true          # ignore case
) | Should -Be 0

The end result was a deep object comparison between a reference object and the object the cmdlet being tested returned. It is by no means perfect as if the properties or contents of any arrays in the object are out of order the comparison will report that there are differences, but because we control the format of these objects this shouldn’t be a problem and should enable some very test strict cmdlet tests.

How the the Final Pester Test in Visual Studio 2015 (with POSH tools)

Edit: after writing a number of Pester tests using the approach I realized it could be simplified slightly by replacing the generation of the comparison object with the actual JSON output produced by the reference object embedded inline in a variable. For example:

Performing the object comparison using JSON in a variable in the test.

The JSON can be generated manually by hand (before writing the function itself) to stick to the Test Driven Design methodology or it can be generated from the object the function being tested created (once it it working correctly) and then written to a file using:

# Generate reference JSON file
Set-Content -Path "$env:TEMP\Switches.json" -Value ($Switches | ConvertTo-Json -Depth 4)

The $switches variable contains the actual object that is produced by the working command being tested.

A Word of Caution about CRLF

I have noticed that when opening the JSON file in something like Notepad++ and copying the JSON to the clipboard (to paste into my Pester test) that an additional CRLF appears at the bottom. You need to ensure you don’t include this at the bottom of your variable too - otherwise the comparison will fail and the objects will appear to be different (when they aren’t).

This is what the end of the JSON variable definition should look like:

And this is what it should not look like (the arrow indicates the location of the extra CRLF that should be removed):

I could have used the Export-CliXML and Import-CliXML CmdLets instead to perform the object serialization and comparison, but these cmdlets write the content to disk and also generate much larger strings which would take much longer to compare and ending up with a more complicated test.

Well, hopefully someone else will find this useful!

Nano Server Technical Preview 3 Available Now!

2015-08-20T00:00:00Z

Naturally, right as I need to be focusing on rebuilding my lab environment (using PowerShell scripts only of course), Microsoft goes and releases Windows Server 2016 Technical Preview 3, which contains lots of cool things like containers. It also meant a new version of the awesome Nano Server.

But this time, Microsoft has released an installer script called new-nanoserverimage.ps1 on the ISO in the Nano Server folder. This means that my script new-nanoservervhd.ps1 isn’t really needed any more. The official Microsoft one contains few more features that the one I wrote and being official should really be used instead of my old one. But as I have a whole lot of scripts to create Nano VM’s already that use my script I thought I’d update my script anyway (and upload it to script center).

So after updating my script I built some new VM’s containing the new packages containers and defender. I noticed something different straight away - the Nano Server now has a minimal head display called Emergency Management Console:

Nano Server Emergency Management Console

This allows you to easily see some basic information about the running Nano Server on a monitor (or, more likely, a VM console). But you do first need to log in to the Nano Server before you can review this information:

Nano Server Authenticate

You can’t actually do much once inside the Emergency Management Console except reboot and shut down the server. But this does mean that you no longer need to create a start-up task that shows the IP address and other details of the Nano Server on screen manually. I’ve left it in my script for now, but it could probably be removed once I’m sure it isn’t necessary.

Back to the lab scripting!

Prevent Template Virtual Machines from Accidentally Being Booted

2015-08-19T00:00:00Z

Here’s a quick tip for Wednesday night:

If you have a VM you have sysprepped so that you can use it as a template to create other new VMs, set the VHD/VHDx file(s) for the VM read-only so that you won’t unsysprep (is that a word?) it by accident. I have spent many wasted minutes re-sysprepping VMs because I accidentally booted a template VM.

Replace NETSH TRACE START with PowerShell

2015-08-10T00:00:00Z

Recently as part of studying for my MCSA I have been using Message Analyzer to look at Kerberos exchanges (among other things). Yes, I really know how to party! I usually did this by starting the trace on the KDC (DC) using the good old command:

The NETSH Way

NETSH TRACE START CAPTURE=yes TRACEFILE=e:\mytrace.etl
NETSH TRACE STOP

The PowerShell Way

# NETSH TRACE START CAPTURE=yes TRACEFILE=e:\mytrace.etl
New-NetEventSession               -Name "Capture" -CaptureMode SaveToFile -LocalFilePath "e:\mytrace.etl"
Add-NetEventPacketCaptureProvider -SessionName "Capture" -Level 4 -CaptureType Physical
Start-NetEventSession             -Name "Capture"

To stop the trace:

# NETSH TRACE STOP
Stop-NetEventSession   -Name "Capture"
Remove-NetEventSession -Name "Capture"

Unfortunately this is a bit more verbose than the NETSH equivalent. It is also a bit of a pity the CmdLets aren’t written so the output of one can be piped to the next. But we can’t have everything.

More Features

Add-NetEventPacketCaptureProvider `
    -SessionName "Capture" `
    -Level 4 `
    -CaptureType Physical `
    -EtherType 0x0800 `
    -IPAddresses 192.168.178.3 `
    -IpProtocols 6,17

Will cause the trace to capture only IPv4 traffic to/from 192.168.178.3 for TCP and UDP.

Remote Capture via RPC

Looking at the documentation for the New-EventSession cmdlet, it seems that it is possible to have the trace output sent to a remote host via RPC and then captured directly by Network Analyzer. I haven’t been able to get this to work as yet. Figuring out how this works and getting it going is going to be my next project (between studying for the next exam).

Your account settings are out of date in Windows 10 Universal Mail App

2015-07-31T00:00:00Z

The Problem

Update: This problem is a very long running issue that I’m surprised hasn’t been resolved by Microsoft yet. The fix below does work for some people, but based on the comments on this post and those on the Microsoft Forums this solution doesn’t always work and many other fixes have been suggested (see the forum for a large number of other suggested solutions). Some folk have reported that the fix below can prevent the Mail and Calendar apps from working at all, so I’d recommend that you use this process with care and recommend strongly that you back up the comms folder before deleting the content.

Recently I noticed on my Windows 10 desktop that I have been using to test all the Windows 10 Insider Preview releases the Windows 10 Universal Mail and Calendar app is was no longer working with my Outlook mail account. Every time the app loaded or I clicked on the Outlook account it would show a message at the top of the mail list stating “Your account settings are out of date” and giving me the option to fix or dismiss the problem. Clicking fix just flashed up a black window for a brief moment (too fast to see any details) and the message remained. I messed about with the account settings and everything I could find for the account in Mail settings but couldn’t figure out how to fix it. I tried various suggested “fixes” I found online, but none of them worked for me.

My Gmail account worked fine in the Mail app as well and my outlook account also worked fine in the Windows 10 Universal Mail and Calendar app on my laptop. Unfortunately I didn’t think to screenshot the problem before I managed to resolve it so I can’t post an image of the error message here. After a lot of investigation and lots of messing around I found out where the Windows Universal Mail app stores its account data - as I thought this is probably where the problem probably was. The Windows 10 Universal Mail and Calendar seems to store all information in the folder %LOCALAPPDATA%\Comms\ I thought perhaps if I could get rid of (back it up just in case) this folder the app might recreate it.

The Solution

Open an Administrator PowerShell prompt (enter “powershell” in the start menu search and then right click the Windows PowerShell icon and select Run as Administrator, you might need to confirm a UAC prompt).

Uninstall the Windows 10 Universal Mail and Calendar app:

Get-AppxPackage |
    Where-Object Name -eq 'microsoft.windowscommunicationsapps' |
    Remove-AppxPackage

@Stephen suggests restarting your computer at this point.
Delete the %LOCALAPPDATA%\Comms\ folder (back it up first if you want):
```
Remove-Item -Path "$Home\AppData\Local\Comms" -Recurse -Force
```
Note: you’ll probably find that some files are in use and can’t be deleted – this is OK.
Reinstall the Windows 10 Universal Mail and Calendar app from the Microsoft Store.

Once this had all been done I loaded the Mail up and it asked me to configure all my mail accounts again (and authorize them with the providers). After this the error message had gone away and all my accounts worked normally. I did also however have to re-enter my credentials for both Google Drive and One Drive.

For more information, suggestions and discussion about this issue, take a look at this discussion on the Microsoft Community boards.

I hope this works for someone else as well.

Enable Device Naming on all Virtual Net Adapters on a VM Host

2015-07-29T00:00:00Z

After a couple of bumps upgrading my development laptop to Windows 10, I finally got to update all my Hyper-V lab VMs to the new version of Hyper-V. This included updating the Virtual Machine Configuration version and enabling virtual network adapter Device Naming - see What’s new in Hyper-V in Technical Preview for more information.

But having a number of VM’s running on this Hyper-V host I couldn’t be bothered updating them all by hand. So as usual, PowerShell to the rescue.

First up, to upgrade the configuration of all the VMs on this host so the new features can be used I ran the following command:

Get-VM | Update-VMVersion

Once that was completed (it took about 10 seconds) I could enable the Device Naming feature of all the Virtual Network Adapters on all Generation 2 VMs (this feature isn’t supported on Generation 1 VMs). The feature labels the NIC inside the guest OS (when supported) with the name you assign in the host.

To enable Device Naming on all Generation 2 virtual NICs:

Get-VM |
    Where-Object  -Property VirtualMachineSubType -eq 'Generation2' |
    Get-VMNetworkAdapter |
    Set-VMNetworkAdapter -DeviceNaming On

All in all this was much easier than the endless clicking I would have had to do in the UI. You can even combine the two steps into a single command:

Get-VM |
    Update-VMVersion -Passthru |
    Where-Object -Property VirtualMachineSubType -eq 'Generation2' |
    Get-VMNetworkAdapter |
    Set-VMNetworkAdapter -DeviceNaming On

So now it’s off to try some of the other new Hyper-V features.

PowerShell Modules Available in Nano Server

2015-07-23T00:00:00Z

I have been spending a bit of time experimenting with loading Nano Server into WDS (using capture images, VHDX files and the like) and while doing this I decided to dig around inside Server Nano to see what is missing. The thing that is missing that makes me grumble the most is that lots of PowerShell modules are missing. This of course is because Server Nano doesn’t have the full .NET Framework available, which most PowerShell modules depend.

What Modules are Included?

This to some degree depends on the packages that are installed, but with the OEM-Drivers, Storage and Guest packages installed the following modules are available:

As the screenshot above shows, there are a lot of useful modules missing. Even within some of the modules, many of the CmdLets are not available.

For example, in the Microsoft.PowerShell.Management module on Windows Server 2012 R2, there are 86 cmdlets available. In Nano Server there are only 38:

Obviously, this is only Tech Preview 2, and so will likely change, but it certainly might be the case that some PowerShell scripts won’t work on Nano Server and will need to be re-written.

SysPrep is Missing

One other element that is missing from Nano Server is the SYSPREP tool. The folder
C:\Windows\System32\Sysprep is there, but it is empty. So sysprepping a Nano Server at the moment doesn’t seem to be possible.

Force WSUS to Synchronize Now from PowerShell

2015-07-22T00:00:00Z

After passing my MS 70.410 exam I had a little bit of free time on my hands, so I thought I’d clean up my WSUS servers and prepare them for Windows 10 and VS 2015. So I thought I’d force myself to do the whole thing via PowerShell. The problem is that the UpdateServices PowerShell module doesn’t have cmdlets for some things I wanted to do – force a synchronization was among them.
So I needed to use the Microsoft.UpdateServices.NET components to perform these functions.

Useful Commands

To force a WSUS server to synchronise now:

(Get-WsusServer).GetSubscription().StartSynchronization()

To get the result of the last synchronisation:

(Get-WsusServer).GetSubscription().GetLastSynchronizationInfo()

Pretty simple! I’m sure additional functions will crop up and I’ll try to post any useful ones here as well.

A Minor Irritation with VHDs and Dynamic Disks

2015-07-14T00:00:00Z

As part of my recent studies (and because I’m a bit OCD) I’ve been writing some notes on how to perform various DISKPART commands in PowerShell. You might also need to do this if you’re converting old DISKPART scripts into PowerShell for automation purposes.

In most cases it’s straight-forward to map DISKPART commands over to PowerShell. For example, to use DISKPART to initialise a disk and set the partition format to GPT on disk 6:

SELECT DISK=6
ONLINE
CONVERT GPT

The PowerShell equivalent would be:

Set-Disk      -Number 6 -IsOffline $false
Initialize-Disk -Number 6 -PartitionStyle GPT

The Problems

However, I ran into two situations where PowerShell can’t currently replace DISKPART:

Dynamic disks can’t be created using PowerShell. Therefore spanned, striped, mirrored or parity volumes must still be created with DISKPART. (If you’re on Windows Server 2012/Windows 8 or later, consider using Storage Spaces instead.)
The PowerShell cmdlets to create or mount Virtual Hard Disk files (VHD/VHDx) are unavailable if Hyper-V is not installed.
This is a little annoying because the Hyper-V role can’t always be installed—e.g. inside a guest VM. While it’s unusual to work with VHD/VHDx files inside a guest, the rise of cloud-hosted dev machines means you might hit this limitation.

That’s it for tonight!

New-NanoServerVHD Updated to support changes in Convert-WindowsImage

2015-06-19T00:00:00Z

A new version of Convert-WindowsImage.ps1 script that me New-NanoServerVHD.ps1 script uses was released a few days ago. It fixes the issue with running on Windows 10 and Windows Server 2016. However it was also changed in other ways that caused my New-NanoServerVHD.ps1 script to no longer function.

So I’ve updated the New-NanoServerVHD.ps1 script to support the newer ConvertWindowsImage.ps1 script. I also added support for creating a VHDx (with a GPT partition table format) so that it can be used with Generation 2 VMs.

You can control the format of the VHD that is created by passing in a VHDFormat parameter. This parameter defaults to VHD. If VHD is created it will automatically have a partition format of MBR set. When a VHDx is created it will have a GPT partition format.

Here is an example showing how to create a Nano Server VHDx for a Gen 2 VM:

.\New-NanoServerVHD.ps1 `
    -ServerISO 'c:\nano\10074.0.150424-1350.fbl_impressive_SERVER_OEMRET_X64FRE_EN-US.ISO' `
    -DestVHD   'c:\nano\NanoServer02.vhdx' `
    -VHDFormat VHDx `
    -ComputerName 'NANOTEST02' `
    -AdministratorPassword 'P@ssword!1' `
    -Packages 'Storage','OEM-Drivers','Guest' `
    -IPAddress '192.168.1.66'

The updated New-NanoServerVHD.ps1 script can be downloaded here.

Setting the Computer Name installing Nano Server bug resolved

2015-06-16T00:00:00Z

When I originally wrote the script to help install Nano Server I ran into a problem where I couldn’t get the Computer Name of the Nano Server to set during the OfflineServicing phase. This was supposed to be a new feature of Windows Server 2016 where the computer name could now be set in this phase rather than having to wait for the Specialize phase - which meant one less reboot during installation of the OS - saving precious seconds. And when installing Nano, saving an extra few seconds actually matter. I spent some time trying to get this new feature to work but nothing I tried worked, so I resorted to setting it in the Specialize phase like installations of old.

But thanks to Michael Birtwistle for pointing out this forum post. It points out that the Unattend.xml file in the original Nano Server instructions from Microsoft was incorrect. So I have corrected the Install-NanoServerVHD.ps1 script to reflect this as well. So even faster Nano Server provisioning should be available now.

Installing Windows Management Framework 5.0 with a GPO

2015-06-09T00:00:00Z

The Need

Last week I decided I needed to get to know the new features that come with DSC in the new Windows Management Framework 5.0 (aka PowerShell 5.0) April Preview release (available to download here). I figured I’d also need to look at updating my DSCTools module to use the new features as well. But first up I’d need to update all my lab machines with the WMF 5.0 update. Being a proper lazy nerd I thought I’d just automate it.

The Problem

After downloading the .MSU file for installing WMF 5.0 from Microsoft I decided that WSUS would be the proper way to get this out to all my machines. Nope. Wrong! WSUS only supports pushing updates that appear in the Microsoft Update Catalogue (looking for Windows Management Framework) - but no update packages greater than Windows Management Frameworks 2.0 are available in the catalogue:

[caption id=“attachment_205” align=“alignnone” width=“660”]
Where is PowerShell 5.0? Or 4.0? Or 3.0?

As an aside after doing a bit of reading on the reason for this it appears that updating to later versions of PS can cause problems with some applications and so it is kept as a manual process.

Next on the drawing board was push the update out using GPO’s software installation policies - except it only supports MSI files. So that won’t work (although converting an MSU to an MSI might be possible according to some sources).

The Solution

Finally I settled on pushing the MSU onto the machines by using a GPO Startup Script - of the PowerShell flavour of course. It seemed like I could just adapt the PS script I wrote for installing Office 2013 via GPO. After a bit of coding I had the update installing (in silent mode of course).

The next problem was that I needed some sort of registry key that could be checked to see if the update was already installed so it didn’t try to repeatedly reinstall every time the computer started up. I spent a few hours hunting around for information on where in the registry a record of installed updates was kept and couldn’t seemed to find any.

So instead I just used a simple WMI query to find out if the update was installed:

Get-WmiObject -Class Win32_QuickFixEngineering -Filter "HotfixID = 'KB2908075'"

The above command will return a list of KBs installed with a matching HotfixID.
There will be zero results if the KB is not present, so to use it I needed to count the objects returned:

(Get-WmiObject -Class Win32_QuickFixEngineering -Filter "HotfixID = 'KB2908075'" |
    Measure-Object).Count -gt 0

If the MSU or EXE (did I mention this script will work with hot-fixes in EXE form) does indeed need installing, the script just builds a command line and calls it:

if ([IO.Path]::GetExtension($InstallerPath) -eq '.msu') {
    $Command = "WUSA.EXE $InstallerPath /quiet /norestart"
    $Type    = "MSU $KBID"
}
else {
    $Command = "$InstallerPath /quiet /norestart"
    $Type    = "EXE $KBID"
}

# Call the installer
& cmd.exe /c "$Command"
$ErrorCode = $LASTEXITCODE

The Script

So after putting all the pieces together I had a finished script that seemed to do exactly what I needed. It can be downloaded from the Microsoft Script Center PowerShell Scripts to Install Application (EXE) or Update (MSU) using GPO. It is also available in my GitHub repo (edit: I moved this out of my generic PowerShell tools repo and into it’s own repo).

Warning

Before getting started, I strongly suggest you read my post about the problems encountered using PowerShell parameters in GPO here. There are some really annoying issues with PowerShell script parameters in GPO that will have you tearing your hair out - luckily I tore my hair out for you.

Using It

Download the PowerShell Scripts to Install Application (EXE) or Update (MSU) using GPO from Microsoft Script Center.
Download the MSU/EXE update file and put it in a shared folder all of your machines can get to. I used \\plague-pdc\Software$\Updates\WindowsBlue-KB3055381-x64.msu
Create a new GPO that will be used to install the update - I called mine Install WMF 5.0 Policy:
Locate the Startup setting under Computer Configuration/Policies/Windows Settings/Scripts:
Double click the Startup item to edit the scripts and select the PowerShell Scripts tab:
Click the Show Files button and copy the Install-Update.ps1 file that was in the zip file downloaded from Microsoft Script Center into this location:
Close the folder and go back to the Startup Properties window and click the Add button.
Set the Script Name to Install-Update.ps1.
Set the Script Parameters to be the following (customized the bold sections to your environment):
```
-InstallerPath "**\\\\plague-pdc\\Software$\\Updates\\WindowsBlue-KB3055381-x64.msu**" -KBID "**KB3055381**" -LogPath **\\\\plague-pdc\\LogFiles$\\**
```
You can leave off the -LogPath parameter if you don’t want to create logs stating if the update was installed correctly on each machine.
Click OK.
Click OK on the Startup Properties window.
Close the GPME window and assign the policy to whichever OUs you want to try WMF 5.0 out on.

That was probably a lot more detail than most people needed, but I thought I throw it in there just in case.

If you require more details on the parameters available in the script, use the PowerShell cmdlet Get-Help .\Install-Update.ps1 in the folder that Install-Update.ps1 is installed into - or just look at the PowerShell Scripts to Install Application (EXE) or Update (MSU) using GPO Microsoft Script Center page.

Installing an Application Script

While I was at it I also decided that I wanted to install applications contained in an EXE by this method too (Notepad++ was my excuse). This required a slightly different approach to detect if the application was already installed. Basically depending on the application a registry key and/or value needs to be checked to see if the application needs installation. These registry entries differ for every application being installed so which keys to check for an app need to be passed in via parameters to the PowerShell script.

For example, for Notepad++ version 6.7.8.2 the registry key HKLM:\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\Notepad++ must exist and requires a string value called DisplayVersion to be equal to “6.7.8.2”.

I also wrote a script, Install-Application.ps1 to do all this as well, and it is available in the same package as the Install-Update.ps1 script. I will write a separate blog post on using this one as it can be a little bit trickier thanks to the limitations with passing parameters to PS scripts in GPOs. So I’ll leave this post till next week.

Thanks for reading!

PowerShell Paramters in GPO Scripts

2015-06-03T00:00:00Z

Introduction

This morning I decided I wanted to update all my lab servers to Windows Management Framework 5.0 so I could do some work on the new DSC features that come with it. To do this, I thought I’d use a GPO with a startup PowerShell script that would perform the installation of the WMF 5.0 April hotfix (available here).

A GPO Startup PowerShell script with parameters.

On thinking about this I decided it might also be a good idea to modify the PowerShell script designed to install Microsoft Office 2013 products via GPO (see the post here). After producing the new scripts and testing them by manually running them to ensure they worked correctly, I put them into some GPOs. And that is when things started to go wrong!

Parameter Problems

The issue I ran into was that the parameters set in the GPO PowerShell script parameters seemed to misbehave in several ways. After about 6 hours of investigating and testing I’ve found the following things cause problems when you do them.

Parameter Length Limit

There seems to be a maximum number of characters that will be used in the Script Parameters setting in a GPO Startup/Shutdown/Logon/Logoff PowerShell script. The limit appears to be 207 but I can’t find official documentation of this. If script parameters longer than this limit is entered the additional characters will simply be ignored, leading to the script either failing to run or running incorrectly.

If you do run into this issue, one way around it is to edit the script and wrap all the code in a function definition and then add a call with the parameters to the end of the script after the end of the function definition. For Example:

Function Install-Application {
    # ... Existing script code here ...
}

Install-Application `
    -InstallerPath "\\Server\Software$\Notepad++\npp.6.7.8.2.Installer.exe" `
    -InstallerParameters "/S" `
    -RegistryKey "HKLM:\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\Notepad++" `
    -RegistryName "DisplayVersion" `
    -RegistryValue "6.7.8.2" `
    -LogPath "\\Server\Logfiles$"

The PowerShell script parameters in the GPO can then be dropped as they are contained in the script itself – this is not ideal, but unless Microsoft lifts this limitation it may be required.

Parameter Quotes

There is also some odd behaviour passing parameters with quotes (single or double) to the PowerShell scripts in a GPO. I have run into several situations where the use of quotes causes the parameters to either not be passed to the script or passed with additional quotes in them. I recommend the following:

DO NOT use single quotes around any parameters - use double quotes around string parameters only.
DO NOT end the parameter list with a quoted parameter - this seems to cause the last parameter content to contain an extra quote.

Summary

In short, if you stick to the above when calling PowerShell scripts with parameters from GPO then you might save yourself a lot of time scratching your head.

As a quick aside, the scripts I wrote as part of this (for installing Windows QFE Hotfixes and Applications via GPO) are available on Microsoft Script Center here. I will be writing a full post on these scripts later in the week.

WSUS - Declining all Superceded Updates - NOW!

2015-05-13T00:00:00Z

Just a quick snippet today. I wrote this because I was didn’t want to have to wait for 30 days before unusused superceded updates in my WSUS server were automatically declined - especially those daily “Definition Update for Windows Defender”.

If you’re happy waiting for these unused superceded updates to be declined after 30 days then you can just use the following cmdlet:

Invoke-WsusServerCleanup -DeclineSupersededUpdates

However, if you don’t want to wait you can fire off this little PowerShell script. It is just a single line of PowerShell code that will automatically decline all updates with a status of anything except for declined and has at least one superceding update:

Get-WSUSUpdate -Classification All -Status Any -Approval AnyExceptDeclined |
    Where-Object {
        $_.Update.GetRelatedUpdates(
            [Microsoft.UpdateServices.Administration.UpdateRelationship]::UpdatesThatSupersedeThisUpdate
        ).Count -gt 0
    } |
    Deny-WsusUpdate

The command will take a few minutes to run (depending on how many updates your WSUS Server has) - on my WSUS server it took about 5 minutes. Once the process has completed you could then trigger the cmdlet to perform a WSUS Server cleanup (to get rid of any obsolete content files):

Invoke-WsusServerCleanup -CleanupObsoleteUpdates -CleanupUnneededContentFiles

That is about it for today!

Reporting on File, Folder and Share Permissions and how they Change

2015-05-13T00:00:00Z

Introduction

Late last year I was asked by a friend if I could write a program that could look at the ACL’s defined within a folder structure and report back how they differed from some previously recorded state. This was basically so the administrators could report back what ACL’s had change and verify that the "security Elves’ hadn’t been messing about.

So after several almost complete re-writes, the end result is the ACLReportTools PowerShell module.

Overview

The intended purpose of this module is to allow an administrator to report on how ACL’s for a set of path or shares have changed since a baseline was last created.

Basically it allows administrators to easily see what ACL changes are being made so they keep an eye on any security issues arising. If performing SMB share comparisons, the report generation can be performed remotely (from a desktop PC for example) and can also be run against shares on multiple computers.

The process that is normally followed using this module is:

Produce a baseline ACL Report from a set of Folders or Shares (even on multiple computers).
Export the baseline ACL Report as a file.
… Sometime later …
Import the baseline ACL Report from a stored file.
Produce an ACL Difference report comparing the imported baseline ACL Report with the current ACL state of the Folders or Shares
Optionally, export the ACL Difference report as HTML.
Repeat from step 1.

The comparison is always performed recursively scanning a specified set of folders or SMB shares. All files and folders within these locations will be scanned, but only non-inherited ACL’s will be added to the ACL Reports.

Report Details

An ACL Report is a list of non-inherited ACLs for a set of Shares or Folders. It is stored as a serialized array of [ACLReportTools.Permission] objects. ACL Reports are returned by the New-ACLShareReport, New-ACLPathFileReport and Import-ACLReport cmdlets.

An ACL Difference Report is a list of all ACL differences between two ACL reports. It is stored as serialized array of [ACLReportTools.PermissionDiff] objects that are returned by the Compare-ACLReports and Import-ACLDiffReport cmdlet.

ACL Reports produced for shares rather than folders differ in that the share name is provided in each [ACLReportTools.Permission] object and that the SMB Share ACL is also provided in the [ACLReportTools.Permission] array.

Important Notes

When performing a comparison, make sure the baseline report used covers the same set of folders/shares you want to compare now. For example, don’t try to compare ACL’s for c:\windows and c:\wwwroot - that would make no sense and result in non-sensical output.

If shares or folders that are being compared have large numbers of non-inherited ACL’s (perhaps because some junior admin doesn’t understand inheritance) then a comparison can take a long time (hours) and really hog your CPU. If this is the case, run the comparison from another machine using share mode or run it after hours - or better yet, teach junior admins about inheritance! 😃

You should also ensure that the account that is being used to generate any reports has read access to all paths and all content (including recursive content) that will be reported on and can also read the ACL’s. If it can’t access them then you may get access denied warnings (although the process will continue).

NTFS Security Module

This Module uses the awesome NTFS Security Module to be installed in your PowerShell Modules path.

Ensure that you unblock all files in the NTFS Security Module folder before attempting to Import Module ACLReportTools. The ACLReportTools module automatically looks for and Imports the NTFS Security Module if present. If it is missing, an error will be returned stating that the module is missing. If you receive any other errors importing ACL Report tools, it is usually because some of the NTFS Security Module files are blocked and need to be unblocked manually or with Unblock-File. You can confirm this by calling Import-Module NTFSSecurity - if any errors appear then it is most likely the caused by blocked files. After unblocking the module files you may need to restart PowerShell.

Installing ACLReportTools

Unzip the archive containing the ACLReportTools module into the one of the PowerShell Modules folders (E.g. $Home\documents\windowspowershell\modules).
This will create a folder called ACLReportTools containing all the files required for this module.
In PowerShell execute:

Import-Module ACLReportTools

How to Use It

The basic steps for using this module is as follows:

Create a Baseline ACL Report file on a set of Folders.
Compare the Baseline ACL Report file with the current ACL’s for the same set of Folders.
Optionally, convert the ACL Comparisson Report into an HTML report file.

In the for following examples, the e:\work and d:\profiles are being used to produce an ACL Difference report for. The Baseline ACL Report and the ACL Difference Report will be saved into the current users documents folder.

Step 1: Create a Baseline ACL Report file on a set of Folders

The first step is to create Baseline ACL Report on the folders e:\work and d:\profiles and store it in the b_aseline.acl_ file in the current users documents folder:

Import-Module ACLReportTools
New-ACLPathFileReport -Path "e:\Work","d:\Profiles" |
    Export-ACLReport -Path "$HOME\Documents\Baseline.acl" -Force

Step 2: Compare the Baseline ACL Report file with the current ACLs for the same set of Folders

This step is usually performed a few days or weeks after step 1. In this step the Baseline ACL Report created in step 1 is compared with the current ACL’s for the same set of folders used in step 1. The output is put into the variable $DiffReport which can then be exported as a file using the Export-ACLDiffReport cmdlet or saved as HTML using Export-ACLPermissionDiffHTML for easier review.

Import-Module ACLReportTools
$DiffReport = Compare-ACLReports `
    -Baseline (Import-ACLReport -Path "$HOME\Documents\Baseline.acl") `
    -Path "e:\Work","d:\Profiles"

Step 3: Convert the ACL Comparison Report into an HTML Report File

Once the ACL Difference Report has been produced, it could be simply dumped straight into the pipeline or converted into an HTML using the Export-ACLPermissionDiffHTML cmdlet. The title that will appear on the HTML page is also provided.

$DiffReport | Export-ACLPermissionDiffHtml `
    -Title 'ACL Diff Report for e:\work and d:\profile'

Reporting on Shares Instead of Folders

Instead of specifying a set of folders it is also possible to specify a list of computers and/or SMB shares to pull the ACL Reports from. For example if we wanted to report on the shares Share1 and Share2 on computer Client the following commands could be used for step 1:

# Baseline for Share1 and Share2 on CLIENT
Import-Module ACLReportTools
Compare-ACLReports `
    -Baseline (Import-ACLReport -Path "$HOME\Documents\Baseline.acl") `
    -ComputerName Client `
    -Include Share1,Share2

Then for step 2 we would use:

# Later, create the difference report
Import-Module ACLReportTools
$DiffReport = Compare-ACLReports `
    -Baseline (Import-ACLReport -Path "$HOME\Documents\Baseline.acl") `
    -ComputerName Client `
    -Include Share1,Share2

Step 3 in would be exactly the same as in the Folder scenario.

Final Word

What started as a simple script actually ended up turning into quite a large module that taught me a huge amount about PowerShell. So I hope someone else out there is also able to find a use for this and it helps track down some of those ‘Permission Elves’.

Install Windows Server Nano the Easy Way

2015-05-08T00:00:00Z

All the recent talk about the new Windows Server Nano (Windows Server Core on diet pills) that is available for installation on the Windows Server 2016 Technical Preview 2 ISO got me quite interested. So, I thought I’d give it a whirl to see what all the fuss was about. Well, first up, it really is as small (568Mb with my chosen packages) and fast as Microsoft says. Second, however, it is most definitely a Tech Preview and is missing lots of stuff and has some obvious issues.

Edit - 14 May 2016: The scripts and process on this page have now been updated to support Windows Server 2016 Technical Preview 5. The process will not work for versions earlier than Windows Server 2016 Technical Preview 5. I removed a some parts of this document as they were not correct for TP5. I decided to update this blog post with TP5 information because it is still getting a lot of traffic.

Manual Install

Installing a copy of Server Nano is not quite as straight forward as mounting the ISO into a VM and booting it up - at least not yet. But if you’re still keen to give it a try, Microsoft provides some instructions on how you can install Nano into a VHDx which you can then mount as the boot disk in a Gen-1 or Gen-2 Hyper-V Machine.

The manual instructions to create a VHD with Nano Server on it can be found here:

Getting Started With Nano Server

It is well worth reading this to get an idea of how Nano Server is different from regular Core Server.

Easy Install

As you can see, installing Nano Server requires quite a few steps. None of them are difficult, but I wouldn’t be much of a nerd if I didn’t convert it into a script. So after a quiet Friday night I managed to cobble something together. You can find it here:

Create a New Nano Server VHD

It is fairly straight forward to install and use:

Create a Working Folder on your computer in the case of this example I used c:\Nano.
Download the New-NanoServerVHD.ps1 to the Working Folder. Download the Windows Server 2016 Technical Preview ISO (download here) to the Working Folder.
Open an Administrative PowerShell window.
Change directory to the Working Folder (cd c:\nano).
Execute the following command (customizing the parameters to your needs):

.\New-NanoServerVHD.ps1 `
    -ServerISO 'c:\nano\14300.1000.160324-1723.RS1_RELEASE_SVC_SERVER_OEMRET_X64FRE_EN-US.ISO' `
    -DestVHD   'c:\nano\NanoServer01.vhdx' `
    -VHDFormat 'VHDx' `
    -ComputerName 'NANOTEST01' `
    -AdministratorPassword 'P@ssword!1' `
    -Packages 'Storage','OEM-Drivers','Guest' `
    -IPAddress '192.168.1.65'

Note: If you run this in the PowerShell ISE, a pop-up message appears during execution:

This error can be ignored without it causing a problem.

If this happens, just click Continue. I’m not sure why this happens in the ISE, but the script still functions fine if it occurs. I tried to get a screenshot of this but I couldn’t get it to happen.

Booting it up

Once you’ve the VHD has been created, just create a new or Gen-2 (or Gen-1) Hyper-V Machine and assign this VHDx as the boot disk. Start up the VM and after about a minute (for me anyway) you should be able to use PS remoting (Enter-PSSession) to connect to it and begin playing.

Remember, Server Nano is completely headless (which just sounds cool), so if you try to connect to it using the Hyper-V Console you will see the recovery console:

Observations

Edit - 14 May 2016: These “issues” have been resolved in more recent versions of Windows Server 2016 Nano Server.

One thing I have noted though, is that if you watch the machine with the Hyper-V Console while it is booting it will show the nice little Windows Start up screen for up to 5 minutes - even though the machine appears to be completely booted and can be connected to. I’m not sure why this is, but I’m sure MS will sort it out.

Nano booting up - on my system it can be connected to and used even while the boot screen is showing.

A second thing I found while writing this script was that in the Unattend.xml file the ComputerName is supposed to be set in the offlineServicing phase (according to the MS instructions). But this didn’t seem to work for me so my script sets it in both the offlineServicing phase and the Specialize phase. This actually doubles the first boot time from 5 seconds to 10 seconds because it needs to reboot to apply the ComputerName.

If anyone reads this and has any ideas on how to improve the process (or if I’ve gone wrong somewhere), please let me know!

DSC Tools- Hopefully Making DSC Easier

2015-05-02T00:00:00Z

Introduction

Desired State Configuration (DSC) is definitely one of the coolest features of WMF 4.0. This article however is not about what DSC is or how to implement it - there are already many great introductions to DSC out there. If you’re new to DSC I’d suggest you take a look at the great free e-book from PowerShell.org: The DSC Book.

I’ve been working with DSC for several months now in my lab environments and I really love it. But it can be a little bit tricky to set up, with lots of steps and things to remember (or forget in my case). It is very easy to miss a step and spend many hours trying to figure out what went wrong - I know this from experience.

I’m certain in the future that Microsoft and/or other tool providers will provide easier methods of implementing DSC, but at the moment it is still a fairly manual process. There are products like Chef (see Chef for Windows) that are going to use DSC to provide configuration of Windows operating systems (and perhaps non-Windows as well), but as of writing this these tools aren’t yet available.

So, after about the 20th time of setting up DSC, I figured that it might be an idea to write some simple tools that could make my life a little bit easier. With that in mind I created a PowerShell module (supporting WMF 4.0 and above) that provides some helper functions and configurations that make setting up DSC Pull Servers and configuring Local Configuration Manager (LCM) on the nodes slightly less painful.

The Module

With all of the above in mind I set about creating a module that would combine all of these steps into simple PS cmdlets that could be used without having to remember exactly how to do things such as creating an LCM configuration file for a Pull Server or something like that. This in theory would leave more time for the really fun part about DSC: creating node configuration files and resources.

For example, to set up a DSC Pull Server (in HTTP mode) there are several steps required:

Download and Install the DSC Resources your configurations will use to the PS modules folder.
Publish the DSC Resources to a folder in your Pull Server and create checksum files for them.
Create a DSC Configuration file for your Pull Server(s).
Run the DSC Configuration file for your Pull Server(s) into to create MOF files.
Push the DSC Configuration MOF files to the LCM on the Pull Servers.

These steps are easy to understand and perform, but if you don’t do them often you will quickly forget exactly how to do it. I found myself referring back to long sets of instructions (found many places online) every time I did it.

To perform the above steps using the DSCTools module simply requires the following cmdlets to be executed on the computer that will become the DSC Pull Server.

# Download the DSC Resource Kit and install it to the local DSC Pull Server
Install-DSCResourceKit

# Copy all the resources up to the local DSC Pull Server (zipped and with a checksum file)
Publish-DSCPullResources

# Install a DSC Pull Server to the local machine
Enable-DSCPullServer

The cmdlets can be executed on a computer other than the Pull Server by providing -ComputerName and -Credential parameters.

DSCTool CmdLet Parameters

Like most PS cmdlets, you can also provide additional configuration options to them as well. For example, you might want your DSC Pull Server resources folder to be placed in a different location, in which case you could provide the commands with the PullServerResourcePath parameter:

# Copy the resources to a non-default folder on the Pull Server
Publish-DSCPullResources -PullServerResourcePath e:\DSC\Resources

# Install a DSC Pull Server that uses the same custom folder
Enable-DSCPullServer    -PullServerResourcePath e:\DSC\Resources

Most DSCTools cmdlets have many other parameters for controlling most aspects of the functions such as the type of Pull Server to install (HTTP, HTTPS or SMB), the location where files should be stored. If these parameters aren’t passed then the default values will be used.

Default DSCTools Module Settings

If you’re lazy (like me) and don’t want to have to pass the same parameters in to every cmdlet, you can change the default values by overriding script variables once the module has been loaded. For example you might always want your Pull Servers to use a configuration path of e:\DSC\Configuration. You could pass the PullServerConfigurationPath parameter to each cmdlet that needs it (which could be many), or you could just change the value of the $Script:DSCTools_DefaultPullServerConfigurationPath variable:

# Change the default location where all Pull-Server cmdlets put configuration files
$Script:DSCTools_DefaultPullServerConfigurationPath = 'e:\DSC\Configuration'

# Now the standard commands pick up that new default
Publish-DSCPullResources
Enable-DSCPullServer

Configuring a Node

Configuring a node to use the Pull Server also used to be a more complicated process (not counting the actual creation of the node configuration file):

Copy the configuration MOF file to the Pull Server.
Generate a checksum file for the configuration.
Configure the LCM on the node to pull its configuration from the Pull Server.
Trigger the node to immediately pull it’s DSC configuration from the Pull Server (rather than wait 30 minutes).

This process can now be performed by just two cmdlets:

# Set up NODE01 to pull from the server MYDSCSERVER
# The MOF file is searched for in $Home\Documents\NODE01.MOF (configurable)
Start-DSCPullMode `
    -ComputerName 'NODE01' `
    -PullServerURL 'http://MYDSCSERVER:8080/PSDSCPullServer.svc'

# Force an immediate configuration pull (optional)
Invoke-DSCCheck -ComputerName NODE01

Configuring Lots of Nodes at Once

Once again, this module is all about laziness. So, why configure one node at a time when you can configure lots of them all at once. Many of the cmdlets in this module support a Nodes parameter, which take an array of hash tables. This array of hash tables will contain the definitions of the nodes that need to be set up or have other procedures performed on them (e.g. invoke a configuration check).

For example, to configure seven different nodes for using a DSC Pull Server (with different configurations even) would require the following code:

$Nodes = @(
    @{ Name='NODE01'; Guid='115929a0-61e2-41fb-a9ad-0cdcd66fc2e1'; RebootIfNeeded=$true; MofFile="$PSScriptRoot\Config\NODE01.MOF" },
    @{ Name='NODE02'; Guid='115929a0-61e2-41fb-a9ad-0cdcd66fc2e2'; RebootIfNeeded=$true; MofFile="$PSScriptRoot\Config\NODE02.MOF" },
    @{ Name='NODE03'; Guid='115929a0-61e2-41fb-a9ad-0cdcd66fc2e3'; RebootIfNeeded=$true; MofFile="$PSScriptRoot\Config\NODE03.MOF" },
    @{ Name='NODE04'; Guid='115929a0-61e2-41fb-a9ad-0cdcd66fc2e4'; RebootIfNeeded=$true; MofFile="$PSScriptRoot\Config\NODE04.MOF" },
    @{ Name='NODE05'; Guid='115929a0-61e2-41fb-a9ad-0cdcd66fc2e5'; RebootIfNeeded=$true; MofFile="$PSScriptRoot\Config\NODE05.MOF" },
    @{ Name='NODE06'; Guid='115929a0-61e2-41fb-a9ad-0cdcd66fc2e6'; RebootIfNeeded=$true; MofFile="$PSScriptRoot\Config\NODE06.MOF" },
    @{ Name='NODE07'; Guid='115929a0-61e2-41fb-a9ad-0cdcd66fc2e7'; RebootIfNeeded=$true; MofFile="$PSScriptRoot\Config\NODE07.MOF" }
)

Start-DSCPullMode -Nodes $Nodes -PullServerURL 'http://MYDSCSERVER:8080/PSDSCPullServer.svc'
Invoke-DSCCheck  -Nodes $Nodes

The nodes array could even be populated from a CSV file to make it even easier.

Where to get It

Once the new PowerShell Gallery is available (for public consumption) I’ll upload the module there. But in the mean time it is available on the Microsoft Script Center here:

DSCTools on Script Center

If you want to see some more example files (as well as sample node config files I’ve used in testing the module) you can check out the GitHub repository for the module:

DSCTools on GitHub

The GitHub repository contains additional files not available in the download from the script center including details examples and test configuration files. It is also contained within a Visual Studio 2013 solution so if you’re using VS2013 and the PowerShell VS add-in you can easily load it up.

Future Versions

Over the next few weeks I am planning to update the module so that some of the functions will create the node configuration MOF files by running the actual configuration PS1 files (provided in a nodes array or as parameters). This will eliminate another step when updating any node configuration files.

After creating this module it occurred to me that there might be an even better way to implement a DSC set up: using a basic DSC system XML configuration file that could define all the pull servers and nodes. This XML file could be passed to a cmdlet that could configure all the applicable servers and nodes from the content. So this is the next long-term goal I have for this module.

If anyone out there finds this module useful and has a request for additional features or finds a (shudder) bug, please let me know!

Best New Feature of Windows 10?

2015-04-26T00:00:00Z

Well for me, it has to be the proper standard support for text selection, copy (CTRL+C), cut (CTRL+X) and paste (CTRL+V) in the Command and PowerShell windows. Finally, Microsoft has gotten rid of the completely unintuitive and non-standard method employed for the last 20 or so years. What took them so long?

Although I still don’t like the fact that copying text in a console window de-selects it. This is not standard behaviour anywhere else. Perhaps they’ll fix this too?

Get DSC Configuration from a Remote Host using an SSL Connection

2015-04-23T00:00:00Z

I’ve spent the last day or so working on a module to help with managing DSC Pull Servers and other functions to help making DSC a little bit easier to get up and running. This module isn’t quite finished yet, but I thought I’d share a quick code snippet that I’ve been using a lot to get the DSC configuration from a remote machine when credentials and special port details are required.

Normally, if you want to pull the DSC configuration for a remote computer that won’t require credentials or SSL WSMan then you can just execute:

Get-DSCConfiguration -CimSession 'DSCSVR01'

This cmdlet just pulls the DSC configuration from the remote host using any existing credentials and using HTTP instead of HTTPS.

But if you want to use any alternative connection information – such as forcing the use of SSL WSMan – you need to add some CIM options:

$cimOption  = New-CimSessionOption -UseSsl
$cimSession = New-CimSession -ComputerName 'DSCSVR01' -Credential (Get-Credential) -SessionOption $cimOption

Get-DSCConfiguration -CimSession $cimSession
Remove-CimSession    -CimSession $cimSession

That is about all I’ve got for today. Hope this helps someone.

Clear All Windows Event Logs

2015-04-09T00:00:00Z

Just a quick one this time.

One thing I often like to do on my lab machines (servers and clients) is clear out all event logs. Not just the older style Windows Logs, but the newer Applications and Services Logs as well:

The easiest way I’ve found to do this is just run the following PowerShell command in an Administrator PowerShell console:

Get-WinEvent -ListLog * | ForEach-Object {
    [System.Diagnostics.Eventing.Reader.EventLogSession]::GlobalSession.ClearLog($_.LogName)
}

This will dump the content of every Windows Log and Applications and Services log in one go.

Be aware, this is a one-way ticket - you can’t recover the content of these logs after they’ve been deleted!

So if you’re a bit concerned and want to archive the content before it gets deleted use this command instead:

Get-WinEvent -ListLog * | ForEach-Object {
    [System.Diagnostics.Eventing.Reader.EventLogSession]::GlobalSession.ClearLog(
        $_.LogName,
        "d:\LogArchive\$($_.LogName -replace '/','.').evtx"
    )
}

You’ll want to configure the d:\ArchiveLog to set the path you want the old events saved to. All the events will be saved into this folder with one file for each event log:

Simple as that!

\m/

Multiple VHD/VHDx Optimization using PowerShell Workflows

2015-04-07T00:00:00Z

Like most tech people I have lots of Hyper-V VM’s scattered across various computers at home. Some of these VM’s are running on Server OS hosts (Server 2012 R2) and some running on client OS hosts (Windows 8.1) on my desktop or laptop. These VMs also get varying amount of use - lab and dev machines getting used most of the time while “experimentation” machines getting booted only rarely.

I also like to run my heavily used VMs on fast SSD drives to keep them “snappy”. But like most people I have only a limited amount of SSD space. I’m also quite an obsessive neat freak. These two things combined means I like to keep the VHD/VHDx files used by my VMs as small as possible.

Keeping VHD/VHDx files trim is can easily be performed inside the Hyper-V management tool by clicking the Edit Disk… button and using the Edit Virtual Hard Disk Wizard to Compact the desired VHD/VHDx file:

This of course is all a bit manual and can be really time-consuming when performing this on lots of virtual hard disks. This sounds like it could be performed by a PowerShell command.

After about 5 minutes of investigation I came up with this simple PowerShell command:

Get-VM | Where { $\_.State -eq 'Off' } | Get-VMHardDiskDrive | Optimize-VHD -Mode Full

It basically performs a full optimization on all VHD/VHDx files attached to all Virtual Machines that are in the Off state on the host the command is run on. This does the job quite well but has a few annoyances:

The optimization is performed in series.
Running guests won’t be optimized.
The command only works on VMs on the host the command is run on.

Looking at the list of annoyances it seems a PowerShell workflow might be a good solution. So, after a few hours coding and testing (I can’t tell the number of times my VMs were rebooted and optimized over the day) I managed to complete a module containing a PS Workflow that was up to the task (documentation contained in the link):

Optimize Hyper-V VHDs using PowerShell Workflow

The module can be installed into a PowerShell modules folder (or imported from any location) and the workflow called like in the same way a normal cmdlet would be called:

Optimize-VHDsWorkflow -ComputerName HV-01,HV-02 -VMName NTB01,NTB02,NTB03 -Mode Quick

The above command would optimize all VHD/VHDx files attached to VMs called NTB01, NTB02 or NTB03 on hosts HV-01 and HV-02. It would perform this optimization in parallel meaning all VHDs would be optimized at the same time. Care obviously needs to be taken here, because optimizing too many VHDs running off the same data store could saturate the IO leading to performance of the data store being crippled.

When this workflow is run, any VMs that are running will not have their VHDs optimized. This is probably a good thing for production guests, but for my test lab guest, I want them to be shutdown automatically, have their VHDs optimized and have them automatically started back up. So I implemented a switch called AllowRestart:

Optimize-VHDsWorkflow -AllowRestart -Mode Full -Verbose

The AllowRestart switch will allow a guests that are in a running state to be shut down (not turned off or forced), an optimization performed and then started back up. If a guest is not in a running state it will just be optimized. If the guest can’t be shut down using a normal shut down (because the guest doesn’t have Hyper-V tools running or installed or isn’t running a compatible OS) then it won’t be optimized.

You can also use the Verbose switch to show more information about the workflow process:

There really isn’t much to the process and it could even be scheduled via Task Manager to be performed automatically. If anyone has any comments or feature requests, please let me know - I’m always enjoy a challenge!

\m/ \m/

Using PowerShell to Install/Uninstall Microsoft Office Products by Group Policy

2015-04-06T00:00:00Z

I’ve been recently doing some experimentation with AD RMS templates and AD FS integration in my lab environment. Clearly, I lead a very exciting life. Of course, to test AD RMS templates, one needs a copy of Office installed into the lab environment. This, I thought, would be a good opportunity to configure Office (Office 2013 Pro Plus to be precise) to be installed via Group Policy.

I, of course, read the Deploy Office 2013 by using Group Policy computer startup scripts documentation on TechNet, which directed me to use GPOs that called batch files on computer start-up. This is all simple enough and works well, but being a bit of a PowerShell fiend, I’d prefer to use it whenever I can. Since Windows Server 2008 R2 and above supports PowerShell scripts for GPO startup and shutdown, I thought I’d write some general-purpose PowerShell scripts that could be used in place of the old batch file scripts:

PowerShell Scripts to Install/Uninstall Office 2013 products using GPO

As a side note - I wonder why Microsoft doesn’t support installing Office 2013 products using the standard Software Deployment policies in GPO (assigning, publishing, etc.). But that is a question only Microsoft can answer.

These PowerShell scripts accept parameters that allow key elements of the installation process (product, source folder, config file, log file location) to be specified in the GPO Script parameters themselves:

For example, to install a copy of Microsoft Office 2013 Pro Plus from the \\server\software$\Office15.ProPlus\ share using the file \\server\software$\Office15.ProPlus\ProPlus.ww\SilentInstall.xml to control the configuration, the following parameters could be used:

-ProductId "Office15.ProPlus" -SourcePath "\\server\software$\Office15.ProPlus\" -ConfigFile "\\server\software$\Office15.ProPlus\ProPlus.ww\SilentInstall.xml"

The full script parameters are documented in the PowerShell scripts as well as on the Microsoft Script Repository page along with additional examples. The scripts can also accept a parameter for controlling where a brief log file can be written to so that it is easy to see if the product has been successfully installed on each machine that has been assigned the GPO.

Creating a GPO using the PowerShell Scripts

In Group Policy Management Console, create a new GPO:
Enter a name for the installation policy and click OK:
Right-click on the new policy and select Edit:
Select the Computer Configuration\Policies\Windows Settings\Scripts (Startup/Shutdown) node:
Double-click the Startup item:
Select the PowerShell Scripts tab:
Click Add to add a new startup script:
Click the Browse button to locate the folder where the policy scripts should be stored:
Copy the PowerShell scripts downloaded from the Microsoft Script Repository into this folder and select the one that should be used with this policy.
Enter the PowerShell script parameters into the Script Parameters box:
Click OK to save the PowerShell startup script:
Click OK to save the Startup Properties.
Close the GPME and apply the new GPO.

Important Note about PowerShell Script Parameters

There appears to be a limit to the maximum number of characters supported in the GPO Startup/Shutdown PowerShell script parameters. This limit seems to be 207 characters, but I haven’t been able to confirm this anywhere. Although more than this number of characters can be entered into the Parameter text box, anything above 207 does not get passed through to the script, which either causes the script to run incorrectly or not at all.

If you do encounter this limit but still need additional parameters passed, you could use positional parameters to reduce the overhead or create another script that calls these scripts with the defined parameters.

Hopefully, someone will find this useful. If you have any comments or requests for improvements to the scripts, don’t hesitate to let me know.

Certificate Web Enrollment on a Server and a Misleading Error Message

2015-03-13T00:00:00Z

The Certificate Web Enrollment component of Certificate Services is fairly helpful for allowing easy certificate request and enrollment from any computer.

Requesting a certificate via the Web Enrollment service web page.

It does require Internet Explorer because of an Active X control that runs on the page, but this is acceptable. It also needs to be connected to using HTTPS - which is also fine. Except when it isn’t. Or more accurately, reports that you are not connected via HTTPS when you in fact are.

The following error message appears when connecting to this page from Internet Explorer 11 on a Windows Server 2012 R2 member server or DC:

The page requires an HTTPS connection - but it is connected via HTTPS!

“In order to complete certificate enrollment, the Web site for the CA must be configured to use HTTPS authentication.”

Clearly, Internet Explorer was using HTTPS to communicate with this web site. Clicking OK on the error message and hoping to just ignore it wasn’t possible because most of the drop down boxes on the form were not being populated - preventing it from being submitted:

After clicking OK on the error message the page is broken.

I spent quite some time investigating the cause of this error, including checking the certificate chain that the client was using:

The certificate and certificate chain - nothing wrong here.

Of course if the certificate or the chain was bad then SSL wouldn’t be being used - which it clearly was.

Eventually I identified the cause of the problem. The Active X was being prevented from being run properly because of IE security. This is probably a good thing on a Server operating system, but the error message being presented in this case was very misleading.

The way to fix this is to add the web site to the trusted sites list in Internet Explorer:

Select Internet Options from the Internet Explorer Settings menu:
Select the Security tab and click Trusted Sites:
Click the Sites button.
Enter the https URL of the Certificate Web Enrollment site and click Add:
Click Close.

The web site can now be refreshed and should work correctly at last:

Certificate Web Enrollment form working correctly with SSL.

Hopefully this helps someone out there avoid this annoying problem.

B\M/D

Disk Cleanup and the joys of Features-on-demand

2015-01-28T00:00:00Z

Features-on-demand - it’s a great new “feature” - when it works. However, the rest of the time it is a real headache.

A couple of months ago I decided I wanted to trim down the size of my Windows Server 2012 R2 VM’s. Disk Cleanup (cleanmgr.exe) is one tool that I’ve often found really useful to have on a server install, especially when preparing OS VM images to ensure the install is as lean and clean as possible.

However, by default the tool isn’t installed on Windows Server 2012. To get access to Disk Cleanup on a server OS you need to install the Desktop Experience feature:

Install Desktop Experience using Add Roles and Features Wizard

Because I had used features-on-demand to remove any disabled packages from the system I received a message telling me I might need to specify an alternate location for the source files:

Specify an Alternate Source path

Anyone who has used features-on-demand should be familiar with this:

Desktop Experience Feature-on-demand removed

Because I haven’t got a GPO restricting my servers from downloading updates and packages from Windows Update I thought I wouldn’t have a problem and didn’t need to specify a source.

I was wrong:

Failed to install the Desktop Experience

That is a bit odd - the server has access to Windows Update - it had downloaded updates earlier that day. Other removed features-on-demand features had been installed on this server without an issue, downloading the source files directly from Windows Update. So I was a little puzzled as to why this was different.

No problem, I thought! All that needs to be done is specify a source. In case this is useful, the following TechNet article covers the different ways of specifying a source when installing features that have been removed:

Configure Features on Demand in Windows Server

There are several different sources that can be provided to the Add Roles and Features Wizard:

Specify a WIM file (and index) containing the windows installation files for the OS version that was installed on the server. This is usually a file called install.wim that can be found in the Sources folder of the Windows Server2012R2 installation media.
Install Feature using a WIM source
Specify the windows folder of a working OS install containing the files for this feature. This is usually done by mapping a drive to a share or by mountingaVHD/VHDx file as a drive to the OS.
Install Feature using a shared Windows Folder on a machine with the Desktop Experience feature installed

I tried both of the above methods but neither of them seemed to work for the Desktop Experience feature. The same error occurred every time:

Install-Windowsfeature: The request to add or remove features on the specified server failed.
Installation of one or more roles, role services, or features failed.
The source files could not be downloaded.
Use the “source” option to specify the location of the files that are required to restore the feature. For more
information on specifying a source location, see http://go.microsoft.com/fwlink/?LinkId=243077. Error: 0x800f0906

I also tried installing the feature using PowerShell, using no alternative source, using a WIM source and using a Windows folder source:

Install-WindowsFeature -name Desktop-Experience -IncludeAllSubfeature -Restart -Source z:

But each time I received the same error message:

Install Feature with PowerShell and specifying a valid source - failure.

At this point I had all but given up. Luckily I didn’t. I thought I’d give it one last try - but this time instead of using commands from the PowerShell DISM module I’d use DISM.EXE directly:

DISM /online /enable-feature /featurename:DesktopExperience /all /source:z:\\

Success! DISM worked!

DISM for the WIN! Back of the net!

This screenshot and the one above of the PowerShell install-windowsfeature failing to install the feature are from the same machine with the same source mapped.

Z: drive here was mapped to a share of the c:\windows folder of a server that has the Desktop Experience feature correctly installed.

It looks like DISM may operate in a slightly different way to PowerShell DISM Module and the Add Roles and Features Wizard when it comes to installing features.

So, if Add Roles and Features Wizard and PowerShell Install-WindowsFeature fail, try DISM - it might work!

Additional Notes

I have also run into the same problem when installing the AD DS feature on a different server - so I don’t think this is specific to the machine or the feature. It has also occurred on machines that I want to convert from a core install to a gui install.

I have tried installing the feature on a clean install of the OS and it works fine - but as soon as all the latest windows hotfixes for the OS are installed from Windows Update the feature can no longer be installed (if it has been removed).

From my investigation, many other people have experienced this problem with varying degrees of success in solving it. Some have said that patching the WIM file with all the latest hotfixes worked for them - but it didn’t for me (but it did inspire me to write a PowerShell module to ease the WIM patch process - more on this in another post). I was certainly in the “tearing my hair out” group until I randomly tried this.

Also, I did try DISM without specifying a source as well, and it failed with the same error code as the PowerShell did:

Install Feature using DISM fails with no Source specified.

At first I actually thought the soution was using DISM with the /LimitAccess switch to prevent DISM from using the internet to download the packages, but after further tests it doesn’t seem to make any difference - DISM works with and without this switch. The equivelent to the /LimitAccess switch also doesn’t appear to be available in the PowerShell Install-WindowsFeature cmdlet.

Tiered Storage Spaces Experimentation with PowerShell

2015-01-11T00:00:00Z

I’ve been recently experimenting with the interesting Tiered Storage Spaces features in Windows Server 2012 R2. As part of this, I’ve been going through some of the excellent MSFT File Server team blogs on TechNet.

This one in particular by Jose Barreto was supremely useful, and I highly recommend it. It contains some excellent PowerShell code for setting up a VM for experimenting with Tiered Storage Spaces as well as configuring them within the VM.

I did find the PowerShell code fragmented throughout the long article, which made it a little bit hard to use (lots of ALT+TAB, CTRL+C, and CTRL+V). So I’ve assembled the code into two code snippets that you should be able to paste into a PowerShell ISE script window and then step through—it should save a lot of repetitive keypressing.

I also modified it slightly to have some of the configuration items, such as where to put the VM HDD and SSD VHDX files, in variables at the beginning of the script to make it easier to manage.

Script to Run on the Host OS

This script will install Hyper-V, create the VM, create the test VHDXs, and start it up:

# --------------------------------------------------------------------------
# Execute on HOST OS
# --------------------------------------------------------------------------

# Configure these paths and names
$SSD_VHD_Path = "F:\VM\VHD"  # Path to store the VM SSDs
$HDD_VHD_Path = "E:\VM\VHD"  # Path to store the VM HDDs
$VMName = 'Windows Server 2012'  # Name of the VM to create/use
$VM_OS_Path = "F:\VM\OS\Windows Server 2012.VHDX"  # Path to the VM OS disk (if creating a VM)

# Install required roles and features, restart at the end
# If Hyper-V is already installed, comment out this line:
Install-WindowsFeature Hyper-V -IncludeManagementTools -Restart

# Create 4 VHDX files on the SSD with 10GB each
1..4 | ForEach-Object { New-VHD -Path "$SSD_VHD_Path\VMA_SSD_$_.VHDX" -Dynamic -SizeBytes 10GB }

# Create 8 VHDX files on the HDD with 30GB each
1..8 | ForEach-Object { New-VHD -Path "$HDD_VHD_Path\VMA_HDD_$_.VHDX" -Dynamic -SizeBytes 30GB }

# Create a new VM. Assumes you have a Windows Server 2012 R2 OS VHDX in place
# If you already have a VM, comment out this line:
New-VM -Name $VMName -Path "D:\VMS" -VHDPath $VM_OS_Path -MemoryStartupBytes 2GB

# Add all data disks to the VM
1..4 | ForEach-Object { Add-VMHardDiskDrive -VMName $VMName -ControllerType SCSI -Path "$SSD_VHD_Path\VMA_SSD_$_.VHDX" }
1..8 | ForEach-Object { Add-VMHardDiskDrive -VMName $VMName -ControllerType SCSI -Path "$HDD_VHD_Path\VMA_HDD_$_.VHDX" }

# Start the VM
Start-VM $VMName
Get-VM $VMName | Get-VMHardDiskDrive

Script to Run on the Guest OS

Once you’ve got your VM up and running, you can execute the following script on it to experiment with the actual Tiered Storage Spaces:

# --------------------------------------------------------------------------
# Execute on GUEST OS
# --------------------------------------------------------------------------

# Display physical disks
Get-PhysicalDisk | Sort-Object Size | Format-Table DeviceId, FriendlyName, CanPool, Size, MediaType -AutoSize

# Create a storage pool
$s = Get-StorageSubSystem
New-StoragePool -StorageSubSystemId $s.UniqueId -FriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk -CanPool $true)

# Configure media type for virtual SAS disks
Get-StoragePool Pool1 | Get-PhysicalDisk | Where-Object Size -lt 20GB | Set-PhysicalDisk -MediaType SSD
Get-StoragePool Pool1 | Get-PhysicalDisk | Where-Object Size -gt 20GB | Set-PhysicalDisk -MediaType HDD

# Create storage tiers
Get-StoragePool Pool1 | New-StorageTier -FriendlyName SSDTier -MediaType SSD
Get-StoragePool Pool1 | New-StorageTier -FriendlyName HDDTier -MediaType HDD

# Create virtual disks with tiering
$SSD = Get-StorageTier -FriendlyName SSDTier
$HDD = Get-StorageTier -FriendlyName HDDTier
Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Simple -StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB
Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space2 -ResiliencySettingName Mirror -StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB

# Initialize and format the virtual disks
Get-VirtualDisk Space1 | Get-Disk | Initialize-Disk -PartitionStyle GPT
Get-VirtualDisk Space1 | Get-Disk | New-Partition -DriveLetter F -UseMaximumSize
Initialize-Volume -DriveLetter F -FileSystem NTFS -Confirm:$false

Get-VirtualDisk Space2 | Get-Disk | Initialize-Disk -PartitionStyle GPT
Get-VirtualDisk Space2 | Get-Disk | New-Partition -DriveLetter G -UseMaximumSize
Initialize-Volume -DriveLetter G -FileSystem NTFS -Confirm:$false

Hopefully, this saves someone out there a little time copying and pasting!

Cleaning Up Windows SxS Folder in Preparation for Imaging

2014-12-08T00:00:00Z

In preparation for releasing an operating system image to be used as a VM template, I usually like to perform some “shrinking commands” to make sure the image has as small a footprint as possible.

All the “shrinking commands” are command-line or PowerShell-based because they must also be able to be performed on a Windows Server Core installation.

Remove all Uninstalled Feature Binaries

The first thing I usually do is remove any uninstalled feature binaries (as part of Windows Features on Demand). I’ve covered this in an earlier article here.

Remove Old Service Pack Backup Files

The following command removes any files that were backed up as part of installing a service pack. If you execute this command, you won’t be able to uninstall any service packs.

DISM /online /cleanup-image /SPSuperseded

DISM /online /cleanup-image /SPSuperseded

Remove Superseded Components

This command removes any components in the Windows SxS folder that have been superseded by newer versions.

For Windows Server 2008/2008 R2/2012 and Windows 7/8:

DISM /online /cleanup-image /StartComponentCleanup

For Windows Server 2012 R2 and Windows 8.1 (performs some additional optimization):

DISM /online /cleanup-image /StartComponentCleanup /ResetBase

DISM /online /cleanup-image /StartComponentCleanup /ResetBase

Remove all Uninstalled Feature Binaries

2014-12-05T00:00:00Z

I’m still getting my head around this whole blog thing, but I’m going to write a quick article about trimming down a Windows Server 2012 install while I wait for my partner to recover from a small bout of vertigo.

The “Features on Demand” feature of Windows Server 2012 is really great at trimming down a Windows Server 2012 installation.

To use the “Features on Demand,” you need to “remove” the binaries required by any features that aren’t currently installed by your OS. You can remove them one by one, but this is a real drag.

The following command will remove the binaries for any non-installed features on your system:

# filepath: d:\source\GitHub\PlagueHO\plagueho.github.io\src\posts\2014\12\2014-12-05-remove-all-uninstalled-feature-binaries.md
Get-WindowsFeature | Where-Object -FilterScript { $_.InstallState -eq 'Available' } | Remove-WindowsFeature -Remove

Features on Demand Removal Progress

Features on Demand Removal Complete

After you’ve run this command, you’ll need to have your Windows install source available if you want to install any additional features. You might also need to specify an alternate source location when installing any future features. For more information on installing Windows features from an alternate source, see this article on TechNet:

Install or Uninstall Roles, Role Services, or Features

I usually like to run this command when I’m preparing a new VM template after I’ve installed all the features I think will be needed for this VM. This results in a smaller deployment footprint.

Installing Windows Server 2012 R2 Core - Additional Tools and Scripts

2014-12-03T00:00:00Z

Although Windows Server Core is a great way of ending up with a slim and trim diet server, it can be a little bit tricky when first getting started configuring it. During my experiments running Server Core VMs, I found that there were a few tools either built into Server Core or available separately that can help get over this configuration “hump.”

Built-in Tools

SConfig.exe

SConfig.exe is a built-in command-line tool (with a simple command-line GUI) that allows you to perform some simple configuration tasks on your core installation.

Some of the functions include:

Renaming the computer
Joining a domain or workgroup
Enabling remote management (Win-RM)
Enabling remote desktop
Installing Windows updates
Configuring network settings
Changing system date/time
Shutting down/rebooting the server

Other Tools

When installing a new copy of one of the Windows Server Core versions, it’s quite useful to install several additional tools that will help manage the server from a command prompt and/or PowerShell console.

Corefig for Windows Server 2012 Core and Hyper-V Server 2012

Corefig allows you to configure some of the main settings of a Windows Server Core installation as well as installing updates and Windows features and roles.

Windows Update PowerShell Module

The Windows Update PowerShell module allows you to install Windows updates from a PowerShell command line by executing (after installing the module onto the server):

Get-WUInstall

Remote Server Administration Tools PowerShell Module

The Remote Server Administration Tools PowerShell module is available as an installable feature on Windows Server Core edition. It needs to be first installed by executing the following command at a PowerShell prompt:

Add-WindowsFeature -Name 'RSAT-AD-PowerShell' -IncludeAllSubFeature

PowerShell Community Extensions

The PowerShell Community Extensions project provides various useful PowerShell cmdlets. I always install this onto the server and copy the entire PSCX modules folder:

C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\

…into the system modules folder:

C:\Program Files\WindowsPowerShell\Modules

This makes the PSCX module available via the Import-Module command in PowerShell.

What's All This Then?

2014-11-21T00:00:00Z

This blog is where I will attempt to document my adventures in coding.