Issue with using Windows Server 2012 RRAS as a local VPN device with Microsoft Azure when WAN interface has DHCP assigned IP address

In my personal lab I use a Windows Server 2012 R2 VM running on a local Hyper-V server as my VPN device to connect to a Microsoft Azure Virtual Network gateway. This setup has been working fine for quite some time, and is indeed a supported configuration. Recently though I could not make it connect. Come hear the story…

My ISP (Telenor) here in Norway is gracious enough to provide me with 2 dynamically assigned official IPv4 addresses. I use one for my home network and the other for the Azure gateway. I hadn’t used the lab for some time so the S2S VPN had been down. When I tried to reconnect it wouldn’t work.I knew Microsoft had made some changes recently, these were announced during Build 2014, so I figured I would reset the RRAS configuration and run the VPN setup from scratch. So I disabled the RRAS service and downloaded a fresh VpnDeviceScript.cfg file from the portal an redid everything. This didn’t take more than a couple of minutes, but still I could not connect. I noticed that when the RRAS service started I lost all connectivity on my WAN (Internet) interface. I could ping an address on the Interne, but the ping immediately stopped when RRAS started. I fired up Wireshark and did a trace of the WAN interface during RRAS startup and discovered that when the service starts several DHCP Discover messages were sent out on the WAN interface:

image

Since my ISP only allows 2 IPs this made the original address I had received on the WAN interface invalid. But why did it happen?

Also during RRAS startup I got this warning in the System log:

image

When RRAS is running as a remote access service, which it does when providing Site-2-Site VPN tunneling, it also supports incoming VPN requests from clients. These clients need an IP address to be able to communicate on the local network. They get this address either from the RRAS server itself and a static IP pool, or from a DHCP server on the local network where the RRAS reserves IP addresses in blocks of 10. In my case I had DHCP selected and thus no static IP pool configured. If the RRAS server is multi-homed, which it usually is, you can select which interface RRAS looks for a local DHCP server on.This settings is configured through the RRAS server properties:

image

The default setting is Allow RAS to select adapter. This was the cause of my problem. The RRAS service selected my WAN interface and send out a bunch of DHCP Discover messages to allocated IP addresses for incoming clients. This invalidated my existing DHCP lease and stopped all communication on the Internet. Once I selected the internal (LAN) Interface I could connect fine.

I guess this problem can occur in any S2S or RAS scenario where the WAN IP address is dynamically assigned.

Posted in Uncategorized | Leave a comment

Debugging unattended domain join on Windows Azure VMs

Introduction

One great thing about Windows Azure PowerShell is the ability to join a VM to an Active Directory domain during provisioning, this ability is not available in the portal. Joining a domain during Windows setup is nothing new, and it is accomplished by using the normal Windows unattended setup mechanism; namely unattend.xml. In unattend.xml you can specify the following information about the domain you want to join and the account that has permissions to perform the join operation in the directory (excerpt has been edited for readability):

<component name=”Microsoft-Windows-UnattendedJoin” processorArchitecture=”amd64″>
<Identification>
  <Credentials>
   <Domain><NetBIOS domain name></Domain>
   <Username><NT type (samaccountname) username></Username>
   <Password><password in encrypted form></Password>
  </Credentials>
  <JoinDomain><FQDN of domain to join></JoinDomain>
  <MachineObjectOU><DN of OU to place machine account in></MachineObjectOU>
</Identification>
</component>

You usually don’t create the unattend.xml file manually, but rather use a tool like Windows System Image Manager from the Windows Assessment and Deployment Kit (ADK).

I recently had an issue where none of my new VMs would join the domain during provisioning, leaving them all in a workgroup. That led me to compile the following during my debugging.

Add-WindowsProvisioningConfig

The cmdlet that enabled a Windows Azure VM to join a domain is called Add-WindowsProvisioningConfig. If used with the –WindowsDomain parameter it lets you specify these additional parameters:

Parameter Info
JoinDomain FQDN of domain to join
Domain NetBIOS name of account with permission to join computers to domain specified in the JoinDomain parameter. This is usually always the NetBIOS name of the domain from JoinDomin, but could be from another domain in the forest or a trusted domain.
DomainUserName NT-style username (samaccountname) of account with permissions to join the domain specified in JoinDomain
DomainPassword Password of the account with permissions to join the domain specified in JoinDomain
MachineObjectOU DN of OU where the computer account of the VM should be placed

This info is loaded into the unattend.xml file used to setup the new VM. After that the unattended setup process continues  as with any regular Windows install.

Logs

If domain join does not work there are several logs that should be examined.

Operation logs in the portal

Under Management Services in the Windows Azure portal you find the Operation Logs.

image

The provisioning of a new VM is an AddRole operation, first one with Status Started and then another one with Status Succeeded. By examining the details of these you can see what is passed to the back end. This is particularly useful if you are using scripts which pass variables to the domain join parameters. Here you see exactly what is passed. Note that the password is omitted from the logs, that is not an error.

Windows Setup logs

After the portal has done its job the rest of the provisioning is left to the normal Windows unattended setup process and is handled entirely within the VM. That means that any normal troubleshooting techniques for unattended Windows setup applies. The working folder of Windows Setup is C:\Windows\Panther. Here you will find the unattend.xml file used during setup (if is has not been deleted by setup itself, Azure does not do this), and the logs for the entire setup process:

File Info
unattend.xml The answer file for unattended Windows setup
setuperr.log Any major errors encountered during setup
setupact.log All setup activity

The first thing you should do is check unattend.xml and verify that it contains the correct info.

Domain join during unattended setup is done with the djoin.exe executable, so to debug domin join we need to search for that in setupact.log. Also searching for the words Warning or Error could reveal useful information. This is an excerpt from the setupact.log with the error I encountered:

2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Begin
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Loading input parameters…
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: AccountData = [NULL]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: UnsecureJoin = [NULL]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: MachinePassword = [secret not logged]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: JoinDomain = [corp.mydomain.com]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: JoinWorkgroup = [NULL]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Domain = [corp]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Username = [Administrator]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Password = [secret not logged]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: MachineObjectOU = [CN=Computers,DC=corp,DC=mydomain,DC=com]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: DebugJoin = [NULL]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: DebugJoinOnlyOnThisError = [NULL]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: TimeoutPeriodInMinutes = [NULL]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Checking that auto start services have started.
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Calling DsGetDcName for corp.mydomain.com…
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: DsGetDcName returned [dc2.corp.mydomain.com]
2014-03-11 14:09:45, Info     [DJOIN.EXE] Unattended Join: Constructed domain parameter [corp.mydomain.com\DC2.corp.mydomain.com]
2014-03-11 14:09:45, Warning     [DJOIN.EXE] Unattended Join: NetJoinDomain attempt failed: 0×2, will retry in 10 seconds…
2014-03-11 22:09:56, Warning     [DJOIN.EXE] Unattended Join: NetJoinDomain attempt failed: 0×2, will retry in 10 seconds…

As you can see from this case, the NetJoinDomain function fails and is eventually abandoned. You can also find these errors in the System log:

ID: 4097
Source: NetJoin
Info: The machine SERVER1 attempted to join the domain corp.mydomain.com\DC2.corp.mydomain.com but failed. The error code was 2.

To find out more about this we need to look at another log; C:\Windows\debug\NetSetup.log. This file contains information for alle domain join operations performed on a machine, not just the ones during setup. Here is a section from mine:

03/11/2014 22:24:44:407 NetpDoDomainJoin
03/11/2014 22:24:44:407 NetpMachineValidToJoin: ‘SERVER1′
03/11/2014 22:24:44:407     OS Version: 6.2
03/11/2014 22:24:44:407     Build number: 9200 (9200.win8_gdr.130531-1504)
03/11/2014 22:24:44:407     SKU: Windows Server 2012 Datacenter
03/11/2014 22:24:44:407     Architecture: 64-bit (AMD64)
03/11/2014 22:24:44:407 NetpDomainJoinLicensingCheck: ulLicenseValue=1, Status: 0×0
03/11/2014 22:24:44:407 NetpGetLsaPrimaryDomain: status: 0×0
03/11/2014 22:24:44:407 NetpMachineValidToJoin: status: 0×0
03/11/2014 22:24:44:407 NetpJoinDomain
03/11/2014 22:24:44:407     Machine: SERVER1
03/11/2014 22:24:44:407     Domain: corp.mydomain.com\DC2.corp.mydomain.com
03/11/2014 22:24:44:407     MachineAccountOU: CN=Computers,DC=corp,DC=mydomain,DC=com
03/11/2014 22:24:44:407     Account: corp\Administrator
03/11/2014 22:24:44:407     Options: 0×23
03/11/2014 22:24:44:407 NetpLoadParameters: loading registry parameters…
03/11/2014 22:24:44:407 NetpLoadParameters: DNSNameResolutionRequired not found, defaulting to ’1′ 0×2
03/11/2014 22:24:44:407 NetpLoadParameters: DomainCompatibilityMode not found, defaulting to ’0′ 0×2
03/11/2014 22:24:44:407 NetpLoadParameters: status: 0×2
03/11/2014 22:24:44:407 NetpDisableIDNEncoding: no domain dns available – IDN encoding will NOT be disabled
03/11/2014 22:24:44:407 NetpJoinDomainOnDs: NetpDisableIDNEncoding returned: 0×0
03/11/2014 22:24:44:407 NetpJoinDomainOnDs: status of connecting to dc ‘\\DC2.corp.mydomain.com’: 0×0
03/11/2014 22:24:44:407 NetpJoinDomainOnDs: Passed DC ‘DC2.corp.mydomain.com’ verified as DNS name ‘\\DC2.corp.mydomain.com’
03/11/2014 22:24:44:407 NetpLoadParameters: loading registry parameters…
03/11/2014 22:24:44:407 NetpLoadParameters: DNSNameResolutionRequired not found, defaulting to ’1′ 0×2
03/11/2014 22:24:44:407 NetpLoadParameters: DomainCompatibilityMode not found, defaulting to ’0′ 0×2
03/11/2014 22:24:44:407 NetpLoadParameters: status: 0×2
03/11/2014 22:24:44:407 NetpDsGetDcName: status of verifying DNS A record name resolution for ‘DC2.corp.mydomain.com’: 0×0
03/11/2014 22:24:44:407 NetpProvisionComputerAccount:
03/11/2014 22:24:44:407     lpDomain: corp.mydomain.com
03/11/2014 22:24:44:407     lpMachineName: SERVER1
03/11/2014 22:24:44:407     lpMachineAccountOU: CN=Computers,DC=corp,DC=mydomain,DC=com
03/11/2014 22:24:44:407     lpDcName: DC2.corp.mydomain.com
03/11/2014 22:24:44:407     lpDnsHostName: (NULL)
03/11/2014 22:24:44:407     lpMachinePassword: (null)
03/11/2014 22:24:44:407     lpAccount: corp\Administrator
03/11/2014 22:24:44:407     lpPassword: (non-null)
03/11/2014 22:24:44:407     dwJoinOptions: 0×23
03/11/2014 22:24:44:407     dwOptions: 0×40000003
03/11/2014 22:24:44:407 NetpLdapBind: Verified minimum encryption strength on DC2.corp.mydomain.com: 0×0
03/11/2014 22:24:44:407 NetpLdapGetLsaPrimaryDomain: reading domain data
03/11/2014 22:24:44:407 NetpGetNCData: Reading NC data
03/11/2014 22:24:44:407 NetpGetDomainData: Lookup domain data for: DC=corp,DC=mydomain,DC=com
03/11/2014 22:24:44:407 NetpGetDomainData: Lookup crossref data for: CN=Partitions,CN=Configuration,DC=corp,DC=mydomain,DC=com
03/11/2014 22:24:44:423 NetpLdapGetLsaPrimaryDomain: result of retrieving domain data: 0×0
03/11/2014 22:24:44:423 NetpCheckForDomainSIDCollision: returning 0×0(0).
03/11/2014 22:24:44:423 NetpGetDnsHostName: PrimaryDnsSuffix defaulted to DNS domain name: corp.mydomain.com
03/11/2014 22:24:44:423 NetpGetComputerObjectDn: Cracking DNS domain name corp.mydomain.com/ into Netbios on \\DC2.corp.mydomain.com
03/11/2014 22:24:44:423 NetpGetComputerObjectDn: Crack results:     name = corp\
03/11/2014 22:24:44:423 NetpGetComputerObjectDn: Cracking account name corp\SERVER1$ on \\DC2.corp.mydomain.com
03/11/2014 22:24:44:423 NetpGetComputerObjectDn: Crack results:     Account does not exist
03/11/2014 22:24:44:423 NetpGetComputerObjectDn: Specified path ‘CN=Computers,DC=corp,DC=mydomain,DC=com’ is not an OU
03/11/2014 22:24:44:423 NetpCreateComputerObjectInDs: NetpGetComputerObjectDn failed: 0×2
03/11/2014 22:24:44:423 NetpProvisionComputerAccount: LDAP creation failed: 0×2
03/11/2014 22:24:44:423 NetpProvisionComputerAccount: Cannot retry downlevel, specifying OU is not supported
03/11/2014 22:24:44:423 ldap_unbind status: 0×0
03/11/2014 22:24:44:423 NetpJoinCreatePackagePart: status:0×2.
03/11/2014 22:24:44:423 NetpAddProvisioningPackagePart: status:0×2.
03/11/2014 22:24:44:423 NetpJoinDomainOnDs: Function exits with status of: 0×2
03/11/2014 22:24:44:423 NetpJoinDomainOnDs: status of disconnecting from ‘\\DC2.corp.mydomain.com’: 0×0
03/11/2014 22:24:44:423 NetpJoinDomainOnDs: NetpResetIDNEncoding on ‘(null)’: 0×0
03/11/2014 22:24:44:423 NetpDoDomainJoin: status: 0×2

I have highlighted my particular error. As you can see I specified the DN of the computers container as the location of the new computer object. Since Computers is in fact a container and not an OU, the NetpCreateComputerObjectInDs function fails. After I specified a bona fide OU in my Add-WindowsAzureProvisioningConfig cmdlet the VM was successfully joined to my domain. If you actually want the computer object to be placed in the Computers container just omit the MachineObjectOU parameter from Add-WindowsAzureProvisioningConfig. Since the Computers container is the default location for new computer objects, unless redirected by the admin or overridden, your VMs account will end up there.

Posted in Uncategorized | Leave a comment

Will it run? How to run your favorite legacy OS in a Windows Azure VM

Introduction

Old OSs are very popular, at least judging by the number of, for want of a better word, “legacy” servers I find in customer data centers. The old workhorses Windows Server 2003/2003 R2, Windows 2000 and even good old Windows NT pop up in disturbing numbers. With them, the question “Will Windows X, run in a Windows Azure VM?” often follows. To be able to answer that question we first need to talk about “run” vs. “supported”. A lot of OSs can be made to run in a Windows Azure VM, but that does not mean that Microsoft will support that OS. Unfortunately a lot of people are only interested in the “run” part of the equation, ignoring any problems of supportability down the road. Support is very straight forward; the oldest supported Windows OS running in a Windows Azure VM is Windows Server 2008 R2 x64, as stated in KB2721672: Microsoft server software support for Windows Azure Virtual Machines. So if you want support the buck stops there. But what can we make to run if we throw supportability to the wind…

It’s still Hyper-V

All the physical servers in a Windows Azure data center run a modified version of Windows Server with the Hyper-V role installed. So basically all the capabilities of Hyper-V should be available in Windows Azure, unless they have been specifically blocked. The About Virtual Machines and Guest Operating Systems page lists all the guest operating systems that are supported on Hyper-V, so lets assume that we can run all of those. I’m not going to test all the OSs on that list, but rather focus on some specific golden oldies (all 32-bit editions):

  • Windows 2000
  • Windows XP
  • Windows Server 2003

There is also a lot of rumor on the Internet concerning support for 32-bit (x86) VMs in Windows Azure. Although all the images in the gallery and the fact that the earliest supported server OS (Windows Server 2008 R2) is only available in 64-bit (x64), this is never actually stated anywhere officially. (At least not that I have found.)

Another area where there is a lot of talk is concerning client operating systems in Windows Azure VMs. In this case Microsoft is very clear; client OSs are not supported and it is a violation of the license to install and run one in a Windows Azure VM. For the sake of scientific discovery I will ignore that issue for this post.

So lets see what’s what…

Basic requirements

I’m going to work from these premises during my testing:

  • The legacy OS needs to be on the supported list for Hyper-V
  • Hyper-V Integration Components need to be available
    Either on the vmguest.iso image, already in the OS or available for download.
  • Remote Desktop for Windows VMs and SSH for Linux VMs must be available
  • Only VHD images are supported in Windows Azure, not VHDX
  • None of the selected legacy OSs will support being generalized in a way that Windows Azure can use, so we can only upload OS disks, not images. (Windows Azure uses the setup technology introduced in Windows Vista, codenamed Panther, which legacy OSs don’t.)

Basic steps

Here is what we need to do (at least):

  1. Use a Hyper-V host running Windows Server 2012 R2
    This probably not a requirement, but the latest and greatest is always nice.
  2. Install a VM with the legacy OS, must select Generation 1 and the VHD disk format.
  3. Once OS install completes, install the Hyper-V integration components
  4. Update the OS through e.g. Microsoft Update
    Having the latest updates will maximize the chance of success
  5. If you want remote PowerShell or WinRM make sure to install those optional updates.
  6. Enable Remote Desktop and make sure the Remote Desktop exception is enabled in the Windows Firewall if it is enabled.
  7. If you are using WinRM technology configure that, preferably with HTTPS instead of HTTP, and enable the appropriate firewall exceptions.
  8. If you are actually trying to run a production server with a legacy OS in Windows Azure (something you should not do), remove any special drivers or low level software that might break the VM. You can always install this alter.
  9. Shut down the VM on your Hyper-V Server
  10. Upload the VHD to a Windows Azure storage account (Add-AzureVHD)
  11. Register the new blob as an OS image (Add-AzureDisk)
  12. Start the VM in Azure and log on (hopefully).

The results

Using the above method I was able to obtain these results:

OS Runs Notes
Windows 2000 Yes
  • Professional, Server and Advanced Server all share the same code base and all work
  • Windows Server 2012 removed Windows 2000 support from the Integration components so you have to use integrations components from Windows Server 2008 R2.
  • Performance was not great on my VM…
Windows XP x86 Yes  
Windows Server 2003 x86 Yes Windows Server 2003 R2 is pretty much the same code base so it should run fine too. And in fact it does.

I assume that Windows Server 2003/2003R2 and XP x64 will also work.

Feel free to try some golden oldies yourself and use the comments for your results. Good luck.

Posted in Uncategorized | Leave a comment

Microsoft Product Use Rights (PUR) document update; great news for Windows Azure and hosters

What is Product use Rights (PUR)?

from microsoft.com:

“When you purchase a software license through a Microsoft Volume Licensing program, the terms and conditions for how you can use the software are defined in the Volume Licensing Product Use Rights (PUR) document, Product List document, and program agreement. The PUR is updated quarterly.”

The change

Effective January 1 2014 Microsoft made a pretty significant change to the Software Assurance benefit. This is from the Windows Server 2012 R2 Remote Desktop Service Licensing Data Sheet:

RDS User CALs Extended Rights through Software Assurance
Today, RDS CALs permit remote access to the Windows Server GUI (Graphical User Interface) running on a customer’s on-premise server and RDS SALs (Subscriber Access License) if running on a shared-server environment. Effective January 1 2014, RDS User CALs will have Extended Rights through Software Assurance. In addition to the on-premise access, RDS User CAL customers will also be able to access the Windows Server GUI running on Windows Azure or on a third party’s shared server, without acquiring a separate RDS SAL.

To leverage this benefit customers should meet following requirements:

  • Maintain Software Assurance coverage on the RDS User CALs
  • Use dedicated VOSE (Virtual Operating System Environment) in Windows Azure or third party’s
    shared servers
  • Access Windows Server session-based desktops and/or applications running on shared server
    environments
  • Limit access by internal users only i.e. by company employees, vendors and contractors and not by
    external users such as customers
  • Assign each on-premise RDS User CAL to the same named user on Windows Azure or a third party’s
    shared servers
    This RDS User CAL Software Assurance benefit allows each User to access RDS functionality only on one shared server environment (i.e. Windows Azure or a third party server) in addition to access the respective on premise servers. The customer must acquire extra RDS SALs (Subscriber Access License) if the same User needs to access RDS functionality on additional shared server environments.

This is indeed great news for all customers wanting to access a remote Windows Server Desktop in Windows Azure or at another third party hoster.

More info:

Posted in Uncategorized | Leave a comment

Quickly stop and start your Windows Azure lab

Introduction

I needed  way to easily stop and start my different lab setups in Windows Azure. I don’t want to keep running, and pay for, a set of VMs I use maybe once a month. So here is a PowerShell script to stop and start a set of Azure VMs. One important dimension here is the order in which the VMs start (and possibly; shuts down, depending on your setup). Since all IP addresses are dynamically allocated in Windows Azure I had to make sure that my VMs started in a specific order and that the script executed synchronously. That way whatever IP they had when they were provisioned would most likely be assigned to them when they started. Therefore the script includes code to start and stop VMs in the order you specify them in the input file. It also tries to wait until the current VM is fully started or stopped before proceeding with the next one.

Pre-requisites, input file and usage

The input file is really simple. It is a text file with a Cloud Service name and a VM name on each line, separated by a semi-colon, no header:

<cs name>;<vm name>

The VMs in the file will be started in the order they are listed and stopped in the reverse order!

By default, the script looks for a txt file in its execution directory called Control-AzureVMs.txt. If it is not found PowerShell throws an error. You can override this behavior by specifying your own file with VMs to either start or stop.

You must already have your machine configured to use Windows Azure PowerShell and specify your subscription in the script:

Select-AzureSubscription “<your subscription name here>”

Whether you want to start or stop a set of VMs is controlled by a script parameter. You use either Stop or Start. Optionally you can add your own txt file with VMs after the Start/Stop parameter.

Usage with the default input file:

Control-AzureVMs.ps1 Stop

Control-AzureVMs.ps1 Start

Usage with the custom input file:

Control-AzureVMs.ps1 Stop MySetOfVms.txt

The code

Here is the code. I can think of all sorts of improvements, but I needed this quickly so that will have to wait. Error checking is pretty much non-existent at this point so use at your own risk. I accept no responsibility whatsoever.

UPDATE: With the release of the Windows Azure PowerShell module v0.7.3 you can now create DHCP reservations for your VMs that will persist when the machine is deallocated. Check out Get-AzureStaticVNetIP, Set-AzureStaticVNetIP, Remove-AzureStaticVNetIP and Test-AzureStaticVNetIP.

Posted in Uncategorized | Leave a comment

NIC 2014 Slide deck

Here is my slide deck from the Nordic Infrastructure Conference (NIC) 2014. My talk was called Modern authentication for the Cloud Era and covered claims based authentication and some common scenarios, OAuth and OpenID Connect. Thanks to everyone who attended my session. Hope to see you there next year!

http://www.slideshare.net/MorganSimonsen/nic-2014-modern-authentication-for-the-cloud-era

Posted in Uncategorized | Leave a comment

Bug in Windows Azure PowerShell module v 0.7.0

The latest version of the Windows Azure PowerShell module v 0.7.0 (released 21-10-2013), which features such great improvements as Windows Azure Active Directory logon support, unfortunately has a bug relating to endpoint ACLs.

Any Set-AzureEndpoint or Add-AzureEndpoint command which includes the ACL parameter will seem to succeed, but no ACL changes will be applied to the endpoint. I reverted to the previous release, v 0.6.16.1, and it works correctly.

I submitted an issue on GitHub, hopefully it will be resolved soon.

Posted in Uncategorized | Leave a comment

Troubleshooting ISO file sharing in System Center Virtual Machine Manager 2012 SP1

So I was trying to use sharing of ISO files instead of copying when mounting on virtual machines. The really helpful dialog box in System Center Virtual Machine Manager (SCVMM) 2012 SP1 politely informs you that

image

OK, so what configuration?

I found these steps for VMM 2008: How to Enable Shared ISO Images for Hyper-V Virtual Machines in VMM

In short this is what you have to do:

  • Use a domain account as the VMM service account
  • Grant share and NTFS Read permissions to the VMM library shares that contain ISO files for both the VMM service account and the computer accounts of any Hyper-V hosts that need to access those ISO files
  • Configure constrained delegation for each Hyper-V host to allow delegation of credentials to the VMM library servers
  • Add the Hyper-V hosts to the Windows Autorization Access builtin group

If you use a group to grant access to you Hyper-V hosts to the library share, like I did you have to restart them to update their security tokens. Configuring constrained delegation also requires a restart.

But it wasn’t working! VMM just told med “Update failed” each time I tried to mount an ISO on a VM. This is the error I got:

Error (12700)
VMM cannot complete the host operation on the <Hyper-V host> server because of the error: ‘<VM>’ failed to add device ‘Virtual CD/DVD Disk’. (Virtual machine ID <GUID>)

‘<VM>’: User Account does not have sufficient privilege to open attachment ‘\\<VMM Library server>\MSSCVMMLibrary\ISOs\<ISO file>’. Error: ‘General access denied error’ (0×80070005). (Virtual machine ID <GUID>)
Unknown error (0×8001)

Recommended Action
Resolve the host issue and then try the operation again.

The Internet didn’t know why it didn’t work either. I know because I asked it!

As far as I cloud see my permissions were correct so I had to figure out if there was another account involved. The best way I know to do that is by auditing object access. So first I enabled auditing of object access on my VMM Library server:

image

Next, to catch all security principals trying to access the files in the VMM Library, I added audit System Access Control List entries for both Authenticated Users and Everyone:

image

After setting this up I tried mounting the ISO again and then opened the Security Event Log. The first event of interest was a logon event (Event ID 4624):

image

The account listed here is the Run As account I use for Hyper-V host management. This was an Audit Success event since the account was allowed to log on to the VMM library server.

Next was an attempt to access a share (Event ID 5140):

image

Again it was the RunAs account this time trying to access the VMM Library share with read permissions (ReadData or ListDirectory). Again, this was an Audit Success event since this was just a request for access. This was followed by something called a detailed file share (Event ID 5145):

image

This was our first Audit Failure event. This event is the result of the File Share request in the previous event. As you can see it was not successful. For the user this will result in an access denied message; exactly what I was seeing in the VMM console.

So apparently the RunAs account is also involved in accessing the share. To test to see if this was the solution I added the RunAs account to the share and NTFS permissions with Read access. Immediately it was successful and the VM could access the ISO file directly from the VMM Library.

Happy that I made it work I was still interested in knowing why the RunAs account was used for this. The answer was to be found in the VMM Add Resource Wizard. This is the wizard used to add fabric resources to VMM, like Hyper-V hosts or clusters. On the Credentials page we see this:

image

So the RunAs account is stored and used to access the hosts discovered with the wizard. The universe was once again in balance.

Posted in Uncategorized | Leave a comment

Listing Windows Azure availability sets

Windows Azure guarantees a 99.95 % uptime SLA, but this is only for multiple instance roles. So, for example, one web server by itself will not be guaranteed 99.95 % uptime, but will rather have “best effort”. Whereas two, or more, in a multi instance role will. So how do you make something multi-instance?

You use something called an availability set. An availability set is spanned across multiple fault and update domains in the datacenter. A fault domain is a collection of resources that are expected to fail together, like a server rack. An update domain is a set of resources that are updated at the same time. An availability set ensures that your instances are spanned across multiple update and fault domains so that no single failure or update can impact all the instances. The point here is that all the instances must be the same and be able to offer the same service to your users. So going back to our example with web servers you have to have at least two identical ones in the same availability set for your service to have a 99.95 % uptime guarantee. It is not correct to say that any VM or instance is guaranteed 99.95 % since it is the service that the instance offers that is guaranteed. Furthermore only Internet facing services have the SLA, since you need to use the Windows Azure load balancer to publish your instances, but more on that another day. Back to availability sets.

You add VMs to availability sets either in the Windows Azure portal, with PowerShell or the REST API. In the portal or PowerShell there is no way to explicitly create an availability set, you always create one as a side effect of configuring a VM. In the portal it looks like this:

image

This is an availability set with only one instance. In the drop down list I can either remove the VM from the availability set or create a new availability set and add the VM to that set.

Note: when you either add or remove a VM from an availability set, it will get a new NIC. This will have the same IP address as before but you will see that its number has been incremented.

In PowerShell you use the Set-AzureAvailabilitySet or New-AzureVMConfig cmdlets. Set-AzureAvailabilitySet updates an existing VM while New-AzureVMConfig lets you specify an availability set when you create the VM.

Something I miss is the ability to list all my availability sets. Since there is no (yet) Get-AzureAvailabilitySet cmdlet you have to extract this information from each of your VMs. Here is a quick one-liner to do that:

(Get-AzureService).servicename | foreach {Get-AzureVM -ServiceName $_ } | select name,AvailabilitySetName

This will list the VM names, cloud service names and availability set names (if any) for all your VMs. Feel free to add any kind of formatting or sorting to the output.

Posted in Uncategorized | Leave a comment

A network by any other name…

OK, funny story. A Windows Azure virtual network with the name MDS-vNET1; no problem. A Windows Azure cloud service with a production deployment slot associated with a virtual network with the name MDS-vNet1; no problem. Try to combine those two; problem.

So let me give you a few more details. Everything in this setup was created with PowerShell. I pre-created all the cloud services, affinity groups, storage accounts etc. I created a virtual network XML file and uploaded it with PowerShell too. But the crucial bit came when I created the VMs. On the New-AzureVM cmdlet I specified a vNetName of MDS-vNet1, instead of MDS-vNET1 that was in the network configuration. The command did not give any errors but when I tried to see which subnet a VM was on that information was missing from the Azure portal, and from PowerShell, and from SC App Controller. All I could see in any of those was the incorrect vNetName (MDS-vNet1). The Subnet field was simply missing. If I did an export of the VM with Export-AzureVM, the Subnet section was empty. Needless to say, the VMs were not communicating.

So how to fix it? You cannot change the vNet associated with a deployment once it is up and running. Neither can you move a VM from one subnet to another once it is created. The only time you can select a subnet for a VM is when you create it. In this case I had do to the following:

  1. Export and delete all VMs in the faulty deployment
  2. Delete the cloud service with the faulty deployment
  3. Manually edit all the VM XML files to include the correct subnet names
  4. Re-create the cloud service
  5. Import all the VMs and for the first VM specify the correct vNetName on the New-AzureVM cmdlet.

Problem solved.

As far as I know the only thing case sensitive in Windows Azure are storage account names, which can include only lower case letters and numbers. Not in any other case have I seen an error relating to text case. Strange that it should have such a drastic effect. If you have any experiences similar to mine, please leave a comment.

Posted in Uncategorized | Leave a comment