Constrained Kerberos Delegation - Battling the hound of hades

Battling the what?... it’s interesting to note that the source of the name is from a Greek mythology creature named Cerberus. Often referred to as "hound of Hades" who guards the gates of the underworld, preventing the dead from leaving... Sounds more management to me not an authentication protocol but you’re not here for my ramblings so let get on with it.

Goal: We require NTLM authentication enabled for outlook anywhere clients. This will remove the dreaded outlook prompts. Our LAB environment consists of 2x Exchange 2010 SP3 servers using F5 GTM and LTM technologies to load balance all Exchange services to and from these servers. 

Internally clients connect to an internal virtual server which proxies the connections directly to the exchange servers which authenticates the user.  Internally outlook anywhere is working.

Externally users connect to an external virtual server. Cached NTLM credentials are used to complete pre authentication. If the authentication fails or local creds no do exist the user is prompted for authentication.

Once pre-authentication has completed successfully the F5 APM takes these user credentials and then requests or translates that into a Kerberos service ticket on the user’s behalf from Active Directory, and then present the service ticket to the Client Access server in order to access the user’s mailbox. This service ticket is only for the destination service required and, therefore, ‘constrained’. 

Issue:  Currently we see the APM completing pre-authentication and the Kerberos delegation appears to be completing correctly. Auto discover and Web services are working correctly via a Kerberos authentication backend. But Outlook anywhere is not.

The reply from exchange is *[:status][503 RPC Error: 6ba] 

RPC error 6ba. Based on MS-ERREF documents, means: RPC_S_SERVER_UNAVAILABLE. (RPC server is not available.)

This error happens after BIG-IP authenticates the user, and performs KCD to the CAS server. On the Exchange server we have enabled Kerberos debugging and we still see the following error:

Log Name:      System

Source:        Microsoft-Windows-Security-Kerberos

Date:          31/03/2016 3:15:56 PM

Event ID:      3

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      EXC1.corp.domain.com

Description:

A Kerberos Error Message was received:

on logon session

 Client Time:

 Server Time: 5:15:55.0000 3/31/2016 Z

Error Code: 0xd KDC_ERR_BADOPTION

Extended Error: 0xc0000272 KLIN(0)

Client Realm:

 Client Name:

 Server Realm: XXXXX

Server Name: oaspn@DOMAIN.COM

Target Name: oaspn@DOMAIN.COM@DOMAIN.COM

Error Text:

 File: 9

Line: f09

Error Data is in record data.

Resolution: A couple of issues were found. 

1. FIREWALL RPC issues were stopping RPC traffic from being sent to the the exchange servers. 

2. Incorrect values in the f5 deployment guide. (eg. processmodel.identityType not processodel.identityType)

3. SPN account was not saving the namespace as a allow

I completed a packet capture on the DC and confirmed the ERR_BADOPTION Status: NO_MATCH

The most common scenario is a request for a delegated ticket (unconstrained or constrained delegation). You will typically see this on the middle-tier server trying to access a back-end server. There are several reasons for rejection:

1. The service account is not trusted for delegation

2. The service account is not trusted for delegation to the SPN requested

3. The user’s account is marked as sensitive

4. The request was for a constrained delegation ticket to itself (constrained delegation is designed to allow a middle tier service to request a ticket to a back end service on behalf on another user, not on behalf of itself).

https://blogs.technet.microsoft.com/askds/2012/07/27/kerberos-errors-in-network-captures/

Having a closer look at the APM Delegation account I thought the “No Match” error was related to the url not being listed in the “Services to which this account can present delgated credentuals” After some testing I could see that the setspn is meant to add the url into the msDS-AllowedToDelegateTo AD field on the APM Delegation account which wasn’t happening for what ever reason. So I manually added these in and it started working.

It would appear the order is important even when things are working well. The APM Delegation account must be set to “Trust this user for delegation to specified service only” before the setspn is completed otherwise the SPN url will not be added to msDS-AllowedToDelegateTo field. 

why the setspn isn't apply the namespace I'm unsure and using a number namespaces appears to hit certain limits. hacking the service account isn't ideal but it seems to work for now....  

Generic Mailboxes and Enterprise Vault

Folder Level access can be a bit trick in Enterprise Vault. Depending on how your environment is set up you may run into this issue.

Issue 
Non owner’s users who operate shared mailboxes are unable to open Enterprise Vault Items. Users will see the following error “You do not have access to this vault.” although they have folder level access to the mailbox and are able to see the EV item.

Cause 
Default or Anonymous permissions have been used in order to grant the users access to the mailbox folder. 

Resolution
Grant each user permissions to a mailbox folder. Users will then receive access to Enterprise Vault Items.  

Don’t let a offline exchange server affect your scripts!!!

I’m sure you like most of you have being remote power shelling for some time but if you are running an on prem solution this might be worth adding to your scripts. In short a simple loop based on a static array. 

Such a loop will assist with completing a retry in the event that the remote session is unable to be created to the exchange server.  

$ExchangeServers = ("SRVEXC001","SRVEXC002")

$SessionActive = $false

$Counterlimit = 10

$Counter = 0

DO

{

       Try{

    $ExchangeServersSession = $ExchangeServers | Get-Random

    Write-Warning "Attempting to Connect to Exchange Server: $ExchangeServersSession"

       $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://$ExchangeServersSession/PowerShell/ -Authentication Kerberos -ErrorAction SilentlyContinue

    $Counter++

    $SessionCurrent = Get-PSSession | Where{($_.ConfigurationName -eq 'Microsoft.Exchange') -and ($_.State -eq 'Opened') -and ($_.Availability -eq 'Available')}

    }

    Catch{

   

    }

    Finally{

    If($SessionCurrent){Write-Warning "Connected to Exchange Server: $ExchangeServersSession";Import-PSSession $Session;$error.clear();$SessionActive = $true}

    }

} Until (($SessionActive -eq $true) -or ($Counter -gt $Counterlimit))     

If($Counter -gt $Counterlimit){Write-error "No Exchange Servers are responding to Remote Powershell Requests"} 

You could also automate the static array $ExchangeServers and select the servers you require without too much hassle but I’ll let you guys look after that part… J

Does Session Affinity really make Sense!

Quick post dedicated for my wife who is a word/grammar guru…

In the IT world there is so many things that we say or pick up that creates you might say a new sub language even possibly IT street language. I wonder if I could register a new creole for the IT industry… But does it often make sense…. I know many will say that it doesn’t really matter.

Session Affinity has been defined as:

Most servers use the term "Session Affinity" to indicate that within a cluster of servers, requests from the same client always get routed back to the same server. This eliminates the need to replicate session data like the Http Session or Stateful Session Beans.

Implementations vary, but is usually base on some kind of routing scheme that identifies the client by IP address or a browser cookie.

Affinity Some seven hundred years ago, affinity meant “relation by marriage.” By extension, the proper use of affinity involves mutuality. But that sense of mutual attraction is often absent in contemporary uses of affinity. An online search reveals many examples such as these: “She always had an affinity for growing fruit.” “I have an affinity for vintage chairs.” “My friend has an affinity for making things out of cardboard.” In these examples, “growing fruit,” “vintage chairs,” and “making things out of cardboard” are passive elements, not active components in a relationship. Better to say “a talent for growing fruit,” “a fondness for vintage chairs,” “a flair for making things out of cardboard.”

In the examples above, affinity is followed by the preposition for. But in formal English, the phrase affinity for is despised. The editor Theodore M. Bernstein advised writers to “discard for” and instead “use between, with, or sometimes to.”

Here are three sentences that use affinity correctly: “There is an affinity between the Irish and the Italians that can be hard to explain.” “Some people have a natural affinity with children.” “Two vaccines containing native proteins with affinity to porcine transferrin were tested.”

There is no affinity unless it is shared by both parties.

Hidden Behind the Load Balancer

Of course you would have been working hard to place your Exchange CASArray’s and Webmail services behind a decent load balancer with decent health checks. (BTW you are only as good as your health checks!!)

Never the less, even in the best designed solutions there is always cracks, things that are missed or worse when things kinda fail but kinda still work. Of course for users they speak first to colleagues, neighbours, Twitter, Facebook and then finally the IT service desk. 

It’s always helpful to confirm which back end client access server the user is currently connecting to, adding this information into an existing ticket will assist confirming if you are indeed have an issue with a single server in your farm.

From the outlook client you can output a number of connection setting via the Connection Status Window. (http://technet.microsoft.com/en-us/library/bb123650(v=exchg.65).aspx)

Of course this is helpful with some aspects such as confirming which type of connection is being used or the number of failed connection being made are important but this fails to confirm which client access server the user is actually hitting behind the load balancer.

OWA is a little better. We have to love the “about” button which cuts straight to the chase and provides details for which Exchange Client Access server is in use and even what roles are used by this server.

Better still is the requirement to all the connections in one go. To do this you can use the following: 

Get-LogonStatistics -id UserID | select clientname, servername, username, applicationid | ft

I was disappointed to hear last year at TechEd that this CMDLet is no longer going to be supported and used in Exchange 2013. Microsoft have been asking for business cases for some time but clearly it lost. So what can we do from here? 

Well moving forward you can always trace the IIS and RPC logs for user connections. Here's one I created a while back. It's a simple script to track the users and can be used to find anything in the logs such as throttled users. 

$PathCAS001 = "\\cas001\C$\inetpub\logs\LogFiles\W3SVC1"

$PathCAS002 = "\\cas002\C$\inetpub\logs\LogFiles\W3SVC1"

$Getdays = 1

$outputpath = "C:\Scripts\IISfind\Output"

$Date = Get-Date

$DateShort = (Get-Date).ToString('yyyyMMdd')

$SearchValue = Read-Host "Please enter a value to seach the IISLogs (eg. UserID):"

   

    add-content -path "$($outputpath)\$($DateShort)IISFind_Output.txt" -value "$date --------------- CAS001"

    Get-Item -Path "$PathCAS001\*" | ?{$_.LastWriteTime -gt $($(get-date).adddays(-$Getdays))} |  get-content | Select-String $SearchValue | add-content -path "$($outputpath)\$($DateShort)IISFind_Output.txt"

   

    add-content -path "$($outputpath)\$($DateShort)IISFind_Output.txt" -value "$date --------------- CAS002"

    Get-Item -Path "$PathCAS002\*" | ?{$_.LastWriteTime -gt $($(get-date).adddays(-1))} |  get-content | Select-String $SearchValue | add-content -path "$($outputpath)\$($DateShort)IISFind_Output.txt"

     

Another helpful tool you could also use is Log Parser (http://technet.microsoft.com/en-au/scriptcenter/dd919274.aspx) which assists tracing these logs. 

For RPC searches just change the source directory to the RPC Log directory and rerun the script across all your Client Access Servers.

I spoke to Scott Schnoll a while back about it here is my case for Get-LogonStatistics. 

Subject: Case Study: Get-LogonStatistics

Hi Scott,

We had a quick chat on Friday and I said I would forward you a case we encountered which required the use of the get-logonstatistics CMDlet.

User Issue: Users were experiencing bad outlook performance to a single datacentre with numerous connects and disconnects throughout a single day.

Core Issue: Due to recent Junos firewall upgrade in the core we found that this firewall had enabled ALG for RPC traffic enabled which was dropping active RPC connections to this datacentre. This would result in outlook not responding and making a new connection to the CAS server. These new connections in turn created the problem that users were starting to breach their throttling limits. MS Support recommended increasing throttling limits which in reality made no difference to outlook performance.

[KB18141] - Microsoft Services are unavailable after upgrade to Junos 10.1 and later versions

MS- RPC ALG is available and enabled by default on SRX-Branch and J-Series platforms running Junos 10.0 and later. However beginning with Junos 10.1, the MS-RPC ALG was added for SRX-HE platforms and enabled by default. This may cause issues with Microsoft traffic such as Exchange and Active Directory (refer to PSN-2010-08-912

Overview: Get-LogonStatistics assisted this issue as I could directly compare between the client and server and confirmed that the CAS server thought it was still holding an active connections when in fact it had been closed by the firewall.

Thanks

Josh