Yes my friends, this post has been long overdue. Life, work and all the other good excuses got in the way. However, there is nothing like a friend calling you out with a “WTF I need part 2” to get the motivation and kerberos mana flowing again. So where did we leave off? In part 1 we discussed some of the most common issues with Kerberos authentication and the necessity to break the problem down to client-side vs server-side authentication.
In part 2 we will look at some of the first troubleshooting steps I take to determine why nothing is working – yes, this happens to me too.
ADTest is Your New BF4L
You know those people that open a web browser to see if their Internet works… yea, don’t be that person. Open a terminal, check if you can ping your gateway, public DNS server, etc. and then, only then, open a browser.
ADTest is your equivalent of ping. Don’t assume because you configured an Active Directory AAA object that authentication is just going to work. Please open a console and verify that Kerberos authentication against the AD server is working with ADTest. Check out my APM Troubleshooting with ADTest for more information.
Time is not on Your Side
If you have worked with Kerberos before you know it is supper picky about time drift. If ADTest just won’t work and you can’t figure out why ensure the BIG-IP’s time matches the KDC. If you need to adjust the BIG-IP follow the F5 SOL3381.
These Are Not the KDCs You’re Looking For
Ever been in a multi-domain environment and the AD admin swears the KDC you’re talking to is the correct one; never mind the “Kerberos Principal Unknown” error you keep getting. So if ADTest doesn’t work then we need to ensure the AD server we’re talking to is a KDC for the expected realm. Now, if someone can RDP into the server this can be ruled out pretty quickly, but when in life is anything that easy.
So it’s helpful to use nslookup to find all KDCs for the intended domain and ensure the IP you were give is in this list – example below:
C:\Users\user>nslookup -type=SRV _kerberos._tcp.dc._msdcs.f5demo.com
_kerberos._tcp.dc._msdcs.f5demo.com SRV service location:
priority = 0
weight = 100
port = 88
svr hostname = ad1.f5demo.com
ad1.f5demo.com internet address = 10.1.10.2
ad1.f5demo.com internet address = 10.1.1.3
Playing Go Fish with SPNs
You have to admire how Microsoft took something as complicated as Kerberos and made it trivial to deploy and manage inside of Active Directory. Having managed MIT and MS version of Kerberos myself I felt a little guilty after setting up my first AD server without hours of troubleshooting issues. However, with that ease of use Microsoft also made it easy to shoot your own foot off if you have no basic understanding of Kerberos.
Think of the KDC as a key, value pair database. The KDC will let you store multiple keys of the same value even though you shouldn’t do that. So if you have multiple SPN entries in AD you are not guarantied that a request for a ticket will return the value you’re looking for. This typically presents itself in APM as authentication works onetime and not the other.
An easy way to check this is to log into a domain machine and issue:
This will print out any duplicate SPNs in your KDC. If the SPN you are working with appears in this list then you need to correct this issue. The easiest way I find is to delete the service account you have created for APM and just use the service account the web server application pool is using.
APM caches Kerberos tickets for both client side Kerberos authentication and server side Kerberos SSO. If you’re troubleshooting Kerberos be sure to clear these caches after you’ve made modifications.
For Kerberos AAA:
bigstart restart apd rba
For Kerberos SSO:
bigstart restart websso
I’ll Just Do It Myself
If you are working with Kerberos SSO then you have to sometimes determine if the issue is with APM obtaining the token or the web server not accepting the token. If you have your WebSSO logging set to debug and Kerberos SSO is working then you should see
If not, then there are a few CLI commands you can use to simulate the request APM makes to the KDC.
First, remove all kerberos tickets.
Second, obtain a kerberos ticket as the AD delegation account (if this works you won’t get a response)
kinit -f <SPN of AD delegation account>
Finally, test if the ticket you obtain has delegation capabilities
knvo -C -U <username> <SPN of AD delegation account>
If you receive a key version number on the kvno command then everything is working and it proves that ASREQ and RSREQ work. So the issue more than likely is on the web server side and not the F5 – more on this in part 3.
9 thoughts on “Kerberos is Easy – Part 2”
Good stuff. It is easy to build, hard to troubleshoot.These steps are methodical.
This is a great reference. Thanks.
Thanks for great article… I’ve been trying to get kerberos working with no success in multiple domains.
So we have multiple domains (domain1 and domain2) and both are part of parent domain i.e maindomain.
We have AAA kerberos configuration keytab file from domain1 (where servers and services resides) and SPN setup (HTTP/xyz.com) . It can authenticate users from both domains without any issue so AAA authentication works after 401 response in APM policy.
We have SSO configuration for each domain domain1 with it’s kerberos Realm(domain1), user Account and password and SPN HTTP/xyz.com@DOMAIN1. For Domain2, we have it’s own respective User Account from domain2 and passowrd with Kerberos Realm(domain2) and SPN HTTP/xyz.com@DOMAIN2.
In APM we have irule to check if user come from DOMAIN1 assign SSO from DOMAIN1 and vice versa.
It’s working for DOMAIN1 user but not for DOMAIN2….
Have you ever tried running AAA kerberos and SSO kerberos in APM for different DOmains.
I have been running into the issue with SSO kerberos configuration for different domains.
No, I have not seen issues with this.
Great Stuff… did you ever write part 3? I got to S4U —> Ok but could not get it working any other tips or tricks to debugging server side?
typically this means the backend server is not configured correctly. More than likely it is looking for a different SPN. Sometimes this will be included in the TCP dump but other times you’ll need to look at the Kerberos configuration on the server. By default APM will do a reverse IP lookup on the pool member and create a Kerberos ticket for that FQDN. If the pool member is expecting a SPN different from it’s DNS FQDN then you’ll need to modify the SPN patter in the APM Kerberos SSO configuration.
Did you ever publish part 3?
No, it’s been in my Ulysses draft folder for quite some time. What issues are you running into, maybe I can help.