APM Troubleshooting with ADTest

Overview

When I first started working in IT it drove me crazy when users would verify if their Internet connection was working by opening a browser and try to get to Google.  Ideally they should have used ping and progressed through the process of pinging their gateway then their exit router and then a public DNS server to determine if their Internet connection was working – yea right!

Well, I have this same feeling when someone that’s new to APM configures AD authentication and then immediately opens a browser and tries to authenticate to an APM application only to find it doesn’t work.  The problem with this approach is that the APM login page will give you very little data as to why this did not work.  A better method would be to use a tool on the F5 that can test AD authentication and ensure that your network, DNS and firewall settings are all correct so you can ensure AD authentication stands a good chance at being successful.  Such a tool exists via the CLI and it’s called adtest.

I’ve mentioned adtest before in a few of my posts but I’d like to give you a little more insight into how I use this tool.  When working with Active Directory there are a few things you need to know regarding how APM performs authentication and data retrieval.  When using the Active Directory AAA object APM will use Kerberos for authentication and LDAP for AD Query processes.  APM authenticates the user via their credentials and not via the service account configured in the AD AAA object.  So for Kerberos authentication to work you must have DNS configured correctly on your BIG-IP and you need to ensure the BIG-IP can access port 88 on the Active Directory Domain Controllers.  Something to note, adtest does not use the data in your AD AAA object but instead these options are entered by you via the CLI when executing the adtest command.

So lets get down to it:

adtest -t auth -d 10 -r f5guru.com -u gututest

So what does this command do?

• -t auth tells adtest that we’ll be authenticating against AD versus querying.
• -d 10 sets the debug level to it’s highest setting so we can see all errors that may occur
• -r defines the realm we’ll be authenticating against
• -u defines the username we’re authenticating with

So what can go wrong when using this command?

1. DNS not configured or misconfigured

If adtest states that it can’t find the defined realm then either the BIG-IP DNS configuration is not correct or your DNS infrastructure does not have the correct service records to point Kerberos clients to the correct KDC.  If your DNS infrastructure is missing the correct service records you can use the -h tag to specify the Active Directory domain controllers name.

adtest -t auth -d 10 -h ad01.f5guru.com -r f5guru.com -u gututest

2. Firewall Issues

If the F5 is not on the same layer2 network as the preferred Active Directory Domain Controller then there is a good chance our Kerberos request will traverse through a firewall and/or IPS solution.  While adtest does not have specific error messages that would indicate a firewall issues exists you would probably start to guess this may be the issues once you’ve verified that your routing, DNS and test credentials are all correct.  A definitive way to prove this is take a tcpdump via the BIG-IP CLI.  An easy way to capture and review tcpdump on the BIG-IP CLI is with tshark (on TMOS 11.3 and higher).  The command below listens for all Kerberos traffic:

tshark -i 0.0 -d tcp.port==88,kerberos -R kerberos -nVXs0

Note: tshark will create temporary files in /tmp/ that will need to be delete once you’re done.  Otherwise you may fill up you BIG-IP disk and cause bigger problems than authentication not working

3. ADtest works but APM AD AAA Does Not

While this can be caused by many things typically I see it boil down to the following issues:

1. APM AD AAA configuration does not match adtest CLI arguments
2. APM AD AAA is using a pool of AD servers

Obviously the 1st issue can be fixed by retyping all of your APM AD AAA configuration object over again – I know this seems silly but it happens more than you’d think.  The 2nd issue stems from the fact that adtest may not perform authentication from the same Active Directory Domain Controllers as you’ve specified in your APM AD AAA configuration.  When adtest uses the -r flag it queries DNS to obtain a KDC server for authentication.  An easy way to ensure the Active Directory Domain Controllers you’ve selected work with APM is to perform an adtest against each domain controller using the -h flag:

adtest -t auth -d 10 -h ad01.f5guru.com -r f5guru.com -u gututest

adtest -t auth -d 10 -h ad02.f5guru.com -r f5guru.com -u gututest

adtest -t auth -d 10 -h ad03.f5guru.com -r f5guru.com -u gututest

I hope this post helps give you an idea of how I use this tool and I’ll keep updating this page as I find new ways to troubleshoot with adtest.