Active Directory troubleshooting for Mac OS X

This is not, at this time, an official ITS document. ITS does not currently endorse or support Active Directory binding or logins on Mac OS X systems. RIT has a unique directory and account management infrastructure and we have not qualified Mac OS X against it with a thorough test plan. We do not recommend binding Mac OS X computers to our Active Directory, especially when faculty/staff accounts are involved.

Therefore, these instructions serve as guidelines for internal use/troubleshooting, but do not constitute a recommendation to use Active Directory with Mac OS X systems at RIT.

The goal of troubleshooting Active Directory problems on Mac OS X is to not only get the client system to work quickly when it is being used, but also to collect enough data so that if it doesn't work, we can determine what caused the problem. In some cases, the data collected will need to be compiled into a bug report suitable for sending to a vendor, which in most of these cases will be Apple.

Basic steps

Verify that the computer is bound

If you are having login problems, verify that the computer is already bound. You should check the local system first:

• Use the Directory Access utility, which requires admin privileges through the Mac OS X authentication services framework
• Run dsconfigad from the command line, which requires sudo privileges on a default Mac OS X installation, and will return "You are bound to Active Directory" if the computer thinks that it is bound:
  % sudo dsconfigad -show

You should also confirm that a computer object for the host system you are examining is present in the Active Directory.

Check date and time

If a computer is bound to the Active Directory but its date and time are incorrect, login failures are guaranteed. Check the four major components of date and time:

• date
• time
• time zone (GMT offset)
• daylight savings.

Why? Active Directory uses Kerberos. Kerberized logins will fail on any client if its time is off by more than ±5 minutes from the Active Directory domain controllers. The domain controllers themselves should be synchronizing to the RIT time servers (ntp.rit.edu). Therefore, each AD-bound Macintosh should sync to either the RIT time server or the Apple time servers (if the computer will be leaving the RIT network).

There are times when you will need to set the date and time manually before performing network time sync.

You may synchronize time by unlocking the Date & Time System Preferences with an administrator account.

The following command can also synchronize a workstation's time with the default NTP time server on demand:

% ntpdate -uvs

Verify DNS: forward, reverse, and resource records

We have a script which can verify Microsoft’s recommended forward, reverse, and resource records for Active Directory. This can be run to periodically or manually verify DNS for Active Directory.

Please contact me if you want to try this script, but realize that I already check on the DNS records for RIT's Active Directory domain.

Kerberos auto-configuration interlude

Mac OS X Tiger and Leopard automatically configure Kerberos for Active Directory. Kerberos is the single-sign on (SSO) authentication method used by Active Directory, and happily, Mac OS X shares this critical protocol suite.

The "kerberosautoconfig" tool is used to configure Kerberos automatically on Mac OS X when used with Active Directory.

More to come on this topic later!

Kerberos configuration property lists

[Leopard and later] Check the configuration files for the local directory service, DSLocal, to see if there are any 0-length files that correspond to your Active Directory forest or domain. If there are, this may be part of your problem (as seen on a thread in the MacEnterprise mailing list).

Look for plist files whose names begin with "Kerberos." If one or more are 0 bytes in size, then they are likely corrupt (because they have no data) and thus can probably be removed safely.

These plist files for your domain appear to be constantly updated. This makes sense, as upon examination, you'll see cached domain controller references. Having a stale cache of that information would likely be bad.

Data collection steps

The following instructions assume that you will have the time and in-person access to the computer, and an account admin-level access. Since most of the steps involve command line utilities, you may also adapt them to run remotely with a secure protocol, such as SSH. If you have an Active Directory bound computer with remote access, you should harden your OpenSSH sshd_config file to prevent unauthorized and/or unwanted Active Directory logins, as any account in the directory has the potentially of logging into the system. SSH probes are common all across the Internet.

Whether you are logging in locally or remotely, you should harden the sudoers file appropriately for any technicians, to limit access to only those commands needed for this service, rather than giving full sudo rights.

Get logs and network traffic recordings

Detailed logs and network traffic recordings can help us determine what caused an Active Directory related problem. It can also serve as supporting material we can attach to bug reports to Apple. You should collect this information while troubleshooting all varieties of Active Directory problems.

1. Log in as an administrator.
2. Enable Fast User Switching in System Preferences > Accounts.
3. Launch Terminal.
4. Capture the network traffic during the login process using tcpdump—and optionally, tcpflow.
   1. Begin a tcpdump of the login process that will follow, changing the tcpdump filename to one of your choosing (or accepting the default one below, which uses the namegen utility and mktemp to make a unique descriptive name easily):
      % sudo tcpdump -i en0 -n -s 0 -w /Users/Shared/tcpdump-`namegen`-`mktemp XXXXXX` port domain or port ldap or ldaps or kerberos or kpasswd or msft-gc or msft-gc-ssl
      Note: This step uses namegen, an in-house utility that generates a consistent name for the operating system edition, version, build number, and architecture. This helps identify the tcpdump output, which may reasonably be unique for the version of Mac OS X being tested because Apple does update the software in the DirectoryService architecture in the minor updates to major versions of the system software. The use of mktemp makes the resulting tcpdump filename unique in the /Users/Shared folder. To approximate this without namegen, use this longer form which only uses the system’s built-in “defaults” command:
      % sudo tcpdump -i en0 -n -s 0 -w /Users/Shared/tcpdump-`defaults read /System/Library/CoreServices/SystemVersion ProductName | sed 's/ Server/-Server/g' | tr -d ' ' | tr '[:upper:]' '[:lower:]'`-`defaults read /System/Library/CoreServices/SystemVersion ProductVersion`-`defaults read /System/Library/CoreServices/SystemVersion ProductBuildVersion`-`arch`-`date +%Y%m%d`-`mktemp XXXXXX` port domain or port ldap or ldaps or kerberos or kpasswd or msft-gc or msft-gc-ssl
      Note: This step uses tcpdump, which is built into Mac OS X. Using tcpflow, as below, is optional if it is available. You should always capture to tcpdump, and capture with tcpflow in addition. The options capture on interface en0, disable name resolution in the capture, and turn off the limit on the capture size of the packets.
   2. [Very optional, bordering on don’t bother] Begin a tcpflow of the login process that will follow, changing “tcpflow-name” to a filename of your choosing:
      % sudo tcpflow -i en0 -c port domain or port ldap or ldaps or kerberos or kpasswd or msft-gc or msft-gc-ssl > /Users/Shared/tcpflow-name
      
      Note: This step requires tcpflow — additional software which can be downloaded here.
5. Enter the following command to increase the level of logging for Mac OS X's Directory Services layer, enabling debug-level logging (to the /Library/Logs/DirectoryService/DirectoryService.debug.log):
   % sudo killall -USR1 DirectoryService
6. Run the following command from Terminal, and verify that “Active Directory” appears in the results. (See Unable to log into AD account twice? at AFP548.com for more information)
   % dscl localhost -list .
7. Run the following command from Terminal, and verify that “Active Directory” appears in the results.
   % dscl /Search -read /
8. Run the following command from Terminal, substituting a username from RIT’s Active Directory for “userid,” and verify that it returns accurate results for that user account:
   % dscl /Active\ Directory/All\ Domains -read Users/userid
9. Repeat the previous step with several more Active Directory user accounts, if the first one was successful. Note if any of the accounts cannot be found with the “dscl -read” command.
10. Run the following command from Terminal, substituting a username from RIT’s Active Directory for “userid.” This uses Mac OS X’s Directory Service software to query Active Directory for information related to the user account you specify.
    % id userid
11. Repeat the previous step with several more Active Directory user accounts, if the first one was successful. Note if any of the accounts cannot be found with the “id” command.
12. Run the following command from Terminal, substituting an RIT Active Directory username (whose login credentials you know) for “userid.” You will be prompted for the account’s password, and if successful, you will have authenticated as that user account.
    % su userid
13. Log out of the account you just authenticated as with “su,” using the “exit” command.
    % exit
14. Run the following command from Terminal, substituting an RIT Active Directory username (whose login credentials you know) for “userid.” You will be prompted for the account’s password, and if successful, you will have authenticated as that user account multiple times.
    % dirt -u userid
    Note: The “dirt” command line tool authenticates a user multiple times in succession; the default is 10. Login failures could immediately disable that account by exceeding the threshold defined in Active Directory. Please type the password carefully.
    Note: Most importantly, in Mac OS X Leopard as of version 10.5.2, the tool prints the password of the user in the system log. This has been reported as a security issue to Apple. Until this is fixed to our satisfaction, don’t use a privileged RIT account, change its password through start.rit.edu immediately after the use of “dirt,” and carefully sanitize any log outputs from that timeframe before you send them as bug report data or post them to any online forums.
15. Switch to the Login Window with Fast User Switching.
16. Note the date and time, so you can examine the relevant logs later. With Mac OS X 10.4.5 and later, you can see the date and time at the login window by clicking the grey text to toggle between host name and other attributes.
17. Attempt to log in at the login window. Make several attempts, preferably with different users and classes of users.
18. Proceed to disable the logging and cancel the traffic recording, as outlined below.

Disable enhanced logging and cancel traffic recording

1. Log back into the administrator account.
2. Cancel the increased debug-level logging for the Directory Services layer by sending another signal to toggle it to the "off" state:
   % sudo killall -USR1 DirectoryService
3. End the tcpflow with Control-C and tcpflow with Control-Z. The output will be saved to the administrator account's Desktop folder.
4. Collect the logging data and traffic recordings, as below.

Attempt to determine the problem from the logs and recordings

You must collect the various logs and traffic dumps in order to analyze them. This data should be handled securely because it may contain sensitive information, such as account passwords.

1. Collect portions of the following logs for the relevant date and time:
   • /private/var/log/system.log, being sure to carefully sanitize any passwords found in it after the use of the “dirt” tool on Mac OS X Leopard
   • /Library/Logs/DirectoryService/DirectoryService.*.log
2. Collect the tcpflow or tcpdump output.
3. Gather an Apple System Profiler report in XML format. If we have to report a potential bug to Apple, we will need this information when logging a ticket with them.

The logs can be examined in the Console utility. The tcpdump recordings can be examined in Ethereal.

Solutions

As a reminder, ITS does not currently endorse or support Active Directory binding or logins on Mac OS X systems.

Re-bind the computer

Unbinding and rebinding a computer is a potentially-destructive action. In particular, the original SID of the computer object in the Active Directory can be lost.

If you still cannot log in with any Active Directory accounts or have other problems, you should attempt to rebind the computer to the directory. You should always try collecting data before rebinding a client computer to the directory, so that we can use that data to further examine the problem later. If you rebind the computer right away and it works, we have lost an opportunity for study of what went wrong in the first place. (Granted, as a system administrator or technician, you will have to weigh each situation on its merits and act accordingly.)

While rebinding, you should either continue the above logging/traces, or restart them (as below). This can uncover additional information that may be useful in troubleshooting.

When starting new network traces, please use a new filename for them so you don't overwrite any you have already captured.

1. Complete the steps above to begin logging and traffic recordings.
2. Delete the computer object from the Active Directory. This requires privileges for the directory.
3. Wait one minute for this change to replicate throughout the Active Directory.
4. Re-bind the computer. If you can do this with elevated logging and/or a tcpflow/tcpdump record, that would improve our chances in determining what went wrong — especially if the re-binding fails.
5. End the increased logging and traffic dumps.

Use a VPN connection during login when off-campus

Some Mac OS X Tiger systems bound to our Active Directory exhibit delays or failures with user logins when attempted outside of the campus network. We are investigating, but in general, you can assume that Active Directory services are unavailable to Mac OS X users beyond the friendly confines of the campus network.

In the meantime, creative use of Fast User Switching and a local account on the system can allow you to create a VPN tunnel for long enough to allow an Active Directory user to login. YMMV.

Remove Active Directory from the Authentication path

For off-campus systems bound to the directory, we have found (thanks to Lance Ogletree of Rice University) that removing the Active Directory from the list of authentication sources in Directory Access can resolve login delays and some failures. Each user that will login must have a valid mobile account cached on the computer, which means they will have to log in before the authentication path is modified.

We are investigating automation to assist with this process, and change it based on whether the computer detects whether or not it is on the campus network.

Copyright © 2005-2008 by Rochester Institute of Technology.