Troubleshooting Forms Base Authentication using Secure LDAP Authentication on ISA Server 2006

Author:

Yuri Diogenes - Security Support Engineer, Microsoft CSS Forefront Security Edge Team

Tech Reviewers:

Thomas Detzner - Escalation Engineer, Microsoft CSS Forefront Security Edge Team

Jim Harrison - Program Manager, Forefront Edge CS

Mohit Saxena - Security Tech Lead, Microsoft CSS Forefront Security Edge Team

One feature in ISA Server 2006 that customers are using more often is authenticating users using secure LDAP; also called LDAPS. In some deployment scenarios where ISA Server 2006 is installed in a workgroup, there is a requirement that the traffic between ISA and the internal server is encrypted. For this requirement, LDAPS is the most common authentication method in use.

While this technology is widely used, many customers report some challenges while deploying and troubleshooting this type of authentication. The reason for that is there are three majors’ components to troubleshoot: ISA Server, LDAPS Authenticator Server and the Certificate Services.

Figure 1 - Components involved.

To summarize, the LDAPS authentication process involves the following steps:

1. Clients send their credentials to ISA Server through the FBA Logon Page or in response to an HTTP-Basic authentication prompt.

2. ISA Server receives the credentials and checks which method the listener is configured to use as the credentials authority. In this case it is configured to use a LDAP Server Set that points to one specific server for all domains.

   Note

   The LDAP server needs to have a server authentication certificate in order to be listening on TCP Port 636

3. Cryptography API (CAPI) validates the server certificate used by this LDAPS Server against the ISA server trust list and through a query to the issuing CA CRL URL.

4. The LDAPS connection is established successfully

5. The LDAP user lookup request is sent.

6. The authentication response is sent back to the ISA Server.

7. ISA Server allows access to the resource that it was requested by the client.

This article will discuss a typical scenario where everything was originally implemented correctly; and then stopped working.

The scenario has the following topology:

Figure 2 - Topology used in this example.

The ISA Server in the above scenario is publishing a web server using Forms Based Authentication with LDAPS as the credentials authority for the Web Listener. Clients began complaining that they could not logon. The actual problem description is covered in Problem 1 session.

Users that were accessing ISA Server 2006 from outside were receiving the following error after typing their credentials.

Figure 3 - Error when client tries to authenticate from outside.

This is a generic error message that will be displayed when authentication fails. At this point the following basic things were already reviewed:

• User is really typing the correct credentials.

• ISA Server 2006 does have connectivity to the DC that authenticates the user.

Now it is time get to more data in order to understand what is going on under the hood.

The best way to gather data for troubleshooting ISA problems is to use the ISA Data Packager in repro mode. Follow the steps below to do that:

1. Download the latest version from www.isabpa.com and install it on the ISA Server.

2. Launch ISA Data Packager and choose the Web Proxy and Web Publishing template.

3. After starting the capture, reproduce the problem and stop the capture, ISA Data Packager will save the logs on the desktop (CAB file).

You can extract the CAB file and look for:

• ISA BPA Report (.xml)

• ISA Event Log (.xls)

• Netmon files (.cap)

We will review the netmon file to see what was happening during the authentication process. This netmon trace was taken on the internal network of the ISA Server and the following traces were captured while ISA Server was communicating with the Domain Controller:

First we have the TCP three way handshake:

11:32:29.75210.20.20.110.20.20.20TCPTCP:Flags=......S., SrcPort=1168, DstPort=ldap protocol over TLS/SSL (was sldap)(636), PayloadLen=0, Seq=1255869843, Ack=0, Win=65535 (scale factor 0) = 65535

11:32:29.75210.20.20.20  10.20.20.1TCPTCP:Flags=...A..S., SrcPort=ldap protocol over TLS/SSL (was sldap)(636), DstPort=1168, PayloadLen=0, Seq=3679997905, Ack=1255869844, Win=16384 (scale factor 0) = 16384

11:32:29.75210.20.20.110.20.20.20TCPTCP:Flags=...A...., SrcPort=1168, DstPort=ldap protocol over TLS/SSL (was sldap)(636), PayloadLen=0, Seq=1255869844, Ack=3679997906, Win=65535 (scale factor 0) = 65535

CAPI sends the SSL Client Hello:

11:32:29.75210.20.20.110.20.20.20  SSLSSL:SSLv2RecordLayer, ClientHello(0x01)

Server (DC) sends the SSL Server Hello

11:32:29.75210.20.20.20  10.20.20.1SSLSSL:  Server Hello. Certificate.

Now is a good time to look closely at the details of the Server Certificate. When we expand this packet and see what we have:

+ Tcp: Flags=...A...., SrcPort=ldap protocol over TLS/SSL (was sldap)(636), DstPort=1168, PayloadLen=1460, Seq=3679997906 - 3679999366, Ack=1255869922, Win=65457 (scale factor 0) = 65457
- Ssl:   Server Hello. Certificate.
  - TlsRecordLayer: 
     ContentType: HandShake
   + Version: TLS 1.0
     Length: 4593 (0x11F1)
   - SSLHandshake: SSL HandShake TLS 1.0 Certificate(0x0B)
      HandShakeType: ServerHello(0x02)
      Length: 70 (0x46)
    + ServerHello: 0x1
      HandShakeType: Certificate(0x0B)
      Length: 1281 (0x501)
    - Cert: 0x1
       CertOffset: 1278 (0x4FE)
     - Certificates: 
        CertificateLength: 1275 (0x4FB)
      - X509Cert: Issuer: STAlone Cont,contoso,msft, Subject: dccont.contoso.msft
       - SequenceHeader: 
        + AsnId: Sequence and SequenceOf types (Universal 16)
        + AsnLen: Length = 1271, LengthOfLength = 2
       - TbsCertificate: Issuer: STAlone Cont,contoso,msft, Subject: dccont.contoso.msft
        + SequenceHeader: 
        + Tag0: 
        + Version: v3 (2)
        + SerialNumber: 0x6107af48000000000006
        + Signature: Sha1WithRSAEncryption (1.2.840.113549.1.1.5)
        + Issuer: STAlone Cont,contoso,msft
        + Validity: From: 08/24/07 04:41:07 UTC To: 08/24/08 04:51:07 UTC <---
        + Subject: dccont.contoso.msft
        + SubjectPublicKeyInfo: RsaEncryption (1.2.840.113549.1.1.1)
        + Tag3: 
        + Extensions: 
       + SignatureAlgorithm: UnknownOidExtension (2.83.1625955)

Bingo! the “validity” field indicates the problem. The certificate that the domain controller is using has expired, causing the SSL session setup to fail even before the user lookup can be performed.

A new certificate was issued and installed on the domain controller to resolve problem 1. It is important to mention that after installing a new certificate on the DC you need to restart it for the change to take effect. After the DC is up and running with the new server certificate, users that are trying to logon are still facing the same response from ISA.

After reviewing the new set of data we can see that the certificate the DC is negotiating with ISA Server is still the same as it was before, with the same expiration date (Validity: From: 08/24/07 04:41:07 UTC To: 08/24/08 04:51:07 UTC), this means that the DC picked the old certificate to negotiate. This happened because the old certificate was not deleted from the local machine personal store. You need to delete it and then restart the DC again. The reason why this problem can occur is described in KB 321051, which states:

Multiple SSL certificates

Schannel, the Microsoft SSL provider, selects the first valid certificate that it finds in the local computer store. If there are multiple valid certificates available in the local computer store, Schannel may not select the correct certificate.

After performing this procedure and asking the users to try to logon again, we find that the issue still persists. This forces us to collect another BPA Data set.

For the next effort, we will enable Schannel event logging on the Domain Controller and on ISA Server. To do that add the following registry key on both computers (DC & ISA) and restart them:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL
Value Name: EventLogging
Data Type: REG_DWORD
Value: 3 

After performing this procedure and asking the user to try & logon again we could see the following events in the Domain Controller:

Event Type:Error

Event Source:Schannel

Event Category:None

Event ID:36869

Date:11/30/2008

Time:7:25:49 PM

User:N/A

Computer:DCCONT

Description:

The SSL server credential's certificate does not have a private key information property attached to it. This most often occurs when a certificate is backed up incorrectly and then later restored. This message can also indicate a certificate enrollment failure.

Event Type:Warning

Event Source:Schannel

Event Category:None

Event ID:36872

Date:11/30/2008

Time:7:25:49 PM

User:N/A

Computer:DCCONT

Description:

No suitable default server credential exists on this system. This will prevent server applications that expect to make use of the system default credentials from accepting SSL connections. An example of such an application is the directory server. Applications that manage their own credentials, such as the internet information server, are not affected by this.

These events lead us to believe that the new certificate the administrator requested and imported on the domain controller does not have the private key. In the netmon trace we can see that the DC is gracefully closing the TCP connection. This is the default behavior if there is a problem during the SSL handshake. The socket is closed by the application layer and therefore TCP/IP closes the connection with a FIN message:

10.20.20.1  10.20.20.20TCPTCP:Flags=......S., SrcPort=1042, DstPort=ldap protocol over TLS/SSL (was sldap)(636), PayloadLen=0, Seq=3903131954, Ack=0, Win=65535 (scale factor 0) = 65535

10.20.20.2010.20.20.1TCPTCP:Flags=...A..S., SrcPort=ldap protocol over TLS/SSL (was sldap)(636), DstPort=1042, PayloadLen=0, Seq=3830306308, Ack=3903131955, Win=16384 (scale factor 0) = 16384

10.20.20.110.20.20.20TCPTCP:Flags=...A...., SrcPort=1042, DstPort=ldap protocol over TLS/SSL (was sldap)(636), PayloadLen=0, Seq=3903131955, Ack=3830306309, Win=65535 (scale factor 0) = 65535

10.20.20.110.20.20.20SSLSSL:SSLv2RecordLayer, ClientHello(0x01)

10.20.20.2010.20.20.1TCPTCP:Flags=...A...F, SrcPort=ldap protocol over TLS/SSL (was sldap)(636), DstPort=1042, PayloadLen=0, Seq=3830306309, Ack=3903132033, Win=65457 (scale factor 0) = 65457

Let’s review the issues that we resolved so far:

• Problem 1 – Domain Controller had an expired certificate.

• Problem 2 – The new certificate issued to the DC didn’t have a private key.

When asked to try again, the external user is still receiving the same error that he received in the beginning. We will use the same data gathering approach that was used in session 4.1.

After reproducing the problem the following data were analyzed:

• Event Viewer on the Domain Controller: clear.

• Netmon on ISA Server: TCP handshake established successfully and failed to complete the SSL handshake.

• Event Viewer on ISA Server: error below:

Event Type:Error

Event Source:Schannel

Event Category:None

Event ID:36882

Date:11/30/2008

Time:9:02:00 PM

User:N/A

Computer:ISASTD

Description:

The certificate received from the remote server was issued by an untrusted certificate authority. Because of this, none of the data contained in the certificate can be validated. The SSL connection request has failed. The attached data contains the server certificate.

You can see on the netmon trace that CAPI downloads the CRT file that has information about the certificate authority:

10.20.20.110.20.20.20HTTPHTTP:Request, GET /CertEnroll/DCCONT.contoso.msft_STAlone%20Cont.crt
- Http: Request, GET /CertEnroll/DCCONT.contoso.msft_STAlone%20Cont.crt 
    Command: GET
  - URI: /CertEnroll/DCCONT.contoso.msft_STAlone%20Cont.crt
     Location: /CertEnroll/DCCONT.contoso.msft_STAlone%20Cont.crt 
    ProtocolVersion: HTTP/1.1
    Via:  1.1 ISASTD
    UserAgent:  Microsoft-CryptoAPI/5.131.3790.1830
    Host:  dccont.contoso.msft
    Accept:  */*
    Connection:  Keep-Alive
    HeaderEnd: CRLF

10.20.20.2010.20.20.1HTTPHTTP:Response, HTTP/1.1, Status Code = 200, URL: /CertEnroll/DCCONT.contoso.msft_STAlone%20Cont.crt
- Http: Response, HTTP/1.1, Status Code = 200, URL: /CertEnroll/DCCONT.contoso.msft_STAlone%20Cont.crt 
    ProtocolVersion: HTTP/1.1
    StatusCode: 200, Ok
    Reason: OK
    Cache-Control:  max-age=86400
    ContentLength:  1166
    ContentType:  application/x-x509-ca-cert
    Last-Modified:  Fri, 24 Aug 2007 04:09:19 GMT
    Accept-Ranges:  bytes
    ETag:  "b0c34a8b4e6c71:3b2"
    Server:  Microsoft-IIS/6.0
    Date:  Wed, 26 Nov 2008 16:52:08 GMT
    HeaderEnd: CRLF
  + payload: HttpContentType =  application/x-x509-ca-cert

After successfully downloading the CRT file (as show above) and checking who issued the certificate, the ISA Server triggers the event 36882 saying that the certificate was issued by an untrusted CA. To fix this problem we need to import the CA Certificate in the Trusted Root CA store as show below:

Figure 4 - Importing the certificate from the Root CA.

You might be asking: but why import the Root CA Certificate if it was working? The Root CA Certificate also has a valid date and in this particular case it had expired.

After addressing the core problems that were avoiding the authentication to correctly happen, let’s see the whole traffic, from the authentication to the end along with the answer from the published server:

TCP Handshake with the DC

10.20.20.110.20.20.20TCPTCP:Flags=......S., SrcPort=1218, DstPort=ldap protocol over TLS/SSL (was sldap)(636), PayloadLen=0, Seq=3116790698, Ack=0, Win=65535 (scale factor 0) = 65535

10.20.20.2010.20.20.1TCPTCP:Flags=...A..S., SrcPort=ldap protocol over TLS/SSL (was sldap)(636), DstPort=1218, PayloadLen=0, Seq=1400902735, Ack=3116790699, Win=16384 (scale factor 0) = 16384

10.20.20.110.20.20.20TCPTCP:Flags=...A...., SrcPort=1218, DstPort=ldap protocol over TLS/SSL (was sldap)(636), PayloadLen=0, Seq=3116790699, Ack=1400902736, Win=65535 (scale factor 0) = 65535

SSL Handshake

10.20.20.110.20.20.20SSLSSL:SSLv2RecordLayer, ClientHello(0x01)
10.20.20.2010.20.20.1SSLSSL:  Server Hello. Certificate.
10.20.20.110.20.20.20SSLSSL:  Certificate. Client Key Exchange. Change Cipher Spec. Encrypted Handshake Message.
10.20.20.2010.20.20.1SSLSSL:  Change Cipher Spec. Encrypted Handshake Message.

Authentication itself

22:01:15.26610.20.20.110.20.20.20LDAPLDAP:Encrypted Over SSL
22:01:15.26610.20.20.2010.20.20.1LDAPLDAP:Encrypted Over SSL
22:01:15.26610.20.20.110.20.20.20LDAPLDAP:Encrypted Over SSL

TCP Handshake with the Web Server

10.20.20.110.20.20.7TCPTCP:Flags=......S., SrcPort=1219, DstPort=HTTP(80), PayloadLen=0, Seq=1400038258, Ack=0, Win=65535 (scale factor 0) = 65535

10.20.20.710.20.20.1TCPTCP:Flags=...A..S., SrcPort=HTTP(80), DstPort=1219, PayloadLen=0, Seq=515776089, Ack=1400038259, Win=16384 (scale factor 0) = 16384

10.20.20.110.20.20.7TCPTCP:Flags=...A...., SrcPort=1219, DstPort=HTTP(80), PayloadLen=0, Seq=1400038259, Ack=515776090, Win=65535 (scale factor 0) = 65535

Get Request to the published server

10.20.20.110.20.20.7HTTPHTTP:Request, GET / 
10.20.20.710.20.20.1HTTPHTTP:Response, HTTP/1.1, Status Code = 200, URL: /

The main goal of this post is to show you that a simple authentication process can be hard to troubleshoot when the error message that the user is experiencing is very generic or the same. You may notice that in all scenarios the error message that the users were receiving while logging on the Forms Based page was the same (see Figure 1). For this reason it is important to have the tools in place to gather the correct information to assist you to fix the issue. This post also showed you that although the error is the same, the resolution can include multiple fixes in different places and this is probably the most challenging aspect of troubleshooting LDAPS authentication.

Here some important articles about secure LDAP authentication through ISA Server 2006 and also related topics:

Secure Application Publishing

https://technet.microsoft.com/en-us/library/bb794854.aspx

Troubleshooting SSL Certificates in ISA Server 2004 Publishing

https://technet.microsoft.com/en-us/library/cc302619.aspx

Configuring and Troubleshooting the Password Change Feature in ISA Server 2006

https://technet.microsoft.com/en-us/library/cc514301.aspx