Protecting the Edge Server Against DoS and Password Brute Force Attacks in Office Communications Server

Microsoft Office Communications Server 2007 and Microsoft Office Communications Server 2007 R2 will reach end of support on January 9, 2018. To stay supported, you will need to upgrade. For more information, see Resources to help you upgrade your Office 2007 servers and clients.

Companies that allow employees to sign in to Office Communications Server remotely from the Internet can be susceptible to denial-of-service (DoS) and brute-force attacks. These kinds of attacks involve guessing users’ passwords or locking users out of their accounts when too many incorrect password attempts are made to a valid Active Directory user account when password policy is enforced. Although internal security is not compromised, these types of attacks are disruptive to users and use up internal server resources. To prevent such attacks at the edge, you can install a security filter on the Edge Server that monitors sign-in attempts and enforces account lockout at the network perimeter.

Author: Rui Maximo

Publication date: May 2010

Product version: Microsoft Office Communications Server 2007 R2

When exposing services, and therefore servers, to the Internet to allow employees more flexibility and mobility in working remotely, there’s always the concern of attacks. In Office Communications Server, the Edge Server provides protection against unauthorized access by using industry-standard security measures. All communications are encrypted and authenticated. However, some customers might be concerned about DoS and password brute force attacks, and these are legitimate types of attacks. Although brute force attacks on user passwords can be prevented by using Active Directory’s lockout security policy, and corporate information is not compromised under such attacks, it does pose a nuisance to legitimate employees trying to connect to the corporate service when their Active Directory accounts have been locked out. It also consumes internal server resources as the external user’s sign-in request to Communications Server must be performed by an internal server such as a Director or pool. The Edge Server cannot authenticate the user because it’s not connected to the corporate’s Active Directory. Such an attack can easily be mounted by knowing the victim’s Active Directory username and domain name. What measures can organizations use to defend against such attacks?

For information about DoS and brute-force attacks in Microsoft Lync Server 2010 communications software, see Protecting the Edge Server Against DoS and Password Brute-Force Attacks in Lync Server 2010.

Overview

DoS attacks are indistinguishable from legitimate sign-in requests. The only differentiation is in the frequency of sign-in attempts and origin. A large number of sign-in attempts in rapid succession can be indicative of a DoS attack. DoS attacks attempt to guess the user’s password to gain unauthorized access, and often result in locking out the user account if security policy is enabled in Active Directory.

The Edge Server doesn’t protect against such DoS attacks. However, because Office Communications Server provides a flexible programming platform, you can use the Microsoft SIP Processing Language (MSPL) to create server applications that intercept SIP messages on the server and perform specialized logic. This is exactly what the security filter does. It’s a server application that inspects all incoming sign-in requests. Because the user is not authenticated at the Edge Server, the sign-in request is passed on to the Director or directly to the internal pool, which then performs the authentication process. The response is passed back to the Edge Server. The security filter inspects the response. If the sign-in failed, the security filter tracks the number of failed attempts for each user account. The next time a client attempts to sign in to the same user account, and the number of failed attempts exceeds the maximum number of sign-ins allowed, the security filter immediately rejects the request without passing the request to the Director or internal pool for authentication. By enforcing account lockout at the Edge Server, the security filter prevents DoS attacks at the edge of the network perimeter, and as a result, protects internal Office Communications Server resources.

Design

The security filter is comprised of two parts: an MSPL script and a .NET application. The MSPL script must be registered with the Edge Server. The MSPL script intercepts SIP messages of interest from the Edge Server and passes those messages to the .NET application that inspects the messages and enforces the account lockout logic.

To prevent such attacks, there were several options that I, with the help of the Communications Server Masters community (thanks Adrian Maclean), investigated to determine how to uniquely identify the user. We could have used source IP address, sign-in name (meaning, the SIP URI), or account name. When we investigated each of these options, and we determined that rogue clients mounting a DoS attack could spoof the source IP address eliminating this choice as a way to uniquely identify the user being authenticated. The sign-in name, although necessary to successfully sign in to Office Communications Server, is not used to authenticate the user. The sign-in name can be changed in sign-in requests and still lock out the same user account. Therefore, neither the source IP address nor the sign-in name were good sources to identify the user. Only the account name uniquely identifies the user account.

The account name, which consists of the user name and domain name, can be extracted only from the authentication protocol. Users trying to sign in from the Internet use the NTLM v.2 authentication protocol, not Kerberos. The NTLM protocol uses a three-stage handshake authentication process. The client passes the user’s credentials in the third stage of the NTLM handshake. Because the security filter runs as a trusted server application on the Edge Server, it’s allowed to intercept the sign-in request. The security filter decodes the username and domain name from the NTLM authentication message. Because the account name is not available in the response, the security filter maps the response to the request by the message ID.

When the internal pool or Director sends the authentication response to the Edge Server, the security filter can view the register response. If the sign-in failed, the security filter increments the count of failed attempts. If the sign-in succeeds, the security filter resets the count of failed attempts to zero.

Every time the Edge Server receives a sign-in request, the request is passed to the security filter. The security filter checks whether the sign-in request has exceeded the maximum allowed number for the particular user account. If the request has not exceeded the maximum lockout count permitted, the security filter allows the request to continue its course to the Director or internal pool. If the request exceeds the maximum lockout count permitted, the security filter blocks the request and returns a 403 response rejecting the request. Any further sign-in attempts are rejected for the duration of the lockout period. After the lockout period expires, it is reset to allow new sign-in requests to be authenticated.

Run the Security Filter

To run the security_filter application, open a command prompt window as a local administrator, and then go to where you downloaded the application. Enter “security_filter”. The application will indicate that two input parameters are missing.

Two configuration settings must be passed to the security filter. The first parameter is the account lockout count. This is the number of failed sign-in attempts allowed before an account lockout is triggered. The second parameter is the account lockout period. After an account is locked out, this lockout period specifies how long the account remains locked before another sign-in attempt is allowed. Any sign-in attempts during this lockout period are immediately rejected without verification.

If there’s enough interest in this application, I can build an installer (MSI) for it to simplify the installation process. Hope you enjoy it!

You can download the security filter application from the Microsoft Download Center. More sophisticated logic can be built into this application. If you need further customizations, contact me.

Summary

DoS attacks can easily be mounted against valid Active Directory user accounts from the Internet by targeting the Edge Server. By leveraging the programmable extensibility available in the Office Communications Server platform, I was able to create a security filter application that runs on the Edge Server. This server application inspects SIP requests and responses of interest, and enforces logic similar to the account lockout policy in Active Directory to prevent DoS attacks on user accounts.

Communications Server Resources

We Want to Hear from You