Microsoft Lync Server 2010: Lync Edge Server Security

While Lync Server uses many standard security measures, you can configure it for additional levels of protection.

Rui Maximo

Security risks to enterprise systems are commonplace and constant these days. Security concerns are particularly acute when you need to configure servers to expose corporate services—such as those provided by Microsoft Lync Server 2010—to the Internet for remote users and federated partners.

While exposing services to the Internet allows employees the flexibility and mobility of working remotely, there’s always the concern of attacks. Microsoft Lync Server 2010 uses a host of industry-standard security measures. All Lync Server communications are authenticated to prevent user identity or server spoofing. Network traffic is encrypted to prevent unauthorized disclosure of information or transmission tampering. The Intelligent IM (IIM) Filter helps protect against social engineering, and the Security Filter defends against Denial of Service (DoS) attacks. Taken together, these provide an effective security barrier.

Lync Security Measures

Every communication channel Microsoft Lync Server 2010 uses is encrypted. All server-to-server communications are encrypted and authenticated using Mutual Transport Layer Security (MTLS). Each server is issued a server certificate by a trusted Certificate Authority (CA), which is then used for authentication and to exchange a symmetric key used to encrypt the network session.

Users are authenticated using Kerberos if the user is internal; TLS-DSK or Windows NT LAN Manager (NTLM) if the user is external. TLS-DSK is certificate-based authentication. Upon successfully signing in to Lync Server the first time, the Lync client requests a client certificate. Lync Server issues a self-signed certificate, which the Lync client uses for all subsequent sign-ins. This certificate is renewed every six months. Lync uses other security tactics, as well:

• Lync uses Session Initiation Protocol (SIP) as the signaling protocol, which is encrypted using Transport Layer Security (TLS).
• Because it leverages the secured SIP channel, IM traffic benefits from the same encryption provided by TLS.
• Application Sharing uses the Remote Desktop Protocol (RDP). This TCP traffic uses TLS as the underlying transport. Authentication is established over the secured SIP channel.
• Web conferencing traffic uses Persistent Shared Object Model (PSOM). PSOM also uses TLS as the underlying transport, and authentication is also established over the secured SIP channel.

Audio and video (A/V) traffic traveling to and from Lync Server is protected with Secure Real Time Protocol (SRTP) to prevent any eavesdropping or packet injection. SRTP uses 128-bit Advanced Encryption Standard (AES) stream encryption. Lync Server establishes a media path that can traverse firewalls and network address translations (NATs) before allowing A/V traffic to flow between two endpoints.

To traverse firewalls, Lync Server uses the Internet Engineering Task Force (IETF) standard from the Interactive Connectivity Establishment (ICE) to determine the most direct media path between two endpoints. ICE is based on two protocols, Session Traversal Utilities for NAT (STUN) and Traversal Using Relay NAT (TURN).

STUN reflects the NAT IP addresses of the external user’s endpoint visible to the internal user’s Lync client. This helps the external user’s Lync client determine which IP addresses other clients can see across firewalls. TURN allocates media ports on the external A/V edge of the Edge Server to allow the internal user’s Lync endpoint to connect to the external user’s Lync endpoint. 

The internal endpoint can’t connect directly to the external endpoint because of the corporate firewalls. Therefore, by dynamically allocating a media port on the A/V edge of the Edge Server, the internal endpoint can send media to the external endpoint over this port. You’ll have to pre-configure the outer network perimeter firewall to allow the A/V edge of the Edge Server to initiate outbound Internet connections.

The Edge Server serves as a media relay access server (MRAS). Besides establishing the media path, this ICE negotiation exchanges a 128-bit AES key over the TLS-secured SIP channel. This key helps encrypt the media flow, and is based on a computer-generated password that rotates every eight hours. A sequence number and random generation deter replay attacks.

Enterprise voice calls also use SRTP. Therefore, they share the benefits of the A/V traffic encryption. Web traffic for services such as distribution list expansion, Address Book and conferencing data is also encrypted over HTTPS.

Enforcing Network Isolation

There are several effective practices for protecting the Edge Server from Internet-based attacks: network isolation, firewall configuration, user protection and filtering for DoS attacks. The Edge Server role is specifically designed to be deployed in the network perimeter. It enables remote user access and federation with other organizations.

It’s important to properly protect the Edge Server from Internet-based attacks. One best practice is to isolate the Edge Server with at least one firewall, preferably two, if possible. One firewall protects the Edge Server from Internet traffic and the other firewall isolates the Edge Server from internal traffic.

The Edge Server must have at least two NICs—one NIC connected to the Internet network, and the other NIC connected to the internal network. You’ll have to configure these NICs into separate subnets and not share the same IP address space (see Figure 1). These two subnets must not be routable across each other.

What this means is that an internal Lync client can’t access the external edge (for example, the Internet-facing NIC) of the Edge Server. The external Lync client can’t access the internal edge (for example, the internal-facing NIC) of the Edge Server.

Figure 1 Separate subnets can’t share the same IP address space.

Carefully Design Firewall Rules

Configuring your firewall rules is not a trivial task. To prevent opening more ports than necessary, you’ll need a solid understanding of which ports and protocols Lync Server needs to use. This is particularly true if you need to explain requirements to your security team during an audit. They’ll want to know exactly what purpose each open port serves. They’re just doing their job to ensure that no unnecessary holes are punched through the firewall that can become attack vectors.

A security audit will likely want to know the following information:

• What protocols does each Lync Server role use, in particular the Edge Server?
• Which ports do each of these protocols require?
• In which direction is the network connection allowed?

The protocol poster is a helpful visual aid and valuable source of information (see Figure 2). The color-coded lines indicate the protocols. The ports each protocol uses are shown within each line. The arrows specify the direction in which the network session is initiated.

Figure 2 Lync uses many protocols to secure and enable communications between servers and endpoints.

Protecting Users

Users will often inadvertently click on a link, then realize too late they’ve navigated to a rogue Web site and downloaded a virus or spam payload. Regaining control of your computers after one of these infections is an unpleasant experience, to say the least.

To help counter these situations, the IIM Filter prevents users from sending clickable URLs. It does so by appending an underscore (_) to the URL. This requires the recipient to copy and paste the URL, and remove the underscore before they can navigate to that site. Just that extra effort ensures the user knows what they’re doing and the navigation won’t be an accident.

Information can easily flow in and out of an organization. Users can transfer documents by e-mail, IM, USB drives and cloud-based file shares. You’ll want to enforce a consistent policy across those channels. The IIM Filter is the Lync method of preventing file transfer. If you want to permit file transfer, make sure each employee’s computer maintains an updated anti-virus scanner that will scan files downloaded with Lync.

Bracing for DoS Attacks

DoS attacks are nearly indistinguishable from legitimate sign-in requests. The only difference is in the frequency of sign-in attempts and their origin. A large number of sign-in attempts in rapid succession are indicative of a DoS attack. DoS attacks attempt to guess the user’s password to gain unauthorized access, and often result in locking out the user account if the security policy is enabled in Active Directory Domain Services.

The Edge Server doesn’t protect against such DoS attacks. However, Lync Server lets you create filters that intercept SIP messages to perform specialized logic. The IIM Filter, File Transfer Filter and Security Filter leverage this platform model to provide additional features as well.

The Security Filter is a server application that inspects all inbound sign-in requests on the Edge Server. The remote user is not authenticated at the Edge Server, so the sign-in request is passed to the director or directly to the internal pool. This is where the authentication process happens.

The response is then passed back to the Edge Server. The Security Filter inspects both the request and the response. If the sign-in fails, the Security Filter tracks the number of failed attempts for each user account.

The next time a client attempts to sign in to the same user account, and the number of failed attempts exceeds the maximum number of allowed sign-in attempts, the Security Filter immediately rejects the request without passing the request along for authentication. By enforcing account lockout at the Edge Server, the Security Filter blocks DoS attacks at the edge of the network perimeter and protects internal Lync Server resources.

Protect Your Assets

Securing your corporate data is one of your primary responsibilities. It’s important to know how to properly configure Lync Server 2010 securely and understand any additional necessary measures. This list can serve as a quick reminder for steps to protect your Lync Server deployment:

• Network Isolation: Protect your Edge Server by sandwiching it between two firewalls. Configure separate subnets to prevent loopbacks.
• Know Your Ports, Protocols and Direction (Inbound/Outbound): This will serve you well when explaining to your security team which holes you need to punch through the firewall.
• Leverage the IIM Filter: Block messages that contain clickable URLs or that attempt to initiate file transfers.
• Leverage the Security Filter: Defend against brute-force password attacks and DoS attacks.
• Regularly Patch Windows Server.

Following these security steps should help you configure your Edge Server and Microsoft Lync Server 2010 to defend against Internet attacks as best you can.

**Rui Maximo**has more than 18 years in the technology industry, having worked with Microsoft, IBM, RSA and several startups. His education is in computer science, mathematics and cryptography. His knowledge of Lync Server dates back to 2003, when it was originally called RTC. He worked on the RTC product team as a program manager and then lead program manager. Maximo is the primary author of five books on Microsoft Lync Server and its previous versions.

Related Content

• Microsoft Lync Server 2010 Security Guide
• Perimeter Network Topologies