Collation and Unicode support

Applies to: SQL Server Azure SQL Database Azure SQL Managed Instance Azure Synapse Analytics Analytics Platform System (PDW) SQL analytics endpoint in Microsoft Fabric Warehouse in Microsoft Fabric

Collations in SQL Server provide sorting rules, case, and accent sensitivity properties for your data. Collations that are used with character data types, such as char and varchar, dictate the code page and corresponding characters that can be represented for that data type.

Whether you're installing a new instance of SQL Server, restoring a database backup, or connecting server to client databases, it's important to understand the locale requirements, sorting order, and case and accent sensitivity of the data that you're working with. To list the collations that are available on your instance of SQL Server, see sys.fn_helpcollations.

When you select a collation for your server, database, column, or expression, you're assigning certain characteristics to your data. These characteristics affect the results of many operations in the database. For example, when you construct a query by using ORDER BY, the sort order of your result set might depend on the collation applied to the database, or dictated in a COLLATE clause at the expression level of the query.

To best use collation support in SQL Server, you should understand the terms that are defined in this article and how they relate to the characteristics of your data.

• Collation
  • Collation sets
  • Collation levels
• Locale
• Code page
• Sort order

A collation specifies the bit patterns that represent each character in a dataset. Collations also determine the rules that sort and compare data. SQL Server supports storing objects that have different collations in a single database. For non-Unicode columns, the collation setting specifies the code page for the data and which characters can be represented. The data that you move between non-Unicode columns must be converted from the source code page to the destination code page.

Transact-SQL statement results can vary when the statement is run in the context of different databases that have different collation settings. If possible, use a standardized collation for your organization. This way, you don't have to specify the collation in every character or Unicode expression. If you must work with objects that have different collation and code page settings, code your queries to consider the rules of collation precedence. For more information, see Collation precedence.

The options associated with a collation are case sensitivity, accent sensitivity, kana sensitivity, width sensitivity, and variation-selector sensitivity. SQL Server 2019 (15.x) introduces an additional option for UTF-8 encoding.

You can specify these options by appending them to the collation name. For example, the collation Japanese_Bushu_Kakusu_100_CS_AS_KS_WS_SC_UTF8 is case-sensitive, accent-sensitive, kana-sensitive, width-sensitive, and UTF-8 encoded. As another example, the collation Japanese_Bushu_Kakusu_140_CI_AI_KS_WS_VSS is case-insensitive, accent-insensitive, kana-sensitive, width-sensitive, variation-selector-sensitive, and it uses a legacy code page for varchar.

The behavior associated with these various options is described in the following table:

¹ If Binary or Binary-code point is selected, the Case-sensitive (_CS), Accent-sensitive (_AS), Kana-sensitive (_KS), and Width-sensitive (_WS) options aren't available.

Each collation is combined as a series of suffixes to define case-, accent-, width-, or kana-sensitivity. The following examples describe sort order behavior for various combinations of suffixes.

¹ If Binary or Binary-code point is selected, the Case-sensitive (_CS), Accent-sensitive (_AS), Kana-sensitive (_KS), and Width-sensitive (_WS) options aren't available.

² Adding the UTF-8 option (_UTF8) enables you to encode Unicode data by using UTF-8. For more information, see the UTF-8 Support section in this article.

SQL Server supports the following collation sets:

• Windows collations
• Binary collations
• SQL Server collations

Windows collations define rules for storing character data that are based on an associated Windows system locale. For a Windows collation, you can implement a comparison of non-Unicode data by using the same algorithm as that for Unicode data. The base Windows collation rules specify which alphabet or language is used when dictionary sorting is applied. The rules also specify the code page that's used to store non-Unicode character data. Both Unicode and non-Unicode sorting are compatible with string comparisons in a particular version of Windows. This provides consistency across data types within SQL Server, and it lets developers sort strings in their applications by using the same rules that are used by SQL Server. For more information, see Windows collation name.

Binary collations sort data based on the sequence of coded values that are defined by the locale and data type. They're case-sensitive. A binary collation in SQL Server defines the locale and the ANSI code page that are used. This enforces a binary sort order. Because they're relatively simple, binary collations help improve application performance. For non-Unicode data types, data comparisons are based on the code points that are defined on the ANSI code page. For Unicode data types, data comparisons are based on the Unicode code points. For binary collations on Unicode data types, the locale isn't considered in data sorts. For example, Latin1_General_BIN and Japanese_BIN yield identical sorting results when they're used on Unicode data. For more information, see Windows collation name.

There are two types of binary collations in SQL Server:

• The legacy BIN collations, which performed an incomplete code-point-to-code-point comparison for Unicode data. Legacy binary collations compared the first character as WCHAR, followed by a byte-by-byte comparison. In a BIN collation, only the first character is sorted according to the code point, and remaining characters are sorted according to their byte values.

• The newer BIN2 collations, which implement a pure code-point comparison. In a BIN2 collation, all characters are sorted according to their code points. Because the Intel platform is a little endian architecture, Unicode code characters are always stored byte-swapped.

SQL Server collations (SQL_*) provide sort order compatibility with earlier versions of SQL Server. The dictionary sorting rules for non-Unicode data are incompatible with any sorting routine that's provided by Windows operating systems. However, sorting Unicode data is compatible with a particular version of Windows sorting rules. Because SQL Server collations use different comparison rules for non-Unicode and Unicode data, you see different results for comparisons of the same data, depending on the underlying data type.

For example, if you're using the SQL collation SQL_Latin1_General_CP1_CI_AS, the non-Unicode string 'a-c' is less than the string 'ab' because the hyphen (-) is sorted as a separate character that comes before b. However, if you convert these strings to Unicode and you perform the same comparison, the Unicode string N'a-c' is considered to be greater than N'ab', because the Unicode sorting rules use a word sort that ignores the hyphen.

For more information, see SQL Server Collation Name.

During SQL Server setup, the default installation collation setting is determined by the operating system (OS) locale. You can change the server-level collation either during setup or by changing the OS locale before installation. For backward compatibility reasons, the default collation is set to the oldest available version that's associated with each specific locale. Therefore, this isn't always the recommended collation. To take full advantage of SQL Server features, change the default installation settings to use Windows collations. For example, for the OS locale "English (United States)" (code page 1252), the default collation during setup is SQL_Latin1_General_CP1_CI_AS, and it can be changed to its closest Windows collation counterpart, Latin1_General_100_CI_AS_SC.

When you upgrade an English-language instance of SQL Server, you can specify SQL Server collations (SQL_*) for compatibility with existing instances of SQL Server. Because the default collation for an instance of SQL Server is defined during setup, make sure that you specify the collation settings carefully when the following conditions are true:

• Your application code depends on the behavior of previous SQL Server collations.
• You must store character data that reflects multiple languages.

Setting collations are supported at the following levels of an instance of SQL Server:

• Server-level collations
• Database-level collations
• Column-level collations
• Expression-level collations

The default server collation is determined during SQL Server setup, and it becomes the default collation of the system databases and all user databases.

The following table shows the default collation designations, as determined by the operating system (OS) locale, including their Windows and SQL Language Code Identifiers (LCID):

After you've assigned a collation to the server, you can change it only by exporting all database objects and data, rebuilding the master database, and importing all database objects and data. Instead of changing the default collation of an instance of SQL Server, you can specify the desired collation when you create a new database or database column.

To query the server collation for an instance of SQL Server, use the SERVERPROPERTY function:

SELECT CONVERT (NVARCHAR (128), SERVERPROPERTY('collation'));

To query the server for all available collations, use the following fn_helpcollations() built-in function:

SELECT *
FROM sys.fn_helpcollations();

You can't change or set the logical server collation on Azure SQL Database, but can configure each database's collations both for data and for the catalog. The catalog collation determines the collation for system metadata, such as object identifiers. Both collations can be specified independently when you create the database in the Azure portal, in T-SQL with CREATE DATABASE, in PowerShell with New-AzSqlDatabase.

Server-level collation in Azure SQL Managed Instance can be specified when the instance is created and can't be changed later.

For more information, see Set or Change the Server Collation.

When you create or modify a database, you can use the COLLATE clause of the CREATE DATABASE or ALTER DATABASE statement to specify the default database collation. If no collation is specified, the database is assigned the server collation.

You can't change the collation of system databases unless you change the collation for the server.

• In SQL Server and Azure SQL Managed Instance, the database collation is used for all metadata in the database, and the collation is the default for all string columns, temporary objects, variable names, and any other strings used in the database.
• In Azure SQL Database, there's no server collation, so each database has a collation for data and a collation for the catalog. The CATALOG_COLLATION is used for all metadata in the database, and the collation is the default for all string columns, temporary objects, variable names, and any other strings used in the database. The CATALOG_COLLATION is set upon creation and can't be changed.

When you change the collation of a user database, there can be collation conflicts when queries in the database access temporary tables. Temporary tables are always stored in the tempdb system database, which uses the collation for the instance. Queries that compare character data between the user database and tempdb might fail if the collations cause a conflict in evaluating the character data. You can resolve this issue by specifying the COLLATE clause in the query. For more information, see COLLATE.

You can change the collation of a user database by using an ALTER DATABASE statement similar to the following code sample:

ALTER DATABASE myDB COLLATE Greek_CS_AI;

You can retrieve the current collation of a database by using a statement similar to the following code sample:

SELECT CONVERT (NVARCHAR (128), DATABASEPROPERTYEX('database_name', 'collation'));

When you create or alter a table, you can specify collations for each character-string column by using the COLLATE clause. If you don't specify a collation, the column is assigned the default collation of the database.

You can change the collation of a column by using an ALTER TABLE statement similar to the following code sample:

ALTER TABLE myTable ALTER COLUMN mycol NVARCHAR (10) COLLATE Greek_CS_AI;

Expression-level collations are set when a statement is run, and they affect the way a result set is returned. This enables ORDER BY sort results to be locale-specific. To implement expression-level collations, use a COLLATE clause such as the following code sample:

SELECT name
FROM customer
ORDER BY name COLLATE Latin1_General_CS_AI;

A locale is a set of information that's associated with a location or a culture. The information can include the name and identifier of the spoken language, the script that's used to write the language, and cultural conventions. Collations can be associated with one or more locales. For more information, see Locale IDs Assigned by Microsoft.

A code page is an ordered set of characters of a given script in which a numeric index, or code point value, is associated with each character. A Windows code page is typically referred to as a character set or a charset. Code pages are used to provide support for the character sets and keyboard layouts that are used by different Windows system locales.

Sort order specifies how data values are sorted. The order affects the results of data comparison. Data is sorted by using collations, and it can be optimized by using indexes.

Unicode is a standard for mapping code points to characters. Because it's designed to cover all the characters of all the languages of the world, you don't need different code pages to handle different sets of characters.

Storing data in multiple languages within one database is difficult to manage when you use only character data and code pages. It's also difficult to find one code page for the database that can store all the required language-specific characters. Additionally, it's difficult to guarantee the correct translation of special characters when they're being read or updated by a variety of clients that are running various code pages. Databases that support international clients should always use Unicode data types instead of non-Unicode data types.

For example, consider a database of customers in North America that must handle three major languages:

• Spanish names and addresses for Mexico
• French names and addresses for Quebec
• English names and addresses for the rest of Canada and the United States

When you use only character columns and code pages, you must take care to ensure that the database is installed with a code page that will handle the characters of all three languages. You must also take care to guarantee the correct translation of characters from any of the languages when the characters are read by clients that are running a code page for another language.

It would be difficult to select a code page for character data types that will support all the characters that are required by a worldwide audience. The easiest way to manage character data in international databases is to always use a data type that supports Unicode.

If you store character data that reflects multiple languages in SQL Server (SQL Server 2005 (9.x) and later), use Unicode data types (nchar, nvarchar, and ntext) instead of non-Unicode data types (char, varchar, and text).

Alternatively, starting with SQL Server 2019 (15.x), if a UTF-8 enabled collation (_UTF8) is used, previously non-Unicode data types (char and varchar) become Unicode data types using UTF-8 encoding. SQL Server 2019 (15.x) doesn't change the behavior of previously existing Unicode data types (nchar, nvarchar, and ntext), which continue to use UCS-2 or UTF-16 encoding. For more information, see Storage differences between UTF-8 and UTF-16.

Significant limitations are associated with non-Unicode data types. This is because a non-Unicode computer is limited to using a single code page. You might experience performance gain by using Unicode, because it requires fewer code-page conversions. Unicode collations must be selected individually at the database, column, or expression level because they aren't supported at the server level.

When you move data from a server to a client, your server collation might not be recognized by older client drivers. This can occur when you move data from a Unicode server to a non-Unicode client. Your best option might be to upgrade the client operating system so that the underlying system collations are updated. If the client has database client software installed, you might consider applying a service update to the database client software.

To use the UTF-8 collations that are available in SQL Server 2019 (15.x), and to improve searching and sorting of some Unicode characters (Windows collations only), you must select UTF-8 encoding-enabled collations (_UTF8).

• The UTF8 flag can be applied to:

  • Linguistic collations that already support supplementary characters (_SC) or variation-selector-sensitive (_VSS) awareness
  • BIN2 binary collation

• The UTF8 flag can't be applied to:

  • Linguistic collations that don't support supplementary characters (_SC) or variation-selector-sensitive (_VSS) awareness
  • The BIN binary collations
  • The SQL_* collations

To evaluate issues that are related to using Unicode or non-Unicode data types, test your scenario to measure performance differences in your environment. It's a good practice to standardize the collation that's used on systems across your organization, and to deploy Unicode servers and clients wherever possible.

In many situations, SQL Server interacts with other servers or clients, and your organization might use multiple data-access standards between applications and server instances. SQL Server clients are one of two main types:

• Unicode clients that use OLE DB and Open Database Connectivity (ODBC) version 3.7 or later.
• Non-Unicode clients that use DB-Library and ODBC version 3.6 or earlier.

The following table provides information about using multilingual data with various combinations of Unicode and non-Unicode servers:

The Unicode Consortium allocates to each character a unique code point, which is a value in the range 000000 - 10FFFF. The most frequently used characters have code point values in the range 000000 - 00FFFF (65,536 characters) which fit into an 8-bit or 16-bit word in memory and on-disk. This range is usually designated as the Basic Multilingual Plane (BMP).

But the Unicode Consortium has established 16 additional "planes" of characters, each the same size as the BMP. This definition allows Unicode the potential to represent 1,114,112 characters (that is, 2¹⁶ * 17 characters) within the code point range 000000 - 10FFFF. Characters with code point value larger than 00FFFF require two to four consecutive 8-bit words (UTF-8), or two consecutive 16-bit words (UTF-16). These characters located beyond the BMP are called supplementary characters, and the additional consecutive 8-bit or 16-bit words are called surrogate pairs. For more information about supplementary characters, surrogates, and surrogate pairs, see the Unicode Standard.

SQL Server provides data types such as nchar and nvarchar to store Unicode data in the BMP range (000000 - 00FFFF), which the Database Engine encodes using UCS-2.

SQL Server 2012 (11.x) introduced a new family of supplementary character (_SC) collations that can be used with the nchar, nvarchar, and sql_variant data types to represent the full Unicode character range (000000 - 10FFFF). For example: Latin1_General_100_CI_AS_SC or, if you're using a Japanese collation, Japanese_Bushu_Kakusu_100_CI_AS_SC.

SQL Server 2019 (15.x) extends supplementary character support to the char and varchar data types with the new UTF-8 enabled collations (_UTF8). These data types are also capable of representing the full Unicode character range.

If you use supplementary characters:

• Supplementary characters can be used in ordering and comparison operations in collation versions 90 or greater.

• All version 100 collations support linguistic sorting with supplementary characters.

• Supplementary characters aren't supported for use in metadata, such as in names of database objects.

• The SC flag can be applied to:

  • Version 90 collations
  • Version 100 collations

• The SC flag can't be applied to:

  • Version 80 non-versioned Windows collations
  • The BIN or BIN2 binary collations
  • The SQL* collations
  • Version 140 collations (these don't need the SC flag, because they already support supplementary characters)

The following table compares the behavior of some string functions and string operators when they use supplementary characters with and without a supplementary character-aware (SCA) collation:

GB18030 is a separate standard that's used in the People's Republic of China for encoding Chinese characters. In GB18030, characters can be 1, 2, or 4 bytes in length. SQL Server provides support for GB18030-encoded characters by recognizing them when they enter the server from a client-side application and converting and storing them natively as Unicode characters. After they're stored in the server, they're treated as Unicode characters in any subsequent operations.

You can use any Chinese collation, preferably the latest 100 version. All version 100 collations support linguistic sorting with GB18030 characters. If the data includes supplementary characters (surrogate pairs), you can use the SC collations that are available in SQL Server to improve searching and sorting.

SQL Server can support inputting, storing, changing, and displaying complex scripts. Complex scripts include the following types:

• Scripts that include the combination of both right-to-left and left-to-right text, such as a combination of Arabic and English text.
• Scripts whose characters change shape depending on their position, or when combined with other characters, such as Arabic, Indic, and Thai characters.
• Languages, such as Thai, that require internal dictionaries to recognize words because there are no breaks between them.

Database applications that interact with SQL Server must use controls that support complex scripts. Standard Windows form controls that are created in managed code are complex-script-enabled.

Starting with SQL Server 2017 (14.x), new Japanese collation families are supported, with the permutations of various options (_CS, _AS, _KS, _WS, and _VSS), as well as _BIN and _BIN2.

To list these collations, you can query the SQL Server Database Engine:

SELECT name,
       description
FROM sys.fn_helpcollations()
WHERE COLLATIONPROPERTY(name, 'Version') = 3;

All the new collations have built-in support for supplementary characters, so none of the new 140 collations has (or needs) the SC flag.

These collations are supported in Database Engine indexes, memory-optimized tables, columnstore indexes, and natively compiled modules.

SQL Server 2019 (15.x) introduces full support for the widely used UTF-8 character encoding as an import or export encoding, and as database-level or column-level collation for string data. UTF-8 is allowed in the char and varchar data types, and it's enabled when you create or change an object's collation to a collation that has a UTF8 suffix. One example is changing Latin1_General_100_CI_AS_SC to Latin1_General_100_CI_AS_SC_UTF8.

UTF-8 is available only to Windows collations that support supplementary characters, as introduced in SQL Server 2012 (11.x). The nchar and nvarchar data types allow UCS-2 or UTF-16 encoding only, and they remain unchanged.

Azure SQL Database and Azure SQL Managed Instance also support UTF-8 on database and column level, while SQL Managed Instance supports this on a server level as well.

The Unicode Consortium allocates to each character a unique code point, which is a value in the range 000000 - 10FFFF. With SQL Server 2019 (15.x), both UTF-8 and UTF-16 encodings are available to represent the full range:

• With UTF-8 encoding, characters in the ASCII range (000000 - 00007F) require 1 byte, code points 000080 - 0007FF require 2 bytes, code points 000800 - 00FFFF require 3 bytes, and code points 0010000 - 0010FFFF require 4 bytes.
• With UTF-16 encoding, code points 000000 - 00FFFF require 2 bytes, and code points 0010000 - 0010FFFF require 4 bytes.

The following table lists the encoding storage bytes for each character range and encoding type:

¹ Storage bytes refer to the encoded byte length, not the data-type on-disk storage size. For more information about on-disk storage sizes, see nchar and nvarchar and char and varchar.

² The code point range for supplementary characters.

Choosing the appropriate Unicode encoding and data type might give you significant storage savings or increase your current storage footprint, depending on the character set in use. For example, when you use a Latin collation that's UTF-8 enabled, such as Latin1_General_100_CI_AI_SC_UTF8, a char(10) column stores 10 bytes and can hold 10 ASCII characters in the range 0 - 127. But it can hold only five characters in the range 128 - 2047 and only three characters in the range 2048 - 65535. By comparison, because a nchar(10) column stores 10 byte-pairs (20 bytes), it can hold 10 characters in the range 0 - 65535.

Before you decide whether to use UTF-8 or UTF-16 encoding for a database or column, consider the distribution of string data that will be stored:

• If it's mostly in the ASCII range 0 - 127 (such as English), each character requires 1 byte with UTF-8 and 2 bytes with UTF-16. Using UTF-8 provides storage benefits. Changing an existing column data type with ASCII characters in the range 0 - 127 from nchar(10) to char(10), and using an UTF-8 enabled collation, translates into a 50 percent reduction in storage requirements. This reduction is because nchar(10) requires 20 bytes for storage, compared with char(10), which requires 10 bytes for the same Unicode string representation.
• Above the ASCII range, almost all Latin-based script, and Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic, Syriac, Tāna, and N'Ko, require 2 bytes per character in both UTF-8 and UTF-16. In these cases, there are no significant storage differences for comparable data types (for example, between using char or nchar).
• If it's mostly East Asian script (such as Korean, Chinese, and Japanese), each character requires 3 bytes with UTF-8 and 2 bytes with UTF-16. Using UTF-16 provides storage benefits.
• Characters in the range 010000 - 10FFFF require 4 bytes in both UTF-8 and UTF-16. In these cases, there are no storage differences for comparable data types (for example, between using char or nchar).

For other considerations, see Write International Transact-SQL Statements.

Because in char and varchar or in nchar and nvarchar, the n defines the byte storage size, not the number of characters that can be stored, it's important to determine the data type size you must convert to. The characters that exceed the size are to be truncated.

For example, consider a column defined as nvarchar(100) that stores 180 bytes of Japanese characters. In this example, the column data is currently encoded using UCS-2 or UTF-16, which uses 2 bytes per character. Converting the column type to varchar(200) isn't sufficient to prevent data truncation, because the new data type can only store 200 bytes, but Japanese characters require 3 bytes when encoded in UTF-8. The column must be defined as varchar(270) to avoid data loss through data truncation.

Therefore, you must know in advance what the projected byte size for the column definition is, before converting existing data to UTF-8, and adjust the new data type size accordingly. Refer to the Transact-SQL script or the SQL Notebook in the Data Samples GitHub, which use the DATALENGTH function and the COLLATE statement, to determine the appropriate data length requirements for UTF-8 conversion operations in an existing database.

To change the column collation and data type in an existing table, use one of the methods described in Set or Change the Column Collation.

To change the database collation, allowing new objects to inherit the database collation by default, or to change the server collation, allowing new databases to inherit the system collation by default, see the Related tasks section of this article.

• SQL Server Best Practices Collation Change
• Use unicode character format to import or export data (SQL Server)
• Write International Transact-SQL Statements
• SQL Server Best Practices Migration to Unicode
• Unicode Consortium
• Unicode Standard
• UTF-8 Support in OLE DB Driver for SQL Server
• SQL Server Collation Name (Transact-SQL)
• Windows collation name (Transact-SQL)
• Introducing UTF-8 support for SQL Server
• COLLATE (Transact-SQL)
• Collation precedence
• Contained database collations
• Choose a Language When Creating a Full-Text Index
• sys.fn_helpcollations (Transact-SQL)
• Single-Byte and Multibyte Character Sets