Improving Performance When Converting SAP to Unicode at Microsoft

Note on IT

Published: October 25, 2007

At Microsoft, SAP R/3 is used to handle almost all of the financial operations of the company. Because SAP R/3 downtime costs Microsoft between 2 million and 4 million dollars per hour, minimizing downtime and improving performance was of critical importance when Microsoft converted its 2.8-terabyte SQL Server 2005 back-end database to Unicode. This Note is intended to help other organizations that intend to convert SAP databases to Unicode gain the benefit from the experience at Microsoft IT.

Introduction

Like many other large organizations, Microsoft relies on SAP AG enterprise resource planning (ERP) software products to manage the day-to-day operations of its business. An early adopter of SAP products, Microsoft first deployed SAP R/3 running together with a Microsoft® SQL Server™ 6.5 back-end database in 1996. Since that time, SAP has grown to be a critical part of the financial operations of the company.

The largest part of the SAP implementation at Microsoft is the SAP R/3 deployment. With a 2.8-terabyte back-end database running on Microsoft SQL Server 2005 and with 60,000 users, of whom between 200 and 1,200 are concurrent users, the SAP R/3 implementation at Microsoft is in the top 10 percent of the largest SAP R/3 deployments worldwide.

The SAP R/3 implementation at Microsoft has the following performance characteristics:

• Over 900,000 dialog steps per business day
• 19 million transactions per month
• 48 batch jobs executed per minute
• An average user response time of less than 0.7 seconds

This SAP R/3 implementation is a multiple language, multiple-code page system that uses the Latiin-1 and kanji code pages. Because the SAP R/3 implementation at Microsoft was an earlier implementation, it did not use the Unicode code page. To maintain a supported environment and to create a base to upgrade to more recent SAP ERP releases, Microsoft had to convert its 2.8-terabyte SAP R/3 back-end database to Unicode.

Because of the importance of SAP R/3 at Microsoft, SAP R/3 downtime is critically important. Depending on the time of year, it costs Microsoft between 2 million and 4 million dollars for each hour that SAP R/3 is unavailable to end-users. Therefore, the reduction of bottlenecks that could slow down the conversion process was one of the key aspects of the Unicode conversion operation.

This document contains the lessons learned by the Microsoft IT Unicode conversion team (Enterprise Application Services (EAS) SAP Technical Services) and the best practices that were used when the team converted the SAP R/3 database to Unicode. Specifically, this document is intended to create an understanding of the types of issues that may slow the export and import of data during the conversion to Unicode and to describe steps to take to reduce or remove these bottlenecks. By using these steps, the conversion team was able to reduce the downtime of the SAP R/3 system to less than 22 hours when the team converted its 2.8-terabyte SAP R/3 database to Unicode.

An IT Showcase Webcast called SAP Unicode Conversion Using SQL Server 2005 on Commodity Hardware includes a discussion on this topic. The Webcast is available at http://msevents.microsoft.com/cui/WebCastEventDetails.aspx?culture=en-US&EventID=1032348439&CountryCode=US.

Background

SAP R/3 ECC 5.0 and later versions only support Unicode for multiple languages that use different code pages. To maintain a supported environment, and to prepare for upgrades to the latest SAP ERP releases, Microsoft IT started working on a project to convert its SAP R/3 database to Unicode.

Project Timeline

Microsoft IT started its Unicode conversion project in 2005. However, Microsoft IT first approached the requirement for Unicode in 2003. The following table illustrates the timeline of the Unicode conversion project.

Table 1. Unicode Conversion Timeline

Period	Activity
2003	Microsoft IT's first contact with Unicode
September 2004 through December 2004	Prototype conversion of a copy of the SAP R/3 production data
July 2005	Start of the Unicode conversion project
November 2005	First conversion operation completed
January 03, 2006	Start of risk area testing
February 24, 2006	ABAP adjustments completed
February 28, 2006	End of risk area testing
March 18, 2006	Development system converted to Unicode
June 01, 2006	Start of stress testing in the Unicode sandbox environment
August 25, 2006	UAT* test system converted to Unicode
February 09, 2007	Production system converted to Unicode
August 01, 2007	Non-Unicode reference system removed

* UAT (User Acceptance Testing)

Unicode Conversion Scope

The conversion team restricted the Unicode conversion project to one that performed only a technical conversion of the SAP R/3 database. No application modifications were included in the conversion project, and no business units were involved in the project.

This technical conversion had the following requirements:

• To reduce the downtime of the SAP R/3 system
• To make sure that the performance of the Unicode system equaled or exceeded that of the original system

Converting Custom Objects

As part of defining the overall scope of the Unicode conversion operation, the conversion team had to determine how many custom objects in the SAP R/3 database required conversion to Unicode. This was accomplished by running a standard SAP Advanced Business Application Programming (ABAP) program to return the list of objects in SAP R/3. Then, the conversion team examined the objects to determine whether they required conversion to Unicode. By examining the 2,670 custom objects in the SAP R/3 database, the conversion team determined the following:

• 1,814 objects (68 percent) were already Unicode compliant
• 136 objects (5 percent) were not used
• 43 objects (2 percent) were no longer required (obsolete)
• 677 objects (25 percent) required conversion to Unicode

Many objects were already Unicode compliant. This is because, in 2004, Microsoft IT decided to adjust custom objects to conform to Unicode requirements going forward. The effect of this decision was that after the Unicode project was started, only objects that had not been changed in the last one and a half years required adjustment.

Distribution Monitor

After consulting SAP support representatives, the conversion team determined that the SAP conversion tool, Distribution Monitor, would be best suited to convert the SAP R/3 data to Unicode. This tool provides the following functionality:

• Starts a configured number of R/3 load processes
• Monitors the R/3 load processes
• Controls the R/3 load processes

Distribution Monitor controls the SAP requirement of one R/3 load process to export each table or package and one R/3 load process to import each table or package.

To convert the data to Unicode, the table or package is exported to a file. The conversion operation occurs during this export process. If the language information is stored in the database row, a direct conversion to Unicode occurs. However, if the language information is not stored in the database row, the conversion operation must access the system vocabulary to obtain the language information for the data conversion operation. If data cannot be converted to Unicode, Distribution Monitor generates the failure information in an XML log file for later examination.

After the data is converted to Unicode, Distribution Monitor uses an R/3 load process to import the converted data into a prepared database.

Figure 1 illustrates the conversion process used by Distribution Monitor.

Figure 1. Distribution Monitor

Configuring a Test Environment

To make sure that the operation to convert the SAP R/3 data to Unicode required minimal downtime, the conversion team configured a test environment (sandbox) in which to test the conversion process. It was here that the conversion team detected and resolved network and database bottlenecks that could challenge the conversion operation.

The SAP R/3 Production Environment

The SAP R/3 environment at Microsoft is made up of the following:

• Four application servers
• One Central Instance server
• Two back-end database servers

Microsoft IT runs its whole SAP R/3 production environment on servers that are considered to be commodity hardware. These servers have the following characteristics:

Back-end Database Servers

• Four x64, 2.2-GHz dual-core processors
• 32 GB RAM (expanded to 48 GB RAM for the Unicode conversion)

Application Servers and the Central Instance Server

• Four x64, 2.2-GHz dual-core processors
• 16 GB RAM (expanded to 48 GB RAM for the Unicode conversion)

The following figure illustrates the SAP R/3 production environment at Microsoft.

Figure 2. SAP R/3 production environment

Microsoft IT introduces changes into the SAP R/3 production environment by using the following procedure.

1. Modifications for the SAP R/3 system are developed on an SAP R/3 development system.
2. These changes are moved to an SAP R/3 test environment by using an SAP transport for SAP application changes or program changes. This test environment has the same configuration as the SAP R/3 production environment.
3. After the changes have been thoroughly tested, they are introduced into the SAP R/3 production environment by using an SAP transport for SAP application changes or program changes.

The Unicode Conversion Test Environment

The conversion test environment duplicates the SAP R/3 production environment except that database mirroring between the two SQL Server 2005 back-end servers is disabled. For the Unicode conversion tests, as well as for the conversion operation in the production environment, the conversion team used one of the back-end database servers (DB1) as the export server and the other back-end database server (DB2) as the import database server. After the conversion operation was completed, the conversion team configured SQL Server 2005 database mirroring from the second database server (DB2) to the first database server (DB1).

To create a test environment for the Unicode conversion operation, the conversion team implemented the following procedure:

1. A copy of the production system was introduced into the Unicode conversion sandbox system.
2. The Unicode conversion operation was performed in the sandbox environment. Refreshed copies of the SAP R/3 system were taken from the production environment as required.
3. A copy of the converted Unicode data was taken from the Unicode conversion sandbox environment, and then introduced into a new sandbox development system. This development system is a small installation that is used only for unit testing of ABAP programs.

Figure 3: SAP R/3 Unicode test procedure

The conversion team used the Unicode conversion sandbox environment to streamline and improve the performance of the conversion process. To this end, the conversion team performed multiple Unicode conversion tests. Each test was compared with earlier tests to determine whether sufficient performance improvements had been made. After each test run, the conversion team examined the results to identify and resolve bottlenecks that could slow down the export or import process.

Improving the Conversion Process

Creating a Baseline

To create a baseline with which to compare changes to the conversion process, the conversion team performed the initial conversion operation as a simple "out of the box" test. For this test, no adjustments were made to SAP R/3, and no modifications were made to the sandbox environment.

Identifying Bottlenecks

The conversion team examined the results of the initial test to identify bottlenecks that could be removed or reduced to improve the run times for the export and import operations. To gather information about each conversion run, the conversion team used the Windows Performance tool to gather performance counter information for each server. By reviewing the performance counter information, the conversion team could identify CPU or network bottlenecks that occurred during the export or import operations. Additionally, the database file sizes and transaction log file sizes were logged during the import operation in each test. To do this, the conversion team used the following SQL script:

set nocount on

SELECT 'FileType'=case FileType when 0 then 'Data' when 1 then 'Log' end, sum(size)/128 as 'Size (MB)', sum(Used)/128 as 'Used (MB)', (sum(size)/128)-(sum(Used)/128) as 'Free (MB)' from

(Select FILEPROPERTY (name, 'islogfile') as

FileType, size, FILEPROPERTY (name, 'spaceused') as Used from sysfiles)

as UsedResults

group by FileType

The conversion team configured this script as a custom SQL job on the import database server. This job was configured to run approximately every ten minutes. The job returns the current database size and the current transaction log file size and then appends this information to a file. By examining this log file, the conversion team was able to determine the database growth after the conversion to Unicode together with the size of the transaction log files at various times during the import operation.

This information was helpful in locating tables or packages that caused unacceptably long import run times. Additionally, by using this SQL job, the conversion team was able to determine how much local log file storage to allocate to the import database server for the conversion operation.

After reviewing the results of a test, the conversion team made changes to the SAP R/3 system or environment, as appropriate, and then the team re-ran the conversion test. The team repeated these tests as necessary to optimize each aspect of the conversion process.

Optimizing Network Bandwidth

During the baseline conversion test, it took over one week to convert the SAP R/3 data to Unicode. Had this conversion been performed in a production environment, over one week of SAP R/3 downtime would have occurred. After reviewing the results of the conversion operation, the conversion team noticed certain key factors that affected the export and import operations.

Storage Configuration

When the conversion team initially calculated the storage requirements for the Unicode conversion tests, the conversion team allocated a single 1-terabyte drive on the import database server together with a very large drive for the SQL Server log file of the import database. The 1-terabyte drive was configured as the storage location for the exported (converted) files from all of the export servers.

Because the import database server was already connected to the storage area network (SAN), this allocation was both simple and inexpensive. However, during the export and import operations, the conversion team noticed a crucial factor.

Because the export processes and the import processes shared the network link to the storage drive, the network became flooded with the export and import traffic. This bottleneck was the main cause of the long export and import times for the initial conversion test. The conversion team determined that no changes that it made to the SAP R/3 system would appreciably improve performance in the conversion process until this network bottleneck was resolved.

Local Storage

After examining the results of the conversion test, the conversion team determined that by connecting each server that was running export and import tasks to the storage area network (SAN), it could remove the bottleneck in the conversion process.

To this end, the conversion team added the following local storage drives to the servers in the sandbox environment and later to the servers in the production environment:

Table 2. Local Storage Configuration

Server Name	Storage Size	Description
Export server - DB1	250 GB	Storage for export files (converted data) *
Import server - DB2	1 TB	Storage for SQL Server 2005 transaction logs
Application server A	100 GB	Storage for export files (converted data)
Application server B	100 GB	Storage for export files (converted data)
Application server C	100 GB	Storage for export files (converted data)
Application server D	100 GB	Storage for export files (converted data)
Central Instance server	100 GB	Storage for export files (converted data)

* Export processes for certain large tables were left on the export database server.

To calculate these storage requirements, the conversion team reviewed conversion test results to determine how much data each server exported. Although the conversion team added 100-GB drives to each application server, only half of that space was generally used during export operations.

To add the storage drives, the conversion team had to schedule a short period of SAP R/3 downtime. This was done to add a host bus adapter (HBA) to each server to connect to the SAN. Although the addition of local storage increased the cost of the conversion operation, the cost increase was only temporary. This was because the external storage was removed after the conversion operation was completed.

The conversion team found that the configuration of local storage drives effectively removed the network bottleneck that had affected the initial conversion operation. During subsequent export operations, the team noticed speeds of up to 200 MB/second reading data from the storage area network (SAN). At the same time, the team noticed up to 300 MB/second speed bursts writing data to the SAN while SQL Server executed a checkpoint.

After implementing local storage drives to resolve the network bottleneck, the conversion team was able to concentrate on adjusting the SAP R/3 environment to further improve export and import performance.

Improving Export Times

Reviews of the export tests revealed the following two key issues that slowed the export operations:

• Certain servers had much more data to export. Therefore the export operations on these servers took much longer to complete. After examining the affected servers, the conversion team noticed that a few large tables consumed most of the export time.
• On each export server, certain tables were much larger than others and therefore took longer to export and convert.

To reduce this imbalance in export processes, the conversion team decided to export the largest tables directly from the export database server. The application servers would be used to export the remaining tables. This strategy reduced load on the particular application servers and balanced network usage among the export servers.

Table Splitting

Even with the export operations balanced among the export database server and the application servers, the operation experienced very long run times when exporting certain tables.

The conversion team calculated that to meet the requirement of converting all the SAP R/3 data to Unicode within a single weekend, no table or package could take longer than five hours to export or to import.

By examining the results of the export tests, the conversion team identified the particular tables that exceeded the five-hour time frame. To reduce these export times, the conversion team implemented SAP table splitting to export smaller packages. By using table splitting and by increasing the number of R/3 loads correspondingly to export the additional packages, the conversion team made sure that no export operation exceeded the five-hour export time frame. Through repeated testing, the conversion team determined that it had to split 41 tables to reach the five-hour goal. Twenty-seven of these tables had secondary indexes that the team had to later create manually. For a list of these tables, see Appendix A..

To split a table, the Microsoft IT Unicode conversion team used the SAP AMD64 R3TA Table Splitter tool together with the supporting dbsl files.

During the next Unicode conversion test, the conversion team experienced improved run times. However, the performance did not improve as much as expected. This behavior occurred because of how the SAP R3TA tool built the packages for each table. When building a package, the tool located the column that had the highest selectivity in the clustered index. The tool created the WHERE clause for the package only according to this particular column. These WHERE clauses were not supported by indexes on the table. Because of this, each export process ended up performing a full table scan.

To resolve this problem, the conversion team manually changed all of the WHERE clauses to use several fields in the clustered index instead of only one field. This had the effect of allowing the use of the clustered index during the scan which greatly improved the export run times.

The conversion team used the following strategy to create the WHERE clauses:

• Use as many fields as required from the clustered index to define the selectivity of the export operation.
• Configure the WHERE clause to include all the fields from the clustered index, starting from the left of the index, without leaving out fields, up to the last field that is required to define the selectivity.
• Try to build 10 or more packages, depending on the size of the table. In some cases, a table may have to be split into 20 or more packages.
• Build the packages so that they have very close row sizes.
• If WHERE clauses are built manually, make sure to not exclude any values. Use the greater than (>) sign and the less than (<) sign to define ranges. Also, include the range value itself in one of the SELECT statements. However, do not use the same values in different WHERE clauses.

For example, consider a table (Table X) that has three fields in a clustered index. In this example, you want to create three packages as follows:

• The first field on the clustered index is MANDT. Only one value exists for this column in the database. For example, MANDT='200'.
• The second field on the clustered index is VBELN. The smallest value in VBELN is '0060001234'.

In this example, you can create three equal packages:

• Range VBELN=''0060001234' to VBELN='006084390' containing 10,000 documents
• Range VBELN='006084395' to VBELN=''006096080' containing 10,100 documents
• Range VBELN='006096100' to VBELN=''006201020' containing 10,050 documents

One method of defining the WHERE clauses for these ranges is as follows:

• WHERE MANDT='200' and VBELN >= '0060001234' and VBELN <= '006084390'
• WHERE MANDT='200' and VBELN>='006084395' and VBELN<='006096080'
• WHERE MANDT='200' and VBELN>='006096100' and VBELN<=006201020'

Although these WHERE clauses are valid, they may not be optimal for the SAP environment. This is because if the SAP system is customized, if business processes are changed, or if additional documents are created, the VBELN values that are created are less than the least values. In this scenario, the values would not be exported, and there is a risk of losing data during the Unicode conversion process. A better method of defining the WHERE clauses is as follows:

• WHERE MANDT='200' and VBELN <='006084390'
• WHERE MANDT='200' and VBELN>''006084390' and VBELN<='006096080'
• WHERE MANDT='200' and VBELN>='006096080'

Index Defragmentation in the SQL Server Database

Even with table splitting, the conversion team noticed that for several large tables, especially large cluster tables such as RFBLG (BSEG), the export time did not improve as much as was required to meet the conversion time frame of a single weekend. Additionally, the team noticed that many physical read operations occurred when export processes accessed the data that was associated with these tables. The conversion team determined that this behavior occurred because of fragmentation in the database.

Microsoft IT does not run Index defragmentation on it's SAP tables. Because the Microsoft IT SAP system generally experiences random read activities, defragmenting the database does not benefit the SAP system. However, for a process such as the export operation that occurs during the Unicode conversion operation, sequential read access occurs. A sequential read operation takes advantage of the SQL Server built-in read ahead capabilities. However, the read ahead feature cannot be used effectively if the data is very fragmented.

To resolve this issue, the conversion team identified the most critical tables and executed a one time index defragmentation operation against them. After defragmenting the indexes for these tables, subsequent export operations took less time. To defragment the indexes, the conversion team ran the following command:

DBCC indexdefrag <TABLENAME>

Improving Import Times

Optimizing Import Loads

To convert all the SAP R/3 data to Unicode in a single weekend, the conversion team not only had to optimize the export operations, it had to optimize import operations into the SQL Server 2005 back-end database.

Even with the addition of the 1-terabyte drive for transaction log files, the import process was taking too long. The conversion team determined that this issue was occurring because SQL Server 2005 was performing single row imports instead of bulk imports when it imported the Unicode data.

To resolve this issue, the conversion team modified the Distribution Monitor control file to add the -loadprocedure fast argument to the R3 load processes. To make this change, the team used the following procedure:

1. Open the Distribution_monitor_cmd.properties file by using any text editor such as Notepad.
2. Locate the following section in the file:
   
   #Additional R3load arguments for the LOAD phase
   r3load.export.loadArgs=
   r3load.import.loadArgs=
3. Modify this section so that it appears as follows, and then save the changes to the file:
   
   #Additional R3load arguments for the LOAD phase
   r3load.export.loadArgs=
   r3load.import.loadArgs=-loadprocedure fast

After this change, SQL Server 2005 uses bulk imports to import the Unicode data, dramatically reducing the import run time.

Reducing Log File Growth

After the export and import processes were optimized by splitting tables and modifying the Distribution Monitor control file, export times and import times greatly improved. However, certain tables still took a long time to import.

When reviewing this situation, the conversion team noticed that the transaction log file on the import database server grew larger than expected. Sometimes, the transaction log grew to 1 terabyte or larger. Additionally, when the transaction log file grew larger than approximately 700 GB, the whole import operation appeared to slow down.

After investigating this further, the conversion team found that although the import of certain tables was complete, SQL Server was unable to free the space from the transaction log. This behavior occurred because of open transactions in one or more other tables. These were processes that were started before the tables were finished being processed. Therefore, SQL Server could not free the transaction log information.

To resolve this issue, the conversion team again reviewed the table splitting settings to fine tune them. The conversion team identified the tables that had long import run times. Then, the conversion team split these tables to allow SQL Server to release the transactions that corresponded to these tables. The conversion team performed additional table splitting operations to keep the transaction log import database between 500 MB and 700 MB.

Conclusion

After optimizing the export and import operations, and after recalibrating table splitting requirements because of database growth in the SAP R/3 environment, the conversion team performed the Unicode conversion in the production environment.

To perform this conversion operation, the team ran the following R/3 load processes:

• Eight export processes on each server (four on the database server)
• Six import processes on each server (four on the database server)

The following table shows the import and export times for the servers involved in the Unicode conversion operation:

Table 4. Export and Import Times

Server	Data Exported	Time to Export and Import
Export server - DB1	190.5 GB	14 hours 41 minutes
Server 1	43.0 GB	15 hours 4 minutes
Server 2	20.5 GB	14 hours 54 minutes
Server 3	40.5 GB	14 hours 46 minutes
Server 4	54.0 GB	14 hours 54 minutes
Server 5	43.5 GB	14 hours 13 minutes

Only 22 hours of SAP R/3 downtime occurred during the successful conversion of the 2.8-terabyte production database to Unicode. Out of these 22 hours, only about 15 hours were required for the actual export and import operations. The remaining downtime included the following:

• Time to run a full backup and system verification
• Time to run a script that the conversion team created to verify the converted data

This conversion met both of the following challenges:

• To perform a large SAP Unicode conversion by using a SQL Server 2005 installation running on commodity hardware
• To complete the Unicode conversion operation within a single weekend (22 hours), thereby minimizing end-user downtime in the SAP R/3 system.

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Centre at (800) 563-9048. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information through the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

Appendix A

List of Split SAP R/3 Tables

The following is the list of tables that the team split during the Unicode conversion operation:

• ACCTIT
• ADRC
• ADRV
• ANLC
• ANLP
• BKPF
• BSAD
• BSAK
• BSAS
• BSE_CLR
• BSIS
• CDCLS
• CDHDR
• CMFP
• COEP
• ECMCA
• EDI40
• GLPCA
• IDOCREL
• KOCLU
• MBEW
• MSEG
• NAST
• PCL2
• PCL4
• PPOIX
• REGUC
• REPOSRC
• RFBLG
• SRRELROLES
• STXH
• STXL
• SWFREVTLOG
• VAPMA
• VBAP
• VBFA
• VBFS
• VBPA
• VBRP
• YPUMA2PL_COND
• YPUMAWIP

For each installation, determining which tables to split must be decided on a case-by-case basis.

Unicode Conversion Table Splits

The following table shows the table splits that the team configured for one test in 2005:

Table 4. Table Split Details

Table	Used	Reserved	Data	Rows	Row modctr	Split count	Splits
ECMCA	163,449,216	163,610,504	79,089,224	141,279,480	419,564	10M	15
PCL2	147,801,384	155,577,040	146,542,880	34,353,923	3,232,700	3M	12
GLPCA	90,443,752	93,089,560	46,503,368	54,948,012	247,870	5M	12
BSIS	89,887,544	93,711,176	67,592,992	132,362,429	8,355,812	10M	12
VBPA	74,726,752	85,081,272	21,213,504	131,251,144	344,055	10M	14
CDHDR	68,105,536	79,353,032	24,318,328	146,264,147	13,565,284	10M	16
COEP	61,291,056	63,182,224	36,939,992	74,271,000	246,842	7M	11
BKPF	60,412,568	69,302,448	19,526,368	41,444,576	63,584	5M	8
VBFA	59,260,688	68,337,056	30,932,360	88,232,337	4,724,749	8M	12
CDCLS	53,435,784	53,530,152	52,796,408	146,944,750	13,663,641	10M	16
BSAD	46,573,648	51,245,536	28,224,976	21,371,164	1,018,773	2M	10
PPOIX	45,042,288	53,434,432	33,773,080	152,949,338	15,658,601	10M	16
RFBLG	44,828,576	46,821,504	44,376,576	41,762,294	1,951,373	4M	10
VBAP	44,793,024	45,806,136	38,056,936	27,882,216	3,181,981	3M	10
NAST	43,164,280	53,115,576	14,835,016	26,304,353	1,954,352	3M	10
ADRC	41,944,088	45,628,968	23,676,720	34,566,101	1,347,495	3M	12
ANLC	39,542,008	39,554,544	39,070,528	41,309,062	5,167,196	4M	11
STXH	38,225,816	39,654,632	24,676,664	90,247,816	6,131,474	9M	11
VBRP	36,290,488	38,130,704	32,148,992	27,383,548	60,313	3M	10
BSAK	31,520,968	33,804,072	18,228,336	12,741,369	746,184	1M	12
ACCTIT	31,482,336	32,130,536	28,101,760	29,921,093	1,967,623	5M	7
VBFS	28,252,792	28,263,920	27,864,016	233,814,237	3,872,948	20M	12
REGUH	24,661,504	25,031,408	21,562,768	20,428,981	26,668	2M	11
BSAS	21,810,496	22,364,584	12,937,824	26,523,108	1,239,946	3M	9
CMFP	21,565,824	22,151,648	21,313,064	86,945,886	134,719	8M	12