Chapter 4: Developing Phase: Memory and File Management

This chapter compares the implementation of memory and file management mechanisms in UNIX and Microsoft® Windows® Services for UNIX 3.5. In addition, this chapter discusses the functions available for memory and file management in UNIX and the corresponding application programming interfaces (APIs) available in Windows Services for UNIX 3.5.

Knowing the differences between the memory management and file management routines in the two environments will enable you to identify the memory-specific and file management-specific porting changes required for UNIX applications. You will also learn about the equivalent routines in the Interix environment for the UNIX system APIs in memory and file management.

*

On This Page

Memory Management Memory Management
File Management File Management

Memory Management

Microsoft Interix supports the majority of UNIX memory management calls; therefore, porting code by using memory management is generally straightforward. However, there are a few specific differences. This section discusses how to address these differences in your code.

Heap Management

Interix supports the alloca() function, which allocates size bytes of space in the stack frame of the caller. However, the alloca() function is neither thread-safe nor async signal–safe; there might be performance issues because alloca() allocates memory on the stack frame instead of the heap.

Therefore, it is usually recommended that you use malloc(size_t size) and call free() because space is not automatically freed on return.

Interix does not support the platform-specific, memory-management functions listed in Table 4.1 and therefore alternative functions need to be used in the code.

Table 4.1. Platform-Specific , Memory-Management Functions Not Supported by Interix

Function Name

Description

Suggested Interix Replacement

void cfree(void *)

Deallocates memory allocator.

free(void *ptr)

Int getpagesize(void)

Gets system page size.

Always returns 65536 (64 KB), regardless of the actual Windows page size.

The getconf(_SC_PAGE_SIZE) and sysconf(_SC_PAGE_SIZE) functions also return 65536 (64 KB) always.

void mallocctl(int cmd, long value)

MT hot memory allocator.

No support or equivalent in Interix. There may be open-source versions of other allocators that can be used.

int mallopt(int cmd, int value)

Provides for controls over the allocation algorithm.

No support or equivalent in Interix. The supported malloc() has no controllable options //mallopt.

void *memalign(size_t alignment, size_t size)

Allocates size of bytes on the specified alignment boundary.

No support or equivalent in Interix.

void * valloc(size_t size)

Equivalent to memalign(sysconf(_SC_

PAGESIZE),size).

void *malloc(size_t size)

watchmalloc

Debugs memory allocator.

No support or equivalent in Interix.

Memory-Mapped Files

Interix supports memory-mapped files by using the mmap function. The length of the mapped space (in bytes) is rounded up to the nearest multiple of sysconf(_SC_PAGE_SIZE). This means that the value returned by sysconf(_SC_PAGE_SIZE) or sysconf(_SC_PAGESIZE) is not the virtual-memory page size used by the system, but the value that is used by the mmap call. Code should work without modification unless it assumes the page sizes to be smaller than 64 kilobytes (KB). Most applications are written without any assumption regarding the page size.

Shared Memory Management

Shared memory permits two or more processes to share a region of memory. Data present within the memory region is not copied as part of the communication process. Instead, the same physical area of the memory is accessed by both the client and the server. For this reason, shared memory performance is considered the best of all interprocess communication (IPC) methods.

Interix supports all of the System V IPC mechanisms, including the shmat, shmctl, shmdt, and shmget shared memory routines.

The ipcs and ipcrm command-line interfaces are also provided for the management of shared memory segments. The ipcs interface reports the status of IPC objects. The ipcrm interface removes an interprocess communication identifier, such as a shared memory segment.

Synchronizing Access to Shared Resources

Code that uses shared memory must ensure that the processes accessing the shared memory are not attempting to access the shared memory resource simultaneously. This is particularly troublesome if one or both of the processes are writing to the same shared memory area. To address this problem, UNIX provides the semaphore object. There are two sets of functions for semaphores: The POSIX real-time extensions are used for thread synchronization, and the System V semaphores are commonly used for process synchronization.

Interix supports both POSIX real-time extensions and System V semaphores (all of them), including the shared memory routines semctl, semget, and semop.

The ipcs and ipcrm command-line interfaces are also provided for the management of semaphore objects. These interfaces perform the same actions as they do for the shared memory management described previously.

File Management

The Interix file management system uses inodes to store administrative information about files and directories like UNIX. The inode is a structure that contains information about such things as the file size, its location, last access time, last modification time, and access permissions. Directories are also represented as files and have an associated inode. In addition to the file information, the inode also contains pointers to the data blocks of the file. An inode has 13 block addresses, of which the first 10 are direct block addresses for the first 10 data blocks of the file. The next addresses point to multiple-level address blocks to accommodate large files.

Interix file and data access (including security settings) differs somewhat from UNIX because of the underlying Windows input/output (I/O) system. Consequently, certain UNIX features are different or do not work in Interix.

Differences Between Interix and UNIX File I/O

Interix does not support file I/O with memory caching turned off (O_DIRECT), but it supports the file I/O APIs (and their associated options) listed in Table 4.2.

Table 4.2. File I/O APIs Supported by Interix

Function Name

Description

ssize_t pread(int fd, void *buf, size_t nbytes, off_t offset) 

Reads from a file descriptor at a given offset.

ssize_t pwrite(int fd, void *buf, size_t nbytes, off_t offset) 

Writes to a file descriptor at a given offset.

The Interix ioctl() Function Implementation

The ioctl() interface has many uses. The ioctl() function does not have a single standard. Its arguments, returns, and semantics vary according to the device driver. The call is used for operations that do not cleanly fit the UNIX stream I/O model.

The ioctl() interface in UNIX has historically been used to handle the following:

  • File control (See the “File Control and ioctl()” section later in this chapter.)

  • Socket control (See the “Socket Control and ioctl()” section later in this chapter.)

  • Disk labels

  • Magnetic tape control

  • Terminal control (See the “Terminal Control and ioctl()” section later in this chapter.)

The disk label and magnetic tape I/O requests are not supported in the Interix environment.

The Windows Services for UNIX 3.5 API set contains many ioctl() operations, including terminal, file, and socket ioctl() support. The following sections explain these operations in more detail.

Terminal Control and ioctl()

Interix supports almost all ioctl() requests for terminal control except a few, and some of these can be replaced with other system calls. The following are the only ioctl() requests that are not supported:

  • TIOCNOTTY

  • TIOCGETP

  • TIOCSETP

  • TIOCSETN

  • TIOCSETC

  • TIOCGETC

  • TIOCLBIS

  • TIOCLBIC

  • TIOCLSET

  • TIOCLGET

  • TIOCSLTC

  • TIOCGLTC

  • TIOCGSID

  • TIOCSSID

TIOCGETP and TIOCSETP can be replaced with POSIX Terminal I/O calls supported by Interix cfgetispeed and cfsetispeed. For more information on POSIX Terminal I/O calls, refer to “POSIX Terminal I/O” in Chapter 6, “Migrating the User Interface” of this volume.

The following three bits are not turned on in Interix, although they are held reserved with values.

  • TIOCM_LE

  • TIOCM_ST

  • TIOCM_SR

Note More information on ioctl requests is available at usr/include/sys/ioctl.h.

File Control and ioctl()

The following are the only ioctl() requests that are defined for file control in Interix:

  • FIONREAD. To get the number of bytes available to read.

  • FIONBIO. To set and unset nonblocking I/O.

The FIOCLEX and FIONCLEX requests (usually found in Filio.h) are not provided. You can replace them with the fcntl() FD_CLOEXEC request, as shown in the following example:

Note: The line has been split into multiple lines for readability.However, while trying it out on a system you must enter it as one line without breaks.

#ifndef __INTERIX
(void) ioctl(fd, FIOCLEX, NULL)
#else
(void) fcntl(fd, F_SETFL, fcntl(fd, F_GETFD) | 
FD_CLOEXEC));
#endif
Socket Control and ioctl()

The only ioctl() request defined for sockets in Interix is SIOCATMARK. Other socket control requests are handled by using fcntl() or by using functions such as setsockopt().

Note Additional ioctls for sockets (for example SIOCGIFCONF and SIOCGIFFLAGS) are available using Interop Systems Porting Library. More information on socket control and ioctl() is available at
https://www.interix.com/tools/warehouse.htm.

File Security

This section covers how files created in the two environments—namely Interix and Microsoft Win32® subsystem—differ in their security settings.

Files Created in the Interix Environment

When you create a file in Interix and view it with ls -l, the following permissions and attributes apply:

  • The file is owned by the user who created it.

  • The file inherits its group from the group of the directory.

  • Group names can contain spaces. For example, “domain users” is a valid group name. Although it is possible to create group names with spaces, if you do so, shell and awk scripts might not work properly because these types of scripts parse a group name as a single token.

  • File permissions are dictated by the user mask.

Interix files are given three access control entries (ACEs) in Windows: one for the owner, one for the group, and one for the Everyone group, which represents everyone else. Interix permissions work as follows:

  • The Interix read permission is represented by the Windows Read (R) permission.

  • The Interix write permission is represented by the Windows Write (W) permission. If the read-only attribute of the file is set, Interix does not assign the write permission, regardless of the content of the ACEs. If you use the chmod command to assign write permission to a file that has the read-only attribute set, the read-only attribute is removed.

  • The Interix execute permission is represented by the Windows Execute (X) permission.

  • You can deny access when the Windows permission is too broad. For example, the owner should not be able to read a file with a mode of 077 (---rwxrwx). However, the Windows Everyone group gives the owner access to the file. In this case, Interix adds a deny permission to the Windows ACE for the owner to accurately represent the permissions.

  • The owner of a file can change permissions on a file.

  • The owner of a file can grant or deny the permission to others to take ownership of a file.

  • Some special permissions, such as the setuid bit, are not represented through ACEs and are not visible through standard Win32 tools.

  • Interix and Windows handle uppercase and lowercase letters of file names differently. Two files created under Interix that differ only in the use of case will be visible to Windows Explorer. However, access will only be given to the contents of one of the files when you try to open either from Windows Explorer. This can cause surprising behavior when interoperating between Interix and Win32.

  • The chmod command can be used in the Interix environment to change file modes and set specific access levels for the owner, group, and everyone for a particular file. There is no equivalent of the chattr command in Interix.

Note Case-sensitivity in the Interix environment is the user choice selected during the installation of Windows Services for UNIX 3.5.

Files Created in the Win32 Subsystem

A file created in the Win32 subsystem can have a number of ACEs associated with it. In addition, those ACEs might not fit neatly into the categories of user, group, and everyone else. The Interix tools “assemble” permission from the available UNIX ACEs in the following ways:

  • For all Windows operating systems except Windows XP Professional, the user is the owner of the file unless the user is a member of the Administrators group. In this case, the Administrators group owns the file. For Windows XP Professional, the user who created the file is the default owner. To change the default owner to the Administrators group, use the Group Policy snap-in.

  • The group for the file will be Windows Domain Users on a server or None on a workstation.

  • If an owner has no specific ACE associated with it, the owner-permission bits are empty.

  • If the owner is a group, group permissions are transferred to owner permissions and the group permissions are made empty.

  • If the ACE used to determine the permissions of the owner does not have Change Permissions (P) or Take Ownership (O) permission, the chown, chgrp, and chmod commands might not work as expected.

Note   When a file created in Win32 is viewed from an Interix application, the additional ACE information available is indicated by a flag being set. This flag appears as a plus sign (+) suffix, with the permissions visible by using ls -l.
In addition, the structure st_mode (the structure returned from stat) has a bit set indicating the additional ACE information: bit S_ADDACE in the file sys/stat.h. The prototype of stat function is as follows.
int stat (const char *path, struct stat *sb)
File Explorer should not be used to adjust the permissions on an Interix file or directory. File Explorer rearranges the order of the ACEs in the ACL, which can cause problems when viewing the permissions later from Interix.

Directory Operations

Interix supports a subset of the routines used to access directory entries. Code that uses the calls listed in Table 4.3 does not require any modifications to compile under Interix.

Table 4.3. Routines for Accessing Directory Entries Supported by Interix

Routine Name

Description

int alphasort(const void *d1, const void *d2)

Routine that can be used for the compar argument of the scandir routine to sort the array alphabetically.

int scandir(const char *dirname, struct dirent ***namelist, int (*select)(struct dirent *),

int (*compar)(const void *, const void *))

Scans a directory for matching entries.

Interix does not support the calls and options listed in Table 4.4. Use the suggested Interix replacements instead.

Table 4.4. Directory Operations Routines Not Supported by Interix

Function Name

        

Description

Suggested Interix Replacement

int getdents(int fd, struct dirent

*dirp, int nbytes)

Gets directory entries and puts them in a file system-independent format.

struct dirent * readdir(DIR *dirp)

int getdirentries(int fd, char *buf, int nbytes, long *basep) 

Gets directory entries in a file system-independent format.

struct dirent * readdir (DIR *dirp)

Working Directory

Interix does not support the routines used to obtain the current working directory. Instead, use the suggested Interix replacements listed in Table 4.5.

Table 4.5. Working Directory Routines Not Supported by Interix

Function Name

Description

Suggested Interix Replacement

char * get_current_dir_name (void)

Gets absolute path of current working directory

(define:__USE_GNU).

char *getcwd(char *buf, size_t size)

char *getwd(char* path_name) 

Gets absolute path of current working directory

(define:__USE_BSD).

char *getcwd(char *buf, size_t size)

The getwd API as defined on Berkeley Software Distribution (BSD) systems is particularly dangerous because it is vulnerable to buffer overrun attacks. Replace it with getcwd and a buffer of known size.

File System Operations in Interix

File system operations in Interix differ in a number of ways from file system operations in UNIX. Some functions are not supported, such as sync, sysfs, and ustat. Others have different parameters or use a different set of options for UNIX. Table 4.6 lists the file system information functions that need to be replaced.

Table 4.6. File System Information Functions Not Supported by Interix

Function

Description

Suggested Interix Replacement

int statfs(const char *path, struct statfs *buf), 

int fstatfs(int fd, struct statfs *buf)

Gets file system statistics.

int statvfs(const char *path, struct statvfs *buf)

int fstatvfs(int fildes, struct statvfs *buf)

void sync(void)

int fsync(int fd)

Writes all information in memory that should be on disk, including modified super blocks, modified inodes, and delayed block I/O. fsync synchronizes changes to a file.

int fsync(int fd)

int sysfs(int opcode, const char *filename)

Gets file system type information.

int statvfs(const char *path, struct statvfs *buf)

int ustat(dev_t dev,struct ustat *buf)

Gets file system statistics.

int statvfs(const char *path, struct statvfs *buf)

The Windows NTFS file system is structured and implemented in such a way as to prevent the need for the sync function. NTFS uses a log-based mechanism to ensure that the file system metadata (the equivalent of super blocks and inodes) is updated in sync with changes to file data. In the event of a system failure, automated NTFS recovery guarantees that either the metadata corresponds to data appearing in files or the changes to the files never occur. Chkdsk is the tool on the Windows environment similar to the UNIX fsck command. This command starts and executes on the Windows environment to synchronize the file system in case of a power disruption. Windows NTFS is a journaled file system (JFS) that contains its own backup and recovery capability. It maintains a log, or a journal, of what activity has taken place in the main data areas of the disk. Interix does not have its own file system and uses the Windows NTFS file system for its operations.

When using the statvfs function in Interix, be aware that the statfs structure has some members that are different from those usually found in UNIX. Table 4.7 summarizes these differences. In general, references to the statfs structure can by replaced with references to the statvfs structure.

Table 4.7. Differences Between UNIX statfs and Interix statvfs

UNIX statfs Structure

Interix statvfs

Structure

Description

long f_type

unsigned long f_type

Type of file system.

long f_bsize

unsigned long f_bsize

Transfer block size.

long f_blocks

unsigned long f_blocks

Total data blocks in file system.

long f_bfree

unsigned long f_bfree

Free blocks in file system.

long f_bavail

unsigned long f_bavail

Free blocks available to all users except superusers.

long f_files

unsigned long f_files

Total file nodes in file system.

(Currently returns 0.)

long f_ffree

unsigned long f_ffree

Free file nodes in file system.

(Currently returns 0.)

fsid_t f_fsid

unsigned long f_fsid

File system ID.

long f_namelen

unsigned long f_namemax

Maximum length of file names.

long f_spare[6]

unsigned long f_flag

Bit mask of values describing the file system.

 

unsigned long f_frsize

Fundamental block size.

 

unsigned long f_favail

Total number of file serial numbers available to a process without the required privileges.

(Currently returns 0.)

 

unsigned long f_iosize

Optimal transfer block size.

 

char f_mntonname

[MNAMELEN+1]

Mountpoint for the file system.

 

char f_mntfromname

[MNAMELEN+1]

Mounted file system.

When using statvfs, the file system types supported by Interix differ from those supported by most implementations of UNIX. The following two lists illustrate this by comparing a common subset of UNIX-supported file systems with those supported in Interix. It is rare for these differences to have an impact on the migration of an application.

Commonly Supported File Systems in UNIX
  • AFFS_SUPER_MAGIC 0xADFF

  • EXT_SUPER_MAGIC 0x137D

  • EXT2_OLD_SUPER_MAGIC 0xEF51

  • EXT2_SUPER_MAGIC 0xEF53

  • HPFS_SUPER_MAGIC 0xF995E849

  • ISOFS_SUPER_MAGIC 0x9660

  • MINIX_SUPER_MAGIC 0x137F /* original minix fs */

  • MINIX_SUPER_MAGIC2 0x138F /* 30 char minix */

  • MINIX2_SUPER_MAGIC 0x2468 /* minix V2 */

  • MINIX2_SUPER_MAGIC2 0x2478 /* minix V2, 30 char names */

  • MSDOS_SUPER_MAGIC 0x4d44

  • NCP_SUPER_MAGIC 0x564c

  • NFS_SUPER_MAGIC 0x6969

  • PROC_SUPER_MAGIC 0x9fa0

  • SMB_SUPER_MAGIC 0x517B

  • XENIX_SUPER_MAGIC 0x012FF7B4

  • SYSV4_SUPER_MAGIC 0x012FF7B5

  • SYSV2_SUPER_MAGIC 0x012FF7B6

  • COH_SUPER_MAGIC 0x012FF7B7

  • UFS_MAGIC 0x00011954

  • XFS_SUPER_MAGIC 0x58465342

  • _XIAFS_SUPER_MAGIC 0x012FD16D

Supported File System Types in Interix
  • ST_FSTYPE_UNKNOWN 0 /* unknown */

  • ST_FSTYPE_NTFS 1 /* NTFS */

  • ST_FSTYPE_OFS 2 /* OFS-NT object FS */

  • ST_FSTYPE_CDFS 3

  • ST_FSTYPE_CDROM ST_FSTYPE_CDFS

  • ST_FSTYPE_ISO9660 ST_FSTYPE_CDFS

  • ST_FSTYPE_HPFS 5 /* OS2 HPFS */

  • ST_FSTYPE_SAMBA 6 /* Samba FS */

  • ST_FSTYPE_NFS 8 /* NFS */

File System Mount Entry Management

Interix supports dynamically mounted file systems, but it provides no mechanisms to mount or dismount them. These operations must be performed through Win32 tools or APIs.

Mount tables are stored in different files on different implementations of UNIX. They are usually stored under the /etc directory and have names such as mtab and fstab or mnttab and vfstab. This is unlikely to affect the migration of an application. However, if the application uses dynamically mounted file systems, the code can be altered using the following options:

  • Use the /net file system to refer to the resource.

  • Use the system() API to invoke a Windows command-line tool that can mount a file system or network share.

  • Change the application and its environment so that the file system required is always mounted.

On UNIX systems, additional disk space is made available at some particular point in the file hierarchy by mounting a new disk volume onto a directory or by replacing the directory with a symbolic link to some already mounted volume. The first operation can be performed in Windows through the use of the Windows Disk Management tool. This solution makes the additional space visible to Interix and Win32 applications. If your application environment does not require Win32 applications to see this additional space, you can instead replace the directory with a symbolic link to any directory on any other volume on the system.

Gdbm Library

GNU dbm (gdbm) is a library of routines that manages data files that contain key/data pairs. Applications still use the gdbm database (that is, an indexed file storage system), which is good at storing relatively static indexed data.

The following APIs are used by an application that uses the gdbm database:

  • gdbm_close()

  • gdbm_delete()

  • gdbm_exists()

  • gdbm_fdesc()

  • gdbm_fetch()

  • gdbm_firstkey()

  • gdbm_nextkey()

  • gdbm_open()

  • gdbm_reorganize()

  • gdbm_setopt()

  • gdbm_store()

  • gdbm_strerror()

  • gdbm_sync()

The following example of application code describes the usage of some these functions.

Note: Some of the lines in the following code have been displayed on multiple lines for better readability.

#include <stdio.h>
#include <string.h>
#include <gdbm.h>
GDBM_FILE dbf;
datum key, nextkey, content;
int ret;
/* opening the database See gdbm(2) manual page for 
details. */
dbf = gdbm_open ("test.db", BLOCK_SIZE, GDBM_WRCREAT,
0755, NULL);
if (dbf == NULL) 
{
    /* if the open fails, print a standard error and 
       exit */
    perror ("error in creating db file: test.db\n");
    return 0;
}
/* create a record */
key.dptr = "SWEngineer";
key.dsize = strlen(key.dptr);
content.dptr = "Title: Software Engineer Salary: $3000";
content.dsize = strlen(content.dptr);
/* write the record to the database */
ret = gdbm_store(dbf, key, content, GDBM_INSERT);
if (ret != 0) 
{
    perror ("error in writing to database: test.db");
    return 0;
}
/* make another record */
key.dptr = "Programmer";
key.dsize = strlen(key.dptr);
content.dptr = "Title: Programmer Salary: $2700";
content.dsize = strlen(content.dptr);
/* write it to the database */
ret = gdbm_store(dbf, key, content, GDBM_INSERT);
if (ret != 0) 
{
    perror ("error in writing to database: test.db");
    return 0;
}
memset(&key,0,sizeof(datum));
/* iterate through the set of records */
key = gdbm_firstkey ( dbf );
key.dptr[key.dsize] ='\0';
while ( key.dptr ) 
{
if (gdbm_exists(dbf, key))
    {
        memset(&content,0,sizeof(datum));
        content = gdbm_fetch(dbf, key);
        content.dptr[content.dsize] ='\0';
      if (!strcmp(key.dptr,"Programmer")) 
        {
         gdbm_delete ( dbf, key );
        }
    }
    nextkey = gdbm_nextkey (dbf, key );
    key = nextkey;
    key.dptr[key.dsize] ='\0';
}
/* closing the database */
gdbm_close(dbf);

The gdbm library is available at
https://www.interopsystems.com/tools/warehouse.htm.

The GNUs gdbm has been ported to Windows Services for UNIX 3.5, and the routines work like the UNIX dbm routine.

Download

Get the UNIX Custom Application Migration Guide

Update Notifications

Sign up to learn about updates and new releases

Feedback

Send us your comments or suggestions