Windows Kernel System Calls

System calls are the fundamental interface between user-mode applications and the Windows kernel. They allow applications to request services from the operating system, such as creating processes, managing memory, accessing files, and interacting with hardware. Understanding system calls is crucial for developing robust and efficient Windows applications, especially those requiring low-level access or high performance.

Overview of System Calls

In Windows, system calls are typically accessed through functions provided in the Native API (also known as Win32 Native API or NT API). While many developers interact with the kernel through the Win32 API (e.g., CreateProcess), these Win32 functions often wrap one or more underlying Native API calls.

The Native API is exposed through dynamic-link libraries (DLLs) such as ntdll.dll. Directly using Native API functions provides greater control and access to lower-level kernel functionalities, but it also comes with increased complexity and platform dependency.

Key Concepts:

  • User Mode vs. Kernel Mode: System calls facilitate the transition from user mode (where applications run with restricted privileges) to kernel mode (where the OS has full access to hardware and memory).
  • System Service Dispatcher: A component within the kernel that receives system call requests, validates them, and dispatches them to the appropriate kernel routine.
  • SSDT (System Service Dispatch Table): A table used by the kernel to map system call numbers to their corresponding kernel function addresses.

Why Use Native API?

  • Access to features not exposed by the Win32 API.
  • Potentially higher performance for specific operations.
  • Useful for system-level utilities, debugging tools, and security software.

Common System Call Categories

System calls can be broadly categorized based on the functionality they provide:

1. Process and Thread Management

  • NtCreateProcess / NtCreateThread: Create new processes and threads.
  • NtTerminateProcess / NtTerminateThread: Terminate processes and threads.
  • NtQueryInformationProcess / NtQueryInformationThread: Retrieve information about processes and threads.
  • NtSetInformationProcess / NtSetInformationThread: Set information for processes and threads.

2. Memory Management

  • NtAllocateVirtualMemory: Allocate memory in the virtual address space of a process.
  • NtFreeVirtualMemory: Free previously allocated virtual memory.
  • NtProtectVirtualMemory: Change memory protection attributes (read, write, execute).
  • NtReadVirtualMemory / NtWriteVirtualMemory: Read from or write to the virtual address space of another process.

3. File System Operations

  • NtCreateFile: Create or open a file.
  • NtReadFile / NtWriteFile: Read from or write to a file.
  • NtClose: Close an object handle (like a file handle).
  • NtQueryDirectoryFile: Query the contents of a directory.

4. Object Management

  • NtCreateObject (generic): Create various kernel objects (events, mutexes, semaphores, etc.).
  • NtOpenObject: Open an existing kernel object.
  • NtQueryObject: Retrieve information about kernel objects.
  • NtClose: Close object handles.

5. Registry Operations

  • NtCreateKey: Create or open a registry key.
  • NtQueryValueKey: Query the value of a registry entry.
  • NtSetValueKey: Set the value of a registry entry.

Example: Using NtCreateFile

Here's a simplified conceptual example of how one might interact with NtCreateFile. Note that direct Native API usage often involves manual management of structures and object handles.


#include <windows.h>
#include <winternl.h>
#include <stdio.h>

// Define a typedef for the NtCreateFile function pointer
typedef NTSTATUS(WINAPI *PNtCreateFile)(
    PHANDLE FileHandle,
    ACCESS_MASK DesiredAccess,
    POBJECT_ATTRIBUTES ObjectAttributes,
    PIO_STATUS_BLOCK IoStatusBlock,
    PLARGE_INTEGER AllocationSize,
    ULONG FileAttributes,
    ULONG ShareAccess,
    ULONG CreateDisposition,
    ULONG CreateOptions,
    PVOID EaBuffer,
    ULONG EaLength
);

int main() {
    HANDLE hFile = NULL;
    NTSTATUS status;
    IO_STATUS_BLOCK ioStatusBlock;
    OBJECT_ATTRIBUTES objectAttributes;
    UNICODE_STRING fileName;
    WCHAR filePath[] = L"\\??\\C:\\Temp\\example.txt"; // Use Win32 path with NT prefix

    // Get a pointer to NtCreateFile from ntdll.dll
    PNtCreateFile pNtCreateFile = (PNtCreateFile)GetProcAddress(GetModuleHandle(L"ntdll.dll"), "NtCreateFile");
    if (pNtCreateFile == NULL) {
        fprintf(stderr, "Failed to get address of NtCreateFile.\n");
        return 1;
    }

    // Initialize OBJECT_ATTRIBUTES
    RtlInitUnicodeString(&fileName, filePath);
    InitializeObjectAttributes(&objectAttributes, &fileName, OBJ_CASE_INSENSITIVE | OBJ_KERNEL_HANDLE, NULL, NULL);

    // Call NtCreateFile
    status = pNtCreateFile(
        &hFile,
        GENERIC_WRITE | GENERIC_READ, // Desired access
        &objectAttributes,            // Object attributes
        &ioStatusBlock,               // IO status block
        NULL,                         // Allocation size (optional)
        FILE_ATTRIBUTE_NORMAL,        // File attributes
        FILE_SHARE_READ | FILE_SHARE_WRITE, // Share access
        FILE_OVERWRITE_IF,            // Create disposition (overwrite if exists)
        0,                            // Create options
        NULL,                         // EA buffer
        0                             // EA length
    );

    if (NT_SUCCESS(status)) {
        printf("File '%ws' created/opened successfully. Handle: %p\n", filePath, hFile);
        // You can now use hFile for read/write operations using NtReadFile/NtWriteFile
        // ...

        // Close the handle when done
        NtClose(hFile); // Assuming NtClose is available or use CloseHandle
    } else {
        fprintf(stderr, "NtCreateFile failed with status: 0x%08X\n", status);
    }

    return 0;
}
Note: This is a simplified example. In real-world scenarios, you'd need to handle error codes more robustly, manage memory allocation for structures like UNICODE_STRING, and understand the implications of using OBJ_KERNEL_HANDLE.

System Call Conventions and Best Practices

  • Use Win32 API First: For most application development, stick to the Win32 API. It's more portable, better documented, and less prone to breaking with OS updates.
  • Understand the Need: Only delve into Native API for specific requirements like advanced performance tuning, interacting with undocumented features, or building system-level tools.
  • Use Undocumented Functions Carefully: Many Native API functions are considered undocumented and may change or be removed in future Windows versions without notice.
  • Error Handling: Always check the return status of system calls. NTSTATUS codes provide detailed error information.
  • Handle Security: Be mindful of the security implications when requesting elevated privileges or accessing sensitive system resources.
← Interprocess Communication Security →