Completion Ports

Completion ports provide a scalable mechanism for managing I/O operations in high-performance network applications. They allow an application to associate I/O operations with a port, and then efficiently retrieve the results of those operations as they complete.

Introduction to Completion Ports

Traditional I/O models often involve blocking operations or complex callback mechanisms. Completion ports offer a more streamlined approach by decoupling the initiation of I/O from its completion. When an I/O operation is initiated with a completion port, the calling thread doesn't wait for the operation to finish. Instead, the operating system signals the completion port when the operation is done, and a worker thread can then retrieve the completion details and process them.

Key Concepts

• Completion Port Object: A kernel object that serves as the central point for managing I/O completions.
• I/O Completion Request: A structure that contains information about a completed I/O operation, including a pointer to the original I/O control block (OVERLAPPED structure), the number of bytes transferred, and a completion key.
• Completion Key: An application-defined value associated with each I/O operation. This key can be used to identify the specific operation or the context in which it occurred.
• Worker Threads: Threads that wait on the completion port for I/O completion events.

Core Functions

The following functions are central to using completion ports:

• CreateIoCompletionPort: Creates a new completion port or associates a file handle with an existing one.
• GetQueuedCompletionStatus: Retrieves an I/O completion packet from a specified completion port.
• PostQueuedCompletionStatus: Posts a user-defined I/O completion packet to a completion port.

CreateIoCompletionPort

This function is used to create a completion port and associate a file handle with it. If no file handle is provided, a new completion port is created. If a file handle is provided, it's associated with the port, and the number of concurrent threads that can process I/O operations can be specified.

HANDLE CreateIoCompletionPort(
  HANDLE ExistingCompletionPort,
  HANDLE FileHandle,
  ULONG_PTR CompletionKey,
  DWORD NumberOfConcurrentThreads
);

• ExistingCompletionPort: A handle to an existing completion port, or NULL to create a new one.
• FileHandle: The handle of the file to associate with the completion port. Use NULL to create a completion port without associating any file.
• CompletionKey: An application-defined value to associate with the file handle. This value is passed back to the application in the CompletionKey parameter of GetQueuedCompletionStatus when an I/O operation associated with this file handle completes.
• NumberOfConcurrentThreads: The maximum number of concurrently running threads that may be associated with the completion port. If this parameter is 0, the system allows as many concurrent threads as there are processors on the system.

GetQueuedCompletionStatus

This function waits for an I/O operation to complete on the specified completion port. It blocks the calling thread until a completion packet is available or a timeout occurs.

BOOL GetQueuedCompletionStatus(
  HANDLE       CompletionPort,
  LPDWORD      lpNumberOfBytesTransferred,
  PULONG_PTR   lpCompletionKey,
  LPOVERLAPPED *lpOverlapped,
  DWORD        dwMilliseconds
);

• CompletionPort: A handle to the completion port.
• lpNumberOfBytesTransferred: A pointer to a 32-bit unsigned integer that receives the number of bytes transferred for a I/O operation that has completed.
• lpCompletionKey: A pointer to a variable that receives the completion key associated with the FileHandle whose I/O completed.
• lpOverlapped: A pointer to a pointer to an OVERLAPPED structure that was specified in the initial I/O operation call.
• dwMilliseconds: Specifies the time-out interval in milliseconds. If the interval is zero, the function returns immediately. If the interval is INFINITE, the function will not return until a completion packet is available.

PostQueuedCompletionStatus

This function posts a user-defined I/O completion packet to a specified completion port. This is useful for signaling events or passing custom messages between threads.

BOOL PostQueuedCompletionStatus(
  HANDLE       CompletionPort,
  DWORD        dwNumberOfBytesTransferred,
  ULONG_PTR    dwCompletionKey,
  LPOVERLAPPED lpOverlapped
);

• CompletionPort: A handle to the completion port.
• dwNumberOfBytesTransferred: The number of bytes transferred.
• dwCompletionKey: The application-defined completion key.
• lpOverlapped: A pointer to an OVERLAPPED structure. Can be NULL.

Creating and Managing a Completion Port System

A typical setup involves:

1. Creating a completion port using CreateIoCompletionPort.
2. Creating a pool of worker threads. Each worker thread repeatedly calls GetQueuedCompletionStatus to wait for I/O operations to complete.
3. Associating socket handles with the completion port using CreateIoCompletionPort, providing a unique CompletionKey for each socket.
4. Initiating asynchronous I/O operations on the sockets using functions like WSARecv and WSASend, passing an OVERLAPPED structure that includes a pointer to a context structure.
5. When an I/O operation completes, GetQueuedCompletionStatus returns in the worker thread. The worker thread uses the returned CompletionKey and OVERLAPPED structure to identify the socket and context, and then processes the results (e.g., handles received data, sends more data).

Advantages of Completion Ports

• Scalability: Efficiently handles a large number of concurrent I/O operations with a relatively small number of threads.
• Performance: Reduces thread overhead by minimizing context switching and blocking.
• Simplicity: Provides a consistent model for handling all types of asynchronous I/O.

See Also

• Overlapped I/O
• Winsock Functions
• Windows Kernel Objects