(A thread pool based on completion ports as well as a file copy application is available here)
Having worked in a few places which did a significant amount of IO such as files, sockets and devices I was surprised that only a few of them used IO completion ports. There are a number of books and articles published on the topic. Perhaps the documentation isn’t clear or perhaps it was first touted as being too clever, leading the topic to be neglected. Here rather than present code, I’ll discuss the overall architecture that will help you see the existing documentation in new light.
Every Windows programmer knows about the message pump in a GUI application. The message pump is based on a queue where messages are posted and the messages are then handed to the window for which it was meant. Hence we can have multiple window handles on a single thread and it is guaranteed that the call back function for the window will be executed in the thread that created the window.
An IO completion port is also a handle to a message queue except that multiple threads can wait on the same queue and the Windows API GetQueuedCompletionStatus returns a task to any thread waiting on the completion port. It ensures that only one thread picks up task and every task posted will be executed in the order in which the task is posted (usually). You can post a task using PostQueuedCompletionStatus.
The major advantage, apart from optimizing thread utilization, is that socket IO, file IO and device IO can use the same completion port as the one used for posting tasks. Hence writing asynchronous IO code is not too difficult. Rather than launching a hundred threads that wait for IO to complete we can have one IO completion port with 8 threads on a 4 core machine; the recommendation is to use twice as many threads as there are CPU’s.