Epoll Starvation: Why Edge-Triggered I/O Left Sockets Hanging Under High Load

The Linux epoll API is the foundation of high-concurrency event loops powering Nginx, Node.js (libuv), Envoy, Netty, and Redis. When developers optimize socket notification performance, the discussion inevitably reaches Level-Triggered (LT) versus Edge-Triggered (ET) notification modes.

Edge-Triggered mode (enabled via the EPOLLET flag) is often praised as the faster option because it reduces redundant wakeups. However, in production network servers handling uneven payload distributions, a naive Edge-Triggered event loop can easily cause severe socket starvation, leading to tail latency spikes and timeouts on idle connections.

Level-Triggered vs Edge-Triggered Semantics

The distinction between the two modes lies in how the kernel signals readiness:

• Level-Triggered (Default): epoll_wait() returns an event as long as the underlying file descriptor buffer is ready (e.g. unread bytes remain in the socket receive buffer). If your application reads only half the buffer, the next call to epoll_wait() immediately wakes up again.
• Edge-Triggered (EPOLLET): epoll_wait() delivers a notification only when a state transition occurs (e.g. from no data available to new data arriving). If you do not consume all available bytes, epoll will not notify you again until fresh data arrives from the network.

The Edge-Triggered Contract

Because Edge-Triggered mode notifies only on state transitions, application code is required to read in a loop until receiving EAGAIN or EWOULDBLOCK:

// Standard Edge-Triggered read loop
while (true) {
    ssize_t count = read(fd, buffer, sizeof(buffer));
    if (count == -1) {
        if (errno == EAGAIN || errno == EWOULDBLOCK) {
            // Read buffer is completely drained. Safe to return to epoll_wait.
            break;
        }
        // Handle actual read error
        close(fd);
        break;
    } else if (count == 0) {
        // Peer cleanly closed connection
        close(fd);
        break;
    }
    process_data(buffer, count);
}

The Starvation Vulnerability

The requirement to loop until EAGAIN creates a fundamental fairness problem when a single client sends a continuous, high-bandwidth stream (such as a multi-megabyte file upload or an aggressive websocket burst).

If client A streams data faster than the application can process it, read() never returns EAGAIN. The worker thread remains trapped inside the while (true) loop processing bytes for socket A:

Client A: [==== Streaming 50MB Payload Continuously ====] -> Traps Worker in read() loop
Client B: [Pending SYN/ACK Handshake]                     -> Starving in epoll ready list
Client C: [Small 64-byte JSON query]                       -> Starving in epoll ready list

Meanwhile, hundreds of other connections sitting in the epoll ready list (including connections with small, urgent requests) wait without receiving any CPU cycles. Under moderate concurrency, p99 request latency degrades rapidly, and clients eventually abort with connection reset or read timeout errors.

Engineering the Solution

1. Bounded Batch Reads with Ring Buffers

To prevent any single socket from monopolizing the event loop, never loop until EAGAIN unconditionally. Instead, enforce a maximum read quota (e.g. read up to 64KB or 16 iterations per connection event):

const int MAX_CHUNKS_PER_CYCLE = 16;
int chunks_read = 0;

while (chunks_read < MAX_CHUNKS_PER_CYCLE) {
    ssize_t count = read(fd, buffer, sizeof(buffer));
    if (count == -1) {
        if (errno == EAGAIN || errno == EWOULDBLOCK) {
            break; // Completely drained
        }
        close(fd);
        return;
    }
    process_data(buffer, count);
    chunks_read++;
}

// If we exited due to quota rather than EAGAIN, data remains.
// Re-queue or manually schedule the socket for the next event loop iteration.
if (chunks_read == MAX_CHUNKS_PER_CYCLE) {
    schedule_continuation(fd);
}

2. Multi-Threaded Dispatch with EPOLLONESHOT

In multi-threaded server architectures (such as thread pools or worker pools), multiple threads may call epoll_wait() on the same epoll descriptor. If an Edge-Triggered event fires on socket A, and socket A takes time to process, another thread might receive a new event for the same socket, creating a dangerous concurrent read race.

Adding the EPOLLONESHOT flag ensures that after an event is delivered to one worker thread, epoll disables notifications for that file descriptor until the application explicitly re-arms it using epoll_ctl(epfd, EPOLL_CTL_MOD, fd, &event):

struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET | EPOLLONESHOT;
ev.data.fd = client_fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);

// Inside worker after processing bounded chunk:
epoll_ctl(epfd, EPOLL_CTL_MOD, client_fd, &ev); // Re-arm socket

3. Why High-Performance Servers Often Prefer Level-Triggered

It is instructive that both Nginx (on Linux) and libuv (the networking engine behind Node.js) default to Level-Triggered mode for general HTTP and TCP socket handling. Level-Triggered mode naturally supports fair single-read passes across all active sockets per event iteration without needing artificial continuation queues or risking silent deadlocks if an Edge-Triggered event is partially drained.

Summary

Edge-Triggered epoll reduces kernel-to-user notification overhead, but it shifts the burden of fairness and starvation management entirely to the application layer. If you use EPOLLET, bound your read loops, monitor per-connection quotas, and use EPOLLONESHOT when sharing descriptors across threads.