In August, Google faced an unprecedented challenge when it encountered the largest HTTP/2-based Distributed Denial of Service (DDoS) attack in its history, with a staggering 398 million requests per second. This attack surpassed any previous DDoS event by more than seven times, raising questions about the evolving landscape of cyber threats and the techniques used by attackers. At the heart of this attack lies the HTTP/2 Rapid Reset, a concept we’ll explore in depth. But to truly understand it, let’s first delve into how HTTP/2 works.
The Power of HTTP/2
HTTP/2 is the second major version of the HTTP network protocol, which powers the World Wide Web. Unlike its predecessor, HTTP/1.1, HTTP/2 introduces a revolutionary concept called “streams.” These streams serve as bidirectional abstractions, facilitating the exchange of messages or “frames” between endpoints.
One of the most significant features of HTTP/2 is “stream multiplexing,” which enables the efficient utilization of each Transmission Control Protocol (TCP) connection. This means that multiple streams can be multiplexed over a single Layer 4 connection, leading to a much more efficient use of network resources.
Stream Multiplexing and Concurrency
In HTTP/1.1, requests are processed serially. Each request necessitates a round trip that includes network latency, proxy processing time, and backend request processing time. Consequently, the rate at which requests can be sent over a single connection is quite limited. One request per round trip is the norm.
HTTP/2 changes this by allowing clients to open multiple concurrent streams on a single TCP connection. Each stream corresponds to an individual HTTP request. For example, a client can initiate 100 streams and send a request on each of them in a single round trip. The proxy reads and processes each stream sequentially, but the requests to the backend servers can be processed in parallel. As responses arrive, the client can open new streams, resulting in an exceptionally efficient throughput of around 100 requests per round trip. This is a significant improvement over the performance of HTTP/1.1 requests, achieving nearly 100 times higher utilization of each connection.
The HTTP/2 Rapid Reset Attack in Action
The attack that targeted Google is known as the “HTTP/2 Rapid Reset attack.” This attack leverages a unique characteristic of HTTP/2, which is the ability of an endpoint to send a RST_STREAM frame immediately after sending a request frame. This sequence of actions makes the other endpoint start processing the request and then abruptly resets it. The request is effectively canceled, but the HTTP/2 connection remains open.
In this attack, the client initiates a large number of streams simultaneously, similar to a standard HTTP/2 attack. However, instead of waiting for responses to each request stream, the client immediately cancels each request. Variants of this attack may involve either instantly canceling streams or opening another substantial batch of new streams.
Unraveling HTTP Semantics
To better grasp how HTTP/2 operates and how the Rapid Reset attack functions, it’s essential to understand HTTP Semantics. These semantics are the core principles common to all HTTP versions, encompassing aspects such as request and response messages, methods, status codes, header fields, and more. Each HTTP version defines how these semantics are transformed into a “wire format” for exchange over the Internet.
In HTTP/1.1, request and response messages are exchanged as a stream of ASCII characters over a reliable transport layer. The format of these messages necessitates that each message is sent as a whole. Strict ordering is required to correctly correlate requests with responses. This means that messages are exchanged serially and cannot be multiplexed.
When using HTTP/1.1 to navigate web pages, a browser often encounters a dilemma. If a user quickly navigates to another page, the browser has two options: either wait for all the queued up responses for the page that the user no longer wants, or cancel in-flight requests by closing the TCP connection and opening a new one. Neither of these options is practical. To work around these limitations, browsers often manage a pool of TCP connections, typically up to six per host, and implement complex request dispatch logic over this pool.
Client cancellation of in-flight requests in HTTP/2
Streams have a lifecycle. Below is a diagram of the HTTP/2 stream state machine. Client and server manage their own views of the state of a stream. HEADERS, DATA and RST_STREAM frames trigger transitions when they are sent or received. Although the views of the stream state are independent, they are synchronized. HTTP/2 supports this in a much more efficient way than HTTP/1.1. Rather than needing to tear down the whole connection, a client can send a RST_STREAM frame for a single stream. This instructs the server to stop processing the request and to abort the response, which frees up server resources and avoids wasting bandwidth. Recall that only streams that are in the open or half-closed state contribute to the stream concurrency limit. When a client cancels a stream, it instantly gets the ability to open another stream in its place and can send another request immediately. This is the crux of what makes CVE-2023-44487 work.
Mitigating Rapid Reset and Stream Management
HTTP/2 brings significant improvements over HTTP/1.1, particularly regarding stream management. However, it also presents new challenges, particularly in the context of rapid resets.
HTTP/2 offers a notion of maximum active concurrent streams, managed through the SETTINGS_MAX_CONCURRENT_STREAMS parameter. Servers can advertise this limit, and clients must adhere to it. If a client attempts to open a stream beyond this limit, the server can reject it using a RST_STREAM frame. This mechanism can be useful, but practical factors can limit its effectiveness.
The HTTP/2 Rapid Reset attack capitalizes on the ability to quickly reset an unbounded number of streams. While an HTTP/2 server that processes RST_STREAM frames promptly can mitigate this threat, issues arise when there are delays or lag in handling these requests. The client can overwhelm the server with requests and resets, leading to resource consumption issues and server backlog.
The attack’s efficiency is further underscored by the client’s ability to open new streams immediately after canceling others, bypassing the concurrency limit. This approach is at the core of CVE-2023-44487, enabling the rapid reset attack.
The Concurrency Dilemma
The concurrency limit alone cannot fully mitigate the rapid reset attack’s impact, as clients can churn out requests beyond the server’s defined limit of concurrent streams. This rapid succession of resets creates challenges for server resource management, particularly in deployments where components process HTTP/2 traffic asynchronously.
Conclusion
The largest HTTP/2 Rapid Reset attack on Google’s services highlights the evolving sophistication of cyberattacks and the need for continuous vigilance in cybersecurity. While HTTP/2 brings numerous benefits, including enhanced performance and efficiency, it also introduces vulnerabilities that attackers can exploit. The incident serves as a reminder of the critical importance of maintaining robust security measures and staying informed about emerging threats in the ever-evolving landscape of digital technology.
More
Number of software vendors issued patches alongside the Tuesday disclosure including Apple, Microsoft, F5, NGINX and others.
- HTTP/2 Rapid Reset Vulnerability, CVE-2: https://www.cisa.gov/news-events/alerts/2023/10/10/http2-rapid-reset-vulnerability-cve-2023-44487
- Cloudflare: HTTP/2 Rapid Reset: deconstructing the record-breaking attack
- Google: How it works: The novel HTTP/2 ‘Rapid Reset’ DDoS attack
- AWS: CVE-2023-44487 – HTTP/2 Rapid Reset Attack
- NGINX: HTTP/2 Rapid Reset Attack Impacting NGINX Products
- Microsoft Response to Distributed Denial of Service (DDoS) Attacks against HTTP/2
Leave a Reply