Compressed HTTP Requests with Curl and PHP

Published On20 Oct 2021

PHP Curl - HTTP Compression with ACCEPT_ENCODING
Compression is a vital and effective way to increase the performance of web pages and web apps. For text-based resources such as HTML files, CSS/JS files, SVG files, etc., compressing the resource at the server prior to the transmission, and decompressing it at the browser can greatly reduce the bandwidth and transfer times.

For the server and web browser, this compression step is quite opaque, in that the server compresses the resources prior to sending it to the browser, and the browser decompresses them before rendering them. The server-side software and the front-end developers do not need to handle the compression/decompression steps.

There are few compression algorithms developed through the years, and browsers and servers can negotiate the correct compression algorithm using HTTP headers.

When making an HTTP request, the browser indicates the encoding algorithms it supports in an Accept-Encoding HTTP header. If the server supports any of the specified encoding algorithms, it may decide to compress the response, and indicate that with an Content-Encoding HTTP header. IANA maintains a list of registered encoding algorithms.

PHP Curl - HTTP Compression with ACCEPT_ENCODING

The Wikipedia page for PHP is around 549 KB in size, and when it compressed with Brotli (br), it is only 92 KB. HTML pages, JSON responses, SVG files, CSS/JS files, and other text-based files compress well, and the computational cost it takes to compress/decompress resources is small compared to the network transfer times the compression saves.

Although browsers default to use compression when fetching resources, PHP's HTTP clients do not. Curl, the popular HTTP Client supports encoding negotiation and opaque compressed requests, but it needs to be enabled.

$ch = curl_init('');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

Above is simplified example of making an HTTP request to the PHP page on Wikipedia. Curl does not use HTTP compression by default, and results in larger transfer size and request time:

curl_getinfo($ch, CURLINFO_TOTAL_TIME); // 0.81 sec
curl_getinfo($ch, CURLINFO_SIZE_DOWNLOAD_T); // 548 KB

CURLINFO_SIZE_DOWNLOAD_T value reveals that Curl download 548 KB of data from the remote server. Because there is no compression support indicated by Curl (via the Accept-Encoding header), the server did not compress the response.

Using the CURLOPT_ENCODING option, it is possible to explicitly specify the values for the Accept-Encoding header. It is not necessary manually decode the response, because Curl does it automatically. Note that setting the CURLOPT_ENCODING option is different from setting Accept-Encoding header manually. If set manually, Curl does not automatically decode the response.

CURLOPT_ENCODING value accepts a few types of values which are not known to be intuitive.

  • CURLOPT_ENCODING = null: Resets the value, disables header and automatic decoding.
  • CURLOPT_ENCODING = "": Curl automatically sends the appropriate header based on the supported algorithms, and automatically decodes the response.
  • CURLOPT_ENCODING = "identity": Client expects the server to not encode the response in any way.
  • CURLOPT_ENCODING = "gzip": Client is capable of gzip algorithm.
  • CURLOPT_ENCODING = "br": Client is capable of Brotli br algorithm.

CURLOPT_ENCODING option in fact accepts any string value, and Curl will attempt to decode it if the responding Content-Encoding header contains a known encoding algorithm.

The most useful and appropriate value for CURLOPT_ENCODING is an empty string (""). It enables encoding, but does not explicitly state the algorithms. This effectively enables all algorithms supported and selected by Curl.

  $ch = curl_init('');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
+ curl_setopt($ch, CURLOPT_ENCODING, '');

With the CURLOPT_ENCODING="" option, Curl now makes HTTP requests with an appropriate Accept-Encoding header, listing all algorithms it supports.

The headers sent-out can be later retrieved using curl_getinfo function:

$ch = curl_init('');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLINFO_HEADER_OUT, true);

$headers = curl_getinfo($ch, CURLINFO_HEADER_OUT);
  string(103) "GET /wiki/PHP HTTP/2
  accept: */*
+ accept-encoding: deflate, gzip, br, zstd

The server can opt to compress the response in one of the compression algorithms it supports. This greatly reduces the transfer time and size:

curl_getinfo($ch, CURLINFO_TOTAL_TIME); // 0.31 sec
curl_getinfo($ch, CURLINFO_SIZE_DOWNLOAD_T); // 90 KB

Curl picks the compression algorithms it supports based on the options it was set at the compilation step.

In most distributions, PHP Curl is compiled with gzip, deflat, and br (Brotli). However, it is also possible to add support for zstd by compiling libcurl with zstd support, and recompiling PHP with the new libcurl header files.

Recent Articles on PHP.Watch

All ArticlesFeed 
Get Composer to suggest dev packages to `require-dev`

Get Composer to suggest dev packages to require-dev

Composer 2.4 comes with a feature that it prompts during a composer require to install packages as require-dev dependencies when certain keywords are present. Development tools and libraries now can make use of this feature to mark themselves, so Composer uses the prompt when suitable.
Security considerations when parsing user-provided INI strings and files

Security considerations when parsing user-provided INI strings and files

Security considerations when parsing user-provided INI strings and files using parse_ini_string and parse_ini_file functions.
How to extend lifetime of legacy PHP applications

How to extend lifetime of legacy PHP applications

As PHP continue to evolve with new breaking changes, and while that is great for most PHP applications, there are legacy applications that can't justify the human and financial cost of keeping up. Here is a guide on how to extend the lifetime of legacy PHP applications with security updates and maintenance.
Subscribe to PHP.Watch newsletter for monthly updates

You will receive an email on last Wednesday of every month and on major PHP releases with new articles related to PHP, upcoming changes, new features and what's changing in the language. No marketing emails, no selling of your contacts, no click-tracking, and one-click instant unsubscribe from any email you receive.