PHP and cURL: How WordPress makes HTTP requests

#
By Peter Tasker

cURL is the workhorse of the modern internet. As its tagline says, cURL is a utility piece of software used to ‘transfer data with urls‘. According to the cURL website, the library is used by billions of people daily in everything from cars and television sets, to mobile phones. It’s the networking backbone of thousands of applications and services, including some very interesting places like unnamed NASA projects.

Many PHP projects and libraries that need to send or receive data over the network use cURL as the default network library under the hood. Unsurprisingly, it’s also a core utility used by WordPress’ Requests API as well as most of our plugins WP Migrate DB Pro, WP Offload Media and WP Offload SES.

If you’re curious about the power of the cURL library, how it works with WordPress and what to watch out for (especially on macOS), then you’re in the right place.

What is cURL?

Let’s start by going over what cURL is. cURL is really two pieces, libcurl which is the C library that does all the magic, and the cURL CLI program. Programming languages like PHP include the libcurl library as a module, allowing them to provide the cURL functionality natively.

The libcurl library is an open source URL transfer library and supports a wide variety of protocols. Not just HTTP, but HTTPS, SCP, SFTP, HTTP/2, HTTP/3 and even Gopher. Pretty much every protocol you can imagine – cURL supports.

cURL has been around since 1998 and turns 23 in 2021, close to a quarter of a century old! But it’s still quite powerful and modern, and it is in very active development with no signs of slowing down. While it does have its quirkiness and issues, it’s useful for a developer to know how it works and what it does.

I can hear you saying, “OK Pete, great. You like this cURL thing, why should I learn about it?”. There are a couple of great reasons!

Reason one is that cURL is neat, and I mean really neat. It’s kind of like an internet swiss army knife. Essentially, if you have a piece of software that needs to make a network request – be it an HTTP POST request to a remote URL or an SFTP file download – cURL is often the most straightforward choice.

For example, to send a HTTP POST request with a file upload, using the cURL CLI, run:

curl --form name=Peter --form age=34 --form upload=@/Users/petertasker/photos/image-1.jpg http://httpbin.org/post

How about downloading a large file?

curl -O http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.14.tar.gz

Or, getting the HTTP headers from a server?

curl -I https://deliciousbrains.com

Which will return the HTTP headers:

server: nginx
date: Thu, 11 Mar 2021 14:56:01 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
link: <https://deliciousbrains.com/wp-json/>; rel="https://api.w.org/"
link: <https://deliciousbrains.com/wp-json/wp/v2/pages/28>; rel="alternate"; type="application/json"
link: <https://deliciousbrains.com/>; rel=shortlink
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
content-security-policy: default-src 'self' https: data: wss: 'unsafe-inline' 'unsafe-eval';
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
fastcgi-cache: HIT

The second reason that cURL is worth learning about is that it is available on pretty much every platform and can be installed quickly and easily. If you’ve got a web server, chances are you’ve got cURL.

For example, if you don’t have cURL already installed on macOS, you can quickly get it with Homebrew by running:

brew install curl

So far, we’ve been looking at cURL, the CLI tool, but cURL bindings are also available for most languages, including PHP. If you’re using PHP software that makes network requests, you’ve likely been using cURL!

cURL and PHP

In PHP land, cURL support is like any other module that you may rely on, like mysqli or the GD library.

Most versions of PHP are compiled with cURL by default, but cURL integration is technically an extension, just like mysqli and everything else listed in the extensions section of your phpinfo() output:

cURL phpinfo() output

PHP’s cURL implementation, however, leaves a little to be desired. Whereas the cURL CLI tool is relatively straightforward, PHP’s implementation is a bit more complicated.

When working with PHP’s implementation of cURL you’re required to use the curl_setopt() function. This function lets you set cURL options. For example, setting up an HTTP POST request looks like this:

$curl = curl_init( 'https://httpbin.org/post' );
curl_setopt( $curl, CURLOPT_POST, true );
curl_setopt( $curl, CURLOPT_POSTFIELDS, array( 'field1' => 'some data', 'field2' => 'some more data' ) );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, true );
$response = curl_exec( $curl );
curl_close( $curl );

While not awful, making requests this way can get a little out of hand with larger requests and more complicated CURLOPT_ parameters.

Fortunately, the wonderful PHP community has created libraries that abstract a lot of the complexity away. Two of the more popular networking libraries are Guzzle and Requests. Because Requests supports older versions of PHP and WordPress historically supported really old versions of PHP (😩), the Requests library is used in WordPress core.

cURL in WordPress

Internally, WordPress uses the WP_Http class for network requests, which in turn relies on the Requests library. This means that all of the HTTP utility methods like wp_remote_get() and wp_remote_post() use Requests. At a high level, WordPress updates, plugin downloads, plugin updates, and pretty much any upload/download functionality in WordPress core are using the Requests abstraction of the cURL bindings and options.

Let’s take a quick peek at how the Requests library makes HTTP requests. If you pop open wp-includes/class-http.php, you’ll be able to check out a lot of the internal plumbing that drives HTTP requests within WordPress. As of WordPress 4.6, the WP_Http::request(); method uses the Requests::request() method.

In WordPress 5.7, you can find this call on line 394 of the WP_Http class referenced above.

$requests_response = Requests::request( $url, $headers, $data, $type, $options );

Pretty simple right? Now compare the HTTP POST request below to the raw CURLOPTS method above:

$data = array( 'key1' => 'value1', 'key2' => 'value2' );
$response = Requests::post( 'http://httpbin.org/post', array(), $data );

Much simpler. And if you’re working in the context of a WordPress plugin or theme, you can use the wp_remote_post() function for even more abstraction:

$data = array( 'key1' => 'value1', 'key2' => 'value2' );
$response = wp_remote_post( 'http://httpbin.org/post', array( 'data' => $data ) );

Now we’re talking! wp_remote_post() simply calls WP_Http::request() with POST as the method parameter.

Now, as far as Requests works internally, let’s take a look at wp-includes/class-requests.php around line 359.

In the Requests::request() method, you can see that the code first looks for a $transport option. In the WordPress implementation of Requests, there are only 2 default options, cURL and fsockopen, in that order. fsockopen uses PHP streams, and is a fallback for when the cURL extension isn’t installed.

...
if (!empty($options['transport'])) {
    $transport = $options['transport'];

    if (is_string($options['transport'])) {
        $transport = new $transport();
    }
}
else {
    $need_ssl = (0 === stripos($url, 'https://'));
    $capabilities = array('ssl' => $need_ssl);
    $transport = self::get_transport($capabilities);
}
$response = $transport->request($url, $headers, $data, $options);
...

Once the transport is determined, the request is passed into the chosen $transport class. Since we’re covering cURL in this article, we’ll quickly examine how Requests uses cURL.

Way down in wp-includes/Requests/Transport/cURL.php around line 130 we’ll see how Requests really works. This class highlights how complex working with cURL in PHP can be. Most of the class logic is about verifying and handling request and response headers and setting the correct CURLOPTS based on the parameters passed to the method.

A lot of parameter setting is also handled in the Requests_Transport_cURL::setup_handler() method on line 309, which switches over the options passed in and correctly sets the right CURLOPT_:

...
switch ($options['type']) {
        case Requests::POST:
            curl_setopt($this->handle, CURLOPT_POST, true);
            curl_setopt($this->handle, CURLOPT_POSTFIELDS, $data);
            break;
        case Requests::HEAD:
            curl_setopt($this->handle, CURLOPT_CUSTOMREQUEST, $options['type']);
            curl_setopt($this->handle, CURLOPT_NOBODY, true);
            break;
        case Requests::TRACE:
            curl_setopt($this->handle, CURLOPT_CUSTOMREQUEST, $options['type']);
            break;
        case Requests::PATCH:
        case Requests::PUT:
        case Requests::DELETE:
        case Requests::OPTIONS:
        default:
            curl_setopt($this->handle, CURLOPT_CUSTOMREQUEST, $options['type']);
            if (!empty($data)) {
                curl_setopt($this->handle, CURLOPT_POSTFIELDS, $data);
            }
    }
...

Ultimately this all boils down to a curl_exec() call once all of the options have been set. If it looks complicated, that’s because it is! Different servers and hosts have different requirements for HTTP headers and SSL handling. Requests does a good job of trying to accommodate a wide variety of setups.

Additionally, there are some hooks within WordPress’ networking functions that allow cURL options to be overridden if needed. For example, I have the following commented out in an mu-plugin in my local development environment:

 add_action('http_api_curl', function( $handle ){
    //Don't verify SSL certs
    curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($handle, CURLOPT_SSL_VERIFYHOST, false);

    //Use Charles HTTP Proxy
    curl_setopt($handle, CURLOPT_PROXY, "127.0.0.1");
    curl_setopt($handle, CURLOPT_PROXYPORT, 8888);
 }, 10);

In the above example, I’m using the http_api_curl hook to first disable SSL certificate verification. This is helpful when you’re working with a development site with a self-signed certificate that doesn’t need to be validated. On a side note, a great alternative to overriding cURL options to disable host verification is to create your own Certificate Authority for your local server.

The second block allows me to proxy through Charles to inspect PHP’s network requests as they go over the wire. Charles is an awesome tool for debugging network requests that lets you see the nitty-gritty details of each request in your local environment.

How do I update cURL in WordPress?

Well, you don’t. WordPress doesn’t have its own cURL library but relies on the version of cURL that the PHP installation provides. And the PHP cURL extension is a wrapper around libcurl and will use the version of libcurl that’s installed on the server.

So upgrading cURL for WordPress will almost always be the same as upgrading cURL on the web server. Unless you manage it yourself, this is something you need to ask the server administrator or hosting company to help you with.

If you manage your own Ubuntu server, you can get the latest libcurl version with:

sudo apt update
sudo apt upgrade

If that still leaves you with an outdated cURL version, you probably need to update Ubuntu itself. That’s quite a bit out of scope for this post but it’s good to recognize the signs that cURL might be out of date.

The most obvious sign that the version of cURL on your server is too old is that WordPress can’t connect to remote servers via HTTPS. The error message to look for is cURL Error 60. This error means that cURL can’t verify the SSL certificate of the remote host. This could be caused by an outdated root certificate list on your system.

To check your current version of cURL via the WordPress admin dashboard, go to Tools → Site Health → Info:

Details about your server in WordPress Site Health tool

Compare your cURL version to the list of cURL releases. If it’s more than 3 years old, it’s a good idea to upgrade. If it’s older than 2017 you may have problems connecting to some Amazon services starting March 23, 2021.

cURL on Mac

One of the issues we have seen come up from time to time in support requests for WP Migrate DB Pro revolve around how cURL and SSL work together.

OpenSSL is an industry standard SSL/TLS toolkit for handling encrypted communications. Like cURL, it’s another software library. cURL is compiled with an SSL/TLS toolkit to make connections over the TLS protocol. In the case of WP Migrate DB Pro, this would be when you try to push or pull from a site that has an SSL certificate installed (HTTPS sites). An issue can arise on macOS when cURL isn’t compiled with OpenSSL.

SSL handling with cURL is a huge topic (we have written support documentation on the subject), but with macOS environments the issue is normally that a different SSL/TLS library called SecureTransport is used.

This used to cause issues on Mac computers because Apple’s SecureTransport didn’t implement all the needed SSL/TLS features as quickly as OpenSSL did. For instance, it took Apple until OSX/High Sierra (10.13.4) in 2018 to include support for the HTTP/2 protocol. This could cause problems for local development on Macs. The situation is much better now, but there’s still some rare cases where Local Transport vs. SSL can bite you.

A good test to see which version of OpenSSL that PHP is using is by using grep to search the phpinfo output for the ‘SSL Version’ string.

php -i | grep "SSL Version"

In the output, you should see if you’re using OpenSSL or SecureTransport.

# Ubuntu 20.04:
SSL Version => OpenSSL/1.1.1i

# MacOS 11.2.1 / Big Sur
SSL Version => (SecureTransport) OpenSSL/1.1.1i

As you can see, PHP on an Ubuntu development server uses OpenSSL (which we prefer), but the PHP interpreter on my Mac uses Apple’s SecureTransport, with bindings to OpenSSL (so that’s going to be just fine).

But wait! There is one more thing with cURL and MacOS. Look closely at what the cURL CLI tool reports on my Mac:

curl 7.64.1 (x86_64-apple-darwin20.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.41.0
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

The cURL CLI tool also uses Apple’s SecureTransport library, but this time it’s with bindings to LibreSSL, which is another library for SSL handling. The interesting thing to note here is that on a Mac, cURL in PHP and cURL on the command line may use different libraries for SSL handling. 🙃

If you want to make absolutely sure that your command line cURL behaves the same way as cURL in PHP, you can run this command to get the OpenSSL version:

brew install curl-openssl

There can also be issues if both servers are using different versions of OpenSSL, and it’s generally best to have matching versions. This can lead to the same errors you would see when using SecureTransport.

Closing Thoughts

So now we know what the cURL library is and how to use it. We leveled up by learning the basics of how PHP and WordPress core use the libcurl bindings and covered the details of a common macOS cURL issue.

What are your thoughts about networking and cURL? Do you have any tips or tricks that you use in your workflow? Let us know in the comments section below!

This entry was posted in WP Migrate DB Pro, Code and tagged , , , , , .

About the Author

Peter Tasker

Peter is a PHP and JavaScript developer from Ottawa, Ontario, Canada. In a previous life he worked for marketing and public relations agencies. Love's WordPress, dislikes FTP.

@petetasker petetasker.com