A properly managed public key infrastructure (PKI) is critical to ensure secure communication on the Internet. Surprisingly, some of the most important administrative steps—in particular, reissuing new X.509 certificates and revoking old ones—are manual and remained unstudied, largely because it is difficult to measure these manual processes at scale.
We use Heartbleed, a widespread OpenSSL vulnerability from 2014, as a natural experiment to determine whether administrators are properly managing their certificates. All domains affected by Heartbleed should have patched their software, revoked their old (possibly compromised) certificates, and reissued new ones, all as quickly as possible. We find the reality to be far from the ideal: over 73% of vulnerable certificates were not reissued and over 87% were not revoked three weeks after Heartbleed was disclosed. Our results also show a drastic decline in revocations on the weekends, even immediately following the Heartbleed announcement. These results are an important step in understanding the manual processes on which users rely for secure, authenticated communication.
1. Introduction
Server authentication is the cornerstone of secure communication on the Internet; it is the property that allows client applications such as online banking, email, and e-commerce to ensure the servers with whom they communicate are truly who they say they are. In practice, server authentication is made possible by the globally distributed Public Key Infrastructure (PKI). The PKI leverages cryptographic mechanisms and X.509 certificates to establish the identities of popular websites. This mechanism works in conjunction with other network protocols—particularly Secure Sockets Layer (SSL) and Transport Layer Security (TLS)—to provide secure communications, but the PKI plays a key role: without it, a browser could establish a secure connection with an attacker that impersonates a trusted website.
The secure operation of the web’s PKI relies on responsible administration. When a software vulnerability is discovered, administrators must act quickly and deploy the patch to prevent attackers from exploiting the vulnerability. Similarly, after a potential key compromise, website administrators must revoke the corresponding certificates to prevent attackers from intercepting encrypted communications between browsers and servers. A recent study suggests 0.2% of SSL connections to Facebook correspond to such man-in-the-middle attacks.10 After considerable research into understanding and improving the speed at which software is patched,14,22 much of software patching has become automated. However, the web’s PKI requires a surprising amount of manual administration. To revoke a certificate, website administrators must send a request to their Certificate Authority (CA), and this request may be manually reviewed before the certificates are finally added to a list that browsers (are supposed to) check. Such operations occur at human timescales (hours or days) instead of computer ones (seconds or minutes). An important open question is: when private keys are compromised, how long are SSL clients exposed to potential attacks?
Historically, these manual processes have been difficult to measure: how can one measure, at scale, how long these processes take if we do not know how often, or precisely when, administrators realize their keys are compromised? In this paper, we use a widespread security vulnerability from 2014, Heartbleed, as a natural experiment: the moment Heartbleed was announced, all administrators of vulnerable servers should have initiated their manual processes as quickly as possible.3 This natural experiment allows us to measure at scale the manual administration of the web’s PKI. In particular, this paper focuses on the response to the public announcement of Heartbleed, in terms of how quickly certificates were reissued and whether or not the certificates were eventually revoked.
Our results expose incomplete and slow administrative practices that ultimately weaken the security of today’s PKI. On the positive side, we also identify ways in which the PKI can be strengthened. Our hope is that, through better understanding how the PKI operates in practice, the security and research community can take concrete steps toward improving this system on which virtually all Internet users rely.
2. Background
In this section, we review the relevant background of SSL/TLS and the PKI, and we describe the Heartbleed vulnerability that serves as our natural experiment.
One of the critical components of the PKI is a certificate: a signed attestation binding a human-understandable subject (a domain name or business name) to a public key. Certificates are signed by a CA, who in turn has its own certificate, etc., forming a logical chain that terminates at a self-signed root certificate. By issuing a certificate, a CA is essentially asserting “this subject is the sole owner of the private key corresponding to this certificate’s public key.” Thus, if someone can prove knowledge of the private key (e.g., by using it to sign a message), then this proves that this someone is the subject. As a result, anyone who has knowledge of the private key can pretend to be that subject.
The assumption that only legitimate subjects hold the corresponding private keys is central to the PKI’s ability to authenticate servers. Unfortunately, keys can be compromised. For example, software vulnerabilities in SSL implementations have resulted in predictable keys22 or the ability to read sensitive server-side data.6
When a private key is compromised, a responsible administrator must do at least three things: first, the administrator must patch the vulnerable software. Second, because the certificate’s private key has been compromised, the administrator must generate a new key pair and ask their CA to reissue a new certificate with this new private key. However, the old, compromised certificate would still exist, and it could be used by a malicious party to undetectably impersonate the website. Thus, there is a critical third step an administrator must do in response to a key compromise: revoke the old certificate. It is important to note that the final two steps necessarily involve the CA, and many CAs charge for these actions, which may lead to perverse incentives for site owners.
When a website stops using a certificate—for instance, because the certificate has been compromised, or because it expired—they must obtain a new certificate. This process is referred to as reissuing the certificate. To do so, the system administrator must contact the CA who signed their certificate and request a new certificate and signature. In cases where the private key may have been compromised, the administrator should also choose a new public/private key pair (since reissuing the certificate with the old public key does nothing to mitigate attacks that leverage the leaked private key).
A certificate revocation is another signed attestation from a CA, which essentially states “this certificate should no longer be considered valid.” CAs are responsible for making revocations available for download, and typically do so with Certificate Revocation Lists (CRLs). Browsers (are supposed to) download CRLs to check if a presented certificate has been revoked; the longer an administrator waits to revoke compromised certificates, the longer users are susceptible to man-in-the-middle attacks.
The PKI uses a default-valid model, where potentially compromised certificates remain valid until their expiration date or until they are revoked. The security of any PKI is thus critically dependent on the timeliness of certificate revocations. However, requesting a revocation is a surprisingly manual process, typically requiring an administrator to visit a website, fill in a form, provide a reason for the revocation, and wait for a representative at the CA to manually inspect the request before issuing the revocation.
While it seems natural to assume that certificates are reissued at precisely the moment the old certificate is revoked, in fact today’s PKI protocols make no such requirement. As our study will demonstrate, reissues can happen before, during, or after a revocation—or even without revoking the old certificate at all. To the best of our knowledge, we are the first to correlate revocations with reissues.
Heartbleed is a buffer over-read vulnerability discovered in OpenSSL15 versions 1.0.1 (released March 14, 2012) through 1.0.1f. The vulnerability stems from a bug in OpenSSL’s implementation of the TLS Heartbeat Extension.17 The intended functionality of TLS Heartbeat is to allow a client to test a secure communication channel by sending a “heartbeat” message consisting of a string and the 16-bit payload_length of this string. Unfortunately, vulnerable OpenSSL versions fail to check that the payload_length supplied by the client matches the length of the provided string. This allows a malicious client to craft a heartbeat message containing a 1-byte string and 216 – 1 as the payload_length. In this case, OpenSSL will allocate a 64KB block of heap memory, memcpy() 64KB of data into it, starting with the 1-byte string, and finally send the contents of the entire buffer to the client. This allows the malicious client to read up to 216 – 2 bytes of the server’s heap memory, although the client cannot chose which memory is read.
By repeatedly exploiting Heartbleed, an attacker can extract sensitive data from the server, including SSL private keys.19 To make matters worse, OpenSSL does not log heartbeat messages, giving attackers free reign to undetectably exploit Heartbleed. Given the severity and undetectable nature of Heartbleed, site operators were urged to immediately update their OpenSSL software and revoke and reissue their certificates.3
Why study Heartbleed? Heartbleed was first discovered by Neel Mehta from Google on March 21, 2014. On April 7, the bug became public and the OpenSSL project released a patched version (1.0.1g) of the OpenSSL library.8
The significance of this timeline, and of Heartbleed in general, is that it represents a point in time after which the administrators of all vulnerable servers should have (1) patched their server, (2) revoked their old certificate, and (3) issued a new one. The scope of this vulnerability—it is estimated that up to 17% of all HTTPS web servers were vulnerable13—makes it an ideal case study for evaluating large-scale properties of SSL security in the face of private key compromise. As a result, Heartbleed acts as a natural experiment, allowing us to measure how completely and quickly administrators took steps to secure their keys. While such events are (sadly) not uncommon,22 the intense press coverage surrounding Heartbleed reduces the likelihood that administrators failed to take action because they were unaware of the vulnerability.
3. Data and Methods
We now describe the data sets that we collected and our methodology for determining a host’s SSL certificate, when it was in use, if and when the certificate was revoked, and if the host was (or is still) vulnerable to the Heartbleed bug.
We obtain our collection of SSL certificates from (roughly) weekly scans of the entire IPv4 address space made available by Rapid7.16 We use scans collected between October 30, 2013 and April 28, 2014. There are a total of 28 scans during this period, giving an average of 6.7 days (with a minimum of 3 days and maximum of 9 days) between successive scans.
The scans found an average of 26.9 million hosts responding to SSL handshakes on port 443 (an average of 9.12% of the entire IPv4 address space). Across all of the scans, we observed a total of 19,438,865 unique certificates (including all leaf and CA certificates). In the sections below, we describe how we filtered and validated this data set; an overview of the process is provided in Figure 1.
Figure 1. Workflow from raw scans of the IPv4 address space to valid certificates (and corresponding CRLs) from the Alexa Top-1M domains. The Rapid7 data after February 5, 2014 did not include the intermediate (CA) certificates, necessitating additional steps and data to perform validation.
To focus on web destinations that are commonly accessed by users, we use the Alexa Top-1M domains1 as observed on April 28, 2014. We first extract all leaf (non-CA) certificates that advertise a Common Name (CN) that is in one of the domains in the Alexa list (e.g., we would include certificates for facebook.com, www.facebook.com, as well as *.dev.facebook.com). This set represents 1,573,332 certificates (8.1% of all certificates).
Unfortunately, despite leaf certificates having a CN in the Alexa list, many may not be valid (e.g., expired certificates, forged certificates, certificates signed by an unrecognized root, etc.). We removed these invalid certificates4 by running openssl verify on each certificate (and its corresponding chain). We configure OpenSSL to trust the root CA certificates included by default in the OS X 10.9.2 root store12; this includes 222 unique root certificates.
After validation, we are left with 628,692 leaf certificates (40.0% of all certificates advertising Alexa domains and 3.2% of all certificates). We refer to this set of certificates as the Leaf Set, each of which has a valid chain. We refer to the set of all CA certificates on these chains (not including the leaf certificates) as the CA Set, which contains 910 unique certificates. The Leaf Set certificates cover 166,124 (16.6%) of the Alexa Top-1M domains. This is the set of certificates and chains that we use in the remainder of the paper.
To determine if and when certificates were revoked, we extracted the CRL URLs out of all Leaf Set certificates. We found 626,659 (99.7%) of these certificates to include at least one well-formed, reachable CRL URL. For certificates that included multiple CRL URLs, we included them all. We found a total of 1,386 unique CRL URLs (most certificates use a unified CRL provided by the signing CA, so the small number of CRLs is not surprising). We downloaded all of these CRLs on May 6, 2014, and found 45,268 (7.2%) of the Leaf Set certificates to be revoked.
We also collected the CRL URLs for all certificates in the CA set. We found that 884 (97.1%) of the certificates in the CA Set included a reachable CRL; the union of these URLs comprised 246 unique reachable URLs. We downloaded these CRLs on May 6, 2014, as well. We found a total of seven CA certificates that were revoked, which invalidated 60 certificates in the Leaf Set (< 0.01%).
3.4. Inferring the Heartbleed vulnerability
Finally, we wish to determine if a site was ever vulnerable to the Heartbleed OpenSSL vulnerability (and if it continued to be vulnerable at the end of the study). Doing so allows us to reason about whether the site operators should have reissued their SSL certificate(s) and revoked their old one(s). Determining if a host is currently vulnerable to Heartbleed is relatively easy, as one can simply send an improperly-formatted SSL heartbeat message with a payload_length of 0 to test for vulnerability without exfiltrating any data.6
However, determining if a site was vulnerable in the past—but has since updated their OpenSSL code—is more challenging. We observe that only three of the common TLS implementations have ever supported SSL Heartbeats17: OpenSSL,15 GnuTLS,20 and Botan.2 Thus, if a host supports the SSL Heartbeat extension, we know that it is running one of these three implementations. Botan is targeted for client-side TLS, and we know of no popular web server that uses the Botan TLS library. GnuTLS has support for the SSL Heartbeat extension, but it is not enabled by default. Furthermore, GnuTLS supports the Max Fragment Length SSL extension,7 which is enabled by default, while OpenSSL has never supported this extension. Thus, if we observe a host that supports SSL Heartbeat but not the Max Fragment Length, we declare that host to have been running a vulnerable version of OpenSSL.
To collect the list of sites that were ever vulnerable to Heartbleed, we extracted the IP addresses in the April 28, 2014 Rapid7 scan that were advertising a certificate with a CN in the Alexa Top-1M list. We found 5,951,763 unique IP addresses in this set. We then connected to these IP addresses on port 443, determined the SSL extensions that the host supported, and checked whether the host was still vulnerable to the Heartbleed vulnerability.
4. Analysis
We now examine the collected SSL certificate data, beginning with a few definitions.
We are concerned with the evolution of SSL certificates (i.e., when are new certificates created, old ones retired, etc.). To aid in understanding this evolution, we define the following notions:
Certificate birth: We define the birth of an SSL certificate to be the date of the first scan where we observed any host advertising that certificate.
Certificate death: Defining the death of a certificate is more complicated, as we observe a number of instances where many hosts advertise a given certificate, and then all but a few of the hosts switch over to a new certificate (presumably, the site intended to retire the old certificate, but failed to update some of the hosts). To handle these cases, we calculate the maximum number of hosts that were ever advertising each certificate. We then define the death of an SSL certificate to be the last date that the number of hosts advertising the certificate was above 10% of that certificate’s maximum. The 10% threshold prevents us from incorrectly classifying certificates that are still widely available as dead, even if the certificate has been reissued.
An example of certificate lifetime for m.scotrail.co.uk is shown in Figure 2. All hosts except one switch to a new certificate after February 10, 2014; this lone host finally switches on April 28, 2014. In this case, we would consider the death date of the old certificate to be February 10, 2014.
Figure 2. Example of lifetime, for certificates for m.scotrail.co.uk. All hosts except one switch to a new certificate after February 10, 2014.
Based on these definitions, we can now define the notion of a certificate reissue and revocation:
Certificate reissue: We consider a certificate to be reissued if the following three conditions hold: (a) we observe the certificate die, and (b) we observe a new certificate for the same Common Name born within 10 days of the certificate’s death, and (c) we observe at least one IP address switch from the old certificate to the new. We define the date of the certificate reissue to be the date of the certificate’s death. For the sake of clarity, we refer to the old certificate that was replaced as the retired certificate.
Certificate revocation: A certificate is revoked if its serial number appears in any of the certificate’s CRLs. The date of revocation is provided in the CRL entry.
In Figure 3, we present the number of certificate births, deaths, reissues, and revocations per day over time (please note the y-axis is in log scale). Births almost always outnumber deaths, meaning that the total number of certificates in-the-wild is growing. Furthermore, we see an average of 29 certificate revocations per day before Heartbleed; after Heartbleed, this jumps to an average of 1,414 revocations per day.
Figure 3. The rate of certificate births, deaths, reissues, and revocations increased sharply after the Heartbleed announcement.
We present a brief analysis of the number of certificates hosted by machines that were ever vulnerable to Heartbleed. Of the 428,552 leaf certificates that were still alive on the last scan, we observe 122,832 (28.6%) of them advertised by a host that was likely vulnerable to Heartbleed at some point in time. These certificates come from 70,875 unique Alexa Top-1M domains. Of these certificates, 11,915 (from 10,366 unique domains) were on hosts that were still vulnerable at the time of our crawl (April 30, 2014). This result demonstrates that even in the wake of a well-publicized, severe security vulnerability, around 10% of vulnerable sites did not address the issue three weeks after the fact.
In Figure 4, we present the fraction of domains that have at least one SSL host that was ever vulnerable to Heartbleed (or still was as of April 30, 2014). We can observe a slight increase in likelihood of ever being vulnerable for the most popular sites, but the distribution quickly stabilizes. The increased likelihood of being vulnerable is likely because these sites have larger numbers of hosts. This trend is mirrored in the hosts that are still vulnerable on April 30, 2014.
Figure 4. Fraction of domains that have at least one host that was ever vulnerable to Heartbleed as a function of Alexa rank, as well as domains that continued to be vulnerable at the end of the study.
We now examine the reissuing of SSL certificates in the wake of Heartbleed. Not all SSL certificate reissues that we observe following Heartbleed’s announcement are due to the Heartbleed vulnerability. In particular, reissues can happen for at least two other reasons: first, the old certificate could be expiring soon. For example, before Heartbleed, we observe that 50% of certificates are reissued within 60 days of their expiry date. Second, a site may periodically reissue certificates as a matter of policy (regardless of expiration date). For example, we observed that Google typically reissues the google.com certificate every two weeks.
In this study, we wish to distinguish a Heartbleed-induced certificate reissue from a reissue that would have happened anyway. We define a certificate reissue to be Heartbleed-induced if all three of the following conditions hold:
- The date of reissue was on or after April 7, 2014 (the day Heartbleed was announced).
- The certificate that is reissued was not going to expire for at least 60 days. This eliminates certificates that were likely to be reissued in the near future anyway.
- We do not observe more than two other reissues for certificates with that CN in the time before Heartbleed. This implies that certificates with that name do not typically get reissued more than once every three months.
Thus, for the examples discussed so far, we do not consider the reissue of the retired certificate in Figure 2 to be Heartbleed-induced (as it happened before Heartbleed), and we do not consider any of the google.com reissues to be Heartbleed-induced (because we observed a total of 12 reissues of that certificate prior to Heartbleed).
Heartbleed-induced reissues. Overall, we observe 36,781 certificate reissues that we declare to be Heartbleed-induced in the three weeks following the announcement; this is 8.9% of all certificates that were alive at the time Heartbleed was announced. Figure 5 examines the fraction of sites that have at least one Heartbleed-induced certificate reissue, as a function of Alexa rank; we can observe a strong correlation with Alexa rank. Higher-ranked sites are much more likely to have reissued at least one certificate due to Heartbleed (even though they are only slightly more likely to have been vulnerable, as observed in Figure 4). This result complements previous studies’ findings that more popular websites often exhibit more sound administrative practices.5, 9
Figure 5. Fraction of domains that have at least one Heartbleed-induced reissue/revocation as a function of Alexa rank.
Reissues with same key. System administrators who believe that their SSL private key may have been compromised should generate a new public/private key pair when reissuing their certificate. We now examine how frequently this is done, both in the case of normal certificate reissues and for Heartbleed-induced reissues.
Before Heartbleed, we observe that reissuing a certificate using the same key pair is quite common; up to 53% of all reissued certificates do so. This high-level of key reuse is at least partially due to system administrators re-using the same Certificate Signing Request (CSR) when requesting the new certificate from their CA. After Heartbleed, we observe a significant drop in the frequency of reissuing certificates with the same key, that is, sites are generating a new key pair more frequently. However, if we focus on the Heartbleed-induced reissues, we observe that a non-trivial fraction (4.1%) of these certificates are reissued with the same key (thereby defeating the purpose of reissuing the certificate). In fact, we observe a total of 912 such certificates coming from 747 distinct Alexa domains; these certificates may represent cases where administrators believe they have correctly responded to Heartbleed, but their certificates remain as vulnerable as if they had not reissued at all.
Vulnerable certificates. Finally, we examine the certificates that should have been reissued (regardless of whether they actually were); we refer to these certificates as vulnerable certificates. We declare a certificate to be vulnerable if the following three conditions hold:
- Its date of birth was before April 7, 2014.
- It has not expired as of April 30.
- It was advertised by at least one host that was (or is) vulnerable to Heartbleed.
In other words, these certificates are vulnerable because their private keys could have been stolen by attackers.
Overall, we find 107,712 vulnerable certificates. Of these, only 28,652 (26.7%) have been reissued as of April 30. The remaining 79,060 (73.3%) vulnerable certificates that have not been reissued come from 55,086 different Alexa Top-1M domains. Thus, the vast majority of SSL certificates that were potentially exposed by the Heartbleed bug remain in-use over three weeks after the vulnerability was announced.
We now turn to investigating certificate revocation before, during, and after the revelation of Heartbleed. Recall that it is critical that a vulnerable certificate be revoked: even if a site reissues a new certificate, if an attacker gained access to the vulnerable certificate’s private key, then that attacker will be able to impersonate the owner until either the certificate expires or is revoked. We study both revocation and expiration here, and correlate them with rates of reissue.
Overall revocation rates. Figure 3 shows the number of certificate revocations over time. As noted above, the average jumps from 29 revocations per day to 1,414 post-Heartbleed. However, the spike on April 16, 2014 is somewhat misleading, as it was largely due to the mass-revocation of 19,384 CloudFlare certificates.18
To mitigate this issue, we plot in Figure 6 the number of unique domains that revoked at least one certificate over time. We make three interesting observations: First, the magnitude of the Heartbleed-induced spike is greatly reduced, but we still observe an up-to-40-fold increase in the number of domains issuing revocations per day. Second, we observe that the number of domains issuing revocations falls closer to its pre-Heartbleed level by April 28, suggesting that within 3 weeks most of the domains that will revoke their certificate in direct response to Heartbleed already have.
Figure 6. The rate of domains revoking certificates spiked after Heartbleed, but dropped closer to normal after three weeks.
Third, we observe three “dips” in the post-Heartbleed revocation rate on April 13, April 20, and April 27—all weekends, indicating that far fewer revocations occur on the weekend relative to the rest of the week. This periodicity can also be (less-easily) observed in the pre-Heartbleed time frame. It is reasonable to assume revocations dip on weekends because humans are involved in the revocation process, however it is not clear who is responsible for the delays: is it site administrators or CRL maintainers at CAs (or both) who are not working on weekends? Regardless of who is responsible, these weekend delays are problematic for online security, since vulnerabilities (and the attackers who exploit them) do not take weekends off.
Heartbleed-induced revocations. Similar to certificate reissues, not all certificate revocations after April 7, 2014 are necessarily due to Heartbleed (e.g., the site could have exposed their private key due to a different vulnerability). We therefore define a Heartbleed-induced revocation to be a certificate revocation where the certificate had a Heartbleed-induced reissue (see Section 4.3).
Overall, we observe 14,726 Heartbleed-induced revocations; this corresponds to 40% of all Heartbleed-induced reissued certificates. Thus, 60% of all certificates that were reissued due to Heartbleed were not revoked, implying that, if the certificate’s private key was actually stolen, the attacker still would be able to impersonate the victim without any clients being able to detect it.
Figure 5 presents the fraction of sites that have at least one Heartbleed-induced certificate revocation, as a function of Alexa rank. Similar to reissues, sites with high rank are slightly more likely to revoke. Ideally, the two lines in Figure 5 should be coincident, that is, all sites reissuing certificates due to Heartbleed should also have revoked the retired certificates. This result highlights a serious gap in security best-practices across all of the sites in the Alexa Top-1M.
Finally, we examine revocation delay, or the number of days between when a certificate is reissued and it is revoked. Figure 7 presents the cumulative distribution of the revocation delay for both Heartbleed-induced and non-Heartbleed-induced revocations. To make the distributions comparable, we only look at differences between -10 and 10 days (recall that Heartbleed-induced reissues and revocations can only occur after April 7, 2014, limiting that distribution). We observe that Heartbleed-induced revocations appear to happen slightly more quickly, though not to the extent one might expect, given the urgent nature of the vulnerability. We also observe that revocation almost always happens after reissue, which makes sense, since this preserves the availability of HTTPS websites. This result contradicts previous assumptions5 that revocations and reissues occur simultaneously.
Figure 7. Heartbleed-induced revocations were issued slightly faster than other revocations.
Expirations are not enough. To demonstrate how long the effects of the Heartbleed vulnerability will be felt if sites do not revoke their vulnerable certificates, we analyze vulnerable certificates that, by the end of our data collection, were reissued but never revoked. Although we find that 60% of the certificates expire within a year, there are vulnerable certificates that are valid for up to 5 years after Heartbleed was announced. In fact, 10% of the vulnerable certificates still had over 3 years of validity remaining. We conclude from this that, given the meager rates of revocation, it would be helpful for CAs to shift to shorter expiry times in their certificates.
Reissues and revocation speed. Next, we examine how quickly sites responded to Heartbleed. Figure 8 shows the fraction of vulnerable certificates that were not reissued or revoked over the three weeks following the Heartbleed announcement. In this figure, the initial y values do not all start at 1.0 for reissues: this is because, with the coarse granularity of our data, we know the range of time during which some certificates were reissued, but not the precise day. We therefore provide the most optimistic possibility: if we know a certificate was reissued between days d and d + k, we assume it was reissued on day d.
Figure 8. Many vulnerable certificates were not revoked and reissued after Heartbleed (note that the y-axis does not begin at zero).
This figure presents a bleak view of how thoroughly sites revoke and reissue their certificates (note that the y-axis begins at 0.60). Three weeks after the revelation of Heartbleed, over 87% of all certificates we found to be vulnerable were not revoked, and over 73% of them were not reissued. We also found that the revocation rate follows a pattern previously observed in earlier studies on the spread of patches14, 22: there is an exponential drop-off, followed by a gradual decline. This behavior is even more pronounced when looking farther beyond the Heartbleed announcement: 16 weeks after the announcement, there were still 86% who had not revoked and 70% who had not reissued.
Extended validation certificates. Recall that one of the major roles of a CA is to validate the identity of the subjects who purchase certificates. Extended Validation (EV) certificates are a means by which CAs can express that this identity-verification process has followed (presumably) more stringent criteria. Many browsers present EV certificates differently in the address bar.
EV certificates are standard X.509 certificates that are not, in and of themselves, more secure, but the rationale is that with a more thorough verification process by the CAs, these certificates deserve greater trust. That said, there remains concern as to whether this trust is well-placed. We close by investigating the rate at which vulnerable EV certificates were revoked and reissued as compared all certificates overall.
Overall, Figure 8 shows EV certificates follow similar trends to the entire corpus, with a slightly faster and more thorough response. Interestingly, while EV certificates were revoked more quickly, their non-EV counterparts caught up within 10 days; however, EV certificates were reissued both more quickly and more thoroughly. We expect that the underlying cause of this observation is a self-selection effect, that is, security-conscious sites are more likely to seek out EV certificates in the first place. Nonetheless, there are still many vulnerable EV certificates that have not been reissued three weeks after the event (67%) and that have not been revoked three weeks after (87%).
5. Concluding Discussion
In this paper, we studied how SSL certificates are reissued and revoked in response to a widespread vulnerability, Heartbleed, that enabled undetectable key compromise. We conducted large-scale measurements and developed new methodologies to determine how the most popular one million domains reacted to this vulnerability in terms of certificate management, and how this impacts security for clients.
We found that the vast majority of vulnerable certificates have not been reissued. Further, of those domains that reissued certificates in response to Heartbleed, 60% did not revoke their vulnerable certificates—if they do not eventually become revoked, 20% of those certificates will remain valid for two or more years. The ramifications of these findings are alarming: Web browsers will remain potentially vulnerable to malicious third parties using stolen keys for a long time to come. Additionally, we found that domains with EV certificates performed only marginally better than other domains with respect to reissuing and revocation.
Our results are, in some ways, in line with previous studies on the rates at which administrators patched vulnerable software22—for instance, revocation rates followed a sharp exponential drop-off shortly after the vulnerability was made public, and tapered off soon thereafter. However, unlike software bugs, we find that the vast majority of certificates remain vulnerable to attacks, as they have still not been reissued or revoked. These findings indicate that the current practices of certificate management are misaligned with what is necessary to secure the PKI.
5.1. Surveying system administrators
To help better understand the reasons behind the lack of prompt certificate reissues and revocations, we informally surveyed a few systems administrators. We asked what steps they had taken in response to Heartbleed: did they patch, reissue, and revoke, and if not, then why not? We received seven responses. Most reported manually patching their systems, but some relied on managed servers or automatic updates and therefore took no Heartbleed-specific steps. There was some variance in when patches were applied, due to a combination of scheduled reboots and delayed responses from vendors, but the majority of patches were applied quickly.
For revoking and reissuing, however, we saw a wide spectrum of behavior. The few who reissued and revoked did so within 48 hours. Many neither revoked nor reissued; a common reason provided was that the vulnerable hosts were not hosting sensitive data or services. Along similar lines, others reported having reissued the certificate but not revoking, explaining that the certificate is only for internal use. Finally, others reported that they did not perceive reissuing and revoking as important because they had patched quickly after the bug was publicly announced (recall, however, that the vulnerability was introduced over 2 years prior).
Our results from this small survey should be viewed anecdotally—more extensive surveys on certificate administration would be an important area of future work6—but they do shed light on the root causes of why revoking and reissuing are not on equal footing with patching. While administrators almost universally understand the importance of patching after a vulnerability, many do not appreciate or know about the importance of revoking and reissuing certificates with new keys. Of those administrators who do understand the importance, some reported push-back from others who perceived the process as being overly complex. In sum, this points to the need for broader education on the treatment of certificates, and perhaps more assistance from CAs to help ensure that all the prescribed steps are taken.
Our results suggest several changes to common PKI practices that may improve security. First, low revocation rates and long expiration dates form a dangerous combination. Techniques that automate revocation would vastly reduce the period during which clients are vulnerable to malicious third parties. Similarly, adopting short certificate expiration dates (as suggested by Topalovic et al.21) by default will significantly reduce the validity period of vulnerable certificates. Second, mechanisms that enable simultaneous reissue-and-revoke for certificates will make it less likely that invalid certificates are accepted by clients. Third, we have found that many domains continue to serve old, vulnerable certificates even after they reissue. Given the large number of certificates and hosts using them per domain in our dataset, we believe administrators would benefit from tools that more easily track and validate the set of certificates they are using.
This paper is, we believe, the first step towards understanding the manual process of reissuing and revoking certificates in the wake of a vulnerability. Several interesting open problems remain. Because our data focuses on the server and CA side of the PKI ecosystem, we are unable to draw any direct conclusions as to what clients experience. A host-centered measurement study would allow us to understand not only when revocations were added to CRLs, but when clients actually received the CRLs. Moreover, our study opens many questions as to why the certificate reissue and revocation processes are so extensively mismanaged. Our results reinforce previous findings that site popularity is correlated with good security practices, but even the highest ranked Alexa websites show relatively anemic rates of reissues and revocations. Understanding the root causes for why the PKI is mismanaged is an important step toward developing a secure infrastructure.
Our analysis relied on both existing, public sources of data and those we collected ourselves. We make all of our data and our analysis code available to the research community at https://securepki.org.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment