Log4j and the Thankless High-Risk Task of Managing Software Component Upgrades

The recent Log4j security vulnerability is an extreme example (but unfortunately, not unique) of a zero-day attack where the entire computing world was caught flat-footed.

What made the Log4j issue so pernicious is that the Log4j framework was used so extensively and that the exposure existed for so long. It is a very good thing for a programming language to have a wide library of trusted components, and Log4j is precisely that kind of example. While not part of Java itself, it is written in Java and a part of the widely-used Apache Commons framework.

This post will not describe the details of this security exposure, but rather will use Log4j as a case study of how managing included versions of third-party software components too often put developers in no-win situations: damned if they upgrade, but damned if they don’t.

The primary CVE description (CVE-2021-44832) for the Log4j issue is described as:

“Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack when a configuration uses a JDBC Appender with a JNDI LDAP data source URI when an attacker has control of the target LDAP server. This issue is fixed by limiting JNDI data source names to the java protocol in Log4j2 versions 2.17.1, 2.12.4, and 2.3.2. “

I find reviewing the Log4j releases in table format helpful to better appreciate this exposure both in length of time and number of versions: June 2013 to December 2021 is 8.5 years and *lot* of releases. 

 

https://logging.apache.org/log4j/2.x/changes-report.html

(as of Jan 3, 2021)

 

Release History

Version	Date	Comment
2.17.1	2021-12-27	GA Release 2.17.1        [Security bug fixed]
2.17.0	2021-12-17	GA Release 2.17.0
2.16.0	2021-12-13	GA Release 2.16.0
2.15.0	2021-12-06	GA Release 2.15.0
2.14.1	2021-03-06	GA Release 2.14.1
2.14.0	2020-11-06	GA Release 2.14.0
2.13.3	2020-05-10	GA Release 2.13.3
2.13.2	2020-04-23	GA Release 2.13.2
2.13.1	2020-02-25	GA Release 2.13.1
2.13.0	2019-12-11	GA Release 2.13.0
2.12.1	2019-08-06	GA Release 2.12.1
2.12.0	2019-06-23	GA Release 2.12.0
2.11.2	2019-02-04	GA Release 2.11.2
2.11.1	2018-07-22	GA Release 2.11.1
2.11.0	2018-03-11	GA Release 2.11.0
2.10.0	2017-11-18	GA Release 2.10.0
2.9.1	2017-09-17	GA Release 2.9.1
2.9.0	2017-08-26	GA Release 2.9.0
2.8.2	2017-04-02	GA Release 2.8.2
2.8.1	2017-02-26	GA Release 2.8.1
2.8	2017-01-21	GA Release 2.8
2.7	2016-10-02	GA Release 2.7
2.6.2	2016-07-05	GA Release 2.6.2
2.6.1	2016-06-05	GA Release 2.6.1
2.6	2016-05-25	GA Release 2.6
2.5	2015-12-06	GA Release 2.5
2.4.1	2015-10-08	GA Release 2.4.1
2.4	2015-09-20	GA Release 2.4
2.3	2015-05-09	GA Release 2.3
2.2	2015-02-22	GA Release 2.2
2.1	2014-10-19	GA Release 2.1
2.0.2	2014-08-16	Bug fixes and enhancements
2.0.1	2014-07-29	Bug fixes
2.0	2014-07-12	GA Release
2.0-rc2	2014-06-21	Bug fixes and enhancements
2.0-rc1	2014-02-16	Bug fixes and enhancements
2.0-beta9	2013-09-14	Bug fixes and enhancements
2.0-beta8	2013-07-10	Bug fixes and enhancements
2.0-beta7	2013-06-01	Bug fixes and enhancements.      [Security bug introduced]
2.0-beta6	2013-05-05	Bug fixes and enhancements.
2.0-beta5	2013-04-20	Bug fixes and enhancements
2.0-beta4	2013-01-28	Bug fixes and enhancements
2.0-beta3	2012-11-11	Bug fixes and enhancements
2.0-beta2	2012-10-07	Bug fixes and enhancements
2.0-beta1	2012-09-18	Bug fixes and enhancements
2.0-alpha2	2012-08-24	Bug fixes and minor enhancements
2.0-alpha1	2012-07-29	Rewrite of Log4j

 

Dependency Management Frameworks

There are many dependency management frameworks for utilizing software components, and the Log4j website has thorough documentation on how to include Log4j with several such as Maven, Ivy, Gradle, and SBT.

https://logging.apache.org/log4j/2.x/maven-artifacts.html

Using Log4j in your Apache Maven build

To build with Apache Maven, add the dependencies listed below to your pom.xml file.

pom.xml

<dependencies>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.23.1</version>
  </dependency>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.23.1</version>
  </dependency>
</dependencies>

 

Using Log4j in your Gradle build

To build with Gradle, add the dependencies listed below to your build.gradle file.

build.gradle

dependencies {
  implementation 'org.apache.logging.log4j:log4j-api:2.23.1'
  implementation 'org.apache.logging.log4j:log4j-core:2.23.1'
}

Log4j isn’t unique in having a long release history, and that is typically a positive attribute as all software efforts need to evolve to remain viable. Irrespective of the stylistic differences for configuration of XML vs. JSON, etc. required by any given dependency framework, the key question that developers need to address for each component utilized is which version to include, and when to upgrade.

Component Upgrades:  Damned If You Don’t

Fixing a specific version and sticking with it has some advantages, as developers know exactly what they are getting with a component.

On the other hand, there are several negative consequences when never upgrading a component version.  For starters, developers receive no new functionality or bug fixes.  It is ironic that if developers had specified version Log4j 2.0-beta6 (or before) and stuck with that, they would have been safe from this particular security exposure.  But that scenario is probably unlikely because the phrase “the GA has been released, but we’re sticking with the beta” would raise a lot of eyebrows in most organizations irrespective of the framework.  Especially sticking with a beta for years on end.

An even greater risk of sticking with one version for long periods of time is the risk of the component going end of life. This in fact happened with Log4j Version 1. Another risk are technical incompatibilities that can arise with older versions of frameworks.

Components such as logging libraries are one thing, but the stakes get even higher with data-layer frameworks such as database or search engines. Running on a data-layer framework that is years out of date is asking for trouble, as it not just complicates support for current operations but makes the upgrade process—and the resulting data migrations—that much more difficult.

And in terms of asking for trouble, not staying on top of operating system and other security related patches is arguably at the top of the list.

So it is clear that not upgrading is a bad thing, and that developers should always upgrade, right?

Component Upgrades:  Damned If You Do

There are arguably two main component upgrade strategies:  always getting the latest, and selective upgrades.

Get Latest

Most dependency management frameworks have configuration patterns to “get the latest,” either by omitting the specified version or by allowing a range of allowable major versions and the latest minor version will be automatically pulled when available.

An advantage of the “Get Latest” approach one will always be current in terms of component functionality (yay!), bug fixes (yay!), and new bugs (boo!). With respect to the Log4j issue, this approach one would have unfortunately automatically picked up the security exposure but then also would have automatically picked up the fix—at least when the fix became available.

A cost of this approach is developers must be constantly ready for breaking component changes in whatever form ranging from the obvious (API or downstream library incompatibility) to the more subtle (component doesn’t work quite the way it did before) and build that assessment time into their own development and testing estimates, because every new component release could trigger the component assessment cycle anew, independent of their own project’s release cycle.

But at least one would be continuously upgraded.

Selective Upgrade

The hardest part of a selective upgrade is determining when, because most developers have a thousand things to do other than review and upgrade component versions.

I’ve also lost count of how many times I’ve heard developers say something like “we need to upgrade component X but marketing wants us to do Y instead because they say we can’t sell infrastructure upgrades or security upgrades to our customers.” In this example, the competing ask Y should be assumed to be completely reasonable and value-add. The fundamental issue is a frequently prevailing viewpoint of functional asks existing on one product roadmap, but technical needs existing on a separate roadmap. But it’s never clear who will work on that second roadmap because most development teams have their hands full just keeping up with the first one. Most organizations simply don’t budget or plan well for this type of work. And while it’s true that one cannot generally sell “infrastructure or security upgrades” to customers, that does not mean those efforts are unnecessary. And the result of this dynamic is trying to get time to upgrade any software component becomes a prioritization battle with stakeholders who typically don’t care or even know about the component in question.

The “Get Latest” approach has potential pitfalls, and it doesn’t address components which require a more heavyweight upgrade process (e.g., data-layer components) but the expectation of building in time for component upgrades still arguably leaves development teams in a much better position than doing nothing, but this would come at a cost that only more advanced development organizations would be willing to pay for. But “Get Latest” would have still walked right into the Log4j exposure, so there’s that.

Given Log4j, SolarWinds, and other recent high-profile security incidents, perhaps it might be easier for technical teams to lobby for time and resources to improve software component management in their respective organizations in the future, and for non-technical leadership to be more attuned to the concerns. One can hope, because there are no easy answers.

Finally

Thanks to all open source component developers and technical communities such as the Apache Software Foundation, as they are a boon to developer productivity. Organizations would learn the wrong lesson from the Log4j security exposure if everyone decided to write their own version of X, where X is a well understood and commonly utilized (and tested) framework, as they would likely be trading one set of problems for another.

There might be a software engineering lesson learned from the Log4j incident in terms of whether the functionality that introduced the security exposure in the first place was truly necessary (feature bloat? too complex?), but I will leave that discussion for others.

 

Related Links

Log4j CVE: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832

Log4j CVE Jira: https://issues.apache.org/jira/browse/LOG4J2-3293

Apache Log4j Security Overview: https://logging.apache.org/log4j/2.x/security.html

“Anna Karenina on Development” – A blog I wrote on the importance of release velocity for the evolution and viability of software efforts: https://cacm.acm.org/blogs/blog-cacm/251396-anna-karenina-on-development-methodologies/fulltext

 

Doug Meil is a portfolio architect at Ontada.  He also founded the Cleveland Big Data Meetup in 2010.  More of Doug’s ACM articles can be found at https://www.linkedin.com/pulse/publications-doug-meil