Opening Up the Baseboard Management Controller

 

In 2011, Facebook announced the Open Compute Project to form a community around open source designs and specifications for datacenter hardware. Facebook shared its hardware specs, which resulted in 38% less energy consumption and 24% cost savings compared with its existing datacenters.⁶ What Facebook and other hyperscalers (Google, Microsoft, et al.) donate to the Open Compute Project are their solutions to the agonizing problems that come with running datacenters at scale.

Since then, the project has expanded to all aspects of the open datacenter: baseboard management controllers (BMCs), network interface controllers (NICs), rack designs, power busbars, servers, storage, firmware, and security. This column focuses on the BMC. This is an introduction to a complicated topic; some sections just touch the surface, but the intention is to provide a full picture of the world of the open source BMC ecosystem, starting with a brief overview of the BMC’s role in a system, touching on security concerns around the BMC, and then diving into some of the projects that have developed in the open source ecosystem.

If the CPU is the brain of the board, the BMC is the brain stem. It monitors and manages the physical state of a computer or hardware device. This state includes temperature, humidity, power supply voltage, fan speeds, remote access, and operating system functions. The BMC has historically been a SuperH or ARM-based system on a chip (SoC) with common functionality including but not limited to:

• RMII (reduced media-independent interface) and RGMII (reduced gigabit media-independent interface) for ethernet.
• A boot flash with an SPI (serial peripheral interface) NOR. (NOR and NAND are types of nonvolatile flash memory; the difference is in the type of logic gate used.)
• PCI (peripheral component interconnect) express.
• An LPC (low pin count) bus for communicating with the host. Intel’s successor to LPC is the eSPI (Enhanced Serial Peripheral Interface bus).

The BMC usually communicates to the outside world (or the datacenter control network) using the intelligent platform management interface (IPMI), a message-based, hardware-level interface specification for managing and operating computer systems. It operates independently of the operating system, the server’s CPU, and the firmware that allows admins to manage a system without an operating system or any system management software. Admins can also take advantage of IPMI’s local network to get a console on a remote computer that is otherwise inaccessible.

Back to Top

BMC Security Concerns

The IPMI stack was not designed with security in mind (the IPMI spec requires making the hash of a user’s password available over the stack). The assumption was the datacenter control networks would be segregated and trusted, which is why IPMI is notorious for security vulnerabilities.² Exploits in the IPMI stack and the BMC are devastating because of the many privileged operations for which they are responsible. Improving IPMI security has historically been neglected, as most IPMI software is proprietary.

The BMC has its own problems with largely proprietary software and vulnerabilities. The most recent notable BMC vulnerability is USBAnywhere,³ discovered by Rick Altherr, principal engineer at Eclypsium. On Supermicro servers, an attacker can use USBAnywhere to connect remotely to a server and virtually mount any USB device to the server. As a result, an attacker could load a new operating system image or implant a firmware backdoor to facilitate ongoing remote access. At the time of the disclosure, 47,000 vulnerable systems were found to be exposed to the public Internet. Another fun vulnerability is Pantsdown,1 which allows read and write access to the BMC’s address space from the host. Pantsdown is an example of a requested feature causing a vulnerability.

But wait, it gets worse. As Trammell Hudson pointed out in his Modchips of the State talk at the 35^th Chaos Communication Congress in 2018,⁷ the BMC often has access to the host firmware via serial peripheral interface (SPI) and to host memory through direct memory access (DMA). The BMC gets DMA access because it is on the peripheral component interconnect express (PCIe) bus as a device. This means it can inject code into the host’s firmware. Much BMC firmware also lacks the notion of a secure boot. This makes the BMC a prime target for hackers. Here, I emphasize a point I made in a previous article on open source firmware:5 It’s an alarming problem that the code running with the most privilege has the least visibility and inspectability.

Back to Top

The BMC Becomes Open Source

The trend toward open sourcing the datacenter has led to a number of innovative BMC projects.

OpenBMC. In 2014, Facebook decided to solve the problems with proprietary BMC software by starting an open source BMC software project.⁴ In 2015, IBM and Rackspace collaborated on solving the same problems with their own project.⁹ Both projects were called OpenBMC and ended up merging into the OpenBMC project the firmware community is familiar with today (https://github.com/openbmc/openbmc). The founding organizations of the OpenBMC project, post-merger, were Microsoft, Intel, IBM, Google, and Facebook. OpenBMC has the widest range of support for various BMCs.

The OpenBMC project encompasses u-boot, an open source boot-loader that boots a Linux kernel with a minimal root file system containing all the tools and binaries needed to run OpenBMC. OpenBMC is designed with a service-oriented approach. Services are started and maintained by systemd and communicate with each other over dbus. Designing for services makes sense as an easy way for multiple collaborators and vendors to contribute to a single BMC implementation. This allows each vendor contributing to the codebase to have separate daemons it can turn on to ship in its specific distribution of OpenBMC; however, it also makes the BMC software more complex to debug, audit, and put into production.

U-bmc. After OpenBMC came u-bmc (https://github.com/u-root/u-bmc), a software project started by Christian Svensson of Google. Written in Go, u-bmc aims for a more minimal BMC software architecture, challenging the status quo by replacing IPMI with gRPC. Removing IPMI makes u-bmc provocative from a security perspective since the attack surface area is reduced. Unlike OpenBMC, u-bmc boots a Linux kernel directly from the ASPEED startup code after DRAM initialization, thus removing the need for a bootloader such as u-boot. As of the publication of this article, u-bmc supports BMCs based on the ASPEED AST2400 and AST2500, but plans to support more in the future and always welcomes contributions. If you have a Supermicro X11SSH board that supports coreboot, it is possible to use u-bmc as your BMC software.

RunBMC. Not only has software around the BMC been open sourced, but the hardware has as well. Eric Shobe and Jared Mednick of Dropbox analyzed all the BMC system topologies and their differences on a platform-by-platform basis. The result was RunBMC, a standard hardware interface for BMCs. Dropbox donated version 1 of the RunBMC hardware specs, along with two reference boards for the Nuvoton NPCM75OR and ASPEED 2500 RunBMC modules, to the Open Compute Project in August 2019.⁸

---

OpenBMC set the stage for BMC firmware and hardware to be open sourced. This spawned a series of other innovations being open sourced, and more can be expected.

---

The RunBMC design allows for swapping out BMCs separate from the rest of the board, isolating and locking down the BMC subsystem. Previous to this, the BMC was soldered onto the board. This is compelling from a security perspective since focus is shifted to a single, swappable BMC card, which can easily be replaced if broken, updated with a different version, or integrated with other security features. For example, a root of trust, the trusted source that verifies system software before execution, can secure I/O between the BMC card and the rest of the board. This also allows users to switch easily between the common BMC manufacturers, ASPEED, and Nuvoton. Interesting fact: Sun also had a BMC interconnect with its Integrated Lights Out Manager (ILOM), as did Dell with Dell Remote Access Controller (DRAC), HP with Integrated Lights-out (iLO), and IBM and Lenovo with integrated management module (IMM)—however, most do not ship this way today.

Back to Top

Open Source Moving the Ecosystem Further

OpenBMC set the stage for BMC firmware and hardware to be open sourced. This spawned a series of other innovations being open sourced, and more can be expected. This space is turning out many awesome projects, and I am lucky to be able to shine a light on the amazing work being done. Open sourcing the software at the lowest levels of the stack provides visibility into the code running with the most privileges on systems. We can only hope this will lead to more eyes vetting the code, encourage more minimal architectures, and lessen the risk of systems being caught with their “Pantsdown” in the future.

Back to Top

Acknowledgments

Thanks to the individuals in the open source ecosystem for helping me learn about their projects: Rick Altherr, Chris Koch, Christian Svensson, Ron Minnich, Trammell Hudson, Eric Shobe, and Jared Mednick. If you are interested in helping with any of the projects mentioned here, check out GitHub.

Related articles
on queue.acm.org

Security for the Modern Age
Jessie Frazelle
https://queue.acm.org/detail.cfm?id=3301253

Commercializing Open Source Software
Michael J. Karels
https://queue.acm.org/detail.cfm?id=945125

GNL is Not Linux
Kode Vicious
https://queue.acm.org/detail.cfm?id=2909572

Back to Top

Back to Top