When Mark Kuemerle, vice president of technology at semiconductor manufacturer Marvell, needed to familiarize some new engineering hires with one of the challenges facing their industry, he borrowed a bunch of LEGO blocks from his kids. He presented the group with a select number of mismatched blocks. The pieces, Kuemerle explained to his new initiates, were meant to represent different chips with particular functions.
Traditionally, all the critical components of a semiconductor have been packaged onto a single piece of silicon, an approach called homogeneous integration. The exercise Kuemerle proposed reflected a design strategy called heterogeneous integration, which pulls together different chips, known as chiplets, with their own specific functions. These chiplets can vary not only in terms of what they are designed to do, but also in their size and specifications. A memory chip might be relatively small, for example, and one designed for machine learning tasks much larger.
The task Kuemerle gave his engineers was to fit the various toy pieces together as if they were real chiplets that needed to transfer data and communicate with each other over the shortest possible distances. The difficulty of the task was immediately, comically, evident. “We were taking the blocks and jamming them together, bending them, shoving them,” Kuemerle recalls. “It’s a very complex geometry problem.”
The potential advantages of a chiplet-based approach render this and other challenges worthwhile. The opportunity to situate chiplets closer to one another accelerates data transfer and performance. The approach is more economical, since it is much less likely for a defect to doom the entire package. Mixing and matching chiplets with different capabilities also allows designers to optimize packages for specific functions or use cases.
“You can put individual chips together that can be very, very different. You could have high-performance processors, advanced memory, sensors, photonics technologies, miniature healthcare-on-a-chip laboratories all on the same platform,” said University of Illinois Chicago computer scientist Inna Partin-Vaisband. “The latest machine learning chips are impressive, but they are expensive and require extensive time to design. By utilizing a chiplet approach, you can integrate various components onto a single platform, allowing for a plug-and-play configuration, and potentially surpass current state-of-the-art performance.”
Maintaining Moore’s Law
The last five decades of semiconductor development have been defined in large part by Moore’s Law, the idea that the number of transistors on an integrated circuit will double roughly every two years, and that the associated rise in costs will be minimal. Although Moore’s Law might be coming to an end, according to Ramin Farjadrad, CEO of chiplet startup Eliyan, the resulting gains have been tremendous. “Because of Moore’s Law we now have processors that are almost five orders of magnitude higher performance than what we had two decades ago,” noted Farjadrad. The downside is that the processing gains have far outpaced advances in other critical components, such as data transfer speeds between chips.
Today the economics have also become more complicated. “The cost of the transistor has been going down for the last five decades from one computer generation to the next, but now we’re reaching an inflection point,” said Madhavan Swaminathan, director of the Center for Heterogenous Integration of Micro Electronic Systems (CHIMES) at Pennsylvania State University. “When you get into very advanced nodes and start building these larger and more sophisticated chips, the capital expenditure required to get such very tiny transistors is very high.”
This, in turn, translates into higher risk. One of the more important variables in the economics of semiconductor manufacturing is yield, or the percentage of successful chips made. A single flaw in one of the components on a standard silicon chip may doom the entire package, as these individual parts cannot be swapped out efficiently. Because of its modular nature, a system comprised of multiple chiplets can boast a higher yield and, therefore, better economics. A defect in one component will not necessarily waste the entire chip.
Moore himself envisioned this possibility in his original 1965 paper, noting: “It may prove to be more economical to build large systems out of smaller functions, which are separately packaged and interconnected.”
Gartner Research industry analyst Bob Johnson explained that chiplets are appealing even based on yield alone: “It sort of keeps Moore’s Law going, as you’re still getting more bang for your buck going forward.”
Chiplet Communication Standardization
The general idea behind chiplets has been around for decades, according to Swaminathan, who worked on early variations in the 1990s, when he was an engineer at IBM. The difference was that those massive multichip modules were made up of components from the same manufacturer. Today, chiplets may come from different suppliers. They have different sizes, as Kuemerle demonstrated in his LEGO exercise, and they do not necessarily have the means to communicate.
Kuemerle said solving this communication gap is one of the next challenges for the industry. “There’s so much complexity that goes into making sure these things work together,” he explained. “It’s like we’re using the same letters of the alphabet but someone is speaking French and someone else is speaking English. If we get to the point as an industry where everybody can talk to everybody, then these components will be able to figure out what work they’re trying to do and how to help each other.” On this front, multiple industry leaders are now supporting an open communications standard, the Universal Chiplet Interconnect Express (UCIe), to help formalize the way chiplets transfer information.
Performance and Geopolitics
Once this communication problem is addressed, there is potential for significant performance gains, in part because if you solve the geometry problem, you can situate key components close to one another. “A centimeter really makes a difference when you’re talking about terabytes of data getting whipped around,” explained Johnson. “Chiplets give you faster data transfer because you can locate the key points physically close to each other, and when you’re talking about running at the speeds they run, that can be critical.”
Companies are also working to improve the physical interconnections between chiplets within a package. One of the problems with standard chip design, according to Eliyan cofounder Farjadrad, is that the connection between compute and memory has not been optimized. “Because of Moore’s Law we have processors that can crunch data 90,000 times faster, but the bandwidth between those chips and the memory nodes has only gone up by 100 times at best,” he explained. To address this gap, Eliyan is developing a chiplet interconnect technology that operates in both standard UCIe-compliant and simultaneous bidirectional (SBD) modes, accelerating data transfer by a factor of two in both directions. The increased bandwidth and faster interconnection will allow AI processors to access memory faster, for example.
The demand for technology that can manage AI workloads is steering semiconductor startup Ayar Labs as well. The company, co-founded by University of California, Berkeley electrical engineer and current CTO Vladimir Stojanovic, has pioneered an optical communications chiplet designed to boost data transfer between semiconductor packages and servers or racks distributed over much larger distances. The technology is being developed to adhere to the UCIe standard and works with a variety of chiplets, but Ayar Labs is focusing on AI. The accelerator chips that the leading AI companies rely on today need to work in unison—workloads are distributed across tens of thousands of chips simultaneously—and Ayar Labs’ TeraPHY optical I/O chiplet increases the bandwidth between them. “We’re connecting hundreds to thousands of compute packages together so that they seem directly interconnected and behave as a single virtual chip,” explained Stojanovic.
The AI applications and general potential for improved performance have made chiplets an area of geopolitical interest. The U.S. has limited China’s access to certain state-of-the-art chip technology, but if chiplet-based designs can keep pace with the best chips on the market, then they might prove to be a suitable workaround. China has increased its focus on advanced packaging, and the disruptive potential of this approach has not gone unnoticed in the U.S., as evidenced by the $50-billion CHIPS and Science Act of 2022. Although the vast majority of that funding is devoted to stimulating semiconductor manufacturing in the U.S., the bill earmarked $3 billion for the National Advanced Packaging Manufacturing Program (NAPMP), to promote the domestic development and fabrication of novel designs like chiplets.
Partin-Vaisband, who is developing a novel, vertical chiplet architecture in part to reduce power loss, expects the infusion of research funding from the $3-billion NAPMP program to drive significant progress in academia. Swaminathan agrees, noting that both university researchers and industry will have a vital role to play in advancing these novel approaches to packaging, and shaping the industry in the years ahead.
“With Moore’s Law, we always had a roadmap and we knew exactly what we needed to do,” Swaminathan said. “That has not happened on the packaging side until now. We now have roadmaps that are giving us insight into the metrics we should be going after, and we can take a technical approach to addressing these challenges.”
Further Reading
- International Roadmap for Devices and Systems (IRDS) 2023 Update (irds.ieee.org).
- Krishnakumar, S., Choi, M., Khorasani, R.R., Sharma, R., Swaminathan, M., Kumar, S., and Partin-Vaisband, I.
Vertical Power Delivery for High Performance Computing Systems with Buck-Derived Regulators. IEEE Proceedings of the Electronic Components and Technology Conference (ECTC). May 2024. - Iyer, S.S., Jangam, S., and Vaisband, B.
Silicon interconnect fabric: A versatile heterogeneous integration platform for AI systems. IBM Journal of Research and Development. 2019 Sep 10;63(6):5-1. - Lau, J.H.
“Recent advances and trends in chiplet design and heterogeneous integration packaging.” Journal of Electronic Packaging 146, no. 1 (2024): 010801. - Li, T., Hou, J., Yan, J., Liu, R., Yang, H., and Sun, Z.
“Chiplet heterogeneous integration technology—Status and challenges.” Electronics 9, no. 4 (2020): 670. - “The Vision for the National Advanced Packaging Manufacturing Program,” CHIPS Research and Development Office, Nov. 20, 2023.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment