Reevaluating Google’s Reinforcement Learning for IC Macro Placement

Circuit placement as an optimization task.  After (x, y) locations of all components are known, wires that connect components’ I/O pins are routed. Routes impact chip metrics (for power, timing/speed, and so on). The optimization of (x, y) locations starts with simplified estimates of wirelength without wire routes. Pin locations (x₁, y₁) and (x₂, y₂) may be connected by horizontal and vertical wire segments in many ways, but the shortest route length is |x₁ − x₂ | + |y₁ − y₂ |.

For multiple pin locations {(x_i, y_i)}_i, this estimate generalizes to

HPWL stands for half-perimeter wirelength, where the perimeter is taken of the bounding box of points {(x_i, y_i)}_i.^23,28,33 It is easy to compute and sum over many nets. This sum correlates with total routed wirelength reasonably well. When (x, y) locations are scaled by a factor γ > 0, HPWL also scales by γ, which makes HPWL optimization scale-invariant and appropriate for all semiconductor technology nodes.^b Algorithms that optimize HPWL extend to more precisely optimize routed wirelength and technology-dependent chip metrics, so HPWL optimization is a precursor:^4,10,22,28

• To test new placement methods; once HPWL results are close to the best known, accurate metrics are used for evaluation; or

• Followed by optimizations of advanced objectives that extend HPWL, for example, the RL proxy cost function in Mirhoseini et al.

Widely adopted optimization frameworks for placement do not use ML^{4,22,23,28,33} and can be classified as: simulated annealing, partitioning-driven, and analytical. Simulated annealing, developed in the 1980s^24,25,38 and dominant through the mid-1990s,⁴⁵ starts with an initial layout (for example, random) and alters it by a sequence of actions, such as component moves and swaps, of prescribed length. To improve the final result, some actions may sacrifice quality to escape local minima. SA excels on smaller layouts (up to 100K placeable components) but takes a long time for large layouts. Partitioning-driven methods³ view the circuit connectivity (the netlist) as a hypergraph and use established software packages to subdivide it into partitions with more connections within the partitions (not between). These methods run faster than SA, capture global netlist structures, and were dominant for some 10 years. Yet, the mismatch between partitioning and placement objectives (Equation 1) leaves room for improvement.³ Analytical methods approximate Equation 1 by closed-form functions amenable to established optimization methods. Force-directed placement¹² from the 1980s models nets by springs and finds component locations to reconcile spring forces.²³ In the 2000s, advanced analytical placement techniques attained superiority^10,22,28,33 on all large, public benchmark sets, including those with macros and routing data.¹⁰ RePlAce¹⁰ from UCSD is much faster than SA and partitioning-based methods, but lags in quality on small netlists.

The Nature paper focuses on large circuit components (macros) among numerous small components. The fixed-outline macro-placement problem, which was formulated in the early 2000s,^1,21,44 places all components onto a fixed-size canvas (prior formulations could stretch the canvas). It is now viewed as part of mixed-size placement.³ A 2004 benchmark suite² for testing mixed-size placement algorithms evaluates the HPWL objective (Equation 1) which, as noted above, is apt for all semiconductor technology nodes. The suite has enjoyed significant use in the literature, for example Cheng et al.,¹⁰ Kahng,²² and Markov et al.²⁸

Commercial and academic software for placement is developed to run on modest hardware within reasonable runtime. The methods and software in Mirhoseini et al. consume significantly greater resources, but at least with SA (during comparisons) it is straightforward to obtain progressively better results with greater runtime budget.

Circuit metrics for evaluating optimization results include circuit timing and dynamic power. Unlike power, timing metrics are sensitive to long/slow paths taken by signal transitions in a circuit and are difficult to predict before detailed placement and wire routing. Accurate early estimation of circuit metrics is a popular topic in the research literature but remains an unsolved challenge in physical design because metric values depend on the actual decisions by optimizers. For example, decisions on which wires take the shortest routes and which ones get detoured determine which pairs of wires experience crosstalk and which signal paths become slow.²³ Because of this estimation difficulty, optimization methods with closed-form objectives are fundamentally limited in what they can achieve, and circuit implementation may need to be redone when routing cannot be completed or timing constraints cannot be satisfied.²²

Key sources.  To solve mixed-size placement, the Nature paper first places macros and then places small components with commercial software. It places numerous macros with an RL action policy that is iteratively improved (fine-tuned) at the same time. The RL policy can be pre-trained on prior circuits or initialized “from scratch.” The iterative process runs for a set time (or until no change) and optimizes a fixed (not learned) proxy cost function that blends HPWL, component density, and routing congestion. To evaluate this function, the small components are placed with force-directed placement. The paper claims that RL beats three baselines: (1) macro placement by human chip designers, (2) parallel SA, and (3) RePlAce software from UCSD, which uses no RL.

Among secondary sources discussed in the context of Mirhoseini et al., we prefer scholarly papers^5,11,46 but also draw on open source repositories and include FAQs as needed.^13,27,c Here, all benchmark sets have hundreds of macros per design, compared to only a handful in sets such as ISPD 2015. We crosscheck claims from three nonoverlapping groups of researchers: those associated with Google Team 1 (Mirhoseini et al. and CT), the Google Team 2 (Bae et al.⁵), and the UCSD Team (Cheng et al.¹¹ and the Macro Placement Repo—see Table 1). Consistent claims from different groups are even more trustworthy when backed by numerous benchmarks. Both Google Team 2 and the UCSD Team included highly cited experts on floor-planning and placement with extensive publication records and several key references cited in Mirhoseini et al., (such as Cheng et al.,¹⁰ Markov et al.,²⁸ and others), as well as experience developing academic and commercial floor-planning and placement tools beyond Google.

Unsubstantiated claims and insufficient reporting.  Serious omissions are clear even without a background in chip design.

U1. With “fast chip design” in the title,³⁰ the authors only described improvement in design-process time as “days or weeks to hours” without giving per-design time or breaking it down into stages. It was unclear if “days or weeks” for the baseline design process included the time for functional design changes, idle time, inferior EDA tools, and so on.

U2. The claim of RL runtimes per testcase being under six hours (for each of five TPU design blocks)³⁰ excluded RL pre-training on 20 blocks (not amortized over many uses, as in some AI applications). Pausing the clock for pre-training (not used by prior methods) was misleading. Also, RL runtimes only cover macro placement, but RePlAce and industry tools place all circuit components.

U3. Mirhoseini et al. focused on placing macros but withheld the number, sizes, and shapes of macros in each TPU chip block, and other key design parameters such as area utilization.

U4. Mirhoseini et al. gave results on only five TPU blocks, with unclear statistical significance, but high-variance metrics produce noisy results (Table 2). Using more examples is common (Table 1).

U5. Mirhoseini et al. was silent on the qualifications and level of effort of the human chip designer(s) outperformed by RL. Reproducibility aside, those results could be easily improved (as shown in Cheng et al.¹¹ later).

U6. Mirhoseini et al. claimed improved “area”, but chip area and macro area did not change and standard-cell area did not change during placement (also see the 0.00 correlation in Table 2).

U7. For iterative algorithms that optimize results over time, fair comparisons show per testcase: better-quality metrics with equal runtime, better runtime with equal quality, or wins for both. Mirhoseini et al. offered no such evidence. In particular, if ML-based optimization is used with extraordinary compute resources, then so should be optimization by SA in its most competitive form.

A flawed optimization proxy.  The chip design methodology in Mirhoseini et al. uses physical synthesis to generate circuits for further layout optimization (physical design). The proposed RL technique places macros of those circuits to optimize a simplified proxy cost function. Then, a commercial EDA tool is invoked to place the remaining components (standard cells). The remaining operations (including power-grid design, clock-tree synthesis, and timing closure^4,23) are outsourced to an unknown third party.^30,35 Results are evaluated with respect to routed wirelength, area, power, and two circuit-timing metrics: TNS and WNS.^d Per Mirhoseini et al., the proxy cost function did not perform circuit-timing analysis²³ needed to evaluate TNS and WNS.^e Therefore, it was misleading to claim in Mirhoseini et al. that the proposed RL method led to TNS and WNS improvements on five TPU design blocks without performing variance-based statistical significance tests (TNS and WNS were optimized at later steps unrelated to RL³⁰).

Use of limited techniques.  To experts, the methodology in Mirhoseini et al.³⁰ looked to have shortcomings: Using outdated methods made it harder to improve the state of the art (SOTA).

H1. Proposed RL used exorbitant CPU/GPU resources compared to SOTA. Hence, the “fast chip design” claim (presumably due to fewer unsuccessful design attempts) required careful substantiation.

H2. Placing macros one by one (a type of constructive floor-planning²³) is one of the simplest approaches. SA can swap and shift macros and make other incremental changes. Analytical methods relocate many components at once. One-by-one placement looked handicapped even when driven by deep RL.

H3. Mirhoseini et al. used circuit-partitioning (clustering) methods similar to partitioning-based methods from 20+ years ago.^3,4,23 Those techniques are known to diverge from interconnect optimization objectives.^3,23 By placing macros using a clustered netlist without gradual layout refinement, RL runs into the same problem.

H4. Mirhoseini et al. limited macro locations to a coarse grid, whereas SOTA methods¹⁰ avoid such a constraint. In Figure 1 (left) macros are placed freely, but a coarse grid used by Google’s RL implementation tends to spread macros apart and disallow large regions for cells, such as in the center of Figure 1 (left). Figure 2 illustrates the difference. Even if RL can run without gridding, it might not scale to large enough circuits without coarse gridding.

H5. The use of force-directed placement from the 1980s¹² in Mirhoseini et al. left much room for improvement.

Questionable baselines.  The Nature paper used several baselines to claim the superiority of proposed techniques. We already mentioned that the human baseline was undocumented and not reproducible.

B1. Key results in Mirhoseini et al. and in Table 1 give chip metrics for five TPU design blocks. But comparisons to SA do not report those chip metrics.

B2. Mirhoseini et al. mentions that RL results were post-processed by SA but lacks ablation studies to evaluate the impact of SA on chip metrics.

B3. RePlAce¹⁰ was used as a baseline in Mirhoseini et al. in a way inconsistent with its intended use. As previously explained, analytical methods do well on circuits with millions of movable components, but RePlAce was not intended for clustered netlists with a reduced number of components: It should be used directly sans clustering (for details, see Bae et al. and Cheng et al.^10,11). Clustering can worsen results due to a mismatch between placement and partitioning objectives,³ and by unnecessarily creating large clusters that are hard to pack without overlaps.

B4. Mirhoseini et al. did not describe how macro locations in SA were initialized, suggesting that the authors used a naive approach that could be improved. Later, Bae et al. identified more handicaps in the SA baseline, and Cheng et al.¹¹ confirmed them.

Undisclosed use of (x, y) locations from commercial tools.  Strong evidence and confirmation by Google engineers are mentioned in the UCSD paper¹¹ that the authors withheld a critical detail. When clustering the input netlist, CT merge in Google code¹³ read in a placement to restructure clusters based on locations. To produce (x, y) locations of macros, the paper’s authors used initial (x, y) locations of all circuit components (including macros) produced by commercial EDA tools from Synopsys.¹³ The lead authors of Mirhoseini et al. confirmed using this step, claiming it was unimportant.¹⁷ But it improved key metrics by 7–10% in Cheng et al.¹¹ So, the results in Mirhoseini et al. needed algorithmic steps that were not included, such as obtaining (x, y) data from commercial software.

More undocumented techniques were itemized in Cheng et al.,¹¹ which mentioned discrepancies between the Nature paper, their source code,¹³ and the actual code used for chip design at Google. These discrepancies included specific weights of terms in the proxy cost function, a different construction of the adjacency matrix from the circuit, and several “blackbox” elements¹³ available as binaries with no source code or full description in Mirhoseini et al. Bae et al., Cheng et al.,¹¹ and the Macro Placement Repo²⁷ offer missing descriptions. Moreover, Mirhoseini et al.’s results did not match the methods used because key components were not mentioned in the paper. And neither results nor methods were reproducible from descriptions alone.

Data leakage between training and test data?  Per Mirhoseini et al., “as we expose the policy network to a greater variety of chip designs, it becomes less prone to overfitting.” But Google Team 1 showed later in Yue et al.⁴⁶ that pre-training on “diverse TPU blocks” did not improve quality of results. Pre-training on “previous netlist versions” improved quality somewhat. Pre-training RL and evaluating it on similar designs could be a serious flaw in methodology of Mirhoseini et al. As Google did not release proprietary TPU designs or per-design statistics, we cannot compare training and test data.

Likely limitations.  Mirhoseini et al. did not disclose major limitations of its methods but promised success in broader combinatorial optimization. The Ariane design image in Mirhoseini et al. shows macro blocks of identical sizes: a potential limitation, given that commercial chip designs often use a variety of macro sizes. Yet, they do not report basic statistics per TPU block: the number of macros and their shapes, design area utilization, and the fraction of area taken by macros. Based on peer reviews³⁵ and the guidance from Google engineers to the authors of Cheng et al.,¹¹ it appears that TPU blocks had lower area utilization than in typical commercial chip designs. Poor performance of Google RL on challenging public benchmarks from Adya and Markov² used in Bae et al. and Cheng et al.¹¹ (illustrated in Figure 2) suggests undisclosed limitations. Another possible limitation is poor handling of preplaced (fixed) macros, common in industry layouts, but not discussed in Mirhoseini et al. By interfering with pre-placed macros, gridding (see H4) can impact usability in practice. Poor performance on public benchmarks may also be due to overfitting to proprietary TPU designs.

A middling simulated annealing baseline.  The “Stronger Baselines paper”⁵ from Google Team 2 improved the parallel SA used by Google Team 1 in Mirhoseini et al. by adding “move” and “shuffle” actions to “swap,” “shift,” and “mirror” actions. This improved SA typically produces better results than RL in a shorter amount of time when optimizing the same objective function. Cheng et al¹¹ reproduced qualitative conclusions of Bae et al. with an independent implementation of SA and found that SA results had less variance than RL results. Additionally, Bae et al. suggested a simple and fast macro-initialization heuristic for SA and equalized compute times when comparing RL to SA. Given that SA was widely used in the 1980s and 1990s, comparing to a weak SA baseline contributed to overestimating the new RL technique.

Pre-training.  Cheng et al.¹¹ performed training using code and instructions in Google’s Circuit CT repo,¹³ which states (June 2023): “The results below are reported for training from scratch, since the pre-trained model cannot be shared at this time.”

• Per the MacroPlacement FAQ in the Macro Placement Repo, Cheng et al.¹¹ did not use pretraining because, per Google’s CT FAQ,¹³ pre-training was not needed to reproduce results of Mirhoseini et al. Also, Google did not release pre-training data.

• Google Team 2⁵ evaluated pre-training using Google-internal code and saw no impact on comparisons to SA or RePlAce.

• Google Team 1 showed⁴⁶ that pre-training on “diverse TPU blocks” did not improve results, only runtime. Pre-training on “previous netlist versions” gave small improvement. No such previous versions were discussed, disclosed or released in the CT documentation¹³ or in the paper itself.³⁰

In other words, the lead authors of the Nature paper want others to use pre-training while they did not describe it in detail sufficient for reproduction, did not release code or data for it, and have shown that it does not improve results in the context of their claims. In September 2024 (years after the publication), the authors announced the release of a pre-trained model but not the pre-training data. Hence, we cannot ensure that a particular example used for testing was not used in pre-training.

Old benchmarks.  Another objection³¹ is that public circuit benchmarks² used in Bae et al.⁵ and Cheng et al.¹¹ allegedly use outdated infrastructure. In fact, those benchmarks² have been evaluated with the HPWL objective, which scales accurately under geometric 2D scaling of chip designs and remains appropriate for all technology nodes (Section 2). ICCAD benchmarks were requested³⁵ by Peer Reviewer #3 of the paper. When Bae et al. and Cheng et al.¹¹ implemented this ask, Google RL ran into trouble before routing became relevant: RL lost by 20% or so in HPWL optimization (HPWL is the simplest yet most important term of the proxy cost optimized by CT/RL^13,30).

Not training until convergence in experiments in Cheng et al.11  This concern was promptly addressed in FAQ #15 in the Macro Placement Repo: “ ‘training until convergence’ is not described in any of the guidelines provided by the CT GitHub repo for reproducing the results in the Nature paper.”²⁷ Cheng et al. followed guidelines by Google in the CT. Later, their additional experiments indicated that “training until convergence worsens some key chip metrics while improving others, highlighting the poor correlation between proxy cost and chip metrics. Overall, training until convergence does not qualitatively change comparisons to results of Simulated Annealing and human macro placements reported in the ISPD 2023 paper.” RL-vs-SA experiments in Bae et al. predated the CT framework, so trained until convergence per six-hour protocol from Mirhoseini et al.

Computational resources used the Nature paper were exorbitant and difficult to replicate. Since both RL and SA algorithms produce valid solutions early and then gradually improve the proxy function, the best-effort comparisons in Cheng et al.¹¹ used smaller computational resources than in Mirhoseini et al, with parity between RL and SA. The result: SA beat RL. Bae et al.⁵ compared RL to SA using the same computational resources as Mirhoseini. Results in Cheng et al.¹¹ were consistent with Bae et al.⁵ If given greater resources, SA and RL are unlikely to further improve chip metrics due to poor correlation to the proxy function from Mirhoseini.

The paper’s lead authors mention in Goldie and Mirhoseini¹⁷ that the paper is heavily cited, but they cite no positive reproductions outside Google that cleared all known obstacles. Bae et al. and Cheng et al. do not discuss other ways to use RL in IC design, so we avoid general conclusions.

Can Google CT/RL code13 be improved?  RL and SA are orders of magnitude slower than SOTA (Table 3), but pre-training (missing in CT) speeds up RL⁴⁶ by only several times. The CT repository now contains attempted improvements, but we have not seen serious improvements to chip metrics. Four major barriers to improving the CT repository and the paper remain:

1. The proxy cost optimized by RL does not reflect circuit timing,¹¹ so improving RL may not help to improve TNS and WNS.

2. SA outperforms RL when optimizing a given proxy function.^5,11 Hence, RL may lose even with a better proxy.

3. RL’s placement of macros on a coarse grid limits their locations (Figure 2). When a human ignored the course grid, they found better macro locations.¹¹ Commercial EDA tools also avoid this limitation and outperform Google CT/RL.

4. Clustering as a preprocessing step creates mismatches between placement and netlist partitioning objectives.^3,23

Conclusions about the Nature paper.  Google Team 2 had access to Google internal code whereas Cheng et al.¹¹ reverse-engineered and/or reimplemented missing components. Google Team 2 and the UCSD Team drew consistent conclusions from similar experiments, and each team made additional observations. We crosscheck the results reported in Google Team 2 and the UCSD Team and also account for the CT framework,¹³ Nature peer reviews,³⁵ and Yue et al.⁴⁶, and then summarize conclusions drawn from these works. This confirms many of the initial doubts about the claims and identifies additional deficiencies. As a result, it is clear that the Nature paper by Mirhoseini et al. is misleading in several ways, such that the readers can have no confidence in its top-line claims and conclusions. Mirhoseini et al. did not improve SOTA while the methods and results of the original paper were not reproducible from the descriptions provided, contrary to stated editorial policies at Nature (see below). The reliance on proprietary TPU designs for evaluation, along with insufficient reporting of experiments, continues to obstruct reproducibility of the methods and the results. Attempts by the authors of the Nature paper to invalidate critiques have been unsuccessful. Surprisingly, the authors of Mirhoseini et al. have not offered new compelling empirical results one-and-a-half years since the publication of Cheng et al.¹¹

Implications for chip design.  Our work highlights deficiencies only in the approach of the Nature paper. But a 2024 effort from China⁴³ compared seven techniques for mixed-size placement using their new independent evaluation framework with 20 circuits (seven with macros). End-to-end results for chip metrics show that ML-based techniques lag behind RePlAce¹⁰ (embedded in OpenROAD) and other optimization-based techniques: DREAMPlace (a GPU-based variant of the RePlAce algorithm) and AutoDMP (a Bayesian Optimization wrapper around DREAMPlace). Despite the obvious need to replicate the methods of Mirhoseini et al., the authors of Wang et al.⁴³ were unable to provide such results.

Policy implications.  Theoretical arguments and empirical evidence suggest that numerous published papers across various fields cannot be replicated and may be incorrect.^34,41 As a case in point, the Nature paper aggravated the reproducibility crisis that is undermining trust in published research.^8,34 Retraction Watch tracks 5,000 retractions per year, including prominent cases of research misconduct.^19,34 “Research misconduct is a serious problem and (probably) getting worse”,⁸ which makes it even more important to separate honest mistakes from deliberate exaggerations and misconduct.^6,7,19,40 Institutional response is needed,^40,41 including clarity in Nature retraction notices.³²

Nature Portfolio editorial policies should be followed broadly and rigorously.  Quoting from Nature Portfolio (https://go.nature.com/4dshcXv):

“An inherent principle of publication is that others should be able to replicate and build upon the authors’ published claims. A condition of publication in a Nature Portfolio journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications[…] After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal.”

Specifically for Mirhoseini et al., the Nature editorial¹⁵ insisted that “the technical expertise must be shared widely.” But when manuscript authors neglect requests for public benchmarking and obstruct reproducibility, their technical claims should be viewed with suspicion (especially if they later disagree with comparisons to their work¹⁷). This point has already been made in a Communications news article.¹⁸ Per peer review file,³⁵ the acceptance of the Nature paper³⁰ was conditional on the release of code and data, but this did not happen when Mirhoseini et al.³⁰ was published or later.¹¹ The Nature paper was amended by the authors to claim that the code had been made available (see the “Data and Code Availability” disclaimer). But serious omissions remain in the released code. This is particularly concerning because the paper omitted key comparisons and details, and fraud was alleged under oath in a California court by a Google whistleblower tasked with evaluating the project.³⁷ This makes reproducibility more critical.

It is in everyone’s interest to reach clear and unequivocal conclusions about published scientific claims, free of misrepresentations. Authors, Nature editors and reviewers, and the research community, share the burden of responsibility. Seeking the truth is a shared obligation.^6,40