June 13, 2011
This article is a slightly updated version of a note I posted several years ago on my personal blog, bertrandmeyer.com. At the recent Microsoft Software Summit in Paris I gave a short talk based on that note, and so many people told me they enjoyed it that I thought it would be appropriate to share the ideas with the readers of the CACM blog. Please note that the most enthusiastic text extracts from funding agencies appearing below are meant to be read aloud, with the proper accents of passion.
The world of research funding has of late been prey to a new craze: paradigm-shift mania. We will fund only 10 curly-haired, cranky-sounding visionaries in the hope that one of them will invent relativity. The rest of you"Bit players! Petty functionaries! Slaves toiling at incremental research!"should be ashamed of even asking.
Take this from the U.S. National Science Foundation's description of funding for Computer Systems Research (CSR)1: "CSR-funded projects will enable significant progress on challenging high-impact problems, as opposed to incremental progress on familiar problems."
The European Research Council (ERC) is not to be left behind2: "Research proposed for funding to the ERC should aim high, both with regard to the ambition of the envisaged scientific achievements as well as to the creativity and originality of proposed approaches, including unconventional methodologies and investigations at the interface between established disciplines. Proposals should rise to pioneering and far-reaching challenges at the frontiers of the field(s) addressed, and involve new, groundbreaking or unconventional methodologies, whose risky outlook is justified by the possibility of a major breakthrough with an impact beyond a specific research domain/discipline."
Frontiers! Breakthrough! Aim high! Creativity! Risk! Impact! Pass me the adjective bottle. Groundbreaking! Unconventional! Highly ambitious! Major! Far-reaching! Pioneering! Galileo and Pasteur only, pleaseothers need not apply.
As everyone knows, including the people who write such calls, this is balderdash. First, 99.97% of all research (precise statistic derived from my own groundbreaking research, funding for its continuation would be welcome) is incremental. Second, when a "breakthrough" does happenthe remaining 0.03%it was often not planned as a breakthrough.
Incremental research is a most glorious (I have my own supply of adjectives) mode of doing science. Beginning Ph.D. students can be forgiven for believing that research is done by a lone genius who penetrates the secrets of time and space by thinking aloud during long walks with his Italian best friend3; we all, at some stage, shared that delightful delusion. But every researcher, presumably including those who go on to lead research agencies, quickly grows up and learns that it is not how things happen. You read someone else's solution to a problem, and you improve on it. Any history of science will tell you that for every teenager who, after getting hit by a falling apple, intuits the structure of the universe, there are hundreds of excellent scientists who look at the state of the art and decide they can do a trifle better.
Here is a still recent example, particularly telling because we have the account from the scientist himself. It would not be much of an exaggeration to characterize the entire field of program proving over the past four decades as a long series of variations on Tony Hoare's 1969 axiomatic semantics paper.4 Here is Hoare's recollection, from his Turing Award lecture5: "In October 1968, as I unpacked my papers in my new home in Belfast, I came across an obscure preprint of an article by Bob Floyd entitled 'Assigning Meanings to Programs.' What a stroke of luck! At last I could see a way to achieve my hopes for my research. Thus I wrote my first paper on the axiomatic approach to computer programming, published in the Communications of the ACM in October 1969."
Had the research been submitted for funding, we can imagine the reaction: "Dear Sir, as you yourself admit, Floyd has had the basic idea6 and you are just trying to present the result better. This is incremental research; we are in the paradigm-shift business."
And yet if Floyd had the core concepts right, it is Hoare's paper that reworked and extended them into a form that makes practical semantic specifications and proofs possible. Incremental research at its best.
The people in charge of research programs at the NSF and ERC are scientists themselves and know all this. How come they publish such absurd pronouncements? There are two reasons. One is the typial academic's fascination with industry and its models. Having heard that venture capitalists routinely fund 10 projects and expect one to succeed, they want to transpose that model to science funding; hence the emphasis on "risk." But the transposition is doubtful because venture capitalists assess their wards all the time and, as soon as they decide a venture is not going to break out, they cut the funding overnight, often causing the company to go under. This does not happen in the world of science: Most projects, and certainly any project that is supposed to break new ground, gets funded for a minimum of three to five years. If the project peters out, the purse holder will only find out after spending all the money.
The second reason is a sincere desire to avoid mediocrity. Here we can sympathize with the funding executives; they have seen too many "Here is my epsilon addition to the latest buzzword" proposals. The last time I was at ECOOP, in 2005, it seemed every paper was about bringing some little twist to aspect-oriented programming. This kind of research benefits no one and it is understandable the research funders want people to innovate. But telling submitters that every project has to be epochal will not achieve this result.
"Telling submitters that every project has to be epochal will not achieve this result."
It achieves something else, good neither for research nor for research funding: promise inflation. Being told that they have to be Darwin or nothing, researchers learn the game and promise the moon; they also get the part about "risk" and emphasize how uncertain the whole thing is and how high the likelihood it will fail.
By itself this is mostly entertainment, as no one believes the hyped promises. The real harm, however, is to honest scientists who work in the normal way, proposing to bring an important contribution to the solution of an important problem. They risk being dismissed as small-timers with no vision.
Some funding agencies have kept their heads cool. How refreshing, after the above quotes, to read the general description of funding by the Swiss National Science Foundation7: "The central criteria for evaluation are the scientific quality, originality, and project methodology as well as qualifications and track record of the applicants. Grants are awarded on a competitive basis."
In a few words, it says all there is to say. Quality, originality, methodology, and track record. Will the research be "groundbreaking" or "incremental"? We'll find out when it's done.
I am convinced the other agencies will come to their senses and stop the paradigm-shift nonsense. One reason for hope is in the very excesses of the currently fashionable style. The preceeding short text from the ERC includes, by my count, 19 ways of saying proposals must be daring. Now we do not need to be experts in structural text analysis to know that someone who finds it necessary to state the same idea 19 times in a single paragraph feels rather insecure about it. At some point the people in charge will realize that such hype does not breed breakthroughs; it breeds more hype.
In the meantime, what should we do? Most of us need funding. Also, there is nothing wrong with a little hype. After all, if you are the shy and unassuming type, not convinced that you are smarter than everyone else, you do not become a researcher. Being too demure will hurt you. (I still remember a project proposal, many years ago, which came back with glowing reviews: The topic was important, the ideas right, the team competent. The agency officer's verdict: Reject. The proposers are certain to succeed, so it's not research.) There is, however, a line not to be crossed. Highlighting and extrapolating the benefits of your proposed research is okay; making absurd representations is not.
So, one cheer for incremental research.
Wait, isn't the phrase supposed to be "two cheers"8?
All right, but let's go at it incrementally. One and one-tenth cheer for incremental research.
1. National Science Foundation, Division of Computer and network Systems, Computer Systems Research, http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=13385.
2. European Research Council, Advanced Investigators Grant, http://erc.europa.eu/index.cfmfuseaction=page.display&topicID=66.
7. Swiss National Science Foundation, http://www.snf.ch/E/funding/projects/Pages/default.aspx. (Disclosure: the SNSF has kindly funded several of my research projects over the past years.)
©2012 ACM 0001-0782/12/0900 $10.00
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from email@example.com or fax (212) 869-0481.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.
Bertrand Meyer is right and wrong.
He rightly states that ground-breaking research needs incremental research. Indeed, many first results are hard-to-digest. Incremental research brings these results to society. Stronger, much of the contribution to society by researchers is communication, often as a side-effect of incremental research.
He is right stating that this funding of ground-breaking research is unlikely to achieve its goal. Indeed, many ground-breaking achievements even surprised their inventors. How can external experts judge which research proposal to fund? What are effective evaluation criteria? How fast will these evaluation criteria become obsolete when researchers adapt (analogous to insects with black and yellow stripes without being able to sting)? To what extent will insider knowledge on these criteria and their relative importance falsify the competition? To what extent does even subtle lobbying become decisive in the funding selection?
He is wrong when suggesting that "quality, originality, methodology, and track record" are sufficient grounds to decide on funding research, incremental or otherwise. For society, supporting a scientific community only makes sense because of ground-breaking results. As an analogy, funding a community of Mensa-members inventing evermore complicated manners to perform calculus in roman numbers is a poor investment for our society. In contrast, the invention of Arab numbers makes funding research a very profitable business for society. Here, a key issue is that incremental research communities are failing to recognize ground-breaking results in an embryonic stage and, when presented with ground-breaking research, often are reluctant to switch.
The solution unfortunately goes against the nature of current funding practices. To facilitate ground-breaking research, it suffices to give researchers their academic freedom. Indeed, 99.99% of researchers want to make a difference to society and contribute something extraordinary. There is no need to (micro)manage something that we get for free. It is not necessary to allocate premium funding for researchers on a trail to ground-breaking results. They know that they are on a trail that makes sense and that has extraordinary potential. The only special requirements for such research is access to resources necessary to go where none has gone before. For the most, this involves the ability to go where no evaluation board would allow it. In a fictitious example, experts in the stone age would not approve experiments attempting to insert colored stones in ceramic pots (because thermal stress will make this fail). However, some of those stones are copper ore. Current funding practices prevent transitions of this nature (from the stone age into the bronze age) and thus close the main path along which research is able to contribute where it counts.
Moreover, by emphasizing accountability over academic freedom, funding agencies put constant pressure on researchers. It is precisely such pressures that deny incremental researchers from welcoming ground-breaking but immature and poorly formulated research results. They are denied the time and resources to re-orient themselves. They are not rewarded because accountability is based on whatever happens to be measurable, which are very rough activity level indicators (paper counting, citations, impact factors). The accuracy of current practices is analogous to measuring the climate quality in buildings by looking at the electricity bills (note: accurate measurement of the electricity consumption does not solve the problem). There are no rewards for early detection of ground-breaking results. To the contrary, incremental researchers are induced to smother ground-breaking research to ensure their own success and even survival as a researcher. The unavoidable immaturity of ground-breaking research results makes them easy victims.
The core issue of the above is that allocating the available funding to the applications for funding (asking for much more than available) needs innovation. The base against which novel approaches have to compete shall not be current practice. The base solution is a structured and weighted lottery in which random selection is applied as soon as no information is available to steer the choices. Structure ensures that funding is spread across domains, short/medium/long term research... Weights account for track records... The prices ensure that talent is attracted (e.g. tenure). Once the zero-sum-game is resolved, the policies, evaluations, reviews... will be able to facilitate rather than deny researchers to do what they deem to be most beneficial to society. Indeed, the latter is something we get for free and does not require management.
Displaying 1 comment