Government Funding Not Conducive to Serendipity
(p. 301) Even in the early twentieth century, the climate was more conducive to serendipitous discovery. In the United States, for example, scientific research was funded by private foundations, notably the Rockefeller Institute for Medical Research in New York (established 1901) and the Rockefeller Foundation (1913). The Rockefeller Institute modeled itself on prestigious European organizations such as the Pasteur Institute in France and the Koch Institute in Germany, recruiting the world's best scientists and providing them with comfortable stipends, well-equipped laboratories, and freedom from teaching obligations and university politics, so that they could devote their energies to research. The Rockefeller Foundation, which was the most expansive supporter of basic research, especially in biology, between the two world wars, relied on successful programs to seek promising scientists to identify and accelerate burgeoning fields of interest. In Britain, too, the Medical Research Council believed in "picking the man, not the project," and nurturing successful results with progressive grants.
After World War II, everything about scientific research changed. The U.S. government--which previously had had little to do with funding research except for some agricultural projects--took on a major role. The National Institutes of Health (NIH) grew out of feeble beginnings in 1930 but became foremost among the granting agencies in the early 1940s at around the time they moved to Bethesda, Maryland. The government then established the National Science Foundation (NSF) in 1950 to promote progress in science and engineering. Research in the United States became centralized and therefore suffused with bureaucracy. The lone scientist working independently was now a rarity. Research came to be characterized by large teams drawing upon multiple scientific disciplines and using highly technical methods in an environment that promoted the not-very-creative phenomenon known as "groupthink." Under this new regime, the competition (p. 302) among researchers for grant approvals fostered a kind of conformity with existing dogma. As the bureaucracy of granting agencies expanded, planning and justification became the order of the day, thwarting the climate in which imaginative thought and creative ideas flourish.
Meyers, Morton A. Happy Accidents: Serendipity in Modern Medical Breakthroughs. New York: Arcade Publishing, 2007.