Peer review and R2O

One area of increasing confusion is the difference between the scientific peer review and research to operations processes, and their relative value for a new research byproduct. To some extent, that value depends on the organization or enterprise. But peer review and R2O are two separate processes, and the comparison between them is a matter of “apples and oranges”. They both have important roles but serve different purposes. Scientific peer review exists to make sure that a given research byproduct is valid in the realm of science from which it was derived. While science evolves, the peer review process has a defined starting and ending point, where the end is usually a publication that can be further scrutinized by the broader scientific community. The R2O process, typically internal to an organization, evolves with the progress of a research byproduct and sometimes does not reach a conclusion as it iterates over the lifecycle of a byproduct in operational status.

Allow me to illustrate the difference between peer review and R2O using a common household appliance—the toaster. The research component is the development of the toaster. The transfer to operations is the manufacture and sales of that toaster to a consumer. The analogous peer review process is the engineering analysis of the appliance. For example, there are inevitably certain checks necessary to make sure that the toaster is safe to use, and does not get too hot, or cause a fire, among other things. In comparison, the R2O process for the toaster would assure that the toaster meets the needs of the likely consumers, through focus groups or other evaluations. It could be a perfectly safe toaster, but if the lever is stiff and difficult to operate, or the slots are not large enough to fit a bagel, then that is a failure of the R2O process. A failure in R2O suggests that it is not functional in an operational sense, even if it is engineered to be safe and energy efficient. Of course, if it causes fire, a failure of the “peer review”, then it is neither safe nor useful.

To that end, scientific validity can be proven through peer review, but even if a research byproduct makes it to operations before peer review is complete, the practitioner will most likely notice glaring scientific shortcomings if they exist. If the toaster catches fire, there will be few if any additional consumers. The same applies to R2O. If it is not scientifically valid, the practitioners will realize that and not use it any further, because it does not add any value. Similarly, it is possible that peer review may not lead to a sufficient amount of initial criticism. That may be caught over time as a result of a continual evaluation within the R2O process, if the process incorporates heavy evaluation from users.

For this reason, the relationship between peer review and R2O is a tenuous one. That is because not every component of a research byproduct is necessarily subject to peer review, or even can be reasonably peer reviewed. Sometimes peer review constraints are greater than demands from the R2O process or practitioners. Or, certain components are quite important to operations that are not evaluated during peer review at all. Science is only a part of the research byproduct transitioned to operations; how it is displayed and the manner in which it is applied to operational problems are unique challenges that peer review will generally not resolve, but R2O processes can carefully examine.

While R2O processes can be developed to be onerous, with numerous checks for compliance with information technology standards, budgets, and so forth, that is ultimately unnecessary. This can slow R2O, particularly when compounded by bureaucracy. Adherence to elements of a specific R2O process unrelated to the development of the byproduct and practitioner experience are not essential to prove operational worthiness or scientific validity. There is not a perfect recipe for which R2O demonstrations over a period of time are necessary to establish a certain degree of rigor, just as peer review is not a failsafe way to assure accurate and reproducible scientific results.

Both peer review and R2O have their place and importance in whatever form they come. One does not need to occur before the other, and sometimes one is more beneficial than the other. When both processes are invoked, it can produce the most desirable and defendable research and resultant byproducts that are useful for practitioners, but organizations and enterprises must consider which elements of these processes are necessary in achieving their overall mission and objectives.

Jordan Gerth

Jordan Gerth

is a research meteorologist with a decade of R2O experience, interacting with academia, the federal government, and the private sector on weather satellite and software projects.
Jordan Gerth

Latest posts by Jordan Gerth (see all)