Testbed Misconceptions

Testbeds are an emerging and essential forum in the weather community for bringing researchers and users together with a goal of determining the operational viability and maturity of research products. In practice, testbeds can take many shapes and their organization can vary. There are, however, misconceptions about testbeds that can negatively impact their utility within the context of R2O.

Misconception: Researchers should not have a role in the testbed.

While interactions between researchers and practitioners in the testbed should be limited, researchers must partake in testbed activities (1) so that they can directly observe how the research products are applied to operational challenges, (2) ask relevant questions to help improve understanding about their operational relevance, and (3) ensure that the practitioners understand the research product and are applying the product as designed. Care must be taken, however, to ensure that this interaction is not contravening the goal of the testbed. For example, researchers should not try to sell the research product at this stage or attempt to monopolize the testbed. Liaisons have an important purpose during the testbed in assuring researchers and practitioners maintain their roles.

Misconception: Operational participants of testbeds are test subjects.

The research products are evaluated in a testbed, not the practitioners. In a testbed where research products are evaluated, the practitioners act as peer reviewers as part of a scientific process. The formal feedback collected during testbeds should be scientifically based and defensible. Generally those testbed vehicles that hide the identity of participants do a disservice to the institution of the testbed because they extract the collaborative environment between research and operations that R2O seeks to confer. Namely, anonymity prevents researchers from asking for subsequent clarification. Similarly, surveys that inquire about the absolute usefulness of certain products demonstrated in a testbed, especially with regard to a subjective scale, are not themselves useful to achieving the testbed goal of determining the relative value of a new research product.

Misconception: Testbeds are an appropriate place for rapid prototyping.

In order to ensure the integrity of the testbed and evaluation period, no further development of research product algorithms should occur during the testbed, unless it is of sufficient length. Sustained evaluation periods are required in order to assure feedback is representative of operational use. Researchers should not be on site to develop and implement updates to research products based on cursory feedback.

All testbeds are different. Both research and operational participants in the testbed should establish and agree upon testbed expectations, roles, and responsibilities before it begins.