Questions and comments concerning these pages or
the Rapid Prototyping Center can be addressed to the
Go to SEC index page
The three basic aspects, or stages, of the RPC are illustrated in Figure 1: Stage 1, the initial evaluation to determine the
suitability of a model or data stream to enter the RPC; Stage 2, the evaluation and modification in an operational
environment; and Stage 3, the final graduation or transition to operational use. The requirements to progress
through each of these stages are outlined below. Although each product transition will be unique, the
anticipated maximum time within the RPC is six months. An evaluation of each product under development
will be performed monthly to assess the likelihood that the product will become operational and to
identify the main issues to be resolved.
Beginning at Stage 1, the external models and new data streams are evaluated by the SEC staff, with representation
from the Systems, Operations, and Research Divisions. Upon acceptance into the RPC at Stage 2, operational evaluation occurs
with specific requirements at each step of the process. During this stage
test products are generated and made available on the Web and in the forecast center. In this way, feedback is
obtained from the SEC forecasters and from outside users. Prior to graduation in Stage 3, the documentation and user
manuals are completed and the operational products are finalized.
Procedures for Models Under Initial Evaluation
The basic acceptance requirements are the following:
- Model/data must be relevant to the operational mission of NOAA Space Weather Operations.
Modelers/data providers must demonstrate that the proposed product will provide one or both of the
- Information currently unavailable that is likely to improve services
- Substantive improvement on information currently being used operationally as judged, for
example, by skill-score comparison
- Models must be compatible with the hardware, software, and data resources available at SEC.
- SEC will provide the following information to help a proposer infer the appropriateness of
his/her model or data stream for RPC matriculation:
- Hardware, software, and visualization resources available through SEC
- Data and model resources available for model use
- Modelers will provide the following information:
- Data and/or other input needed to run the model
- Required CPU usage, RAM usage, and disk storage
- Visualization tools used
- Required network connectivity
- A straw-man Concept of Operations to include information such as
frequency of anticipated operational use (hourly, daily, weekly, etc.)
- Additional special considerations
- Verification testing that has been performed and the results
- Model must run reliably using test data provided by SEC. SEC will provide archived, time-series data
to allow the modeler to develop and test the interface to the real-time data. The modeler will then
provide SEC with the model output, including a description of any incompatibilities between the model and
the data. The data provided by SEC could be available on the Web, and may include simulated data with a
minimum amount of gaps, erroneous values, etc.
- Model/data products must be valid. The model/data must operate without unanticipated interruptions,
and "bad" values must be flagged. Where possible, model output or data will be compared by the researcher
and by the RPC staff with other measurements or model output to indicate that the results are valid. This
will be a consideration during the pre-entrance evaluation and throughout the transition process.
- Model/data must be documented.
- Model input requirements: data type, time resolution, software and hardware requirements
- Model output format or data format
- Description of output/data: units, resolution, known limitations, all the raw variables that
could be used
- Range of validity: What does it do well, where does it break down? Is there a range where you
do not expect it to be valid, where hasn't it been tested?
- It is desirable that the model be documented internally (not just the part with which SEC will
- Larger models should be modular to support model modification as well as to allow output from
intermediate stages to be extracted for operational products.
- A brief development plan must be provided. SEC will contribute to this plan to best direct the model
development toward operational use.
The modeler will provide a plan for the model development within the RPC to meet the final operational
requirements. This will include a description of the model/data comparisons that will be made, the
possible products that will be available to the forecasters, and the suggested steps that will be taken to
improve the reliability and the accuracy while in the RPC. This plan should also include an estimate of
the staff support required to implement the model and the time required to adequately train the staff in
the maintenance of the model and to properly interpret the model results.
Procedures for Models and Data That Have Entered the RPC
Models/data submitted to RPC proceed through a sequence of decision points along the path to operational acceptance.
The four main decision points are outlined below. However, some of these points may not apply in all
cases. For example, models dealing with predictions of sunspot maximum cannot be expected to undergo
verification in real time, since the time scales involved are so long that only testing against historical
data is practical. Data streams taken into the RPC may be evaluated using similar data sets if such exist
or using historical and real-time data to assess the consistency and accuracy.
Each entrant into the RPC will follow a customized plan through the four basic decision points. That is, a
statement of criteria will be developed for each step. Success in graduating from the RPC consists of
passing through all pre-defined decision points. Failure may occur at any step, resulting in either
recycling to a previous step or elimination from further consideration. At each step, an estimate will be
made of the resources necessary to continue. If, for example, a major modification becomes necessary
during one of the steps that would require substantial future costs, it may be decided to reject the
model altogether at that point, rather than to reinsert it at the beginning of the current step. Some
flexibility in adjusting the decision criteria may be appropriate during the execution of the adopted development plan,
but final acceptance of any such proposed changes will rest with the RPC staff.
The following main decision points are identified within the RPC:
- Bring model up on local or remote RPC system:
- Self-contained models to run on SEC workstations and peripherals alone:
- Port code to local SEC systems (model/data provider responsibility
with SEC support).
- Preliminary benchmarking, assessment of resources consumed
- Bring up visualization in at least a diagnostic mode.
- Models run remotely that need an interface and post-processing at SEC:
- Set up links with remote system to SEC system and resolve interface issues including
data ingest, simulation output, run-time diagnostics.
- Bring up post-processing tools and visualization software on SEC system.
- Arrange for long-term access.
- Verification with historical data (no external dissemination of experimental products):
- Begin with best cases, closest to model concept; proceed on to more general difficult cases.
- Collect statistics on performance.
- Establish measures of accuracy.
- Establish epochs or broad classes of applications (e.g., stratify performance according to
phase of solar cycle).
- Compile statistics on resources consumed.
- Verification with real-time data (internal use of experimental products, but also begin external
dissemination, e.g. through a New Products Web site, to show progress and to encourage outside comment):
- Establish links to real-time data for model input and develop strategy for filling in missing
data, correcting errors, etc.
- Develop and implement preliminary operational visualization tools.
- Issue preliminary assessment of "value" in terms of internal consistency/accuracy, operator
assessments of day-to-day utility, likelihood that model/data will see further application/development.
- Bringing system fully online:
- Finalize Concept of Operations for full implementation.
- Produce documentation, define training for operational use.
- Finalize full set of production visualization tools and diagnostics.
- Document verification, statistical performance, etc, within scope of predetermined criteria.
- Finalize outside links (WWW).
- Document record of operator questions, difficulties, etc.
- Produce statement of scope, known limitations (e.g. verification truth table).
- Produce clear statement of ownership of final RPC product, what can/cannot be freely accessed
and modified, and what part remains the property of the applicant.
Disposition of Models and Data After Transition to Operational Use
SEC will have full rights to run and distribute output from all models that graduate from the RPC. In
- Access to the operational version of model code will be through SEC staff only.
- Routine access to the model output will be through the standard SEC products.
- All upgrades to the operational version of the model will be implemented through the RPC.
- The modeler will have no further obligations to SEC.
In addition, model verification such as skill-score will be made publicly available, as with any operational model at SEC.