Accrediting models for the TMD COEA: A case study in face assessment

This paper describes a face validation process developed to assess computer models used in Phase I of the Theater Missile Defense (TMD) Cost and Operational Effectiveness Analysis (COEA) and to aid the COEA Study Director in making accreditation decisions for those models. The process was applied to five BMDO models, only two of which had a formal configuration management program. The process was expedient, efficient and proved to be an effective method of distilling essential information for the decision maker. Further, the process identified model deficiencies which provided a road-map for modeling improvements needed for follow-on COEA studies. The paper describes each phase of the process: preparation (including validation team membership, analyst interviews to develop needs, preparation of developer questionnaires and solicitation of preliminary responses, and the selection of assessment criteria); conduct of the panel/development team interaction; report preparation; and review cycles. The paper concludes with a list of limitations for face validation and lessons learned from the authors experience, including what did and did not work well during the model assessments. This same process has since been used by BMDO to collect information needed for planning and decision making, and to orient the continued development of TMD models consistent with user priorities.