Detailed program reviews are also carried out by individual agencies via their management review structures, including official advisory committees. This section highlights recent and current oversight and review activities and describes existing review mechanisms within each of the agencies.
The Defense Advanced Research Projects Agency (DARPA) reviews its HPCC efforts at many different levels to ensure consistent program evaluation in a dynamic R&D environment. Projects produce reports on a regular basis that are reviewed by program managers. An annual process of updating accomplishments, milestones, and project plans is tied to the incremental funding process. Site visits, project meetings, principal investigator meetings, and regular interactions using the Internet as an information infrastructure enable DARPA staff to fulfill their program management responsibilities. In addition, contracting details are handled by contracting agents that work with DARPA as part of the program management process. Office directors and program managers develop plans and milestones that are approved by senior management during the planning and budget cycle. New programs and ideas are proposed during this process. In conjunction with yearly funding decisions by DARPA and DoD, senior DARPA technical management critically review program areas, plans, and accomplishments. Guidance is provided to reflect programmatic, technical, and funding directions. At the DoD level, programs are described through a formal process that requires agency, DARPA Comptroller, DoD Comptroller, senior DARPA management, and senior DoD approvals. Once approved, these descriptions become part of the Defense budget submitted to Congress for approval. In addition to other internal Federal HPCC reviews, there are Congressional briefings, HPCC agency crosscuts, technical working groups, DoD advisory panels, and several National Academy of Science studies that contribute to the planning process.
The National Science Foundation (NSF) HPCC program long-term goals and objectives consist of numerous processes and mechanisms that contribute to the definition and review of the supported projects. In addition, more specific goals and objectives are established for each of the activities within the program. The National Science Board, panels and committees commissioned to study and recommend program activities, and external advisory committees contribute primarily to the more general, long-term goals. The establishment of goals, objectives, implementation mechanisms, evaluative measures, and the broader value of specific activities is accomplished through such means as external peer review (mail reviews, panel reviews, and site visits), workshops for developing research agendas, program committees of visitors, technical oversight teams, ongoing site visits by program staff and outside experts, program officer review of final project reports, and the bodies of opinion held by the community of researchers themselves. Evaluation of progress is integrated into the management process as appropriate to each class of HPCC activities. Examples are:
 |
Federal Networking Council Advisory Committee for the Internet
|
|
Directorate Advisory Committees
|
|
Committee of Visitors for each research program |
The Department of Energy (DOE) HPCC program focuses on basic mathematics and computational research and on developing and delivering technology for use by other scientists and engineers in DOE and associated U.S. academia and industry. Performance evaluation is an integral part of these programs. Because of this focus, external review by prospective users of the technology is a critical component of measuring performance. In previous years this has been most explicitly present in the use of committees of users such as the ESNet (Energy Sciences Network) Steering Committee and the ER (Energy Research) Supercomputer Users Group to evaluate the effectiveness of the access and networking programs. Many of the education programs established under the DOE HPCC program have evaluation built in. In software technology, employing prospective users of technologies as reviewers has ensured that the technology developed is that required by users. DOE further formalized these procedures to include program-wide reviews of the basic technology components of the program by significant prospective users of those technologies. These include reviews of effectiveness in all categories as well as specific numerical targets.
The National Aeronautics and Space Administration (NASA) HPCC program evaluations take place at several levels. At the agency level, the NASA Advisory Council (NAC) has established the Ad Hoc Task Force on Supercomputing, which completed a review and report on the NASA HPCC program. It is expected that other such bodies will be permanently chartered under the Aeronautics Advisory Committee to advise the NASA HPCC program. Within the program, annual comprehensive reviews are conducted for each of the projects. In addition to appropriate NASA personnel, representatives from other Federal agencies, academia, and industry may be invited to participate. Annual reviews of program progress and plans will also be conducted by the NASA HPCC Working Group. In addition, quarterly reviews of the NASA HPCC program are conducted by program managers, the associate administrator of aeronautics, and the director of the NASA HPCC Office.
The National Institutes of Health (NIH) HPCC program goals are enhancements of existing NIH program missions to support biomedical science and expand biomedical knowledge. Program objectives are developed by Institute Directors, advisory bodies, and senior program staff, and are peer reviewed for determination of merit. Each of the participating NIH components has one or more standing external advisory committees that review new and existing programs. These include the National Library of Medicine (NLM) Board of Regents, NLM Board of Scientific Counselors, National center for Research Resources (NCRR) Advisory Council, NCRR Biomedical Research Technology Review Committee, the National Cancer Advisory Board, the National Cancer Institute Division of Cancer Biology and Diagnosis Board of Scientific Counselors, and the Division of Computer Research and Technology Advisory Council. The final decision regarding individual HPCC programs within each of the participating Institutes rests with the Director of that Institute. Within each of the participating NIH Institutes, mechanisms exist to ensure objective evaluation of progress and results and identification of possible new activities.
The National Security Agency (NSA) reviews its HPCC support efforts on a yearly basis in several separate reviews. A steering group composed of senior managers from the technical components provides high level guidance prior to the formal budget process. The steering group receives individual project assessments from the project managers and determines whether any major shifts or changes are needed to the NSA program. Senior management has the flexibility to sponsor HPCC efforts in several budget reviews. Individual projects are proposed and budgeted within the technical components, and are constantly evaluated by the project managers. Monthly status reports are evaluated and meetings held with the project staff to ensure that the correct focus is maintained. During the budget review cycle, the projects are evaluated and terminated or retained based on their performance, importance relative to other initiatives, and priority based on the steering group guidance. New projects can be proposed by the technical components each year during the NSA Technology Program review process.
The National Institute of Standards and Technology (NIST) has the National Academy of Science review annually all the programs and activities of each operating unit, as part of its normal operation. Assessment includes relevance, performance measures, and achievements. Consequently, programs and activities that are a part of the Federal HPCC Program are subject to review and comments of an external panel of experts from academia and industry. A written report is presented to the Director of NIST, the Administration, and Congress. Selection of individual projects and subsequent progress reviews are conducted by the program manager. These are reviewed by an inter-operating unit panel composed of senior managers for relevance to agency mission and the Program, and for continued acceptable performance.
The National Oceanic and Atmospheric Administration (NOAA) HPCC Program supports and enhances NOAA programs in environmental prediction and stewardship. Comprehensive NOAA science reviews of these programs are held periodically. Quarterly reviews of NOAA HPCC program progress are conducted as an integral part of NOAA-wide quarterly reviews by the NOAA Administrator and other NOAA senior line and program managers. Overall NOAA HPCC program goals and plans are reviewed annually as part of the NOAA strategic planning process.
The Environmental Protection Agency (EPA) HPCC program is focused on incorporating advances in computing and communications technology into critical environmental assessment applications and transferring those advanced tools to key state, Federal, and industrial users. EPA senior management officials review the EPA HPCC program annually to assess its relevance to the agency mission and program achievements. Agency guidelines require an external peer review of the EPA HPCC program every two years. The external review panel is composed of representatives from other Federal agencies, academia, and industry. Within the program, each major project is reviewed at least twice a year to evaluate progress toward the program objectives.
The Department of Education (ED) technology initiatives are focused on expanding the use of advanced computer and telecommunication technologies to help students achieve challenging academic standards. ED has awarded 5-year grants to 43 consortia of local school systems, colleges, universities, and private businesses to develop creative ways to use technology for learning. These and other R&D efforts also focus on the needs of handicapped learners, migrant students, and other populations with special needs, as well as to strengthen the links between home and school, teachers and their colleagues, and students over wide geographic areas and among heretofore diverse communities of learners.
The Agency for Health Care Policy Research (AHCPR) HPCC program focuses on health care technology applications of computer based patient records, computerized clinical decision support systems, and patient care data standards. These HPCC activities are predominantly technology applications support for promoting the development and evaluation of systems to foster their economic and medical feasibility.
The Department of Veterans Affairs (VA) HPCC program focuses on data capture, information systems, local area and wide area networks connecting a significant number of its member facilities, which includes 171 VA medical centers, 362 outpatient clinics, 129 nursing homes, and 35 domiciliaries.
 |
 |