Evaluation is a critical component of the STARS Alliance, to assess both the effectiveness of the Alliance and the success of Alliance activities and to motivate other institutions to initiate similar actions. Both formative and summative evaluation measures will be used to inform and refine Alliance activities through the course of the project, and to determine the overall success of the Alliance in reaching its goals and outcomes.

The Alliance offers a unique opportunity to build an outstanding evaluation and assessment team. The STARS Alliance evaluation team includes Stephen Rollins, Juan Gilbert, Sarah Berenson, Nathan Thomas, and Laurie Williams. Stephen Rollins, an educational psychologist and former FSU College of Education Executive Associate Dean, offers a psychology perspective for designing and evaluating all Alliance intervention programs. Juan Gilbert, an Associate Professor of Computer Science and Engineering at Auburn University, provides insight in evaluating the effectiveness of interventions based on his well-designed programs and studies to broaden participation in STEM careers, including a 2004 ITWF grant. Sarah Berenson, Director of the Center for Research in Science and Mathematics Education at North Carolina State University, provides insight and the educational research design perspective, along with experience from ITWF-funded interventions and research projects designed to affect and understand the participation of women in IT. With success in obtaining more than 40 grants, publishing more than 85 research articles, and giving 60 presentations at national and international meetings, she was awarded the 2005 Alexander Quarles Holladay Medal for Excellence, the highest faculty award presented by the University and its Board of Trustees. Nathan Thomas, Director of Diversity at the University of South Florida at Lakeland, has a background in ecology and psychology. He will serve as the content expert for evaluating the SLC based on his ôecological modelö [Tho05] for retention of students in computing programs. Laurie Williams, an Assistant Professor of Computer Science at North Carolina State University, provides expertise and guidance in designing and evaluating the pair-programming interventions for the Alliance. This outstanding team will collaborate to design, refine, guide, and evaluate the effectiveness of the Alliance and its activities.

Daniel Stufflebeam's Context, Input, Process, Product (CIPP) model [Mad83,Stu71] will be the fundamental evaluation design employed by the evaluators. This model is wellrecognized and broadly used by evaluation researchers to assess projects in K-16 educational settings. Using this model, evaluators record and assess the following:

• Context - the larger setting in which the project is being carried out.
• Input - all crucial staff, materials, and resources that are devoted to the project.
• Process - strategies, activities, practices, and procedures used to carry out the project. The process component, which constitutes formative evaluation, is used to solicit information to determine modifications and adjustments needed to improve how a project operates
• Product - the ultimate result obtained that can be attributed to interventions carried out through the project. The product component, which constitutes summative evaluation, is used to determine if a project should be continued, modified, or terminated.

The evaluation will include both quantitative and qualitative components. The quantitative component will examine those factors that may be readily quantified, such as number of students involved; number of faculty and professionals in computing disciplinary areas involved; hours contributed by the students and faculty; and number of students taught by teachers involved in the project. The qualitative component will involve electronic journaling by all SLC students and select interviewing or focus groups of key individuals and groups involved in the STARS programs (e.g., mentors, faculty, students). Qualitative data will be solicited from all these individuals and compared using the process called triangulation. Electronic journal entries will be analyzed using Content Analysis techniques to categorize journal entries. These entries should provide insight into the impact of the SLC program, and should reflect affective, behavioral and cognitive (ABC) change in participants. Affective changes reflect changes in attitude or perception, such as expressing an increased interest in majoring in a computing discipline. Behavioral changes reflect actual change in participant behavior, such as enrolling in a computing degree program. Cognitive changes reflect change in knowledge, such as becoming aware of previously unknown programs or scholarships for computing students [Ajz01,Pet97]. Since we seek to effect change in all areas, survey instruments will be designed to capture change relative to computing in all three of these areas.

For the SLC and Pair Programming programs, and any interventions designed and implemented through the Alliance, all participants will take pre- and post-tests to measure changes in these areas. We view the Alliance as an experimental testbed in the southeast, where ideas for BPC can be incorporated and studied on a multi-institutional scale. Therefore, we design our analysis as an exploratory study to determine the variables that contribute to observed outcomes. We plan to augment this analysis through requesting additional funding for REUs or consultants in statistics to explore and analyze the findings we collect throughout the project.

Our evaluation plan works in the following way: Alliance activities are developed to align with research evidence that shows that such activities will impact the overall goals of the Alliance. Evaluation of each activity is tailored to measure the particular factors that contribute to BPC goals. Overall assessment of Alliance success will be determined through a careful comparison of baseline data, disaggregated by race, ethnicity, gender, and physical ability for each institution and observing gains or losses in each of the specific outcomes identified.