Performance, Metrics & Accountability
Metrics, Dollars, and Systems Change: Learning From Washington State's Student Achievement Initiative to Design Effective Postsecondary Performance Funding Policies
This brief, jointly produced by IHELP and the Community College Research Center (CCRC) at Columbia University, examines the development and implementation of the Washington State Student Achievement Initiative (SAI), a policy adopted by the Washington State Board for Community and Technical Colleges that draws on intermediate measures of student progress to reward colleges for improvements in student achievement. This brief, based on findings from a three-year evaluation, offers lessons for leaders in states looking to design performance funding policies for community colleges. The brief offers recommendations for several design principles for an effective performance funding system including designing a useful and relevant system of performance metrics, linking performance to budgets (baseline funding rather piecemeal funding from allocated funds), and supporting college environmental conditions through more strategic communication and emphasis on institutional research to produce systemic institutional change.
Washington State Student Achievement Initiative Policy Study: Final Report
This report, jointly produced by IHELP and the Community College Research Center (CCRC) at Columbia University, analyzes the impact of the Washington State Student Achievement Initiative (SAI) on college efforts to improve student outcomes and on student outcomes. SAI, a policy adopted by the Washington State Board for Community and Technical Colleges, draws on intermediate measures of student progress to reward colleges for improvements in student achievement. Funded by the Bill and Melinda Gates Foundation, this three-year evaluation includes both data analysis and extensive interviewing of faculty and staff and is intended in part to draw lessons for leaders in other states who are considering adopting performance incentive policies for community colleges. Report findings include that SAI is viewed by the colleges as one force among others pushing them to improve student success, and its intermediate milestone framework as helpful in focusing collective efforts on student progression and in publicly accounting for college performance. The funding mechanism has proved unpopular, however, as SAI funding has come from reallocated base funds rather than as additional funds as originally intended. While larger colleges earn more awards than smaller colleges, there is little evidence that colleges serving more at-risk, low-income students are penalized by the SAI awards method. Consistent with the SAI’s goals, the basic skills metric appears to have encouraged enrollment from traditionally underserved groups.
On Balance: Lessons in Effective Coordination from the Washington State Board for Community and Technical Colleges – An Organizational Perspective
This report examines the importance of effective coordination of postsecondary education in boosting educational attainment and economic competitiveness. This report is a case study of the Washington State Board for Community and Technical Colleges (SBCTC), and analyzes the key strategies and conditions that have led to the effectiveness of the Board as a coordinating agency over locally governed colleges. The study, sponsored by the Bill and Melinda Gates Foundation, found that three sets of factors -- state political and economic context, institutional design, and organization and leadership strategies -- explain the success of the Board. The report includes a self-assessment instrument intended for use by other states that seek to improve the effectiveness of their own postsecondary education coordination to better serve students and meet state needs. Click here for the self-assessment instrument.
Measuring Institutional Conditions that Support Student Success in the California Community Colleges
This report, prepared by IHELP for the University of California All Campus Consortium on Research for Diversity (UC/ACCORD), looks at the opportunities and challenges in measuring institutional conditions that support student success. The report, part of the Pathways to Postsecondary Success project funded by the Bill & Melinda Gates Foundation, draws from the literature and ongoing research to develop a set of indicators and metrics to assess the institutional conditions related to student success in the California community colleges. The report describes the significant challenges in identifying, defining and measuring indicators of supportive institutional conditions in the community colleges, but offers a list of possible indicators and existing sources of data that could be used as a “starting point” in defining a set that could fairly and accurately capture the conditions at a particular institution.
Dollars and Sense: Analysis of Spending and Revenue Patterns to Inform Fiscal Planning for California Higher Education
This report uses data from the Delta Project on Postsecondary Education Costs, Productivity, and Accountability to analyze revenue and spending patterns across California higher education. The report compares patterns across California’s three public higher education systems (University of California, California State University and the California Community Colleges), documents changes over a seven-year period for which data are available (2002-2009), and compares California to the rest of the nation. The report documents several unique characteristics of California higher education including the largest disparity in the nation in educational expenditures per student between the research university sector and the community college sector and a very low share (one-tenth) of educational costs covered through community college tuition compared to the national average of one-third. Documented trends within California higher education include steep declines in state subsidies for UC and CSU and large tuition increases, such that students at those two segments are paying a much larger share of their educational costs, and incremental gains in degree productivity at all three segments that fall well short of state needs. Researchers conclude that policymakers’ traditional approach to fiscal planning is inadequate for today’s challenges and recommend greater transparency in spending and revenues and a more strategic and state-wide approach to financing higher education.
Sense of Direction: The Importance of Helping Community College Students Select and Enter a Program of Study
This report examines the importance of declaring and entering an academic program of study for community college student success and completion. Researchers studied an entering cohort of more than 430,000 community college students and followed their progress over a six-year period through programs of study to completion of a certificate, associate degree or transfer to a university. The study used student course patterns to identify those who entered a program of study in 21 program areas across the liberal arts and sciences and career technical education. Researchers found that entering a program of study is an important milestone on the path to college completion that only half of entering community colleges students reach, and that students who enter a program of study in their first year are twice as likely to complete a certificate, degree, or transfer as those who enter a program of study in the second year or later. (Click here for the appendix) Click here for the brief.
Consequences of Neglect: Performance Trends in California Higher Education
This report uses national data to take an in-depth look at California higher education performance in relation to other states and by region and by race/ethnicity. Researchers analyzed California’s postsecondary performance in the categories of preparation, participation, affordability, completion, benefits and finance, and found that the state’s performance is average, at best, and trending downward. This report is the fourth in a series of reports previously titled The Grades are In, which followed the publication of the National Center for Public Policy and Higher Education’s bi-annual Measuring Up reports that graded the 50 states on their higher education performance.
Performance Incentives to Improve Community College Completion: Learning from Washington State's Student Achievement Initiative
A policy brief, jointly produced by IHELP and the Community College Research Center (CCRC) at Columbia University, offers lessons to date about the Student Achievement Initiative (SAI), a policy adopted by the Washington State Board for Community and Technical Colleges that draws on intermediate measures of student progress to reward colleges for improvements in student achievement. The policy brief examines policy choices that Washington faced in designing and implementing SAI, the choices that leaders in other states will confront when considering adopting performance incentive policies as a means to improve student outcomes. Funded by the Bill and Melinda Gates Foundation, the three-year evaluation will include an examination of the impact of SAI on college efforts to improve student outcomes and on student outcomes.
Divided We Fail: Improving Completion and Closing Racial Gaps in California's Community Colleges
This report, sponsored by the Campaign for College Opportunity, and co-released by 14 organizations, analyzes the progress and outcomes of degree- and certificate-seeking students in the California Community College system. The study tracks the 2003-04 entering cohort (more than a quarter of a million students) over six years, analyzing their progress along a series of intermediate milestones and completion outcomes by race/ethnicity. The report emphasizes that low completion rates and continued racial/ethnic disparities pose serious risks to the state’s future prosperity and offers recommendations for changes to policy and practice with a goal of improving student success, especially among underrepresented minority populations. The recommendations include coordinating a systemwide effort to analyze and publicly report cohort data on enrollment patterns and student progress for each college, by race/ethnicity, and adopting a new funding model that rewards colleges for helping students progress through key milestones – with extra incentives for key attainments achieved by students who entered under-prepared. See Regional Profiles
Divided We Fail in LA: Improving Completion and Closing Racial Gaps in the Los Angeles Community College District
This report was sponsored by Alliance for a Better Community, a nonprofit organization dedicated to enhancing opportunities for Latinos to ensure the well-being of the Los Angeles region. It applies the approach used in our recent report, Divided We Fail, to the nine colleges within the Los Angeles Community College District. The study tracks the 2003-04 entering cohort of degree- and certificate-seeking students (more than 18,000 students) over six years, analyzing their progress along a series of intermediate milestones and completion outcomes by race/ethnicity. The report points out the risks posed by racial/ethnic disparities and offers recommendations for changes to policy and practice with a goal of improving student success, especially among underrepresented minority populations.
Taking the Next Step: The Promise of Intermediate Measures for Meeting Postsecondary Completion Goals
This report, sponsored by Jobs for the Future and written by IHELP Research Specialist Jeremy Offenstein and IHELP Director Nancy Shulock, examines system, state and multi-state efforts and multi-institution initiatives to develop and use intermediate measures of student success as a tool to improve accountability and guide institutional efforts to improve student success. The report distinguishes between milestones that must be attained in order to get to completion and success indicators that increase a student’s chances of completion. The report analyzes the differences in approach, definitions and uses of the data on intermediate measures and offers recommendations on the collection, reporting and effective use of the data and the need for common practices and definitions.
Advancing by Degrees - A Framework for Increasing College Completion
This report, produced by IHELP for The Education Trust, offers higher education leaders guidance on using data to monitor student progress and applying the results to inform changes in policy and practice to help more students earn degrees. The report describes a framework of milestones, or intermediate educational achievements that students reach along the path to degree completion, and on-track indicators, or academic and enrollment patterns that are related to a greater likelihood of graduation. The report uses data from the State University System of Florida and the California Community Colleges to demonstrate how the framework can be used in two-year and four-year institutions to diagnose where and why students fall off the path to success and to make changes in policy and practice to increase degree completion. The Education Trust, working with the National Association of System Heads, will distribute the report to leaders of higher education systems across the country.
Strategies for Improving Higher Education in California: Some Lessons from Florida for California's Higher Education Policy
This policy brief, sponsored by the Campaign for College Opportunity, examines the public higher education policies and practices of the state of Florida in order to determine possible lessons for California in its efforts to increase student success and degree completion. Florida was chosen for evaluation due to its large and diverse higher education system, the importance of community colleges in its higher education system, and its participation in national projects and education reform efforts. While Florida still faces significant challenges, some of the state’s policy approaches for public higher education warrant consideration in California’s quest for improvement, specifically the comprehensive student data system, policies related to student transfer from community colleges, statewide Career Technical Education program standards, and standardized policies for assessment, placement and remediation.
Student Flow Analysis: CSU Student Progress Toward Graduation
The above is a linked excerpt of the report to the California State University as part of its one-year planning grant from the Lumina Foundation’s Making Opportunity Affordable project. The report studies the 23-campus system’s efforts to improve graduation rates, analyzes systemwide data on student progress toward degrees, and makes recommendations for future steps. IHELP researchers drew on their previous work on student success in Steps to Success in order to use the “milestone” framework to analyze student patterns of progress toward completion.
Steps to Success: Analyzing Milestone Achievement to Improve Community College Student Outcomes
This report offers a framework, based on the research literature, for guiding educators in using available knowledge and tools to improve student outcomes. Using data for the California Community Colleges, the report illustrates the framework, which consists of milestones, or intermediate educational achievements that students reach along the path to degree completion, and indicators of success, or academic patterns students follow including remediation, gateway courses, and credit accumulation. The report shows how the framework can be used to diagnose where and why students fall off the path to success, to suggest appropriate interventions, and to improve accountability in community colleges.
Student Progress Toward Degree Completion: Lessons from the Research Literature
This report reviews the research literature on student success to identify intermediate outcomes, sometimes called “milestones,” along the college pathway that give students momentum toward degree completion. It points to academic behaviors and patterns that have been found to predict student progress and success that can, therefore, be tracked to identify where and why student progress stalls and how changes to policies and practices might increase degree completion. The report was prepared as background for upcoming reports on milestone achievement among community college and university students.
Community College Student Outcomes: Limitations of the Integrated Postsecondary Education Data System (IPEDS) and Recommendations for Improvement.
This report on the IPEDS analyzes the value and effectiveness of the system for understanding student outcomes in community colleges. The report discusses the system’s shortcomings such as the limitation of the graduation rate data to full-time students, the difficulty in discerning student intent in order to report on the appropriate outcomes, and the limitations for using the data to make comparisons across colleges. Recommendations are made for improving the data collected and for better use of the data.
The Grades are In - 2008: Is California Higher Education Measuring Up? February, 2009
This report is the third in a series of reports analyzing the performance of California higher education in the areas of preparation, participation, completion, affordability, and benefits. It presents data related to these categories of performance by region and by race/ethnicity, and discusses key issues and policy recommendations for each category. It also describes California's performance relative to other states as presented in the National Center's Measuring Up 2008 report card. See the regional performance profiles.
The Grades are In - 2006: Is California Higher Education Measuring Up? October, 2006
This report is the second in a series of reports analyzing the performance of California higher education in the areas of preparation, participation, completion, affordability and benefits. It presents data related to these categories of performance by region and by race/ethnicity.
The Grades are In - 2004: Is California Higher Education Measuring Up? June, 2005
This report analyzes California's performance on the 50-state report card called Measuring Up. It includes discussion of the state's score in each of the five categories of performance, and presents a breakdown of related data by region and by race/ethnicity.
A Framework for Incorporating Public Trust Issues in States' Higher Education Accountability Plans
This report explores the relationship between public trust and public accountability. It offers a framework for incorporating the concept of public trust into state accountability initiatives in higher education.
Facing Reality: California Needs a Statewide Agenda to Improve Higher Education Outcomes
This report examines seven states that share California's high rates of growth and demographic change to see what California can learn about how to improve access to and success in postsecondary education.
California Community College Transfer Rates: Policy Implications and a Future Research Agenda
This statistical study identifies factors that explain observed differences in transfer rates among California's community colleges.
An Accountability Framework for California Higher Education: Informing Public Policy and Improving Outcomes
This report responded to a request by the state Senate to begin developing an accountability system for higher education to measure progress toward definable state policy goals. The report formed the foundation for legislative efforts still underway to establish a state accountability system.
Concerns About Performance-based Funding and Ways that States are Addressing the Concerns
As the nation’s colleges and universities struggle with limited budgets and demands for increased degree completion, states are once again looking at performance-based funding as a way to increase efficiency and reward institutions for access and success. This IHELP brief examines concerns that have arisen about performance-based funding and summarizes the ways that states are addressing, or could address, the concerns. This four-page brief was written by IHELP Director Nancy Shulock as part of her work with the California Community Colleges’ Student Success Task Force and is intended to inform discussions for states that are considering changes to postsecondary education funding models.
Prerequisite Policy in the California Community Colleges
This two-page brief written by IHELP Director Nancy Shulock examines proposed changes to prerequisite policy in the California Community Colleges. The brief discusses the challenges faced by the CCC regarding under-prepared students and the current trends in other states to increase the success of under-prepared students. A written statement containing this information was presented to the CCC Board of Governors
Beyond the Rhetoric: Improving College Readiness Through Coherent State Policy
This policy brief, jointly produced by the Southern Regional Education Board and the National Center for Public Policy and Higher Education, examines the college readiness problem and offers recommendations to help government and educational leaders strengthen their efforts to lessen the college readiness gap. IHELP Director Nancy Shulock participated in the workgroup that produced the report, which analyzes the causes of the college readiness gap and discusses how states could better address the problem. The brief recommends specific policy changes and offers state leaders a model college readiness agenda consisting of state-wide college readiness standards developed and adopted jointly by P-12 and postsecondary education. It describes assessment, placement, curricular, teacher development, and accountability practices carefully designed to reflect and implement the standards.
A Fundamentally New Approach to Accountability: Putting State Policy Issues First
This report was prepared for the Forum on Public Policy of the Association for the Study of Higher Education.