This post is part of a new blog series called Transformative Change Initiative (TCI) Featured Evaluator, that includes interviews with members of TCI’s Evaluation Collaborative. This community of evaluators has a wealth of knowledge, experience and insights into evaluation of the TAACCCT grants that are being implementation throughout the United States. Want to be profiled or know someone who would make a great feature? Email us at occrl@illinois.edu.
Name: Derek V. Price
Current position: Principle Owner, DVP-PRAXIS, LTD.
Short bio: Dr. Derek V. Price is principle owner of DVP-PRAXIS LTD, an action oriented consulting focused on higher education and the workforce. His firm specializes in mixed-method formative and summative evaluation services to inform implementation and measure impact. They also provide strategic advising services for project development and implementation, and conduct research and policy analysis on critical issues facing higher education and the economy. Dr. Price is a national leader in strategic thinking for institutional transformation and systems change. He has a strong policy, research, and evaluation background and works on a broad range of policy and practice issues around postsecondary education and workforce development. Before starting DVP-PRAXIS, Dr. Price was the director of higher education research at Lumina Foundation for Education. In that role, he helped launch the Foundation’s premier national initiative, Achieving the Dream.
Questions:
Q. What is the design and predominant methods for your TAACCCT evaluation?
A. In each of the three consortium evaluations I am leading, we are using Propensity Score Modeling to identify comparison groups for the impact study. While we will look at overall participant outcomes against a matched comparison, this broad approach is unlikely to yield statistically significant impacts consortium-wide, because of the considerable variability among the consortium colleges. In addition to the program of study differences (e.g., welding, machine tool, and industrial maintenance for the TAACCCT2 Wisconsin consortium), colleges have modified their respective programs in many ways. Some colleges have implemented bridge programs for low-skilled adults as program onramps; other colleges have embedded enhanced instructional supports into program classes; and some colleges have simply added program capacity.
This program variability elevates the importance of our qualitative methods – primarily in-depth site visits, semi-structured interviews and participant focus groups. Our qualitative methods allow us to document implementation progress, and also to examine factors that can lead to sustainability of the TAACCCT programs, and potentially to transformative change. We are also using our qualitative evaluation to identify more discrete treatments within a college or subset of colleges, and will use PSM to identify comparison groups in these instances. In doing so, we hope to make our impact evaluation more theoretically informed and relevant to the field.
Q. What advice do you have for new TAACCCT evaluators?
A. Do not get hung up on the extensive performance reporting requirements from DOL. As a third-party evaluator, I believe it is our responsibility to discern which program modifications – new curriculum models; enhanced academic and non-academic supports; team-teaching; etc. – are most likely to yield improved participant outcomes. And then design an impact study that can further the field’s understanding of these effective practices. We also need to move beyond documenting implementation progress, and begin to identify and validate the factors that can influence sustainability and transformation. TCI is a great venue for these broader conversations.
Q. What questions do you have for others about TAACCCT evaluation?
A. We are finding very few TAA clients enrolling in these programs. We also are finding that referrals from the workforce system are quite small. Most participants enroll due to college recruitment techniques. Is this true for other TAACCCT projects?
Q. Do you have tools, reports or other products you are willing to share?
A. As we near the end of TAACCCT2 in September 2016, we expect to have reports and briefs we can begin sharing with the field. Where do evaluators and practitioners want to see (or hear) results? Which publications or conferences do folks read or attend?
Derek can be reached at derek@dvp-praxis.org.