Which outcomes can be fairly measured and when will depend on how the Promise program is intended to work and how any program eligibility criteria are enacted. Developing a Theory of Change, as described in a prior section, is an important first step in thinking about what needs to happen in order for a particular Promise program to realize its intended outcomes. Since Promise programs represent long-term efforts at systemic change, careful consideration of when various targeted outcomes are even possible to detect is suggested. In order to do that, one must clearly understand when the key components of the Promise intervention are fully implemented or have had sufficient time to be experienced by the students.
The Pittsburgh Promise, for example, made scholarship dollars available beginning with the high school graduating class of 2008. This Promise program has grade point average, attendance, and residency requirements that were phased in over 3 cohorts of students (a programmatic choice to give students graduating soon after announcement more of an opportunity to access the scholarship). In addition, it took time for relevant supports to be developed and implemented by the Pittsburgh Promise, the Pittsburgh Public Schools, families, and community partners. Below, we show an example table that was used to help think about the Pittsburgh Promise implementation timeline. Each Promise program may have its own constraints and conditions that need to be considered. Please feel free to use this matrix as a starting point that can be adapted for your particular context.
This table shows quite clearly that the 2008 and 2009 graduating classes were not recipients of the full Promise model (as articulated in the Pittsburgh Promise Theory of Change), but did benefit from up to $5,000 a year toward post-secondary education (the full award of $10,000 annually became available to the 2012 class). If we consider looking at college enrollment and degree attainment rates of the 2008 cohort we are measuring the impact of the financial award exclusively since there was not sufficient time from announcement of the Promise for the school system, parents, and/or students to make attitudinal and/or behavioral responses before that cohort matriculated. Students who graduated from high school in 2014, however, experienced all of middle and high school with the knowledge of the Pittsburgh Promise and with corresponding supports that were rolled out in the several years after the program start.
Other programs with simpler requirements, such as the Kalamazoo Promise, made the full amount of benefits available to the first eligible graduating class (Class of 2006), but students who were seniors when the program was announced in November 2005 had very little time to adapt their high-school experience in light of the new financial aid resources now available to them. Successive graduating classes have had more years of exposure to the Kalamazoo Promise, so one would expect to see greater impacts in both the K-12 setting and post-secondary outcomes. Not until 2018 will students who spent their entire K-12 years knowing about the availability of the Kalamazoo Promise graduate from high school. This cohort will potentially represent the full "Kalamazoo Promise effect."
Some communities may announce the creation of a Promise program to take effect several years down the road, giving students time to adjust their expectations in light of future benefits. The roll-out of benefits in these cases may lead to different timing in terms of when results might be captured. Similarly, early commitment programs that ask students to enroll in 6th or 7th grade provide several years in which both students and systems can adjust to an impending scholarship opportunity.
Completion of a table such as this exemplar may help determine when it is appropriate to measure outcomes and for which groups of students.