Impact & Evidence
Impact & Evidence
We measure the work, study the results, and use what we learn.
Across multiple years of federally supported work, Life that Counts has built, delivered, adapted, and evaluated programming with schools. The record includes positive participant experience, encouraging pre/post findings, results that were null or unfavorable, and practical lessons that now shape stronger program and evaluation practice.
This page presents findings from two external evaluation sources. Each study is reported with its population, period, measure, denominator, method, and limitation kept together. Historical findings demonstrate experience, participant response, and lessons for design. They are not carried forward as automatic evidence for a different or current program.
Two evaluations, reported separately
The evaluations involved different age groups, program structures, measures, delivery conditions, and analytic methods. Their statistics are not pooled, averaged, or used to rank one project against the other.
Historical middle-school evaluation
Washington Park Research and Evaluation (WPRE)
Period represented: 2020–2022. Population and setting: Primarily middle-school students in Alabama and Montana across three historical curriculum levels. Evidence available: Entry and exit survey records, post-program participant feedback, post-program self-assessment items, staff implementation interviews, and evaluator recommendations. Evidence status: Descriptive participant and implementation evidence. Entry and exit respondents were not matched, and no comparison group or causal analysis was reported.
Historical high-school evaluation
ITMESA
Period represented: 2019 and 2020–2021. Population and setting: High-school mentees and mentors in a multi-state peer-mentoring initiative; Life that Counts led Alabama implementation. Evidence available: Adjusted pre/post participant-survey estimates from 2019, unmatched pre/post analyses from 2020–2021, post-program participant experience, and open-ended feedback. Evidence status: Preliminary participant-survey and implementation evidence. The source packet does not document an untreated comparison group or support causal claims.
Three questions require three kinds of evidence
1. Was the program delivered?
Implementation evidence documents who was reached, what was delivered, attendance and survey flow, schedule changes, delivery conditions, and where records need improvement.
2. What did participants report?
Participant-experience evidence describes clarity, respect, engagement, perceived usefulness, attitudes, knowledge, skills, intentions, and immediate reflections.
3. Did the program cause a measured difference?
Causal impact evidence requires an appropriate comparison, linked follow-up records, prespecified measures, documented implementation, and uncertainty estimates. The historical reports below do not meet that standard, so their findings are presented as descriptive or pre/post evidence.
Evidence earns trust when the finding and its limitation appear together.
Middle-school programming across Alabama and Montana
WPRE | Final report compiled September 2022 | Descriptive evaluation
Washington Park Research and Evaluation compiled a final report on three levels of historical programming delivered primarily to middle-school students in Alabama and Montana. The report combined completed survey records with staff interviews about implementation and contextual conditions.
Life that Counts continues to work with WPRE, led by Dr. Galena Rhoades, on independent evaluation planning. The historical report summarized in this section is attributed to WPRE as an organization.
Survey records—not a participant-retention count
The report lists 2,541 entry surveys and 2,204 exit surveys across the three levels. Dividing those totals produces an aggregate exit-to-entry survey-record ratio of 86.7%. Because entry and exit records were not linked to the same individuals, that ratio is not a participant completion rate, retention rate, or longitudinal follow-up rate.
Historical level | Survey window | Entry surveys | Exit surveys — Level 1 | September 3, 2020–May 19, 2022 | 1,910 | 1,681 — Level 2 | January 25, 2021–April 13, 2022 | 526 | 414 — Level 3 | May 17, 2021–February 7, 2022 | 105 | 109
Table note: These are completed survey records, not verified unique participants or matched entry/exit pairs. Level 3 recorded more exit surveys than entry surveys; the report does not identify the precise source of that difference.
What respondents said about the program experience
Across the three levels, most respondents said the material was clear, activities helped them learn, and they felt respected all or most of the time. The graph reports each level separately and uses percentages calculated from the exact category counts in the report.
Participant experience — Level 1
Percent selecting “all of the time” or “most of the time.”
Interested in sessions — 70.8% (1,113 of 1,573)
Material was clear — 85.3% (1,341 of 1,572)
Activities helped learning — 82.9% (1,301 of 1,569)
Had a chance to ask questions — 80.5% (1,260 of 1,566)
Felt respected — 83.5% (1,305 of 1,562)
Participant experience — Level 2
Percent selecting “all of the time” or “most of the time.”
Interested in sessions — 76.2% (301 of 395)
Material was clear — 88.0% (346 of 393)
Activities helped learning — 86.5% (341 of 394)
Had a chance to ask questions — 85.2% (333 of 391)
Felt respected — 88.0% (346 of 393)
Participant experience — Level 3
Percent selecting “all of the time” or “most of the time.”
Interested in sessions — 74.8% (80 of 107)
Material was clear — 86.9% (93 of 107)
Activities helped learning — 87.7% (93 of 106)
Had a chance to ask questions — 85.0% (91 of 107)
Felt respected — 85.0% (91 of 107)
Respondent experience across three historical curriculum levels. Percentages are calculated from the report’s category counts; item denominators vary.
What respondents said participation made them more likely to do
The report also asked post-program respondents whether participation made them more likely to resist peer pressure, manage emotions, plan toward goals, and understand healthy relationships. These were retrospective self-assessments collected after programming. They were not observed behaviors or matched changes from baseline.
Post-program retrospective self-assessment — Level 1
Percent selecting “much more likely” or “somewhat more likely.”
Resist or say no to peer pressure — 72.2% (1,142 of 1,581)
Manage emotions in healthy ways — 68.1% (1,074 of 1,578)
Make plans to reach goals — 79.8% (1,259 of 1,578)
Better understand healthy relationships — 81.7% (1,326 of 1,623)
Post-program retrospective self-assessment — Level 2
Percent selecting “much more likely” or “somewhat more likely.”
Resist or say no to peer pressure — 75.4% (230 of 305)
Manage emotions in healthy ways — 71.5% (218 of 305)
Make plans to reach goals — 86.8% (264 of 304)
Better understand healthy relationships — 84.5% (257 of 304)
Post-program retrospective self-assessment — Level 3
Percent selecting “much more likely” or “somewhat more likely.”
Resist or say no to peer pressure — 78.5% (84 of 107)
Manage emotions in healthy ways — 73.8% (79 of 107)
Make plans to reach goals — 73.6% (78 of 106)
Better understand healthy relationships — 79.2% (84 of 106)
Source: Washington Park Research and Evaluation, final three-year report, September 2022. Historical programming in Alabama and Montana. Percentages were calculated from reported category counts. Item denominators vary. Entry and exit surveys were not linked at the individual level. The four retrospective items were measured only after programming; 96 early Level 2 exit respondents were not asked them. No comparison group, confidence intervals, effect sizes, or statistical-significance tests were reported for these items.
What WPRE recommended
The evaluation documented schedule interruptions, differences in facilitation arrangements, and discrepancies between classroom counts and submitted surveys. WPRE recommended stronger roster reconciliation, make-up-survey documentation, cohort assignment, session-exposure tracking, and entry/exit matching. The report also recommended additional data collection for the two smaller curriculum levels.
What this evidence supports
The WPRE report supports statements about historical implementation, participant experience, perceived likelihood at post-program, and organizational learning. It does not establish individual change, long-term behavior change, or causal effectiveness.
High-school peer-mentoring evidence
ITMESA | Reporting materials assembled September 2021 | Pre/post and post-program evidence
Life that Counts participated in a multi-state high-school mentoring initiative examined through participant entry and exit surveys, post-program feedback, and open-ended responses. Life that Counts led implementation in Alabama. ITMESA reporting materials assembled in September 2021 summarize evidence from 2019 and 2020–2021; the source packet does not provide Year 2 outcome data.
The materials do not document an untreated comparison group, an assignment process, or a causal counterfactual. They should therefore be described as historical participant-survey evidence—not as a controlled impact evaluation.
2019 Alabama findings
The 2019 Alabama records include 434 pre-surveys and 343 post-surveys, producing a 79.0% exit-to-entry survey-record ratio. The report describes multilevel repeated-measures regression adjusted for gender, race, Hispanic ethnicity, age, mentor/mentee role, and state. It does not report the exact analytic sample for each outcome, confidence intervals, standardized effect sizes, complete missing-data procedures, or a no-program comparison. The reported Alabama model estimates were favorable for risk-avoidance attitude, risk-reduction knowledge, and risk-reduction skill. The reported estimate for risk-avoidance behavioral intention was not statistically significant.
Reported 2019 Alabama adjusted difference estimates
Risk-reduction knowledge — +10.9 points — p=.0001
Risk-avoidance attitude — +6.6 points — p=.0003
Risk-reduction skill — +4.8 points — p=.0002
Risk-avoidance behavioral intention — +3.2 points — Not statistically significant
STATE COMPARISON · 2019
Alabama and Indiana: concurrent peer-mentoring results
ITMESA reported the same four principal constructs for concurrent implementations in Alabama and Indiana. See the construct-by-construct comparison, exact estimates and p-values, study context, and public evidence brief.
Explore the Alabama–Indiana comparison
Source-reported, covariate-adjusted pre/post estimates for the 2019 Alabama survey records. These are model estimates, not percentages of participants who improved.
Source: ITMESA 2019 high-school participant-survey analysis. Alabama records include 434 pre-surveys and 343 post-surveys. The source describes multilevel repeated-measures regression with participant and state covariates. It does not report the exact outcome-specific analytic sample, confidence intervals, a contemporaneous comparison group, or complete missing-data procedures. Findings describe adjusted pre/post associations and do not establish causation.
What participants reported about the experience in 2020–2021
In the 2020–2021 post-program file, 79.8% reported that the material was clear all or most of the time, 73.2% said discussions or activities helped them learn all or most of the time, and 84.5% said they felt respected all or most of the time. The same file reports that 63.9% felt interested all or most of the time and 69.4% had opportunities to ask questions all or most of the time. The source lists 247 post surveys across both states but does not provide the item-specific denominator for each percentage. These participant-experience findings are displayed without assigning 247 as the denominator for every item.
2020–2021 participant experience
Interested in sessions — All 25.5% | Most 38.4% | Some 29.2% | None 6.8%
Material was clear — All 42.7% | Most 37.1% | Some 18.0% | None 2.1%
Activities helped learning — All 34.5% | Most 38.7% | Some 21.1% | None 5.7%
Had a chance to ask questions — All 46.6% | Most 22.8% | Some 22.3% | None 8.3%
Felt respected — All 57.7% | Most 26.8% | Some 12.4% | None 3.1%
Post-program participant experience in the 2020–2021 combined survey file. The file reports 247 post surveys overall; item-specific denominators are not supplied.
Results were not uniform across years
The 2020–2021 pre/post analysis used unmatched groups and reported mixed, predominantly nonsignificant results. The later files did not reproduce a uniform favorable pattern in Alabama’s principal measures. In the combined two-state analysis, the report also listed unfavorable changes in a risk-reduction behavior-intention measure and the school-success score. Delivery conditions, participating cohorts, and learning modes changed during this period. These differences do not prove why the results changed, but they demonstrate why mode, implementation consistency, session exposure, baseline context, measurement, and participant linkage must be documented. Life that Counts does not treat the 2019 findings as guaranteed results for later cohorts or current programs.
Participant voice
Open-ended responses discussed school effort and future goals, confidence and self-respect, kindness toward others, relationship decisions, emotional regulation, and healthier choices. Some respondents reported no change or uncertainty. The response file does not provide a formal coding protocol, unique-respondent denominator, or publication review. These themes are presented only as illustrative context; no participant quotations, word cloud, or percentages are published.
What this evidence supports
The ITMESA materials support statements about historical implementation, participant experience, and reported pre/post estimates within the surveyed cohorts. They do not establish long-term behavior change, a program-versus-no-program difference, or effectiveness of a current Life that Counts program.
Positive findings matter. So do null findings and limitations.
Taken together—but never statistically combined—the historical evaluations show that many participants described clear instruction, respectful treatment, useful activities, and greater readiness to make thoughtful decisions. They also show that results were not uniform across measures, cohorts, years, or delivery conditions. That full record strengthened five priorities:
Keep participant records linkable and private. Use privacy-protected identifiers that allow approved evaluators to connect measurement points without exposing identities to program staff.
Document what was actually delivered. Record the planned sequence, sessions offered, sessions attended, make-up work, delivery mode, adaptations, and major disruptions.
Separate implementation, experience, and impact. A well-received program may still require a controlled study before effectiveness can be claimed.
Protect evaluator independence. Research instruments, identifiable outcome data, analysis, and scientific reporting should remain under independent evaluator authority.
Report the full result. Favorable, null, and unfavorable findings should be interpreted with the same attention to method, context, and limitation.
Historical evidence informs the next study; it does not replace it.
The historical evaluations informed current BELIEVE development and evaluation planning, but they do not establish BELIEVE as effective. Current planning calls for a defined twenty-component sequence, qualified adult-led core instruction, a limited and supervised peer-reinforcement role, privacy-protected longitudinal matching, documentation of comparison programming and disruptions, clearly defined follow-up points, and independent data custody and analysis.
Life that Counts works with Washington Park Research and Evaluation, led by Dr. Galena Rhoades, to strengthen evaluation planning, data governance, and scientific reporting. Effectiveness claims are reserved for findings produced by an appropriately controlled evaluation.
Review the source and the method
WPRE final report
Final three-year report on historical middle-school programming in Alabama and Montana, compiled September 2022.
ITMESA public evidence summary
Reviewed summary of the 2019 and 2020–2021 high-school participant-survey methods, findings, and limitations.
Institutional review
Schools, funders, researchers, and institutional partners may contact Life that Counts to discuss source documentation, implementation history, evaluation planning, and the limits of the available evidence.
Bring evidence into the planning conversation
Tell us whom you serve, what decisions your young people are facing, what support is already in place, and what you need to learn. We will help you consider program fit, implementation responsibilities, and an appropriate approach to measurement.