Pilot Implementation and Evaluation of “Choices Enhanced” An Abstinence-Only Demonstration Project by Catholic Charities of Onondaga County

Tonya Roloson, B.S., Catholic Charities of Onondaga County, Syracuse, NY
Deborah Monahan, Ph.D., Vernon Greene, Ph.D. Mi Ditmar, B.A., Syracuse University, Syracuse, NY

Introduction

Choices Enhanced is an abstinence-only, health and wellness program designed and implemented by Catholic Charities of Onondaga County (CCOC). Children ages 9-15 (grades 4-10) and their parents are recruited from five CCOC neighborhood centers, located in the city of Syracuse. The youth program, delivered by a team of educators, consists of two hours of interactive classroom intervention each week for 14 weeks. The curriculum, “Families United to Prevent Teen Pregnancy” (FUPTP), includes: (1) foundations for abstinence education (self-concept and setting goals), (2) communication (reflective listening, assertiveness, and resistance skills), (3) developing healthy relationships (friendships, developing rules and relationships), (4) sexually transmitted infections and AIDS, and (5) skills for life. Parents of youth participants are encouraged to attend workshops facilitated by two parent educators. These workshops use the “Parent-Child Connectedness: New Interventions for Teen Pregnancy” (PCC) curriculum. The program evaluation will examine whether there are statistically significant differences between youth randomly assigned to the Choices Enhanced program, and those assigned to control status. Outcome variables include: (1) favorable attitudes toward positive health and emotional benefits of abstaining from premarital sexual activity, (2) favorable attitudes toward healthy lifestyle choices, including avoiding alcohol and drugs, and (3) the intent to practice healthy lifestyle choices and to remain abstinent until marriage. Evaluation of these objectives will be based on analysis of pre-post change scores in the intervention versus control group on these AFL Core Baseline instrument outcome items, detailed in the methods section. The evaluation of the parent/guardian intervention will assess parents’ levels of (1) involvement in the lives of their children, (2) satisfaction with their relationship with their child, and (3) ability to appropriately monitor and control their child’s behavior. Also examined will be whether parental gains are associated with youth gains.

Methods

Choices Enhanced is offered at the five neighborhood centers of CCOC. Four of the five zip codes served by these neighborhood centers rank in the top 10% for New York state teen pregnancy rates. An experimental two-condition randomized cluster design is being used to evaluate the program. With reference to the outcome variables in the Introduction above, the evaluation of youth outcome Objective #1 will analyze treatment versus control group change scores on AFL Core Baseline items 24, 25a, 25b, 25c, 26a, 26b and 26c (measured as the difference from the scores on the identical items on the Follow-up instrument). Evaluation of youth outcome Objective #2 will analyze treatment versus control group change scores on AFL Core Baseline items 19a, 19b, 19c, 24, 26a, and 26b. The evaluation of youth outcome Objective #3 will analyze treatment versus control group change scores on AFL Core Baseline items 30, 31, and 32. During each phase, three randomly determined centers (the “treatment” sites) will run the Choices Enhanced curriculum program in addition to their regular programs, while the two remaining centers (“control” sites) will run only their regular youth programs. The trainers who designed the curriculum were from Rosalie Manor (Milwaukee, Wisconsin), and training was conducted on site in Syracuse, New York to ensure fidelity of the intervention.  The AFL Core Instrument for Youth along with the Supplemental Scales as well as the Parent Survey Instrument received IRB approval from Syracuse University in Year 1.

Operationally, in Year 1, a pre-test baseline survey instrument was administered to subjects at the treatment and control sites during the first week of the Choices Enhanced program. The post-test version of the survey instrument will be administered to both groups at the end of the 14-week intervention period, which will occur in Year 2. Follow-up with both groups will occur at 12 and 36 months post-intervention. The survey instrument consists of the AFL Core Instrument, plus items making up the following three summative composite scales: Self-Concept Scale for Children (Lipsitt, 1958), the Parental Nurturance Scale (Buri, 1989), and the Adolescent Coping Orientation for Problem Experiences (A-COPE), (McCubbin & Thompson, 1991) The Cronbach’s alpha coefficients for these scales in the pilot pre-test data are respectively .73, .64 and .77. These scales were selected as assessments of known protective factors against teen pregnancy, including high self-esteem, high quality parent relationships and robust coping skills. The Core Instrument was modified for participants in the 9-10 age group to make it more appropriate for their developmental level. The parent/guardian survey instrument includes the Family Functioning Scale (Tavitian, Lubiner, Green & Grebstein, 1987), the Family Organized Cohesiveness scale (Fisher, Ransom, Terry & Burge, 1992) and the Kansas Parental Satisfaction scale (James, Schumm & Hall, 1985). 

Completed survey instruments were scanned into an SPSS database using SNAP Professional Edition software (also used to create the survey instruments, using identical structure and wording from the current AFL instruments), from which data will be converted for further analysis using STAT/Transfer 9.5. When post-test data become available in Year 2, hypotheses about program efficacy will be tested, based on the AFL survey instrument outcome items detailed above.  Hypotheses about the youth outcome variables that the program is intended to influence (as described in the introduction), will be statistically tested using hierarchical linear modeling (HLM) carried out using the HLM6 software package (Scientific Software, 2008). The use of HLM, rather than conventional regression analysis or ANOVA, is necessary to control for site-specific (clustering) effects that would otherwise lead to underestimating the true risk of Type-1 error (spurious rejection of the null-hypothesis). Process evaluation measures, which include both qualitative and quantitative data from youth and parent educator logs, participant enrollment forms, and attendance rosters, are also being collected.

Results

In Year 1 of the project, 121 youth (26% self-identifying as white, 41% as African American and 31% as Hispanic or Latino) provided baseline data through completing the pre-test survey instrument. Eighty-five of these are at treatment sites and are currently participating in the Choices Enhanced program and 36 are at the control sites. All participants were recruited from the five zip codes served by the CCOC neighborhood centers where child poverty rates range from 31% to 59%. There were no substantial baseline differences between the treatment and control sites on age, sex, ethnicity, smoking, alcohol or drug use, or previous sexual activity, though there were substantial differences in cluster-specific characteristics such as racial composition, living arrangements and other factors related to neighborhood characteristics, and hence characterizing the catchment areas for the neighborhood centers (clusters). These differences will attenuate with time as the number of randomizations grows. Of the youth who completed the full baseline survey, 60% “agreed a lot” or “agreed a little” that only married people should have sex, and 71% “strongly agreed” or “agreed” that they intend to wait until they are older before they have sex.

Discussion

The pilot implementations of both the youth and the parent interventions are currently underway in Year 1, and administration of their respective baseline (pre-test) surveys has been completed.  Follow-up surveys will yield post-test data early in Year 2. Timing the start of the pilot phase during the summer months rather than during regular after school programming complicated the recruitment and scheduling. Further, it resulted in treatment group students returning to other after school programs rather than completing the intervention. Efforts to pursue these post-test surveys for these youth are underway, though their dosage on the intervention will be lower than originally planned.  Other challenges included language barriers, participants’ reading levels, and the greater than expected heterogeneity of the neighborhood centers in programming, demographics and educational levels that bear on survey comprehension. We are working actively to minimize these challenges in future implementations with intensive planning for scheduling and for maximizing completed curriculum sessions during the school year. We will also use the Spanish-language AFL Core Instrument when available, use verbal presentation of the surveys for those with reading difficulties or vision impairment, and possibly restructure the intervention randomization process from cluster to individual randomization if we find that power attenuation using cluster randomization is too great to be overcome with available sample sizes.

Implications

Since we have only pre-test data from Year 1, our data do not yet permit us to statistically estimate program effects. While we are still in our first (pilot) cycle in the field, it appears that our procedures for recruitment and curriculum delivery, as well as data collection and database entry, have all gone as planned with pre-testing, data collection and curriculum delivery occurring essentially on schedule. However, this pilot implementation of Choices Enhanced has revealed that even established community organizations such as CCOC benefit from greater flexibility in devising more innovative means of recruiting families and becoming more engaged in conducting neighborhood outreach to underserved youth. Strategies under consideration include “family nights” to inform parents about Choices Enhanced, accelerated recruitment for the parent intervention, development of a condensed “retreat” format for the PCC training, and greater use of developmentally tailored interactive lessons.

Contact Person

Tonya Roloson  
Telephone: (315) 396-4267 
E-Mail Address: troloson@ccoc.us