AFL Prevention Programs - Core Evaluation Instrument User Guide

back to Evaluation Instruments main page

User Guide

Contents

AFL Prevention Core Evaluation Overview
AFL Prevention Questionnaires
      Overview of questions
          Core Baseline Questionnaire
          Core Follow-up Questionnaire
      Administration
          Consent
          Incentives
      Questionnaire question-by-question specifications
Data Security and Human Subjects Approval Guidelines
Analysis of AFL Core Questionnaire Data
      Initial anticipated data requests from OAPP
      Crosswalks and comparisons to other national datasets
      Cross-walk tables to other national data collection instruments
Summary and next steps

Appendix A: Informed consent for clients

Appendix B: Informed consent for parents/guardians of clients

AFL Prevention Core Evaluation Overview

The AFL prevention core evaluation instrument was developed for use by AFL prevention demonstration projects funded by the Adolescent Family Life (AFL) program that is administered by the Office of Population Affairs (OPA). The AFL program was enacted in 1981 as Title XX of the Public Health Service Act. Prevention demonstration projects are designed to educate adolescents on the importance of abstinence from premarital sexual activity as a means to prevent pregnancy and sexually transmitted infections, as well as to promote a successful transition to adulthood. Additionally, AFL prevention projects stress the importance of family and parent involvement in the delivery of funded services for adolescents.

AFL Prevention demonstration projects are conducted in a wide variety of sites, including schools, community centers and hospitals and focus primarily on young adolescents between the ages of 9-14 years.

These instruments have been developed for a number of reasons, first among them, the repeated requests by AFL grantees and evaluators to do so. Additionally, because the evaluations of AFL demonstration projects are conducted independently, the data collected from one project to another vary and the Office of Adolescent Pregnancy Programs (OAPP) has no way to track performance of the program, as a whole, on a number of indicators that are particularly relevant in terms of the purpose of the Title XX statute and responsiveness to the A – H definition of abstinence education contained in the Welfare Reform Act of 1996. As the core instruments are integrated into individual project evaluations, OAPP will be better able to monitor the direction and progress of the program and direct future funding resources to approaches that have demonstrated effectiveness, as well as track the performance measures that have been developed in response to recommendations from the Office of Management and Budget’s recent evaluation of the program.

AFL Prevention Questionnaires

All AFL Prevention demonstration projects funded in fiscal year 2005 and beyond will be required to use this core evaluation instrument. They may add additional questions if they choose. Projects funded prior to fiscal year 2005 will be encouraged to use the core instrument in their evaluations, but will not be required to do so.

Both the baseline and follow-up AFL Prevention core evaluation instruments contain a section that asks a variety of questions to determine, not only attitudes about abstinence, but also opinions about the consequences of premarital sexual activity. Because these questions, and the definition of sexual activity given in the section, may be considered sensitive, particularly for the youngest clients served in AFL Prevention demonstration projects, the OPA will consider granting a waiver to its use on a case basis and with adequate written justification by the grantee.

Overview of questions

Core baseline questionnaire

The Prevention core evaluation instrument consists of six sections covering, in addition to basic demographic information: family composition and communication patterns; activities the adolescent participates in; health behaviors; decision making; self confidence; future orientation; and attitudes about abstinence.

Section 1. Demographics, family composition and communication patterns

Questions 1.1 to 1.17 collect data on age, gender, race/ethnicity, family composition and communication patterns. A single intervention can have different outcomes for younger adolescents than for older adolescents, for females than for males, and for adolescents of different races or ethnicities. Moreover, parent involvement and communication are a central focus of the Title XX AFL statute. In addition to establishing that the AFL client lives with a parent or adult guardian, this section collects data on the nature of the relationship, parental boundary setting, and patterns of communication.

Section 2. Activities

Question 2.1 asks about involvement in extra curricular activities, including sports, clubs, community service, and faith-based groups. As the AFL program has matured, it has moved away from a curriculum-driven approach to abstinence education and encouraged funded demonstration projects to include youth development concepts in their interventions.

Section 3. Decision-making

Questions 3.1 through 3.10 collect data about what the adolescent thinks about dating, their ability to resist pressure, and whether they engage in health risk behaviors. Being able to make responsible decisions around issues such as relationships and health behaviors is an important component of a successful transition to adulthood. Moreover, early sexual activity is associated with other health risk behaviors, as well as peer group influence.

Section 4. Self Confidence

Questions 4.1 through 4.3 collect data measuring an adolescent’s perception of him or her self and his/her ability to negotiate the challenges of growing up. An important part of AFL Prevention interventions is to help adolescents gain in self-confidence, building on their competencies to make good choices, including abstinence from premarital sexual activity.

Section 5. The Future

Questions 5.1 through 5.6 collects data to assess the importance of future events to adolescents enrolled in the intervention. Young people who have hopes for their future are more likely to make good health and relationship choices during the adolescent years. The OPA encourages AFL Prevention grantees to address issues around the importance of education, career and marriage to give project clients sound reasons to delay initiation of sexual activity and prevent unintended pregnancy and sexually transmitted infections, including HIV.

Section 6. Attitudes about Abstinence

Questions 6.1 through 6.6 collect data to determine, not only attitudes about abstinence, but also opinions about the consequences of premarital sexual activity. The overarching goal of AFL Prevention demonstration projects is to promote premarital abstinence for adolescents. To that end, instilling and/or maintaining positive attitudes toward abstinence is an essential measure of an intervention’s effectiveness.

Core follow-up questionnaire

This instrument is designed to collect information on attitudes, beliefs and behaviors at both baseline and follow-up, and consequently repeats the same questions as the core baseline questionnaire.

Administration

Project staff or a project evaluator will administer the baseline core instrument at intake and at completion of the program, or at 12-months, whichever comes first. Completion of the questionnaires will be voluntary; project clients will be informed that they may refuse to answer any or all of the questions. The instruments are designed to be age-appropriate for clients ages 12-19 with low-literacy levels, and to be administered as pencil-and-paper surveys. Although the surveys are intended to be self-administered, program personnel may also administer them if respondents find the questionnaire too difficult to read.

The respondents should be provided a quiet private area in which to complete the questionnaire. A staff person who is knowledgeable about the questionnaire and administration procedures should be nearby and available to answer any question respondents may have. This includes reading the questions to the respondent, should they request it.

Once the survey is completed it should be immediately placed in the secure area designated by the site IRB and HIPAA procedures. The staff person who oversees the administration of the questionnaire should check with the client to see if they have any questions, or if they need to discuss any feelings or issues brought up by completing the questionnaire.

Consent

Prior to administering the instrument to any clients, the program must obtain active consent from the client to participate in the questionnaire. Additionally, clients under the age of 18 must have the consent of a parent or legal guardian to participate in the questionnaire. To this end, all respondents must read (or have read to them) and sign an IRB approved consent form. Sample consent forms that contain all of the OAPP required information is contained in Appendix A (for clients) and Appendix B (for parents of client under the age of 18). The appended consent forms represent the minimum protections required. Local IRBs may choose to add additional language and protections.

If local evaluators choose to use this questionnaire with control/comparison groups, they are required to obtain consent from all of the parties discussed above using the same protocols.

Incentives

OAPP authorizes AFL Prevention programs to offer non-cash incentives (e.g., gift cards, incentives) of a value of up to $10.00 to each program participant who participates in filing out the core questionnaires. The incentives are to be offered at both baseline and follow-up data collection. OAPP will consider this an approved use of grant funds. If a program’s local IRB determines that incentives are not to be offered, OAPP will defer to the local IRB’s determination.

Questionnaire question-by-question specifications

The following section provides question by question (QxQs) clarification on the meaning and intent of each of the survey questions. This will allow program service providers to accurately respond to any queries that respondents have regarding how they should answer particular questions. The version of the instrument used for the QxQs is the baseline data collection instrument. This instrument contains all of the questions present across both instruments. Although the item numbers may vary between the instruments, the specifications remain the same. There are several questions for which clarification does not seem necessary; therefore, none is provided.

1.  Client ID: ____  ____  ____  ____  ____  ____

2.  Site Number: ____  ____  ____  ____  ____  ____

3.  Entry Date: ____  ____  ____  ____  ____  ____

Q1.  The site should create a unique ID for each client that does not directly identify the client.  This ID should be maintained in the client’s protected confidential file, so that they can be matched for baseline and follow-up data collection.  The site IRB and HIPAA privacy board will need to determine additional site-specific confidentiality protections.

Q2.  The site ID number will be provided by OPA, OAPP AFL Grantee Project Officer.

Q3.  Enter as MM, DD, YY

Core Baseline Questionnaire

AFL Pregnancy Prevention Programs

Section 1

These first questions ask about you and your family.

1.1  How old are you (in years only): _________

Q1.  Round to the nearest whole year.  Do not use fractions or decimals.

 

1.2  Are you a girl or boy?

  Boy

  Girl

 

1.3  Who do you live with now? CHECK ALL THAT APPLY.

PARENTS (Includes step parents, adoptive parents and foster parents):

          Mother

          Father

ADULT GUARDIANS

          Other adult relatives

          Other adults I am not related to

 

1.4  Answer the next few questions about your mother or female guardian. How close do you feel to her?

  Not very close

  A little close

  Pretty close

  Very close

Q1.4 – 1.6.  If the respondent has both a mother and someone they view as a female guardian, they should answer for the individual with whom they have the closest relationship.

 

1.5  Does she spend enough time with you?

  Not enough time, but I don’t want more

  Not enough time, I wish she spent more time with me

  She spends enough time with me

 

1.6  Here are some statements about you and your mother or female guardian.  For each one, check how true the statement is:

				Mostly True		Some-times True		Hardly Ever True			Does Not Apply
a.		I am comfortable talking with her about things that happen in school
b.		I am comfortable talking with her about things that happen in my life

 

1.7  Answer the next few questions about your father or male guardian. How close do you feel to him?

  Not very close

  A little close

  Pretty close

  Very close

Q1.7 – 1.9.  If the respondent has both a mother and someone they view as a female guardian, they should answer for the individual with whom they have the closest relationship.

 

1.8  Does he spend enough time with you?

  Not enough time, but I don’t want more

  Not enough time, I wish she spent more time with me

  She spends enough time with me

 

1.9  Here are some statements about you and your father or male guardian.  For each one, check how true the statement is:

				Mostly True		Some-times True		Hardly Ever True			Does Not Apply
a.		I am comfortable talking with him about things that happen in school
b.		I am comfortable talking with him about things that happen in my life

 

1.10  What is your ethnic background?

  Hispanic or Latino

  Not Hispanic or Latino

 

      1.10 a.  Check the box or boxes that describes your race. CHECK ALL THAT APPLY.

        American Indian

        Asian

        Black or African-American

        Native Hawaiian or Other Pacific Islander

        White

      

Q1.10a.  Any and all combinations of categories can be chosen

 

These next few questions are about your family rules and family communication.

 

1.11  For each of the following, tell us if your family has strict rules, some rules, or no rules about…

				Strict Rules		Some Rules		No Rules
a.		The people I hang out with
b.		Dating and going to parties with boys or girls
c.		Telling my parents or guardian where I am

Q 1.11.  “Family” refers to parent(s) or guardian(s)

 

1.12  Which of the following would your parent(s) or guardian(s) do if they found out that you had broken an important rule? CHECK ALL THAT APPLY.

  Discuss it calmly with you

  Ignore it, pretend that it didn’t happen, or let you get away with it

  Sulk, pout, or give you the silent treatment

  Take away a privilege, ground you, or give you a chore

  Make threats that won’t be kept

  Yell, shout, or scream at you

  Use physical punishment

  None of the above

Q1.12.  These responses need not have actually occurred. It is a question about what the respondents’ expectations would be if their parent(s) or guardian(s) found out they had broken an important rule.

 

1.13  Have you ever talked to one or both of your parents or guardians about any of these things?

				Yes		No
a.		Puberty – that is, the physical changes in your body and your emotions that happen when young people develop
b.		How someone gets pregnant
c.		Why people your age should not drink or do drugs
d.		Why people your age should not have sex
e.		How to say “no” to alcohol, drugs, or sex

Q1.13.  This questions refers to communication initiated by the parent(s) or guardian(s).

 

1.14  How often during the past year have you asked one or both of your parents or guardians questions about the changes in your body, dating, or alcohol and drugs?

  Never

  1 or 2 times

  A few times

  Many times

Q1.14.  This questions refers to communication initiated by the respondent.

Section 2

These next questions are about activities you may do.

 

2.1  Please mark yes or no for each activity that you do in school or after school.

				Yes		No
a.		Band, orchestra, chorus, choir or other music group
b.		School play or musical
c.		Student government or student council
d.		Hobby club like photography or chess
e.		Sports team
f.		Dance or gymnastics
g.		Girl Scouts, Boy Scouts, Girls Clubs, Boys Clubs, or 4-H
h.		After school tutoring program
i.		Community service or volunteer activity
j.		Church or religious youth group
k.		Mostly watch TV or hang out
l.		Some other club

Q2.1.  If the respondent engages in any of these activities, but does not do so presently because of issues of such as seasonality they should still indicate that they do engage in that activity. For example, it is not the soccer season when the questionnaire is administered, but the respondent plays soccer each season (and intends to do so when the season restarts) the respondent should indicate that they participate in a “Sports team”.

Section 3

In this section, there are questions about things people your age have to make decisions about.

 

3.1  At what age do you think it is appropriate for someone to go on a “alone” date, that is, not in a group?

  Less than 14 years old

  14-15 years old

  16 years old

  Don't Know

Q3.1 & 3.2.  Respondents need not have actually done of the activities indicated in each of the questions. The focus is on the respondents’ perception of what they think is appropriate.

 

3.2  How much do you agree with the following statement?
It’s okay for teens to date someone three or more years older or younger than they are.

  Agree strongly

  Agree

  Disagree

  Disagree strongly

  Don't know

The next few questions are about some things that adolescents think about or do.

 

3.3  People your age often feel pressure from a lot of different places and for different reasons. Thinking about yourself, how much pressure have you personally felt to do things you might get in trouble for?

  A lot

  Some

  Not much

  None

  Don't Know

Q3.3.  Respondents need not have actually done anything that got them in trouble. The focus is on the respondents’ perception of the level of pressure they felt to do things they believe are wrong.

 

3.4  How much would you say that the following statement is true about you?
You can say no to activities that you think are wrong.

  Not at all like you

  Little like you

  Mostly like you

  Very much like you

  Don't Know

 

3.5  How much would you say that the following statement is true about you?
You have learned to stay away from people who might get you in trouble.

  Almost never

  Some of the time

  Usually

  Almost always

  Don't Know

 

3.6  During the past month, how often have you smoked cigarettes?

  I have never smoked cigarettes

  Not at all

  Only a few times

  1 or 2 times a week

  Several times a week or more

 

3.7  How often do you drink alcohol, like beer, wine, or liquor?

  I have never had a drink

  I have had alcohol in the past, but do not drink alcohol anymore

  1 or 2 times a month

  Several times a month or more

 

3.8  How many of your friends drink alcohol like beer, wine, or liquor?

  None

  A few

  Some

  Most

Q3.8.  If the respondent is unsure of what their friends have done or do when they are not around, they should answer what they believe is most likely to be true.

 

3.9  How often do you use marijuana or other drugs?

  I have never used marijuana or other drugs

  I have used marijuana in the past, but do not use marijuana anymore

  1 or 2 times a month

  Several times a month or more

 

3.10  How many of your friends have tried marijuana or other drugs?

  None

  A few

  Some

  Most

Q3.10.  If the respondent is unsure of what their friends have done or do when they are not around, they should answer what they believe is most likely to be true.

Section 4

These next questions ask about a variety of things that people your age might think about themselves.

 

4.1  Here are some opinions that adolescents sometimes have about themselves. Please tell us how much you agree or disagree with each one.

				Agree A Lot		Agree A Little		Disagree A Little			Disagree A Lot
a.		I don’t have enough control over the way my life is going.
b.		For me, good luck is more important than hard work for success.
c.		I can't do things as well as most other people.
d.		My plans hardly ever work out.
e.		When I make plans, I know I can make them work.
f.		Chance and luck are important for what happens in my life.
g.		I can do just about anything I really set my mind to.

 

4.2  Please tell us how much you agree or disagree with each of these statements.

				Strongly Disagree		Disagree		Agree			Strongly Agree
a.		You know at least one adult you could talk to about personal problems.
a.		You know adults who encourage you often.

 

4.3  Comparing yourself with your other friends in the group you hang out with, would you say you have more self-confidence, less, or about the same amount?

  More confidence

  Less confidence

  About the same

  Don't Know

Q4.3.  If the respondent is unsure of how their friends feel, they should answer what they believe is most likely to be true.

Section 5

These next questions ask about your future.

 

5.1  As you look to the future, how important is it for you to get a good education?

  Not important at all

  Not too important

  Somewhat important

  Quite important

  Very important

Q5.1 & Q5.2.  There is no set level of academic or professional attainment implied here. This should be answered with regard to what the respondent believes a good education or good job is.

 

5.2  As you look to the future, how important is it for you to get a good job or to be successful in a career?

  Not important at all

  Not too important

  Somewhat important

  Quite important

  Very important

 

5.3  As you look to the future, how important is it for you to remain abstinent (not to have sex) until marriage?

  Not important at all

  Not too important

  Somewhat important

  Quite important

  Very important

 

5.4  As you look to the future, how important is it to you for the person who becomes your spouse to remain abstinent (not to have sex) until marriage?

  Not important at all

  Not too important

  Somewhat important

  Quite important

  Very important

Q5.1 & Q5.2.  This question focuses on what the respondent would expect from someone they would consider marrying.

 

5.5  As you look to the future, how important is it for you to have a good marriage and family life?

  Not important at all

  Not too important

  Somewhat important

  Quite important

  Very important

 

5.6  Here are some things that people your age might say when they think of their lives in the future. In general, do you agree or disagree with each statement?

				Agree		Disagree
a.		I don’t know what I want out of life.
b.		I have a clear picture of what I'd like to be doing in the future.
c.		I don’t know what my long-range goals are.

Section 6

Section 6:  Because these questions, and the definition of sexual activity given in the section, may be considered sensitive, particularly for the youngest clients served in AFL Prevention demonstration projects, the OPA will consider granting a waiver to its use on a case basis and with adequate written justification by the grantee.

The next few questions ask for your opinions on topics about having sex. “Having sex” means oral, anal, and vaginal sex.

 

6.1  Does having sex as a teenager make it harder for someone to study and stay in school in the future?

  No, not harder at all

  Yes, somewhat harder

  Yes, much harder

  Haven't thought about it yet

 

6.2  Does having sex before marriage make it harder for someone to have a good marriage and a good family life in the future?

  No, not harder at all

  Yes, somewhat harder

  Yes, much harder

  Haven't thought about it yet

 

6.3  Does having sex as a teenager make it harder for a teen to grow and develop emotionally and morally?

  No, not harder at all

  Yes, somewhat harder

  Yes, much harder

  Haven't thought about it yet

 

6.4  Is there a problem with unmarried teens having sex if no pregnancy results from it?

  No problem at all

  Some problem

  A big problem

  Haven't thought about it yet

Q6.4.  “Problem” refers to the participants’ belief of right and wrong behaviors.

 

6.5  For each of the following questions please tell us how much you agree or disagree with each one.

				Agree A Lot		Agree A Little		Disagree A Little			Disagree A Lot
a.		Only married people should have sex.
b.		It would be OK for teens who have been dating for a long time to have sex.
c.		Having a good marriage does not seem realistic for me.
d.		I admire teens who remain abstinent (do not have sex) until marriage.
e.		My friends admire teens who remain abstinent until marriage.
f.		Remaining abstinent is the only certain way to avoid pregnancy, STDs, and other related health problems.

 

6.6  How many of your 5 closest friends think someone should wait until marriage before having sex?

  None of them

  One or two of them

  Three or four of them

  All of them

Q6.5e & 6.6.  If the respondent is unsure of how their friends feel, they should answer what they believe is most likely to be true.

Thank you for participating in the questionnaire!

Data Security and Human Subjects Approval Guidelines

All AFL Prevention sites must submit the questionnaire to their site IRB (and HIPAA Privacy Board if the site is a Covered Entity) prior to initiating data collection. The questionnaire data are to be treated as confidential and maintained in a manner that satisfies the confidentiality requirements set forth by their site IRB (and HIPAA Privacy Board if the site is a Covered Entity). To facilitate confidentiality protections, none of the 18 HIPAA designated identifying data elements are collected on the instruments.

Any and all transmission of individual or case level data must also be done in accordance with confidentiality requirements set forth by their site IRB (and HIPAA Privacy Board if the site is a Covered Entity).

Analysis of AFL Core Questionnaire Data

These instruments are designed to meet several research needs. They will allow comparisons of aggregate data across all AFL Prevention Project sites, as well as comparisons of clients’ attitudes, knowledge and behavioral intentions with those collected and reported through other national studies. While the comparisons will be limited in scope because the samples are drawn in different ways, the information gleaned will be valuable for the AFL Prevention Project service providers. Analysis of the data for any site’s required independent evaluation of each project will vary, and be determined by the individual grantees and their evaluators.

Initial anticipated data requests from OAPP

The OPA will require AFL Prevention Demonstration Projects to provide summaries of data on basic demographics and selected questions in the core evaluation instrument in their end-of-year reports. These aggregate data will be used to track progress on the performance measures currently being developed by the OPA in response to OMB’s recommendation.

Crosswalks and comparisons to other national datasets

Many AFL Prevention grantees struggle to find the resources and expertise to support rigorous evaluations that incorporate random assignment or quasi-experimental comparison group designs. Evaluations, in general, and those of programs working with adolescents, in particular, require a good comparison group. Absent this group, it is impossible to tell whether changes in clients are attributable to program participation or other factors (e.g., maturation).

One technique for minimizing the costs and burdens associated with establishing a rigorous control group (i.e., random assignment) or identifying a local comparison group, is to examine program data in light of national survey statistics or norms. Although this approach has its own weaknesses (e.g., local population characteristics and norms may not correspond to characteristics found in a national database), it can inform evaluation designs, such as the pre-post designs with the same participant groups that are found in many AFL Prevention grantee evaluations. For example, although solely descriptive in nature, sites can compare attitudes, knowledge and behavioral intentions on key factors such as risk-taking behaviors pre- and post- program services and see how their populations look compared to other similar surveys.

This is one reason most items in the core evaluation instrument have been drawn from large national surveys that have been successfully administered to youth across the country for many years. Additionally, the instruments were pilot tested at an AFL Prevention Grantee site to ensure that the questions in the instrument are understood by the respondents. The majority of the questions were taken from the Mathematica Policy Research Title V Evaluation Teen Activities and Attitudes Instrument, which, in turn, drew its questions from other national instruments. The national instruments from which the core evaluation instrument questions were drawn are:

• Mathematica Policy Research Title V Evaluation Teen Activities and Attitudes Instrument;
• The National Longitudinal Survey of Adolescent Health (Add Health; 1998);
• Annual National Survey, National Campaign to Prevent Teen Pregnancy (NCPTP);
• National Longitudinal Survey of Youth (NLSY; 1979-98);
• Youth Asset Survey (YAS; 2002)
• National Survey of Children (NSC; 1976-87)
• Kaiser Family Foundation: Talking with Kids about Issues (1996-98)
• National Survey of Adolescent Males (NSAM)
• Handbook of Adolescent Sexuality and Pregnancy: Research and Evaluation Instruments (Sage Publications, 1993)
• National Education Longitudinal Study (NELS; 1988)
• Worth the Wait Survey Instrument (Worth the Wait, Pampa, TX);
• National Survey of Adolescents and Young Adults (NSAYA); and
• National Survey of Teens (NST).

Each of these surveys is regularly administered to adolescents and has publicly available data sets (i.e., for the purpose of examining comparison statistics for specific questions).

Abt and OAPP also worked jointly to develop several items that could not be found in preexisting surveys. Where this occurred, the items were written in a manner that reflected the language level and tone of the items drawn from the other preexisting surveys.

Cross-walk tables to other national data collection instruments

Exhibits 1-3 below detail the original source for each of the questions selected for the instruments. The items are grouped by the primary domains of interest identified by the Title XX legislation:

• Strengthening Family, which includes items about family composition and communication patterns;
• Strengthening Youth Assets, which includes items about adolescent activities, future orientation, self-efficacy, and decision-making; and
• Reducing Sexual Risk Behaviors, which includes items about health behaviors and attitudes about abstinence.

The question numbers are the same for the baseline and follow up surveys, with the exception of Section 1, where some of the demographic items are dropped from the follow up questionnaire. These question numbers are found in the first column of each table for all sections except Section 1, where the question numbers are found in the second and third columns.

Exhibit 4 illustrates the association between the definition of abstinence education in the Title XX legislation (a-h) and its corresponding indicators in the core evaluation instruments.