A Pilot Trial of a Web-based Data Collection Instrument for an Adolescent School-based Intervention: A Promising but Imperfect Fit-Results of a Supplemental OAPP Funding Grant

Linda J. Hulton, RN, Ph.D
Associate Professor of Nursing, James Madison University
Kim Hartzler-Weakley, MS, Assistant Project Director
Jennifer Rea, BSW,   Communications Specialist

Introduction

In recent years, area schools in the Shenandoah Valley implemented a grant-funded school-based educational intervention aimed at improving adolescent decision-making.  Past challenges with the data collection methods have been identified and include lengthy paper/pencil questionnaires, time-consuming data entry, and costly printing.  Previous research from electronic data collection studies with adolescent populations have found that participants answered more honestly in sensitive topics than comparison groups completing paper/pencil questionnaires or face to face interviews. Other advantages of electronic data collection have been reduced turn around time for data analysis and enhanced survey item completion rates.  The purposes of this study were: 1) to develop a web-based self-administered questionnaire and pilot it with two different samples of adolescents; 2) to determine if the scores of the web-based survey were equivalent to those obtained by paper/pencil methods, and 3) to compare longitudinal outcomes of the project using the web-based questionnaire. 

Methods

A one group before-after time series design was used for this study:

            O₁        X        O₂      →      6 mo.  →      O₃

After receiving Institutional Review Board (IRB) approval, a convenience sample of middle school students (n = 25) and alternative school students (n = 30) received a 10 session “Vision of You” interactive theory-based teaching program.  This intervention educated students on the social, psychological, emotional, and physical benefits to be gained by abstaining from sexual activity, alcohol., tobacco, and other drug use.  Parental consent and student assent were required.  The Adolescent Family Life (AFL)  Core Instrument is a 58 item questionnaire which measures specific social and self-control constructs targeted in the curriculum, including Self-Management, Self-efficacy, Future Orientation, STD Knowledge and Substance Use. An electronic survey was developed using Websurveyor© software and hosted on a secure website with SSL protocol.  The web-based questionnaire was administered at the beginning and the end of the 10 session program and again six months post intervention.  Students took the electronic survey in the school computer lab during school hours and then took the paper/pencil survey within the same week.

Results

Several indicators of data quality were examined including response rates, absenteeism, refusals, and data processing time. Results of the scores on the outcome variables at Data Collection #2 demonstrated no significant differences between the web-survey and the paper/pencil surveys. The websurvey had better completion rates overall with less missing values. Both methods demonstrated similar results with the completion of sensitive questions.  Student self-reported scores showed modest positive movement in most of the variables immediately after the intervention, but then had a fade out effect at six months postintervention. Using ANOVA, middle school students had statistical fade out differences in self-efficacy (F=4.39; p=.027) and increased substance use (F=78.9; p=<.00), but increased in self-management scores (F=6.49; p=<.008). Alternative school students had fade out in all constructs, but showed positive effects in decreased use of substances. However, these were not statistically significant.

Discussion and Implications

Documented advantages of the websurvey included automatic data entry, branching patterns for questions, a skip logic, and the potential for the adolescents to feel that their responses were more confidential. There was an estimated cost savings of over $3000/semester due to the efficiency of the web-based methodology. However, limitations were some mismatching of the pre-tests/posttests, a change in the software vendor, and some missing data due to confusion with the student PIN. Also, there remained some significant personnel time in creating the websurvey variables and data cleaning before the analysis. The resulting fade out effects mirror previous longitudinal studies with adolescents.  Implications from the longitudinal findings include the need to provide booster interventions and follow up sessions in adolescent populations that can be difficult to track. These types of psychosocial short-term interventions may yield short term results and multi-year programs are more likely to have more sustained results. 

Contact Information:

Linda Hulton, RN, Ph.D
Associate Professor of Nursing
MSC 4305, James Madison University
Harrisonburg, VA 22807
hultonlj@jmu.edu 
540-568-6883