Efficacy of a multi-risk internet prevention program: Safety.net

Abstract

There are increasingly more prevention and intervention programs concerning internet risks. However, most of them do not consider the breadth of the existing risks of both the relational and dysfunctional use of technology. The main objective of this study was to confirm the effectiveness indicators of the multi-risk internet prevention program (Safety.net). This program, consisting of 16 sessions (1 hour/session), promotes general skills for adequate internet use and prevents eight risks: cyberbullying, sexting, online grooming, cyber dating abuse, problematic internet use, nomophobia, internet gaming disorder, and online gambling. It has a networked instructional design to remember the contents already addressed as as the program progresses. The sample comprised 726 adolescents (54% girls) between 11-14 years (M = 12.11, SD = 0.89). For its evaluation, a pre/post-test repeated-measures design was used with an intervention group (n = 450) and a control group (n = 276). The intervention group showed significant improvements compared to the control group in peer cybervictimization, cyber dating victimization, sexual solicitation/interaction with adults, problematic internet use, and nomophobia. These results suggest that the Safety.net program effectively prevents the increase of most internet risks assessed through a reduced number of sessions. This is a potential psychoeducational tool to be integrated into tutorial action plans.

Resumen

Cada vez existen más programas de prevención e intervención sobre riesgos de internet. Sin embargo, la mayoría de ellos no tienen en cuenta la amplitud de los riesgos existentes tanto en el uso relacional como disfuncional de la tecnología. El principal objetivo de este estudio ha sido confirmar los indicadores de eficacia del programa de prevención multirriesgo de internet (Safety.net). Este programa, que consta de 16 sesiones (1 hora/sesión), promueve competencias generales para un adecuado uso de internet, y previene ocho riesgos: ciberacoso, sexteo, ciberembaucamiento, abuso online en la pareja, uso problemático de internet, nomofobia, trastorno por juego en internet y trastorno de apuestas online. Tiene un diseño instructivo en red para recordar los contenidos ya abordados a medida que avanza el programa. La muestra está formada por 726 adolescentes (54% chicas) entre 11–14 años (M = 12.11, DT = 0.89). Para su evaluación se ha utilizado un diseño de medidas repetidas pre/post-test con un grupo de intervención (n = 450) y un grupo de control (n = 276). El grupo de intervención muestra mejoras significativas en comparación con el grupo de control en cibervictimización entre iguales y en la pareja, solicitación/interacción sexual con adultos, uso problemático de internet y nomofobia. Estos resultados sugieren que el programa Safety.net puede ser eficaz para prevenir el aumento de la mayoría de los riesgos de internet evaluados mediante un número reducido de sesiones. Se trata de una herramienta psicoeducativa potencial para integrar en los planes de acción tutorial.

Introduction

The widespread use of relationship, information, and communication technologies (RICT) and their continuous increase worldwide has implied a change in interpersonal communication in adolescence. According to the results found by the Pew Research Center, 95% of adolescents have their own smartphone or access to this device, even if it is not their own, and 45% claim to be online almost constantly (Anderson & Jiang, 2018). In 2022, the average consumption was almost eight daily hours of the internet (WeAreSocial, 2022). In Spain, 96% of 15-year-olds have access to the internet from their smartphone (Instituto Nacional de Estadística, 2021); of them, 61% have access throughout the day, including nighttime (Díaz-López et al., 2020). In addition, 33% admit to using it at all hours (Díaz-López et al., 2020). Although the internet offers a window of possibilities with multiple potential benefits for these "always on" generations, it also involves many risks. González-Cabrera and Machimbarrena (2023) divide these risks into two blocks: relational and dysfunctional. The former are generated by the minor’s interaction with other people through the internet, in many cases producing victimization problems. In this sense, a recent UNICEF report indicates a prevalence of 22.5% in cybervictimization and 7.9% in cyberaggression among peers (Andrade et al., 2021). In addition, according to this report, 8% and 26.8% send and receive sexting (respectively), and one in ten adolescents received a sexual proposition from an adult (Andrade et al., 2021). In the case of cyber dating abuse, the perpetration and victimization ranges between 8.1%–93.7%, and 29%-94.8%, respectively (Caridade et al., 2019). On the other hand, dysfunctional risks are those derived from the inappropriate use of RICTs that can generate negative consequences for the person (González-Cabrera & Machimbarrena, 2023). Regarding general problematic internet use, its prevalence varies between 14.3% and 54.9% of minors (Laconi et al., 2018), and between 6.1% and 26.7% present severe cases of nomophobia (intense, irrational, and disproportionate fear of not being able to use the smartphone) (León-Mejía et al., 2021). According to the meta-analysis of Fam (2018), 4.6% of adolescents have an internet gaming disorder, and between 0.77% and 57.5% have some degree of problem with online gambling (persistent gambling with money that can cause clinically significant discomfort) (Montiel et al., 2021).

In recent years, the relationship and overlap of some cyber risks, such as peer cybervictimization, sexting, and online grooming, have also been revealed (Calvete, Fernández-González, González-Cabrera et al., 2020; González-Cabrera et al., 2021; Machimbarrena et al., 2018; Montiel et al., 2016). Ortega-Barón et al. (2023) show that 2.7% of adolescents (10-15 years) simultaneously overlapped three types of online victimization (peer cybervictimization, cyber dating abuse, and solicitation and sexualized interaction with adults). Although there is no record of studies evaluating the overlap of all the dysfunctional risks discussed, the scientific literature also shows a significant relationship between some of these (Beranuy et al., 2020; González-Cabrera, Machimbarrena, Beranuy, Pérez-Rodríguez et al., 2020) and the overlap between Problematic Internet Use (PIU) and Internet Gaming Disorder (IGD) (Machimbarrena et al., 2023).

In the educational context, many initiatives have been carried out nationally and internationally to address several of these problems. However, most existing programs have focused on prevention or intervention in a single specific risk, such as cyberbullying (Gaffney et al., 2019), cyber dating abuse (Carrascosa et al., 2019), online grooming (Davidson et al., 2009), PIU (Díaz Salabert & Gómez Torres, 2019), IGD (Giménez Lozano & Morales Rodríguez, 2022) or nomophobia (Khosravi et al., 2021). In this sense, few programs focus on preventing the combination of some risks; for example, Cyberprogram 2.0, Prev@cib or Brief Preventive Intervention prevent cyberbullying, sexting and/or online grooming (Cortazar et al., 2021; Garaigordobil & Martínez-Valderrey, 2014; Ortega-Barón et al., 2019). Do not cover the whole spectrum and do not address relational and dysfunctional risks jointly. Moreover, most of them do not focus on highly recommended age ranges for primary prevention work (11-13 years) (Yeager et al., 2015). To fill this gap in the literature, the Safety.net program was created (Ortega-Barón et al., 2021).

Safety.net program

The program’s objective is to primarily prevent a set of relational (cyberbullying, sexting, online grooming, cyber dating abuse) and dysfunctional risks of internet use (PIU, IGD, online gambling, and nomophobia) in adolescents aged 11 to 14 years. The effectiveness of the Safety.net program were examined in a pilot study conducted by Ortega-Barón et al. (2021) with a sample of 165 adolescents aged 11-14 years. The results of this study indicated lower scores in the intervention group, which had received the program's backbone modules, compared to the control group on several of these risks (online grooming, problematic internet use, internet gaming disorder, and nomophobia). This study was considered exploratory due to its small sample and the impossibility of implementing it in its entirety due to home confinement during the COVID-19 pandemic. Nonetheless, the program's effectiveness was observed in such an adverse context when adolescents spent most of their time connected to the internet (Ortega-Barón et al., 2021). That said, and given these preliminary positive results, the necessary application of the complete program to a larger sample and in a context of greater educational normality is warranted.

The Safety.net program is based on four theoretical frameworks, which are set out in more detail in the pilot version of Ortega-Barón et al. (2021): (1) the theory of planned behavior (Ajzen, 2011); (2) the model of social co-construction (Subrahmanyam & Smahel, 2011): (3) the cumulative risk model (Evans et al., 2013); and (4) the theory of empowerment (Zimmerman, 2000). This program consists of 16 one-hour sessions, divided into four modules: (1) Digital competences (general module); (2) Relational risks; (3) Dysfunctional risks; and (4) Change of attitudes and thoughts (to see all the information, consult Ortega-Barón et al., 2021). Each session contains an activity to be carried out to internalize the concepts and promote the change of thoughts, attitudes, and behaviors. It also has some recall sessions recorded by students of the same age in key sessions that act as cybertips. Concerning the previous version of the program, after the pilot version, we followed the suggestions for improvement collected at a qualitative level (Ortega-Barón et al., 2021), such as the reduction of some content and the improvement in the explanation of cyber dating abuse or online gambling.

Due to the reduced number of sessions for each content and to allow acquiring knowledge about risks and preventive skills, the program follows an instructional design (Merrill, 1987). The principles of the program are: (1) activate prior knowledge in each session in a networked format; (2) focus on the most important elements in each session; (3) distribute the cognitive load intrasession and intersessions; and (4) use meaningful materials/activities (Clark, 2008). The recommendations of Garaigordobil and Martínez-Valderrey (2014) for inter-sessional consistency, staff, and session structure were also followed to ensure adequate program implementation.

Considering these antecedents, this study aimed to analyze the effectiveness of the Safety.net program. After the pilot study by Ortega-Barón et al. (2021), we hypothesized that decreasewould be found in the intervention group compared to the control group in the following risks: online grooming, PIU, IGD, and nomophobia. For the rest, we hypothesized that the remaining relational and dysfunctional risk scores would not increase in the intervention group compared to the control group. We also expected that adolescents in the intervention group would have more knowledge about cyber risks than the control group and that the intervention group would express positive satisfaction with the learning/improvement achieved through the program.

Method

Design and participants

To evaluate the effectiveness of the Safety.net program, a two-stage repeated-measures (pre-test and post-test) study was designed with an intervention group (n = 450) that received the program and a control group (n = 276) that only completed the questionnaires in both stages (pre-test and post-test). The total sample of this study was composed by 726 adolescents aged 11-14 years (M = 12.05, SD = 0.86) from schools in five Spanish regions (Aragon (9.8%, n = 71), Castilla la Mancha (13.9%, n = 101), Castilla y León (14.2%, n = 103), Community of Madrid (31.6%, n = 230), and Principality of Asturias (30.5%, n = 221). The sample was obtained by convenience sampling (Creswell, 2003). For each center, the participating classes were paired in which one group was intervention and one control (if the number of classes per class was odd, it was balanced between the participating centers). All information about the participants (intervention group and control group) at T1 (pre-test) and T2 (post-test) can be seen in Table 1 and Figure 1. There were only significant differences depending on the academic grade, χ²_{(2, 726)} = 52.58, p < .001). There was a lower percentage of students from 2nd grade of Compulsory Secondary Education (CSE) because we tried to increase the group of 6^th grade of Primary Education to maximize primary prevention.

Table 1.

Characteristics of the intervention and control group according to age, sex and academic year: frequency and (percentage)

Variables	Intervention group (n = 450)	Control group (n = 276)	T test and Chi Square
Age M (SD)	11.98 (0.88)	12.15 (0.82)	t = -2.612, p = .009
Sex			χ^2 = .802, p = .370
Boys	211 (46.9%)	120 (43.5%)
Girls	239 (53.1%)	156 (56.5%)
Academic grade			χ^2 = 52.58, p < .001
6th grade of PE	235 (52.2%)	70 (25.3%)
1 st grade of CSE	145 (32.2%)	126 (45.7%)
2nd grade of CSE	70 (15.6%)	80 (29%)

Note. Age (M = Mean; SD = standard deviation); PE = Primary Educations; CSE = Compulsory Secondary Education; χ² = Chi Square; t = T test.

Figure 1.

Diagram of the sampling process between pre-test and post-test.

Based on the results obtained in the pilot test (Ortega-Barón et al., 2021), a calculation of the minimum sample size necessary to detect differences between the groups with a mean effect size and a statistical power of 95% was performed using the GPower program. The participation of at least 155 adolescents in each group was estimated to be necessary.

Instruments

Adolescents in this research reported sociodemographic aspects (age, sex, or academic grade). In addition, eleven items (correct/incorrect) were designed to assess the essential content to be learned in Safety.net (Table 2). Eleven items were also designed for the occasion to assess the intervention group's self-assessment of learning/improvement and satisfaction after the application of Safety.net. The response was rated on a range of 0-100. The rest of the evaluation instruments were:

Table 2.

Correct answers in the knowledge of the internet risks addressed in Safety.net

Item	Pre-test f (%)	Post-test f (%)	McNemar test (χ^2, p, η^2)
Intervention group	Correct answer	Correct answer
1. Netiquettes are the options that are available to label other people on social networks.	7.8%	37.6%	χ^2 = 101.661; p < .001; η^2 = .01
2. Privacy is controlling the information you want your contacts to see on social networks.	64%	73.8%	χ^2 = 11.139; p = .001; η^2 = .04
3. In a case of cyberbullying, cyber-observers are also part of the problem.	57.8%	84.2%	χ^2 = 73.672; p < .001; η^2 = .01
4. Sexting is taking sexy photos or videos of oneself to send to someone.	31.3%	74.2%	χ^2 = 143.440; p < .001; η^2 = .01
5. Online grooming is when someone impersonates you to harass others on social media.	5.8%	30.9%	χ^2 = 88.965; p < .001; η^2 = .04
6. Cyber-dating is when two people meet on the internet, and they go out on a date.	4%	16.2%	χ^2 = 32.764; p < .001; η^2 = .10
7. Playing online video games for more than three hours a day and almost every day of the week is problematic.	71.3%	81.1%	χ^2 = 13.207; p < .001; η^2 = .05
8. Online gambling is when you play an online video game with other people.	11.6%	25.1%	χ^2 = 30.769; p < .001; η^2 = .08
9. Nomophobia is defined as the intense fear of not being able to use your mobile.	19.6%	24%	χ^2 = 226.982; p < .001; η^2 = .03
10. FoMO (Fear of Missing Out) is when you are afraid of being excluded from a group on the internet.	3.3%	11.3%	χ^2 = 19.141; p < .001; η^2 = .03
11. Whatever is published on the internet is forever; it can never be erased.	42.9%	68%	χ^2 = 76.957; p < .001; η^2 = .19
Control group
1. Netiquettes are the options that are available to label other people on social networks.	10.9%	15.6%	χ^2 = 3.512; p = .061
2. Privacy is controlling the information you want your contacts to see on social networks.	62%	68.1%	χ^2 = 2.639; p = .104
3. In a case of cyberbullying, cyber-observers are also part of the problem.	61.2%	70.7%	χ^2 = 8.013; p = .005
4. Sexting is taking sexy photos or videos of oneself to send to someone.	46.7%	45.7%	χ^2 = 0.037; p = .848
5. Online grooming is when someone impersonates you to harass others on social media.	10.9%	10.5%	χ^2 = 0.000; p = 1.00
6. Cyber-dating is when two people meet on the internet, and they go out on a date.	6.9%	5.8%	χ^2 = 0.148; p = .700
7. Playing online video games for more than three hours a day and almost every day of the week is problematic.	67.4%	69.9%	χ^2 = 0.493; p = .483
8. Online gambling is when you play an online video game with other people.	12.7%	9.8%	χ^2 = 1.167; p = .280
9. Nomophobia is defined as the intense fear of not being able to use your mobile.	29%	35.1%	χ^2 = 3.241; p = .072
10. FoMO (Fear of Missing Out) is when you are afraid of being excluded from a group on the internet.	7.2%	3.3%	χ^2 = 3.704; p = .054
11. Whatever is published on the internet is forever; it can never be erased.	43.8%	52.5%	χ^2 = 8.817; p = .003; η^2 = .36

Note. χ² = Chi Square; η² = Partial square eta.

Cyberbullying Triangulation Questionnaire (CTQ; González-Cabrera et al., 2019). This scale is composed of nine items assessing peer-to-peer cybervictimization and nine items assessing peer-to-peer cyberaggression. Responses on this Likert-type scale range from 0 = never to 4 = almost every week. For the cybervictimization dimension the Cronbach's alpha coefficient (α) was .71 (pre-test) and .74 (post-test); the Omega was .74 (pre-test) and .77 (post-test), the AVE was .47 (pre-test) and .56 (post-test) and the CR was .89 (pre-test) and .92 (post-test). For cyberaggression dimension the Cronbach's alpha coefficient (α) was .68 (pre-test) and .80 (post-test); the Omega was .71 (pre-test) and .81 (post-test), the AVE was .51 (pre-test) and .64 (post-test) and the CR was .90 (pre-test) and .94 (post-test).

Sexting questionnaire (Gámez-Guadix et al., 2015). This scale is composed of three items that assess whether minors send information or messages of sexual or intimate content to three possible recipients: (1) partner, (2) friend or acquaintance, and (3) someone they have met online, but not in person. The response range of this scale is from 0 = never to 4 = seven or more times. Cronbach's alpha coefficient (α) was .79 (pre-test) and .68 (post-test); the Omega was .75 (pre-test) and .77 (post-test), the AVE was .58 (pre-test) and .79 (post-test) and the CR was .80 (pre-test) and .92 (post-test).

Questionnaire for Online Sexual Solicitation and Interaction of Minors with Adults (Gámez-Guadix et al., 2018). This 10-item scale assesses both solicitation and sexual interactions by an adult with a minor. The response range of this scale is from 0 = never to 3 = six or more times. Cronbach's alpha coefficient (α) was .92 (pre-test) and .86 (post-test). the Omega was .92 (pre-test) and .83 (post-test), the AVE was .86 (pre-test) and .81 (post-test) and the CR was .98 (pre-test) and .98 (post-test).

Online Intimate Partner Abuse Questionnaire (Calvete, Fernández-González, Orue et al., 2020). This 11-item scale evaluates behaviors of control and direct aggression to the partner through the internet and cell phones, both from the perspective of victimization and aggression. The response options of this Likert-type scale range from 0 = never to 3 = almost always. For the victimization dimension the Cronbach's alpha coefficient (α) was .83 (pre-test) and .89 (post-test). The Omega coefficient, AVE and CR of the Online Intimate Partner Abuse Questionnaire victimization and aggression scale were not calculated as the variance of some items was equal to 0 and CFA could not be performed.

Generalized and Problematic Internet Use Scale (GPIUS2; Gámez-Guadix et al., 2013). This 15-item scale assesses the dimensions compulsive use, preference for online social interaction, cognitive preoccupation, mood regulation, and negative consequences. Responses on this Likert-type scale range from 0 = totally disagree to 5 = totally agree. Cronbach's alpha coefficient (α) was .92 (pre-test) and .94 (post-test); the Omega was .92 (pre-test) and .94 (post-test), the AVE was .60 (pre-test) and .67 (post-test) and the CR was .96 (pre-test) and .97 (post-test).

Internet Gaming Disorder Scale–Short-Form (IGDS9‐SF; Beranuy et al., 2020). This 9-item scale assesses internet gaming disorder in adolescents according to DSM-5 criteria. The response options of this Likert-type scale range from 0 = never to 4 = very often. Cronbach's alpha coefficient (α) was .85 (pre-test) and .90 (post-test); the Omega was .85 (pre-test) and .90 (post-test), the AVE was .54 (pre-test) and .67 (post-test) and the CR was .91 (pre-test) and .95 (post-test).

Nomophobia Questionnaire (NMP-Q; León-Mejía et al., 2020). It consists of 20 items that assess the irrational and disproportionate fear of not being able to use your cell phone or running out of coverage and/or battery. Response options range from 0 = totally disagree to 6 = totally agree. Cronbach's alpha coefficient (α) for this study was .95 (pre-test) and .96 (post-test); the Omega was .95 (pre-test) and .97 (post-test), the AVE was .61 (pre-test) and .73(post-test) and the CR was .97 (pre-test) and .98 (post-test).

Online Gambling Disorder Questionnaire (OGD‐Q; González-Cabrera et al., 2020). This 11-item scale evaluates the problem of online gambling disorder in adolescence. The response range of this scale is from 0 = never to 4 = every day. Cronbach's alpha coefficient (α) for this study was .85 (pre-test) and .95 (post-test); the Omega was .85 (pre-test) and .95 (post-test), the AVE was .62 (pre-test) and .85 (post-test) and the CR was .94 (pre-test) and .98 (post-test).

Procedure

All the necessary agreements were made with the educational centers to carry out the study. Families whose children were included in the intervention groups received an informed consent form that had to be signed by all legal guardians in order to participate. In the case of families with students in the control group, passive consent was obtained through an official channel. This consent also indicated the sponsor of the study, the funding, the purpose of the study, the risks (in this case there are none), the duration of the study, the principles of voluntariness, anonymity, and disinterestedness were respected (participants were not paid). An email address of a researcher was also provided and three online meetings were proposed. If a family did not agree to the child’s participation, they could return it signed (this occurred in slightly less than 2% of the cases). The children involved in the control group were able to participate in the program in the following school year (when the schools included it in their tutorial action program in Compulsory Secondary Education). The Research Ethics Committee of the Universidad Internacional de La Rioja approved both the program's implementation and the performance of this research. The juvenile prosecutor's offices in each region were informed of the study.

Initially, the teachers underwent online training of approximately 30 hours to implement the program correctly, and a website was created where all the integrated materials (guide, sessions, activities, etc.) were located: http://www.programasafety.net. The Safety.net program was implemented between January and June 2021 in schools during tutoring hours but without affecting any teaching activity. During this period, there was no confinement due to COVID-19, but there were social restriction measures and a lower ratio of students per class. To verify the effectiveness of the Safety.net program, a battery of instruments was administered both to the intervention and control groups before (January 2021) and after (June 2021) implementing the program (approximately 20-30 days after completing the program). The questionnaires were completed online through Survey Monkey© under the supervision of a teacher in the school. In addition, monthly follow-up meetings were held with each school to ensure proper implementation of the program.

Data analysis

Statistical analyses were conducted using the Statistical Package for the Social Sciences (SPSS) 25 (IBM®) program. To determine the internal consistency of the instruments employed, the Cronbach’s alpha, McDonald’s Omega, Average Variance Explained (AVE) and Composite Reliability (CR) coefficients were estimated. For the calculation of the differences in the socio-demographic variables, the chi-square statistic and Student's t-test were used. No approximations were made using hierarchical models due to the almost zero variability at level two (schools) (Hox, 2002). Hence, repeated-measures analysis of variance (ANOVA 2 × 2) was used with an inter-group factor (intervention group and control group) and an intra-subject factor (before and after the program: pre-test and post-test) to evaluate the effectiveness of the Safety.net program. The variable age has been introduced as a covariate to control for possible effects. This statistical analysis is recommended when the selected groups are natural and unequal at pre-test (Weinfurt, 2000), as is the case of sexting (t = 2.60, p = .010) and PIU (t = 3.75, p < .001), where the intervention group obtained higher scores. The eta-square statistic (η²) was used as an indicator of the amount of improvement gained (η² < .06 was considered small, η² between ≥ .06 and ≤ .13 was intermediate, and η² ≥ .14 was large) (García et al., 2008). Additionally, to analyze changes in knowledge, a McNemar test was conducted to compare correct and incorrect answers in the pre-post and to determine possible differences between them. The effect size was calculated by transforming the odds ratio into the eta-square statistic (η²) (Lenhard & Lenhard, 2022). A value of p ≤ .05 was considered significant.

Results

Effects of the Safety.net program on internet risk knowledge

Table 2 shows that the percentage of correct answers increased significantly between the pre- and post-test in the intervention group. This trend was not observed in the control group, except for item 11. These results suggest that the intervention group retained the key contents of each session. Specifically, as can be seen in the McNemar test in Table 2, the increase in knowledge in the intervention group is more significant in item 6 (χ² = 32.764, p < .001) and item 8 (χ² = 30.769, p < .001) with an intermediate effect size (η² = .10 and η² = .08, respectively), and in item 11 (χ² = 76.957, p < .001) with a large effect size (η² = .19).

Effects of the Safety.net program on relational internet risks

Table 3 shows that the time x group interaction had a significant effect on peer cybervictimization and solicitation and sexualized interactions with adults, F_{(1, 719)} = 12.10, p = .001; F_{(1, 698)} = 4.23, p = .040, respectively; with a small effect size (η² = .02; η² = .01, respectively). Thus, whereas the intervention group decreased its scores on these variables after the program, the control group increased them. Regarding the variable cyber dating abuse victimization, the effect of time x group was significant, F_{(1, 79)} = 3.78, p = .050, η² = .05. Specifically, both groups increased their scores, but the increase in the intervention group was much less pronounced (Figure 2). There were no effects in peer cyberaggression or the romantic partner context (Table 3). However, there was a tendency to decrease these scores in the intervention group and increase them in the control group (Figure 2). Finally, concerning sexting, neither was the time x group interaction significant, F_{(1, 692)} = 3.20, p = .074; η² = .01. Concretely, it can be observed that although in the pretest the intervention group had higher scores compared to the control group, in both groups there was a slight increase, less pronounced in the intervention group.

Table 3.

Intergroup effects and repeated-measures analysis of variance (ANOVA 2 × 2) in online relational risks controlling for the age variable

Variables	M (SD)			F (η^2)
	Group	Pre-test	Post-test	Time effect	Group effect	Interaction effect
Cybervictimization	Intervention	2.13 (3.07)	1.64 (2.79)	4.43* (.01)	0.60 (.00)	12.10** (.02)
	Control	1.95 (2.75)	2.22 (3.12)
Cyberaggression	Intervention	0.92 (1.73)	0.78 (2.00)	0.02 (.00)	0.25 (.00)	2.20 (.00)
	Control	0.88 (1.85)	1.01 (2.74)
Cyber dating abuse (victimization)	Intervention	0.67 (1.56)	0.92 (2.12)	0.00 (.00)	3.18 (.04)	3.78* (.05)
	Control	0.68 (1.49)	2.35 (4.59)
Cyber dating abuse (aggression)	Intervention	0.81 (2.20)	0.64 (1.82)	0.52 (.01)	0.27 (.00)	1.63 (.02)
	Control	0.26 (0.58)	0.87 (2.46)
Solicitation and sexual interaction with adults	Intervention	0.49 (2.36)	0.29 (1.48)	0.02 (.00)	0.40 (.00)	4.23* (.01)
	Control	0.27 (1.28)	0.44 (1.76)
Sexting	Intervention	0.12 (0.75)	0.37 (0.41)	5.73* (.01)	1.76 (.00)	3.20 (.01)
	Control	0.03 (0.27)	0.53 (0.47)

Note. η ² = Partial square eta.

*

p < .05.

**

p < .01.

Figure 2.

Means obtained by intervention and control group in online relational internet risks.

Effects of the Safety.net program on dysfunctional internet risks

Regarding PIU, a time x group interaction, F_{(1, 688)} = 16.22, p < .001 and significant main effects in group and time effect (Table 4), with a small effect size (η² = .03; η² = .01, η² = .01, respectively), were observed (Figure 3). For IGD, the effect of time x group was not significant, F_{(1, 560)} = 2.27, p = .133. Thus, although there was a decrease in scores in both groups, it was more pronounced in the intervention group. In nomophobia, a significant time x group interaction effect was also observed, F_{(1, 674)} = 16.02, p < .001 with a small effect size (η² = .03), and significant effects were also observed in the main time and group effects (Table 4). No significant effect, either main or interaction, was observed in online gambling (Table 4 and Figure 3).

Table 4.

Intergroup effects and repeated-measures analysis of variance (ANOVA 2 × 2) in online dysfunctional risks controlling for the age variable

Variables	M (SD)			F (η^2)
	Group	Pre-test	Post-test	Time effect	Group effect	Interaction effect
Problematic Internet use	Intervention	22.48 (16.46)	19.46 (15.86)
	Control	18.05 (13.57)	19.34 (16.63)	5.25* (.01)	7.29** (.01)	16.22*** (.03)
Internet gaming disorder	Intervention	5.18 (6.01)	3.97 (5.90)
	Control	4.47 (5.11)	3.93 (5.09)	1.53 (.00)	1.17 (.00)	2.27 (.00)
Online gambling	Intervention	5.17 (5.77)	4.39 (6.29)
	Control	6.00 (10.12)	1.20 (1.64)	2.73 (.12)	0.48 (.00)	0.66 (.03)
Nomophobia	Intervention	34.88 (25.65)	23.73 (24.40)
	Control	35.70 (27.86)	33.05 (30.34)	10.06** (.02)	5.53* (.01)	16.02*** (.03)

Note. η² = Partial eta squared.

*

p < .05.

**

p < .01.

***

p < .001.

Figure 3.

Means obtained by intervention and control group in online dysfunctional internet risks.

Self-assessment of learning and improvement and participant satisfaction with the Safety.net program

Table 5 shows that in all the items, the adolescents who received the program generally perceived learning and improvement after its completion (the highest percentages were 80 — 100%). Regarding satisfaction with the program, 73.6% stated, “Yes, I liked it a lot” (331 participants), 9.8% said, “No, I did not like it” (44 participants), and 13.8% said, “It's all the same to me” (62 participants).

Table 5.

Self-assessment of the intervention group's learning/improvement and satisfaction after the implementation of the Safety.net

Items	Score 0-100 [number of participants (%)]
1. After the program, I am more aware of the risks of the internet and how they can affect me or other classmates.	LSL: 0-50 [51 (11.3%)]; MSL: 51-79 [62 (13.8%)];
	HSL = 80-100 [205 (45.6%)]
2. Thanks to the program and the cyber tips, I have learned recommendations to be safer on the internet and know what to do in the face of the different cyber risks that currently exist.	LSL: 0-50 [58 (12.9%)]; MSL: 51-79 [63 (14%)];
	HSL = 80-100 [193 (42.9%)]
3. I am now more aware that netiquette helps us avoid misunderstandings and helps us to have better relationships with the people we talk to on the internet, social networks, or mobile.	LSL: 0-50 [71 (15.8%)]; MSL: 51-79 [80 (17.8%)];
	HSL = 80-100 [162 (36%)]
4. I have learned that setting privacy on our social networks allows us to control who sees our publications, and that makes us safer and helps to avoid problems.	LSL: 0-50 [45 (10%)]; MSL: 51-79 [62 (13.8%)];
	HSL = 80-100 [207 (46%)]
5. I am now more aware of the damage and negative consequences that cyberbullying has on victims and aggressors, and, in addition, I have a better understanding of the observers’ critical role.	LSL: 0-50 [41 (9.1%)]; MSL: 51-79 [39 (8.7%)];
	HSL = 80-100 [233 (51.8%)]
6. I have learned what sexting is and why it is dangerous to send compromising and sexual photos to other people.	LSL: 0-50 [31 (6.9%)]; MSL: 51-79 [33 (7.3%)];
	HSL = 80-100 [246 (54.7%)]
7. I understand better what grooming is, and what I can do if an adult blackmails me with photos of me of sexual content on the network (for example: save the evidence and go immediately to an adult).	LSL: 0-50 [39 (8.7%)]; MSL: 51-79 [59 (13.1%)];
	HSL = 80-100 [212 (47.1%)]
8. I am more aware that the control of and contempt for one's partner through the internet sometimes begins in very subtle ways, but that, little by little, they aggravate the situation.	LSL: 0-50 [45 (10%)]; MSL: 51-79 [73 (16.2%)];
	HSL = 80-100 [197 (43.8%)]
9 I am more aware of the behaviors that can cause problems when we connect too much to the internet, and I have learned some recommendations to avoid being so hooked.	LSL: 0-50 [54 (12%)]; MSL: 51-79 [68 (15.1%)];
	HSL = 80-100 [192 (42.7%)]
10. I have learned that, although playing a video game is O.K., it can also generate situations in which you obsess and play too much, producing negative consequences.	LSL: 0-50 [48 (10.7%)]; MSL: 51-79 [63 (14%)];
	HSL = 80-100 [201 (44.7%)]
11. Now I understand better that if a classmate is dealing with a risk on the internet and having a hard time, you should help them solve this problem.	LSL: 0-50 [50 (11.1%)]; MSL: 51-79 [72 (16%)];
	HSL = 80-100 [189 (42%)]

Note. LSL (0-50) = Low self-perception of learning or improvement; MSL (51-79) = Intermediate self-perception of learning or improvement; HSL (80-100) = High self-perception of learning or improvement. The highest percentage of students' improvement/learning is indicated in bold.

Discussion

The main objective of this study was to evaluate the effectiveness of the Safety.net program in a more normalized context and with a larger sample of participants than its previous implementation in a pilot sample. It was designed to prevent relational and dysfunctional internet risks jointly. To date, the authors are not aware of other similar initiatives since the study by Ortega-Barón et al. (2021) that have covered such a broad spectrum of internet risks to carry out a global prevention of the most prevalent and problematic problems for adolescence. In addition, the multi-risk prevention approach is increasingly necessary because more and more studies refer to the overlap or accumulation of internet risks (Calvete, Fernández-González, González-Cabrera et al., 2020; González-Cabrera et al., 2021; Machimbarrena et al., 2018; Montiel et al., 2016; Ortega-Barón et al., 2023). Moreover, each risk is addressed in one session (minimal temporal impact), and the networked instructional design allows critical content to be remembered and saturated as the sessions progress.

In a previous program implementation, positive preliminary results were observed in some risks, such as online grooming, PIU, IGD and nomophobia (Ortega-Barón et al., 2021). The results suggest that efficacy indicators have been found in more problems than those hypothesized. On the one hand, cybervictimization among peers was significantly reduced in the intervention group (compared to an increase in the control group). This represents an improvement in the program given that in the previous pilot study, although there were no significant differences, the increase in this problem in the intervention group was less pronounced than in the control group (Ortega-Barón et al., 2021). Along the same line, there was also a post-test decrease in online grooming regarding sexual solicitations/interactions with adults in the intervention group. In this second study, there are also indicators of a reduction in cyberdating abuse victimization in the intervention group compared to the control group. In online partner abuse, cybervictimization increased at post-test, but was less pronounced than in the control group. This could be explained by the inexperience in this type of affective-sexual relationship in early adolescence, where certain myths of romantic love are normalized (Cava et al., 2020). No significant effects are observed after the Safety.net program on reducing sexting. This may be largely due to the fact that as age increases experiences and exploration of sexuality are typical at this stage (Englander & McCoy, 2018). There was also no significant group effect in the pre/post-test of cyberaggression among peers and in the partner context. This shows the need to provide the Safety.net program with more content on the role of the aggressor in these two sessions, given that the current version places more emphasis on the role of the victim.

Concerning dysfunctional risks previous data are ratified, the scores of PIU and nomophobia are reduced in the intervention group when comparing pre-test and post-test (Ortega-Barón et al., 2021). For IGD, no significant effects were observed after the implementation of the program, in contrast to the pilot study that did show a significant effect. This may be due to the fact that in the pilot study during confinement the time spent playing video games was longer compared to the post-pandemic situation (Hodgetts et al., 2023). Neither in the pilot study nor in this version were there positive results for online gambling. This can be explained because this program mainly targets students aged 11-13 years, where betting behaviors are residual (González-Cabrera, Machimbarrena, Beranuy, Pérez-Rodríguez et al., 2020). We have almost no data for these age ranges, as the national reports collect data on ages between 14-18, with a prevalence of online gambling of 9.4% for this age range (Observatorio Español de las Drogas y las Adicciones, 2022). One way to improve this could be to connect online gaming with loot boxes (as they are a covert bet within video games and are frequent in this age bracket) (González-Cabrera et al., 2022). The most significant effect of Safety.net is the reduction of nomophobia between pre-test and post-test in the intervention group compared with the control group. According to Khosravi et al. (2021), nomophobia is a booming problem in adolescence and, therefore, the Safety.net program can have a very positive effect on its prevention, considering that the age range with the highest prevalence is usually 12-16 years (León-Mejía et al., 2021).

Although a universal primary prevention program should have as a critical objective the prevention of the intervention group’s developing the addressed behaviors/risks, in the Safety.net program, we also observed that it behaves potentially as a secondary prevention program by reducing some problems. It should be noted that indicated interventions should pre-identify and select potential participants based on the presence of symptoms (Horowitz & Garber, 2006), and Safety.net has been universally implemented without this prerequisite. However, it has shown a reduction in problems that were already prevalent in the intervention group at pre-test. In general, in Spanish adolescents, the prevalence of these risks is high (Andrade et al. 2021), and there is also an overlap of many of these risks (Calvete, Fernández-González, González-Cabrera et al., 2020; Machimbarrena et al., 2018; Ortega-Barón et al., 2023). In general, many internet risks also tend to increase as adolescence progresses (as also seen in this study with the control group). This is particularly remarkable in the transition from early adolescence (11-13 years) to middle adolescence (14-17 years). This may at least be due to increased smartphone use (in time and type of use) and reduced parental mediation actions by families (Díaz-López et al., 2020).

After the implementation of the Safety.net program, the results indicate an improvement in the intervention group's knowledge. This aspect is also crucial, as reflected in other programs on internet risks, such as Webquest or Survivors, although they are mainly focused on cyberbullying (Amse, 2014; Lee et al., 2013). According to Espelage and Hong (2017), although adolescents have more truthful information, it is not always associated with less risky online behavior. In this sense, besides the knowledge of cyber risks, attitudinal and behavioral changes are also needed for healthy internet use.

This study also has some limitations: (1) the results were based on self-reports, so there may be bias in the responses; (2) although there was no confinement due to COVID-19, there were social distancing measures in Spain that did not prevent minors from attending school; (3) some groups that, due to restrictions, were quarantined for weeks had to be eliminated; moreover, there were problems with T1-T2 (pre-test and post-test) pairings in several classes where they were control groups (as well as lower participation in T2 (post-test) in the centers in the control groups); (4) the study used a large sample, but it should be noted that the sampling was non-probabilistic and it was impossible to assign the participants to the intervention and control group randomly; (5) Numerous reliability indicators have been calculated for the questionnaire in relation to the pre-test and post-test scores of our sample. Only the Online Intimate Partner Abuse Questionnaire instrument was impossible to calculate because there was no variance in items 6 and 8. This situation is understandable as the sample is in early adolescence; (6) the study presents numerous statistical comparisons, and some of the (statistically significant) results may contain type I errors; and (7) no analyses were carried out that could consider the grouping by schools or classrooms, because of the low variability of level 2 (schools) (Hox, 2002) and because there was a large number of groups/classes with fewer than ten students (Lai & Kwok, 2015). Although several limitations of the initial pilot study of Ortega-Barón et al. (2021) were improved, the interpretation and generalization of the results should still be done with caution. Finally, it was impossible to obtain an additional measure six months after completion of the program. These limitations also suggest future lines of research to obtain more indicators of the possible effectiveness of the program and to determine changes in other blocks, such as the fourth one, which addresses cyber-observers or viral challenges.

In conclusion, the Safety.net program has very relevant implications in the educational and social fields. This multi-risk program effectively could prevent and reduce many internet risks in a minimum number of sessions during early adolescence. Its structure and length allow it to be applied in a four-month period in schools.

Funding

This work has received funding from Ministerio de Ciencia, Innovación y Universidades (RTI2018-094212-B-I00), Universidad Internacional de La Rioja [UNIR Research Plan (2020–2022 and 2022–2024)] and [Instituto de Transferencia e Investigación (ITEI-B23-006 & ITEI-B24-001)].

Conflict of interests

There are no financial, work, or other relationships that may constitute a conflict of interest concerning this work.