Home and group-based implementation of the “Growing Up Happily in the Family” program in at-risk psychosocial contexts

Abstract

“Growing Up Happily in the Family” is a program to prevent child maltreatment targeted at parents of children aged 0–5 years old in at-risk psychosocial contexts. The program is delivered via either a group-based or a home-visit format. The objective of this study was to evaluate the impact of various implementation components in the home and group versions on changes in parental attitudes about child development and education. At-risk and non at-risk parents participated in the group-based (196 participants in 26 groups) and home-visit (95 participants) versions of the program delivered through local social services. We analyzed program adherence, adaptations, participant responsiveness, quality of delivery, and implementation barriers as predictors of changes in parental attitudes. The results showed that greater program adherence, better quality of delivery and participant responsiveness, and positive climate predicted changes in parental attitudes in both formats. Therefore, it is important to take into account the quality of the implementation process when testing the effectiveness of early group-based and home-visit interventions in at-risk families.

Resumen

«Crecer Felices en Familia» es un programa para prevenir el maltrato infantil dirigido a padres y madres con hijos/as de entre 0 y 5 años que se encuentran en contextos de riesgo psicosocial. El programa cuenta con una modalidad grupal y una modalidad domiciliaria. El objetivo de este estudio fue evaluar el impacto de varios componentes de la implementación de la versión grupal y domiciliaria en el cambio de actitudes parentales relacionadas con el desarrollo infantil y la educación. Padres y madres en situación de riesgo y de no riesgo participaron en la modalidad grupal (196 participantes en 26 grupos) y en la modalidad domiciliaria (95 participantes) desarrolladas a través de los servicios sociales. Se analizaron la adherencia al programa, las adaptaciones, la respuesta de los participantes, la calidad del desarrollo y las barreras de la implementación como predictores de los cambios en actitudes parentales. Los resultados mostraron que la adherencia al programa, la calidad del desarrollo, la respuesta de los participantes y el clima positivo predicen cambios en las actitudes parentales en ambas modalidades del programa. Es importante tener en cuenta la calidad del proceso de implementación cuando se evalúa la efectividad de las intervenciones grupales y domiciliarias para las familias en situación de riesgo.

In 2006, the Council of Europe issued a recommendation encouraging states to develop programs and plans to support the strengths and capacities of families to exercise the parental role, with a new focus on taking a positive view of family intervention. Several studies have shown the effectiveness of parenting programs in promoting parenting skills and childhood wellbeing (Barlow, Smailagic, Huband, Roloff, & Bennett, 2012; Barth, 2009; Johnson et al., 2010; Prinz, Sanders, Shapiro, Whitaker, & Lutzker, 2009). Particularly, in the field of early child maltreatment prevention, there is increasing interest in the importance of evidence-based intervention to support vulnerable parents (Barth et al., 2005; Rodrigo, Byrne, & Álvarez, 2012; Webster-Stratton & Reid, 2010). The present study examined the implementation of the home and group-based versions of the Growing Up Happily in the Family program for at-risk parents of children aged 0–5, delivered through local social services in Spain. This study may contribute to increasing our knowledge of the differential impact on program outcomes of the various implementation components of group-based and home-visit programs.

Child maltreatment is associated with multiple risk factors related with the parents, including a lack of knowledge or inappropriate expectations of child development and child management (Azar & Weinzierl, 2005; Burke, Chandy, Dannerbeck, & Watt, 1998; Reid, Kavanagh, & Baldwin, 1987), a strong belief in the value of punishment (Bower-Russa, 2005; Bower-Russa, Knutson, & Winebarger, 2001; Vittrup, Holden, & Buck, 2006), an inability to be empathically aware of the child's needs (Crittenden, Lang, Claussen, & Partridge, 2000; Shahar, 2001), or significant role reversal in which the parent looks to the child for the satisfaction of their own emotional needs (Bavolek, 1989). Stressful events faced by parents (i.e., unemployment, high life stress, low education, illness, etc.) can have deleterious effects on parenting, including the development of abusive disciplinary practices (Collins, Maccoby, Steinberg, Hetherington, & Bornstein, 2000).

In the area of child maltreatment prevention, parent education programs are among the most commonly used interventions in child welfare contexts (Barth et al., 2005; Rodrigo et al., 2012; Rodrigo, Byrne, & Álvarez, in press). Thus, parenting programs are often designed to increase parents’ knowledge of child development and to support them in developing parenting skills and addressing the challenges and difficulties inherent in parenting (Sanders, Markie-Dadds, Tully, & Bor, 2000). Several reviews have shown positive results of parenting programs for families at risk (e.g., Johnson et al., 2010; Kaminski, Vallew, Filene, & Boyle, 2008; Macleod & Nelson, 2000; Sandler, Schoenfelder, Wolchik, & MacKinnon, 2011). Parenting programs are now being offered in a variety of settings, such as clinics, community-based settings and in the home; and in home-visit and group-based formats.

Programs with a group format are delivered in prevention-oriented groups (involving eight to twelve participants on average) and guided by a facilitator. This approach is more cost-effective and user-friendly than individual interventions (Kumpfer, Whiteside, Greene, & Allen, 2010). Group parent education programs utilize a range of different techniques to support parents: videotape vignettes, didactic training, role-playing, modeling, group discussion, and homework (Sampers, Anderson, Hartung, & Scambler, 2001). In recent years, group programs have been integrated into the community through partnership schemes between child protection agencies, institutions, and university teams that develop programs, thus assuring their sustainability (Rodrigo, Màiquez, Martín, & Byrne, 2008).

Programs delivered via the home-visit format are usually provided to vulnerable families with specific needs or risks, and are generally carried out in the private family sphere. Home programs have been shown to have a positive impact on neglect and abuse (Chaffin, Hecht, Bard, Silovsky, & Beasley, 2012; Lowell, Carter, Godoy, Paulicin, & Briggs-Gowan, 2011; Olds et al., 1999; Silovsky et al., 2011). However, several reports of home-visit programs pointed out the lack of information concerning the theoretical framework supporting the programs, the implementation procedure followed, what happens during the visits, and the main results achieved (Carroll et al., 2007; Fixsen, Naoom, Blase, & Friedman, 2005; Vimpani, 2000). In general, a wide gap still exists between available evidence-based interventions and practices for treating and preventing child abuse and neglect and methods of effective dissemination, implementation, and sustainment of those interventions (National Research Council, 2013).

It has been recommended that parent education programs be manualized, with a defined structure and a theoretical basis (Kaminski et al., 2008), and that they be applied with high levels of adherence to the original model designed by the authors, which should ideally undergo previous testing (Durlak & Dupre, 2008; Elliot & Mihalic, 2004). It is very important to evaluate the quality of implementation to understand which aspects of a program contribute to its good functioning when applied in real-life conditions (Domitrovich & Greenberg, 2000; Durlak & Dupre, 2008). Several reviews (Dane & Schneider, 1998; Durlak & Dupre, 2008; Dusenbury, Brannigan, Falco, & Hansen, 2003) have identified eight dimensions of program implementation: fidelity, dosage, quality, participant responsiveness, program differentiation, monitoring of control conditions, program reach, and adaptation. More recently, Berkel, Mauricio, Schoenfelder, and Sandler (2011) have proposed a comprehensive model to analyze the relation between implementation components and program outcomes. Their proposal is based on four implementation components: fidelity (i.e., adherence to the program curriculum, dosage, and duration), quality of delivery (i.e., the skill with which facilitators deliver material and interact with participants), program adaptation (i.e., changes made to the program), and participant responsiveness to the program (i.e., participants’ level of enthusiasm for and participation in a program, program satisfaction). The evidence indicates that variability in the implementation of prevention programs in large-scale interventions has an impact on the outcomes achieved by these programs (Chiapa et al., 2015; Fixsen et al., 2005; Smith, Dishion, Shaw, & Wilson, 2013).

The intervention: Growing Up Happily in the Family. This is a promotional and preventive program focusing on the promotion of parental warmth, sensitivity, positive expectations of child development, strategies for the child's self-regulation, and family-school support as protective factors for child development. The content of this program is based on the research on attachment (Bowlby, 1969; De Wolff & van IJzendoorn, 1997), parental childrearing practices (Grusec & Goodnow, 1994), self-regulation (Shonkoff & Phillips, 2000), parental sense of competence (Coleman & Karraker, 2003; Jones & Prinz, 2005), and family stress and social support (Ceballo & McLoyd, 2002; McCubbin, McCubbin, & Thompson, 1995). To avoid attrition a non-directive and participative process of co-construction with emotional involvement is promoted instead of the unidirectional transmission of expert knowledge.

The program focuses on promoting child development through improved parenting skills and family learning environments. It has a number of specific objectives: (a) to promote the development of secure attachments, positive interaction between parents and children, and parenting satisfaction; (b) to help parents identify and attend to the needs of their children, encouraging them to react positively to their development; (c) to guide parents in establishing daily routines and acquiring skills in child care and safety; (d) to help parents differentiate between and understand their children's mental states and to stimulate their physical and emotional autonomy; (e) to help parents identify interaction efforts and stimulate their children's verbal and nonverbal communication skills; (f) to provide parents with strategies for regulating their children's behavior in accordance with the rules and to provide them with alternative guidelines for childrearing; (g) to promote protective factors and reduce risk factors through the promotion of parenting skills, stress reduction, and increased social support.

Program characteristics and participant profiles. A number of characteristics make this intervention different from other parent education programs: (a) the program combines group-based intervention with home visits, thereby encompassing a wide range of family needs. It offers a highly flexible format (families can attend the group-based version, the home-visit version, or a combination of the two); (b) the group activities are coordinated and streamlined by a facilitator who also offers accompaniment, counseling, modeling, and support for parents; and (c) the program establishes a framework for collaboration with parents based on participation, training, and involvement in their childrearing tasks.

The program is recommended for parents with a history of neglect or physical and/or emotional maltreatment; teenage mothers and/or cases of unwanted pregnancies; parents with poor parenting skills; children with health problems or a difficult temperament; parents with health problems or emotional instability; families with low socioeconomic level combined with risk conditions and social vulnerability; and immigrant parents. An experiential methodology was designed, already validated in other parenting programs, that helps at-risk parents verbalize their interpretations of a variety of family situations in the group format, enrich their interpretations with others’ views on parenting, reflect upon the consequences of their actions on family life, and reach compromises of change (Byrne, Rodrigo, & Máiquez, 2014; Rodrigo et al., 2012; Rodrigo, Correa, Máiquez, Martín, & Rodríguez, 2006).

Contents and structure. The group version of this intervention is a community-based, multisite program delivered through a series of 90-min weekly group meetings in municipal social services and lasting four to five months (22 sessions, 4 per module and 2 for evaluation). The program has five modules (see Table 1). Given the participants’ low educational level and diverse cultural backgrounds, materials include vignettes, videos, case studies, guided fantasies, puzzles, games, and group discussions.

Table 1.

Structure and content for group and home versions of the program.

Group version

Home version

- Session 0a: Presentation - Session 0b: Pre-test assessment - Module 1: Sensitive and responsive parenting (4 sessions) - Module 2: Coming to know our children (3 sessions) - Module 3: Regulating child behavior (4sessions) - Module 4: First family-school relationships (4 sessions) - Module 5: Parenting: A solitary task? (4 sessions) - Post-test assessment and celebration

- Preparatory session - Session 1: Presentation of the program; pre-test assessment - Session 2: Identification of family needs and concerns; pre-test assessment - Session 3: Feeding routine - Session 4: Grooming routine - Session 5: Playing routine - Session 6: Walking routine - Session 7: Bathing routine - Session 8: Sleeping routine - Session 9: Sharing learning contents - Session 10: Assessing parenting skills; post-test assessment - Session 11: Reflecting on progress; follow-up assessment

The home version of the program is delivered as a series of home visits, and offers individualized information, guidance, advice, practical help, and emotional support to families. This version consists of nine weekly sessions lasting approximately 90 min each and two monthly follow-up sessions, all held in the families’ homes, for a total program duration of four months (see Table 1). The program content involves interactive activities and stimulation sequences aimed at strengthening the parent-child relationship and improving child development. The program allows for some flexibility in the session timing to take into account participants’ needs, and sessions may also address other cross-cutting topics, depending on the age of the children in question.

In sum, this study evaluates the outcomes of this intervention program, as delivered in the group-based and home-visit formats, measured in terms of the changes in parental attitudes in at-risk parents and comparing the particular implementation components that predicted the program outcomes in each case.

Method

Participants and procedure

In the group program, the participants were 133 parents who attended the Growing Up Happily in the Family program in ten social services in the Autonomous Communities of Castile and Leon, Canary Islands, and Catalonia. In the home version, the participants were 96 parents referred by the municipal social services of Tenerife. The participants in both programs were mainly young mothers living in urban areas, with primary education, on welfare, and unemployed, with half of the participants living in a two-parent family and half in a situation of low-medium risk (see Table 2).

Table 2.

Description of participants in group and home versions of the program.

	Group version (N = 133)	Home version (N = 96)
	M (SD)/%	M (SD)/%
Sex
Mother	90.3	96.9
Age	32.85 (8.36)	30.86 (7.54)
Family structure
One parent	46.3	42.7
Two parent	53.7	57.3
Area
Rural	40.25	39.4
Urban	61.25	60.4
Educational level
No studies	14.1	29.2
Primary level	62.4	70.8
Secondary/high-school studies	23.5	–
Financial situation
On welfare	60	62.5
Employment situation
Unemployed	81.3	69.8
Employed	18.7	30.2
Psychosocial risk status
Non at-risk	23.9	–
Low-medium risk	55.4	57.3
High risk	20.7	42.7

In the group program there were 30 facilitators, all women, with an average age of 34.33; all had graduate degrees: just over one third (34.8%) were social educators, 21.7% were social workers, 21.7% were psychologists, and 21.6% held a degree in pedagogy. In the home version there were 31 facilitators, 91.7% of whom were women, with an average age of 26.3; 43.7% were social workers, 34.4% held a degree in pedagogy, and 21.9% were psychologists.

As part of their normal casework, social services personnel had to identify families with a minor who was at risk; a minor is declared to be at risk when he or she is in a situation that could be potentially harmful to his or her healthy development according to several psychosocial family and personal factors. The program was offered as part of their case plan. In the group program, social services personnel also interviewed the non-referred parents that attended the program on voluntarily basis to clarify their motivations to participate and to make sure that they did not have any problematic situation that put their children at risk.

An intensive training program of 25 h was given to the group and home facilitators and also to the coordinators responsible for each of the local social services to better integrate the program within the service. This training program covered the core principles, methodology, and evaluation of the program, as well as guidance on how to implement it successfully and integrate it into the social workers’ existing casework plan. There was also one training session conducted halfway through the program, to assure the supervision of the facilitators. One important aspect that facilitates the application of a program with fidelity is the standardization of intervention content, structure, and materials. In Growing Up Happily in the Family, all components relating to the implementation of the programs are described in detail.

In the group version, two warm-up sessions were necessary to create a group feeling and to establish the group roles. Part of the first session was also used to complete pre-test questionnaires. Post-test questionnaires were completed within a week of the program completion in the last session. In the home version of the program, family pre-test data were collected as part of the first home visits. The post-test questionnaire was completed directly after the intervention and at two follow-up sessions. Session checklists, interviews, and other instruments were also used to collect data, as described below. Special care was taken in establishing online connections with the group facilitators and coordinators to assure that such a variety of implementation data was correctly collected. Written consent was obtained from all the participants according to the protocol approved by the Ethics Committee of the University of La Laguna.

Measures and instruments

Implementation measures

Table 3 shows the components of the implementation process in the group and home versions. A variety of qualitative and quantitative measures and informants were used according to the type of component and taking into account the characteristics of the implementation in the group and home formats. A detailed description of the implementation measures follows:

• -

  Implementation measures in group version

  • (a)

    Dosage. This refers to the number of sessions performed. The complete dose includes 22 sessions and the partial dose includes 14 sessions. At the initial session, the service coordinators asked groups which dose they wished to opt for, according to their preferences and availability.

  • (b)

    Session checklist. At the end of each session, group facilitators filled out a checklist that included the following information:

    • •

      Duration of session. This was recorded in minutes. As the recommended duration was 90 min, sessions lasting between 80 and 100 min were coded as having an adequate timing (coded as 1), whereas sessions with durations above or below this interval were coded as having inadequate timing (coded as 0).

    • •

      Adaptations. We computed for each group the number of crucial modifications performed in each session, i.e., those that involved critical changes affecting the program fidelity, such as changes in the methodology, contents, and objective of the activities (coded as 1). Modifications affecting the order of the activities, language adaptations, or the use of other group dynamics were not considered to affect the program fidelity (coded as 0). A higher percentage indicates a higher number of crucial modifications.

    • •

      Didactic assessment. This recorded, using a 0–5 Likert scale: (a) the didactic quality of the material resources provided in the sessions, (b) the extent to which the activities were related to the session goal, (c) whether clear guidelines were provided for the facilitators and the group, and (d) whether the objectives were reached. A higher score indicates better didactic quality.

    • •

      Group dynamics assessment. The same response scale (0–5) was used for the assessment of the quality of group dynamics: (a) participation and interest, (b) group cohesion and positive climate. A higher score indicates better group dynamics.

    • •

      Parental attendance. This was recorded on an individual basis. At the end of the program each participant's rate of attendance was computed as a percentage of the total sessions, and an average was computed per module.

  • (c)

    Final interview. At the end of the program, facilitators and coordinators were interviewed by the program staff to identify any obstacles or barriers that had threatened or hindered the implementation process. Several barriers were identified and coded as: Motivational (e.g., participants do not show interest in the program; participants get bored during the sessions); Engagement (e.g., irregular participant attendance, lack of punctuality); Adaptation (e.g., participants do not understand the activity, abstract contents difficult to grasp); Organizational (e.g., lack of organization at the initial of the program, timetable or location changes); Coordination with the agency (e.g., lack of coordination with the professionals in charge of the families, lack of communication with the agency). Open-ended responses were coded by two independent judges, yielding an inter-rater agreement of 90–95%, and a Kappa index of .80 for motivational barriers, .82 for attendance barriers, .79 for the adaptation barriers, .78 for organizational barriers, and .81 for coordination barriers. For each barrier reported by the facilitators/coordinators, percentages were computed with respect to the total number of barriers reported, taking into account the fact that more than one barrier could be reported.

  • (d)

    Satisfaction scale (Almeida et al., 2008), translated ad hoc into Spanish. Consists of 44 items with a 0–4 Likert scale, involving self-reports of participants in the following dimensions: logistic, program structure, contents, group dynamics, facilitator behavior, and parental changes observed. An average total score for each participant was used.

• -

  Implementation measures in home version

  • (a)

    Session checklist. Facilitators completed a checklist after each session that included the following information:

    • •

      Timetable. This indicated when the sessions were conducted. It was considered that an appropriate time is one that matches the corresponding ideal for the routine in question, and an inappropriate time is one that does not match the routine schedule.

    • •

      Duration of session. Same as in group version.

    • •

      Participants in the session. This recorded who attended the session-parents, children, or other family members—and their degree of participation in the session. Afterwards, we calculated the average of child participation, with high participation considered to be when children were involved in more than half of the sessions, and low participation to be when they participated in fewer than half of the sessions.

    • •

      Didactic assessment. This recorded: (a) whether clear guidelines were provided for the facilitators, (b) the setting and negotiation of objectives, (c) the defining of steps to be followed, (d) whether activities were coherent and sufficient in number, (e) the didactic quality of the material resources provided in the sessions, and (f) the estimated time. Also, participant responsiveness was measured in terms of participation and interest.

  • (b)

    Participants’ session assessment. After each session, parents completed an information sheet indicating their assessment of the session: its applicability to daily life, the climate, and the emotional response of participants.

  • (c)

    Client satisfaction questionnaire (CSQ-8) (Larsen, Attkisson, Hargreaves, & Nguyen, 1979; Spanish version by Echeburúa & Corral, 1998). This measures client satisfaction with the intervention, based on the quality, quantity, and value of the intervention received. It included eight items presented in a 4-point Likert scale, with higher ratings being indicative of greater satisfaction.

Table 3.

Components of the implementation process in group and home versions of the program.

Components	Group version			Home version
	Indicators	Measures	Informants	Indicators	Measures	Informants
Adherence	- Dosage - Parental attendance - Duration of session	Register at initial Session checklist Session checklist	Coordinator Facilitator Facilitator	- Timetable - Duration of session - Participation of children	Session checklist	Facilitator
Adaptations	- Number and type of modifications	Session checklist	Facilitator	–	–	–
Quality of delivery	- Material resources - Goal-related activities - Clear guidelines - Objectives reached	Session checklist	Facilitator	- Adaptation to family characteristics - Defining steps - Clear guidelines - Setting objectives	Session checklist	Facilitator
	Implementation barriers	Final interview	Facilitator/Coordinator
Group and participant responsiveness	- Group participation and interest	Session checklist	Facilitator	- Family participation and interest	Session checklist	Facilitator
	- Group cohesion and positive climate			- Applicability to everyday life - Positive climate	Session checklist	Participants
	- Participant satisfaction	Final satisfaction scale	Participants	- Participant satisfaction	Final satisfaction scale	Participants

Measures of parental attitudes

Adult-Adolescent Parenting Inventory (AAPI-2) (Bavolek & Keene, 2001; ad hoc Spanish version, using a double translation procedure). This measures parental attitudes and behavior using two forms (Form A at initial session and Form B at completion), each including 40 items presented on a 5-point Likert Scale (1 = agree; 5 = strongly disagree). The AAPI-2 provides five subscales: inappropriate expectations of children, parental lack of empathy toward the child's needs, belief in the use of corporal punishment, parent-child role reversal, and oppressing the child's independence. As the scale is reversed, higher mean scores for the AAPI-2 subscales indicate less negative outcomes.

Plan of analysis

Repeated ANOVAs were used to examine differences in parental attitudes between the pre-test and post-tests. Change scores for each factor were calculated by subtracting the pre-test score from the post-test score, so that a higher score indicated an increase in the factor and a lower score reflected a reduction of the factor. The effect size was explored using the R²statistic; the clinical relevance of this statistic was classified as negligible when R² < .01, small when R² > .01 and R² < .09, medium when R² > .09 and R² < .25, and large when R² > .25 (Cohen, 1988). Finally, hierarchical linear regression analyses were run separately for the group and home versions of the program, to study the influence of the implementation components on the changes in parental attitudes. All the variables included in the regression models were standardized (Tabachnick & Fidell, 2007).

Hierarchical regression models were calculated to determine the progressive impact of the implementation components on changes in attitudes in both the group and home versions. We checked for collinearity, normality of residuals, linear relationship between variables, and homoscedasticity of variances. To interpret the global significance of the model, at each step we examined the statistic F, the values for the Adjusted R² (Adj R²) and the change in R(ΔR²), as well as the specific contribution of each variable to the total variance explained by the model through the significance and the value of the squared semi-partial correlation (rs²). All analyses were conducted using the SPSS 18.0 statistical software assuming a confidence level of 95% for Type I error.

Results

Pre-post changes in parental attitudes

The results of the pre-and post-tests on parental attitudes are presented in Table 4. In both versions of the program we observed positive results. In the group version, parents showed positive changes in their inappropriate expectations toward the child, empathy to their children, belief in corporal punishment, and role reversal. In the home version, there were statistically significant changes for all five of the AAPI subscales, indicating that parental attitudes significantly improved between pre- and post-test on each of these measures. In the group version, effect sizes were large for empathy to their children (.58), medium for role reversal (.10), and small for inappropriate expectations (.02) and belief in corporal punishment (.02). In the home version, effect sizes were large for empathy to their children (.53), belief in corporal punishment (.34), and role reversal (.29), medium for oppressing children's independence (.13), and small for inappropriate expectations (.07).

Table 4.

Mean differences in parental attitudes before and after participation in the Growing Up Happily program, group and home version. Higher mean scores indicate fewer negative outcomes (i.e., less role reversal).

	Group version			Home version
	Pre-test M (SD)	Post-test M (SD)	F(1,132)	Pre-test M (SD)	Post-test M (SD)	F(1,95)
Parental attitudes
Inappropriate expectations	2.58(0.73)	2.68 (0.69)	2.62^*	2.62 (0.69)	2.90 (0.82)	6.99^**
Lack of empathy	3.09(0.65)	3.89 (0.67)	206.35^***	3.07 (0.62)	3.84 (0.58)	105.41^***
Belief in corporal punishment	3.71(0.64)	3.79 (0.63)	2.01^*	3.71 (0.69)	4.17 (0.61)	49.29^***
Parent-child role reversal	2.92(0.73)	3.12(0.85)	9.61^***	2.89 (0.67)	3.40 (0.66)	39.63^***
Oppressing child's independence	3.76(0.68)	3.62(0.68)	2.71	3.44 (0.75)	3.76 (0.72)	13.88^***

*

p < .05.

**

p < .01.

***

p < .001.

Regression models in the group version

To study the predictive capacity of the implementation variables on changes in parental attitudes, we carried out hierarchical regression analyses in three steps: in step 1, we introduced the variables of adherence (dosage, duration, crucial modifications). In step 2, we included the variables related to the assessment of the sessions by the facilitators (material resources; goal-related activities; clear guidelines; objectives reached; participation and interest; group cohesion and positive climate). In step 3, we included the variables related to the final evaluation by the facilitators (motivational barriers, engagement barriers, adaptation barriers, organizational barriers, coordination barriers) and the level of participant satisfaction with the program. After checking for collinearity, normality of residuals, linear relationship between variables, and homoscedasticity of variances, it was decided not to include the variables “group cohesion & positive climate”, “goal-related activities” and “clear guidelines”, for failure to comply with cases of non-collinearity. Also, a general measure of satisfaction with the program was calculated due to the high positive correlation between the components of this measure.

The regression model for the change scores in inappropriate expectations was not significant in step 1 (F(3,129) = 1.16, p = .33) or step 2 (F(6,126) = 1.86, p = .093), but was significant in step 3 (F(12,120) = 2.15, p = .019), explaining 18% of the variance. As the scale is reversed, higher mean scores for the subscales indicate less negative outcomes. Increased participation and interest (rs² = .02) and satisfaction with the program (rs² = .02) predicted fewer inappropriate expectations. The model of lack of empathy was not significant in step 1 (F(3,129) = .42, p = .73), but was significant in step 2 (F(6,126) = 2.31, p = .038) and step 3 (F(12,120) = 2.07, p = .023), explaining 17% of the variance. Better evaluation of material resources (rs² = .03) as well as fewer problems with engagement barriers predicted less lack of empathy. The regression model for role reversal was significant in step 1 (F(3,129) = 4.87, p = .003), step 2 (F(6,126) = 3.43, p = .004), and step 3 (F(12,120) = 2.75, p = .002), explaining 21% of the variance. Fewer motivational barriers (rs² = .05), full dosage, lower number of adaptations, better evaluation of material resources and fewer organizational barriers predicted less parent-child role reversal. The regression model of the change scores in oppressing the child's independence was significant in step 1 (F(3,129) = 3.25, p = .024), step 2 (F(6,126) = 2.21, p = .046), and step 3 (F(12,120) = 2.35, p = .009), explaining 19% of the variance. The appropriate duration of the sessions (rs² = .04) as well as program satisfaction predicted less oppressing the child's independence (see Table 5).

Table 5.

Regression models of implementation variables on changes in parental attitudes in group version of the program. Higher mean scores for the parental attitudes indicate fewer negative outcomes (i.e., less role reversal).

	Inappropriate expectations			Lack of empathy			Role reversal			Oppressing independence
	β	Adj R^2	ΔR^2	β	Adj R^2	ΔR^2	β	Adj R^2	ΔR^2	β	Adj R^2	ΔR^2
Step 1		.03			.03			.10^**			.07^*
Full dosage	.13			−.05			.23^**			.01
Appropriate duration	−.07			.00			.10			.23^**
Crucial modifications	.04			−.08			−.24^**			.15
Step 2		.08	.06		.10^*	.09^*		.14^**	.04^**		.10^*	.03
Full dosage	.02			−.09			.20^*			.09
Appropriate duration	.11			.02			.13			.11
Crucial modifications	−.01			.03			−.18			.16
Material resources	−.12			.29^**			.15			.04
Participation and interest	.26			.12			.08			−.18
Objectives reached	.17			.06			.10			−.12
Step 3		.18^*	.10^*		.17^*	.07^*		.21^**	.08^**		.19^**	.10^*
Full dosage	.04			−.15			.17			.04
Appropriate duration	−.10			−.23			−.09			.35^*
Crucial modifications	.00			.13			−.29^*			.07
Material resources	−.21			.26^*			.22^*			.04
Participation and interest	.24^*			.12			.03			−.06
Objectives reached	.14			.09			.21			−.16
Motivational barriers	−.15			−.06			−.47^**			.06
Engagement barriers	−.16			−.30^*			.16			.32
Adaptation barriers	−.04			−.11			−.20			.06
Organizational barriers	−.12			−.29			−.34^*			.13
Coordination barriers	.16			.04			−.02			.11
Program satisfaction	.17^*			.10			.01			.17^*

*

p ≤ .05.

**

p ≤ .01.

Regression models in the home version

To study the predictive capacity of the implementation variables on changes in parental attitudes, we carried out hierarchical regression analyses in three steps: in step 1, we introduced the variables of adherence (timetable, duration, child participation). In step 2, we included the variables related to the quality of delivery (adaptability to the family, defining steps to be followed, clear guidelines, setting objectives). In step 3, we included the variables related to the participant's responsiveness (participation and interest, positive climate, applicability, participants’ satisfaction with the program).

The regression model for the change scores in inappropriate expectations was significant in step 1 (F(3,92) = 3.20, p = .27), step 2 (F(7,88) = 3.42, p = .003), and step 3 (F(11,84) = 5.25, p = .000), explaining 41% of the variance. As the scale is reversed, higher mean scores for the subscales indicate less negative outcomes. Increased defining of steps (rs² = .03) and positive climate (rs² = .18) predicted fewer inappropriate expectations. The regression model for lack of empathy was not significant in step 1 (F(3,92) = 1.21, p = .31) or step 2 (F(7,88) = 1.48, p = .18), but was significant in step 3 (F(11,84) = 2.15, p = .03), explaining 21% of the variance. Increased adaptability (rs² = .04) and positive climate (rs² = .10) predicted less lack of empathy. The regression model of the change scores in belief in corporal punishment was significant in step 1 (F(3,92) = 6.29, p = .001), step 2 (F(7,88) = 6.42, p = .000), and step 3 (F(11,84) = 4.71, p = .000), explaining 38% of the variance. Increased defining of steps (rs² = .04), clear guidelines (rs² = .03), and participation and interest (rs² = .04) predicted less support for the use of corporal punishment. The regression model for role reversal was not significant in step 1 (F(3,92) = .66, p = .577), step 2 (F(7,88) = 1.84, p = .089), and step 3 (F(11,84) = 2.55, p = .008), explaining 25% of the variance. Increased adaptability (rs² = .05) and positive climate (rs² = .09) predicted less parent–child role reversal. The regression model for the change scores in oppressing the child's independence was significant in step 1 (F(3,92) = 8.89, p = .000), step 2 (F(7,88) = 4.21, p = .000), and step 3 (F(11,84) = 2.94, p = .002), explaining 28% of the variance. The appropriate timetable of sessions (rs² = .09) predicted less oppression of the child's independence (see Table 6).

Table 6.

Regression models of implementation variables on changes in parental attitudes in home version of the program. Higher mean scores for the parental attitudes indicate fewer negative outcomes (i.e., less role reversal).

	Inappropriate expectations			Lack of empathy			Belief in corporal punishment			Role reversal			Oppressing independence
	β	Adj R^2	ΔR^2	β	Adj R^2	ΔR^2	β	Adj R^2	ΔR^2	β	Adj R^2	ΔR^2	β	Adj R^2	ΔR^2
Step 1		.09^*			.04			.17^***			.02			.23^***
Appropriate timetable	−.06			−.07			.22			.13			.50^***
Appropriate duration	.19			.20			−.06			−.04			−.12
Child participation	.24^*			.08			.13^*			−.06			.08
Step 2		.21^**	.12		.11	.07		.34^***	.17		.13	.11		.25^***	.03
Appropriate timetable	−.06			−.05			.11			.14			.45^***
Appropriate duration	.17			.19			−.09			−.06			−.13
Child participation	.08			−.00			.13			−.09			.02
Adaptability	.26^*			.32^*			.01			.38^*			−.04
Defining steps	.22			−.17			.23^*			.01			.07
Clear guidelines	.03			.05			.32^*			.09			.13
Setting objectives	−.18			−.06			.02			−.17			.05
Step 3		.41^***	.19		.21^*	.11		.38^***	.04		.25^*	.12		.28^**	.03
Appropriate timetable	−.11			.08			.06			.11			.40^**
Appropriate duration	.13			.17			−.07			−.09			−.14
Child participation	.11			.02			.04			−.02			−.02
Adaptability	.19			.29^*			−.05			.31^*			−.09
Defining steps	.23^*			−.14			.23^*			−.01			.06
Clear guidelines	.04			.08			.28^*			.09			.11
Setting objectives	−.22			−.05			.03			−.21			.02
Participation and interest	−.26			−.22			.34^*			−.21			.07
Positive climate	.53^***			.40^***			−.07			.37^**			.15
Applicability	.06			−.06			−.10			.20			−.01
Program satisfaction	.02			−.04			−.05			−.06			.04

*

p ≤ .05

**

p ≤ .01

***

p ≤ .001

Discussion

This study evaluates the outcomes of Growing Up Happily in the Family program measured in terms of the changes in parental attitudes and the impact of implementation components on parental changes. Concerning changes in parental attitudes, completion of both the group and home versions of the program led to positive outcomes. Participants exhibited positive changes immediately following completion of the program, with increases in positive parental attitudes that showed medium to large effect sizes; this was especially observed in the home version, despite the fact that most of these parents had a medium to high psychosocial risk status. Other parent education programs have also led to improvements in parental attitudes related to prevention of child maltreatment (Almeida et al., 2012; Breitenstein et al., 2012; Cowen, 2001; Estefan, Coulter, VandeWeerd, Armstrong, & Gorski, 2013; Gross et al., 2009). Changes in these dimensions are key to developing positive parenting skills, as improvements of this type are related to the promotion of child wellbeing and protective factors in the family (Barlow et al., 2012; Barth, 2009; Johnson et al., 2010; Prinz et al., 2009).

For the second aim, to examine the impact of implementation components on the outcomes, the results show that the quality of the implementation process is an important contributing factor to the changes brought about by parenting programs. To determine this, we undertook a detailed examination of the implementation of the Growing Up Happily in the Family program, taking into account a range of components to reflect a more comprehensive and systemic approach to implementation (Berkel et al., 2011). The results showed that the implementation components included in the analysis were able to predict the pre-post changes measured at the end of the program, in both the group and home versions. These implementation components explained a large part (17–41%) of the variation in the changes in parental attitudes in both versions of the program. Moreover, the analyses showed that the implementation components affected the change dimensions differently, implying that some outcomes are more sensitive to certain implementation components than others.

Adherence was associated with changes in parental attitudes in both the group and home versions of the program. Ensuring an appropriate duration of the group sessions and home visits and scheduling the home visits at an appropriate time of day predicted positive changes in the promotion of the child's independence. These results underline the importance of ensuring program fidelity, as elements of fidelity are key to producing the desired changes (Barth, 2009; Baydar, Reid, & Webster-Stratton, 2003; Spoth, Goldberg, & Redmond, 1999). Crucial modifications made to the program by the facilitator, on the other hand, constitute an element of adherence that is negatively associated with changes in parental attitudes in the group-based format. This means that any modifications made by the facilitator beyond those accepted as part of the program core would lead to a negative change in program outcomes (Elliot & Mihalic, 2004).

In the home-visit version of the program, the ability to adapt to the family's characteristics was related to positive changes in expectations, empathy, and the use of appropriate roles. Some authors suggest that in the case of programs delivered in the home, the intensity of the support, the number and type of strategies employed, and the activities offered to the family should be individualized, tailored to the family's interests and priorities, and integrated into the family's customs and routines (Jung, 2003; Keilty, 2008; Woods, Kashinath, & Goldstein, 2004). These findings highlight the importance of factoring flexibility into structured programs, especially those delivered in the home. They support the idea that it is important to strike the right balance between strictly adhering to program protocols and adapting evidence-based programs to meet clients’ needs (Fixsen et al., 2005).

With respect to the quality of the program delivery, in the assessments of each group session, a significant relation was observed between positive evaluations of the resources available in the session and improvements in parental attitudes. This shows that facilitators must be given the materials and resources they need to allow them to run the sessions properly. In the home version of the program, it was also shown that defining the steps to be followed and providing clear guidelines were associated with more positive outcomes with respect to expectations toward the child and beliefs about corporal punishment. Manualizing group or home programs allows the facilitator to run sessions in a structured manner, thus facilitating implementation, which in turn leads to more positive outcomes (Dusenbury et al., 2003; Keilty, 2008; Luborsky & De Rubeis, 1984).

Parents’ participation and interest were also associated with improvements in parental attitudes in both the group and home versions of the program. The positive effect of group dynamics in group-based programs, measured in terms of participation levels and their effects on individual change, has already been confirmed in other studies (Nix, Bierman, & McMahon, 2009; Ogrodniczuk & Piper, 2003). It follows that parent education programs need to include specific elements to encourage participant motivation, as negative perceptions of participation and interest will affect the quality of the facilitator's implementation of the program, thereby also affecting program outcomes. In home-visit programs as well, when families are encouraged to show interest and participate and a positive climate is created between the facilitator and the families, participants may be more inclined to participate and change their attitudes (Jones, 2010). This is in line with other findings suggesting that the quality of participation is related to program outcomes (Lieberman, Weston, & Pawl, 1991; Roggman, Boyce, Cook, & Jump, 2001). One notable finding is that the variable that best predicted change was positive climate, suggesting that it is essential to create an appropriate, relaxed, and intimate climate between the facilitator and participants.

Another implementation component related to participant responsiveness was satisfaction with the program. In the group-based version, high levels of satisfaction were associated with an increase in appropriate expectations and greater promotion of the child's independence. This finding is in line with those of other studies that observed a link between high levels of participant satisfaction and positive program outcomes (Garvey, Julion, Fogg, Kratovil, & Gross, 2006; Prado, Pantin, Schwartz, Lupei, & Szapocznik, 2006; Tolan, Hanish, McKay, & Dickey, 2002). This effect was not observed in the home-visit version, where satisfaction with the program did not predict changes; this contrasts with the greater weighting observed for the quality of interactions with the family and the positive climate created during the home visits.

Finally, in the group version, the quality of implementation decreased and program outcomes worsened as the number of barriers to program implementation encountered by the facilitator increased. This is in line with the findings of Eames et al. (2009) and Forgatch, Patterson, and DeGarmo (2005). This illustrates the importance of ensuring that program design takes into account the difficulties that a facilitator may encounter during implementation, as such barriers will affect program quality and outcomes. These barriers can even threaten a program's continuity and sustainability.

Study limitations include the fact that it was not possible to use observational data or recordings as assessment tools, as such tools and techniques are extremely cost-intensive in real-life conditions such as these (Berkel et al., 2011). While they do improve the validity of such studies (Durlak & Dupre, 2008), their use in this case proved too costly and logistically complex to implement.

In sum, few studies have examined the implementation process from a comprehensive standpoint that analyzes all the components involved (Durlak & Dupre, 2008). Even fewer have made a comparative analysis of the differential impact of these implementation components in programs with both group-based and home-visit formats aimed at parents of very young children. This study presents an assessment model that includes a number of implementation components of both the group and home versions of the program, thus permitting a more complete analysis of the process. The main findings include observations that the components of adherence, program delivery, parent participation and interest, and positive climate are the best predictors of program effectiveness. Future assessments of evidence-based programs delivered in both formats will need to take into account the important contribution that these implementation components make to program effectiveness.

Financial support

The evaluation of the program was supported by the Spanish Ministry of Economy and Competitiveness through a project (PSI2015-69971) and research grant to the first and second author.

Conflict of interest

The authors have no conflict of interest to declare.

Acknowledgments

We would like to express our appreciation to all the social agents of the municipalities of the Autonomous Communities of Castile and Leon, Canary Islands, and Catalonia in Spain, as well as the parents who participated in this study. The evaluation of the program was supported by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund under the Grant PSI2015-69971-R (MINECO/FEDER).