Item properties of the DCFS-TW were analyzed using ceiling/floor effects (i.e., how many people responded in the extreme scores of 4 and 1), corrected item-total correlations, and the test-retest reliability for each item score. We proposed the acceptable item properties as follows: ceiling/floor effects < 20%; corrected item-total correlations > .3; test-retest reliability illustrated by Pearson correlation > .3. In addition, we assessed the ceiling/floor effects, Cronbach's α, and test-retest reliability for the entire DCFS-TW score to examine the reliability of the instrument in scale-level.

We examined the construct validity of the DCFS-TW score using the three competing CFA models. Model 1 is the one-factor model without controlling wording effects. Model 2 is the three-factor model found by Struening et al. (2001). Model 3 is based on Model 1 but also controls wording effects; that is, we correlated the item uniqueness of the three negatively worded items. Diagonally weighted least squares estimator was used in all CFAs to account for the categorical data (i.e., four-point Likert scale). In addition, we adopted the following fit indices to help us determine whether the structure of the DCFS-TW score is supported: comparative fit index (CFI) and Tucker-Lewis index (TLI) > .95; root mean square error of approximation (RMSEA) < .06; and standardized root mean square residual (SRMR) < .08 (Hu & Bentler, 1999; Lin, Strong, Tsai, Lin, & Fung, 2018). Moreover, we used χ2 difference test to determine the factorial structure of the DCFS-TW. If Model 3 does not significantly outperform Models 1 and 2, we would not use Model 3 for the following analyses because a factor (or subscale) is recommended to consist of more than three items (Bollen, 1989).

After confirming the structure of DCFS-TW, concurrent validity was determined using Pearson correlation and the comparisons between the correlation coefficients. We first computed the correlations between the DCFS-TW score with other instruments’ scores, including the ASS, MSPSS, CES-D, and RSES. Then, the correlation coefficients derived from the DCFS-TW and the ASS scores were compared with all the other coefficients using a Z test (Steiger, 1980). Because all the instruments measured the psychological aspects of the participants, we anticipated their scores being correlated. However, the correlation between the DCFS-TW and the ASS scores would be the highest because they share more similar psychological concept (i.e., the stigma) than do other instruments.

In order to understand the extent of the DCFS-TW score linking to the clinical characteristics of PWMI, an important concern for mental health professionals, we investigated the known-group validity. Three clinical characteristics (had been hospitalized, had suicidal behaviors, and had been compulsorily admitted) were adopted in the known-group validity because they are related to the level of self-stigma (Chang, Wu, Chen, & Lin, 2016; Rüsch et al., 2014). Before conducting the known-group validity, we tested the measurement invariance of the confirmed DCFS-TW structure using χ2 difference test across each clinical characteristic. Specifically, we tested the differences between three nested models (configural model; model with factor loadings constrained equal across group; model with factor loadings and item intercepts constrained equal across group). Non-significant differences indicate measurement invariance. Several independent t tests with effect size were used to care for the known-group validity; an effect size = 0.2, 0.5, and 0.8 means small, medium, and large, respectively.

The CFAs were analyzed using lavaan package in the R software (Rosseel, 2012), the comparisons of the correlations coefficients analyzed using online tool (Lee & Preacher, 2013), and other data were analyzed using IBM SPSS 23.0 (IBM Corp., Armonk, NY.).

The characteristics of the participants (i.e., the family caregivers) and those of the PWMI are demonstrated on Table 1. The item properties of the DCFS-TW scores were acceptable in the ceiling/floor effects, corrected item-total correlation, and test-retest reliability. However, two items had relatively low test-retest reliability (r = .27 for Item DCFS2 and .23 for DCFS8). The negatively worded items (i.e., items DCFS2, DCFS5, and DCFS6) had the lowest corrected item-total correlations. The entire DCFS-TW also demonstrated satisfactory psychometric properties (Table 2).

The CFA results showed that all the models had all fit indices satisfactory, and all the factor loadings were significant (p < .001). The χ2 difference tests showed that Model 1 was inferior than were Models 2 (Δχ2 = 29.88, Δdf = 2; p < .001) and 3 (Δχ2 = 32.38, Δdf = 3; p < .001). As comparing Models 2 to 3, the χ2 difference test was not significant between the two models (Δχ2 = 2.50, Δdf = 1; p = .11). The correlations among the uniqueness of the three negatively worded items were .13, .21, and .31 (Table 3).

Concurrent validity of the DCFS-TW score was illustrated by the correlations in the anticipated directions between DCFS-TW total score and the scores of other instruments: r=.58 with ASS score; = -.34 with MSPSS score; = .36 with CES-D score; and r = .20 with RSES score. In addition, the correlation between the DCFS-TW score and the ASS score was the highest as compared with other correlations (Z=6.01 to 9.45; p < .001).

The nested models for measurement invariance in Model 3 were not significantly different from each other (Table 4). Hence, we examined the known-group validity as we described in data analysis section. The known-group validity of the DCFS-TW score was supported by two clinical characteristics: hospitalization experience and compulsory admission experience. The caregivers reported higher total scores in the DCFS-TW if the PWMI they cared had been hospitalized (2.30±0.41 vs. 2.14±0.36; effect size = 0.41; p < .001) or had been compulsorily admitted (2.39±0.44 vs. 2.18±0.36; effect size = 0.51; p < .001). However, the caregivers of the PWMI who had suicidal attempt had similar DCFS-TW score to these of the PWMI who had not suicidal attempt (2.26±0.41 vs. 2.22±0.38; effect size = 0.10; p = .23).

This is, to the best of our knowledge, the first study applying series of psychological tests to examine the psychometric properties of the DCFS-TW score in a relatively large sample of Taiwanese caregivers (N=511). Our results showed that the DCFS-TW structure fit the one-factor solution when considering the wording effects. Moreover, the DCFS-TW score is related to the instrument score on stigma more than the instrument scores on psychological health; the DCFS-TW score has adequate reproducibility. In addition, the DCFS-TW score is able to differentiate family caregivers of PWMI who had been hospitalized or who had compulsory admission from those who had not.

Although no studies have used CFA to test the DCFS score, some clinical research did report the internal consistency of the DCFS score. Therefore, we could compare our internal consistency results with others. The satisfactory internal consistency in our study (α = .85) echoes the internal consistency reported previously: α = .86 in parents of children with autism spectrum disorder (Mak & Kwok, 2010); α = .71 and .77 in two samples of caregivers of PWMI (Struening et al., 2001); α = .73 in primary caregivers of people with major depressive disorder (Sher et al., 2005); α = .81 in caregivers of people with psychosis (Sideli et al., 2016). Based on the promising internal consistency, we tentatively concluded that the DCFS items, with or without the two additional items generated by Mak and Kwok (2010), are highly correlated.

In addition to the internal consistency, our findings of CFA support both three-factor and on-factor structures for the DCFS-TW. Unfortunately, our factorial structure results are hard to compare with previous studies because of the dearth in such information. Nevertheless, we believed that our findings echo what Struening et al. (2001) have found in their exploratory factor analysis: a three-factor solution. However, it is commonly viewed as suboptimal for a factor (or subscale) to consist of fewer than three items (Bollen, 1989). Therefore, we believe that our one-factor model, which also shows satisfactory fit indices in the CFA, is adequate and support the commonly practical use of DCFS: Struening et al. (2001) suggested using the DCFS as a uni-dimensional instrument, and the uni-dimensional structure has been applied in all other studies (Mak & Kwok, 2010; Sher et al., 2005; Yang & Singla, 2011).

As we justified earlier that the three factors generated from the exploratory factor analysis in the study of Struening et al. (2001) is due to the wording effects, our CFA results somewhat corroborate our postulation. Specifically, the one-factor CFA model significantly improved in the fit indices after we controlled for the negatively worded items. However, the factor loadings of the three negatively worded items were relatively low, and the correlations between the uniqueness of the three items were weak. Also, improvement was found in the three-factor structure. Therefore, we were unsure whether the three negatively worded items are embedded together with all the positively worded items or the three items belong to other factor(s). Because models with very few items tend to yield better fit indices than longer scales and the rules of thumb generally recommend a minimum of 3 (or even 5) items per factor for CFA (Kenny & McCoach, 2003), we tentatively concluded that the one-factor solution accounting for wording effects is suitable for the DCFS-TW.

Known-group validity supported the DCFS-TW score in two clinical characteristics: hospitalization and compulsory admission. However, the DCFS-TW score was unable to differentiate the family caregivers of PWMI who had suicidal attempt and those of PWMI who had no suicidal attempt. The significant differences of the DCFS-TW score between yes vs. no in the clinical characteristics of hospitalization and compulsory admission can be explained by the elevated public stigma on the two characteristics (Link, Cullen, Frank, & Wozniak, 1987): Most of community residents think those who have been previously hospitalized and involuntarily admitted are dangerous and tend to avoid them. Therefore, family members of PWMI with one of the two clinical characteristics might consequently perceive higher levels of stigma than their counterparts did.

In terms of the non-significant results in relation to suicidal behaviors, we postulated that suicidal behaviors only link to self-stigma and not to perceived stigma (Chang, Wu et al., 2016; Reynders, Kerkhof, Molenberghs, & Van Audenhove, 2014). PWMI with higher levels of self-stigma may consider help from healthcare providers as unimportant (Pattyn, Verhaeghe, Sercu, & Bracke, 2014), which decreases their attempts to seek help when encountering psychological problems, and might subsequently lead to suicidal behaviors. However, such a link between suicidal behaviors and stigma seems only appear in PWMI who has self-stigma, and may not be generalized to the perceived stigma of family caregivers. Nevertheless, future studies are warranted to corroborate our findings.

Because our results demonstrated sound psychometric properties (e.g., the satisfactory test-retest reliability, acceptable internal consistency, and validity) for the DCFS-TW score, we are confident to recommend using the DCFS-TW to assess perceived stigma towards family caregivers of PWMI. Another benefit of the DCFS-TW is its briefness with short administrative time (with a range of 5 to 10minutes), which means that it is efficient to use even in a busy clinical setting. Therefore, we highly recommend that healthcare providers monitor perceived stigma among families of PWMI regularly and design interventions to prevent family caregivers from negative consequence caused by perceived stigma, which may also have a positive influence on the health of PWMI.

Our study findings provided several implications. First, currently only ASS and CESQ can be used to assess self-stigma and experienced stigma, respectively, of a family member who takes care of PWMI. Our DCFS-TW findings filled in the lack of a validated instrument on measuring perceived stigma of a family member. Second, the two added items in the DCFS-TW cover more comprehensively than the original DCFS in the perceived stigma concept. Moreover, the international comparisons using DCFS-TW and DCFS are applicable because we can simply remove the two added item scores from the DCFS-TW for the comparison. Third, given that both Hong Kong and Taiwan use the traditional Chinese characters in the DCFS-TW with the same nine items, we can directly compare the perceived stigma in family caregivers between Hong Kong and Taiwan. Fourth, the DCFS-TW can be used as an important outcome to assess the effectiveness of a stigma-reduction program in the policy level. If the policy is effective, we may detect the reduction in the DCFS-TW score. Moreover, we may approach those family members who have high DCFS-TW score to prevent them from negative consequences due to high levels of perceived stigma.

There are some limitations to this study. First, because we conducted a convenience sampling in the Southern Taiwan in a very specific sample, our results were unable to generalize to all of Taiwan. Moreover, as we did not randomly select our participants, the sample could be biased because the access possibilities of our study were accidental and regional directed. Second, most of the participants were primary family caregivers. Given that primary caregivers are very likely to have different levels of perceived stigma than other caregivers; cautions should be made when applying our results to non-primary caregivers. Third, the DCFS-TW version is slightly different from the original DCFS designed by Struening et al. (2001). Therefore, researchers who want to use the original DCFS should acknowledge that the psychometric properties of the DCFS-TW score might be slightly different from the original DCFS score. Future studies are thus warranted to test the psychometric properties of the different versions of DCFS. For example, we need evidence to determine whether the DCFS-TW really works better than the original DCFS in Taiwanese populations. Suppose that DCFS-TW is more suitable than DCFS to be used in Taiwanese population, a further question is whether the DCFS-TW score in Taiwan is comparable to the original DCFS score in the other countries.

In conclusion, the DCFS-TW is a brief instrument for healthcare providers to efficiently measure the perceived stigma of family caregivers of PWMI. We suggest simply summing up the item scores to represent perceived stigma, which is easy and quick for healthcare providers to do in busy clinical setting. Moreover, the DCFS-TW is a useful instrument which provides valid findings and relevant information to health care providers.