Assessing neighborhood disorder: Validation of a three-factor observational scale

Abstract

This study presents data on the development and preliminary validation of an observational scale assessing neighborhood disorder. Independent observations by trained raters of neighborhood disorder were conducted in 552 census block groups in the city of Valencia (Spain). Intraclass correlation coefficients assessing inter-rater reliability indicated fair to substantial levels of agreement among raters. Confirmatory factor analyses supported a final three-factor model scale measuring physical disorder, social disorder, and physical decay. Results for the internal consistency showed large composite reliability indices indicating good reliability for all neighborhood disorder factors. Evidence of criterion-related validity was found by exploring associations between neighborhood disorder factors and three neighborhood characteristics: neighborhood socioeconomic status, immigrant concentration, and residential instability. Also for criterion-related validity, Moran's I test results for spatial correlation showed that the three types of neighborhood disorder tend to cluster in space and are not randomly distributed across the city. In general, this paper provides evidence of a reliable and valid observational measure to assess neighborhood disorder.

Resumen

Este estudio presenta el desarrollo y validación preliminar de una escala observacional para la evaluación del desorden en los vecindarios. Se realizaron observaciones independientes del desorden por evaluadores entrenados en 552 sectores censales de la ciudad de Valencia (España). Los coeficientes de correlación intraclase para la evaluación de la fiabilidad interjueces indicaron unos niveles adecuados de acuerdo entre jueces. Los resultados del análisis factorial confirmatorio apoyaron un modelo final de tres factores: desorden físico, desorden social y deterioro físico. La evaluación de la consistencia interna mediante composite reliability indices mostró valores elevados para todos los factores. La validez de criterio fue determinada mediante la exploración de las asociaciones entre los factores de desorden del vecindario y tres características del mismo: estatus socioeconómico, concentración de inmigrantes e inestabilidad residencial. Además, como medida de validez de criterio, el test de Moran que evalúa la correlación espacial mostró que los tres tipos de desorden tienden a agruparse espacialmente y no se distribuyen aleatoriamente en la ciudad. En general, este artículo proporciona evidencias de la fiabilidad y validez de una escala para la medida del desorden en los vecindarios.

In recent decades, a large body of literature has examined the influence of neighborhood characteristics on a wide range of outcomes, including health, violence, or crime (Diez-Roux & Mair, 2010; Kawachi & Berkman, 2003; O’Campo et al., 2015; Sampson, 2012; Sampson, Raudenbush, & Earls, 1997). Among these neighborhood characteristics, the concept of neighborhood disorder has played a central role and has received the attention of scholars from different disciplines like sociology, criminology, social psychology or epidemiology. Neighborhood disorder can be defined as “observed or perceived physical and social features of neighborhoods that may signal the breakdown of order and social control, and that can undermine the quality of life” (Gracia, 2014, p. 4325). Examples of neighborhood disorder may include behaviors such as prostitution, drug dealing, and fighting in the streets, or physical characteristics such as abandoned cars, vandalized buildings, or litter in the streets (Sampson & Raudenbush, 1999; Skogan, 1990; Taylor, 2001; Wilson & Kelling, 1982).

The concept of neighborhood disorder can be linked to social disorganization theories and their idea that structural characteristics of neighorhoods, like concentrated disadvantage, can undermine social control and increase levels of violence, crime, and other negative outcomes (Gracia, 2014; Kingston, Huiziga, & Elliot, 2009; Kubrin & Weitzer, 2003; Maimon & Browning, 2010; Park, Burgess, & McKenzie, 1925; Sampson et al., 1997; Shaw & McKay, 1942; Wilson, 1987). Also, the Broken Windows Theory of urban decay has been of particular relevance for the wide appeal of the concept of neighborhood disorder (Wilson & Kelling, 1982). According to this perspective, physical and social cues of neighborhood disorder signal the breakdown of formal and informal social controls leading to further disorder and crime (Gracia, 2014; Perkins, Meeks, & Taylor, 1992; Sampson & Raudenbush, 1999; Skogan, 1990; Taylor, 1997, 2005; Toet & van Schaik, 2012; Wei, Hipwell, Pardini, Beyers, & Loeber, 2005; York Cornwell & Cagney, 2014). According to Gracia (2014) “as neither residents nor external agencies (e.g., police and other authorities) are able or willing to intervene and maintain social order, more disorder is facilitated, and criminal activity is attracted” (p. 4325). Neighborhood disorder would also trigger a number of community processes like fear, insecurity, powerlessness, or mistrust that lead residents to disinvest in and withdraw from community life, increasing social disorganization and neighborhood decline (Geis & Ross, 1998; Kawachi, Kennedy, & Wilkinson, 1999; Kim & Conley, 2011; Ross, Mirowsky, & Pribesh, 2001; Skogan, 1986, 1990). In this regard, neighborhood disorder has been linked to urban decay, concentration of social problems, racial or ethnic segregation, social integration, confidence in the police, or public social control strategies like reporting crime (Gracia, Garcia, & Musitu, 1995; Gracia & Herrero, 2006a, 2006b, 2007; Perkins et al., 1992; Perkins & Taylor, 1996; Ross & Mirowsky, 1999; Skogan, 1990; Taylor, 1997; Toet & van Schaik, 2012).

Although neighborhood disorder has traditionally been studied in relation to street-level outcomes, an increasing body of literature has also examined its influence on processes and outcomes that occur “behind closed doors” (Wright & Benson, 2011), such as parental socialization practices (Gracia, Fuentes, García, & Lila 2012; Lila & Gracia, 2005; McDonell, 2007; Roosa et al., 2005; Tendulkar, Buka, Dunn, Subramanian, & Koenen, 2010; White, Roosa, Weaver, & Nair, 2009; Worton et al., 2014), child maltreatment (Coulton, Crampton, Irwin, Spilsbury, & Korbin, 2007; Coulton, Korbin, & Su, 1999; Freisthler, Bruce, & Needell, 2007; Freisthler, Merritt, & LaScala, 2006; Garbarino & Sherman, 1980; Gracia & Musitu, 2003; Lila & Gracia, 2005; Martin-Storey et al., 2012), or intimate partner violence (Cunradi, 2007, 2009; Gracia, Herrero, Lila, & Fuente, 2009; Gracia, López-Quílez, Marco, Lladosa, & Lila, 2014, 2015; Kirst, Lazgare, Zhang, & O’Campo, 2015; see Beyer, Wallis, & Hamberger, 2015; Pinchevsky & Wright, 2012, for reviews).

More recently, research on social disorder has also examined its influences on individual well-being indicators like subjective well-being, psychological distress, anxiety, or depression (García-Ramírez, Balcázar, & de Freitas, 2014; Herrero, Gracia, Fuente, & Lila, 2012; Hill & Angel, 2005; Hombrados-Mendieta & López-Espigares, 2014; Latkin & Curry, 2003; Latkin, German, Hua, & Curry, 2009; O’Campo et al., 2015; Ross & Mirowsky, 2009), and how this may affect negative health behaviors such as low physical activity, heavy drinking, smoking, or obesity (Burdette & Hill, 2008; Echeverría, Diez-Roux, Shea, Borrell, & Jackson, 2008; Hill, Ross, & Angel, 2005; Keyes et al., 2012; O’Campo et al., 2015; Ross & Mirowsky, 2001). Research has also examined the association between neighborhood disorder and different public health issues such as health service usage, low body weight at birth in children, injuries, sexually transmitted diseases, loss of physical function in older adults, and mortality risk (Balfour & Kaplan, 2002; Cohen et al., 2000, 2003; Martin-Storey et al., 2012; Pearl, Braveman, & Abrams, 2001; Winkleby & Cubbin, 2003).

Assessing Neighborhood Disorder

Assessment of neighborhood disorder tipycally considers two types of disorder, physical and social (Robinson, Lawton, Taylor, & Perkins, 2003; Skogan & Maxfield, 1981; Taylor & Shumaker, 1990). Physical disorder refer to urban landscapes with high levels of decay and deterioration. For example, abandoned houses, graffiti, trash on the streets, abandoned cars, used needles, and vacant lots would exemplify physical disorder (Brunton-Smith, 2011; Garvin, Cannuscio & Branas, 2013; Robinson et al., 2003; Sampson & Raudenbush, 1999; Skogan, 1990; Taylor, 2001, Toet & van Schaik, 2012). Some scholars, however, make a further distinction between physical disorder and physical decay: physical disorder would refer to features like dirt in the streets (litter, bottles, condoms), graffiti, abandoned cars, etc. (i.e., behavioral manifestations), whereas physical decay would refer to structural characteristics that can arise from lack of institutional investments and have long term effects, such as abandoned buildings, burn-out houses, badly deteriorated recreational facilities, etc. (Sampson, 2009; Sampson & Raudenbush, 2004). As Sampson and Raudenbush (2004) argue, it is important to make this distinction because physical disorder is “limited to behavioral manifestations (e.g., graffiti, garbage in the streets) that can be conceptually decoupled from structural resources” (p. 326). Social disorder refer, on the other hand, to events in public places seen as potentially threatening, and can be exemplified by the presence of people taking drugs or alcohol in the street, drug dealing, fights and arguments, presence of homeless people, public drunkenness, street prostitution, high levels of police activity, and other criminal or not criminal activities that create a sense of danger (Gracia, 2014; Gracia & Herrero, 2007; Robinson et al., 2003; Ross & Mirowsky, 2001; Sampson, 2009; Sampson & Raudenbush, 2004). Despite some studies suggesting that physical and social disorder may overlap, being order and disorder two ends of a single continuum (Ross & Mirowsky, 1999; Xu, Fielder, & Flaming, 2005), most studies support the distinction between physical and social disorder (Brunton-Smith, Sindall, & Tarling, 2010; LaGrange, Ferraro, & Supancic, 1992; Sampson & Raudenbush, 2004; Taylor & Shumaker, 1990).

In order to assess neighborhood disorder, researchers generally use three different approaches (McDonell & Waters, 2011; Mooney et al., 2014). One approach, based on a more objective perspective, draws from neighborhood information from governmental or commercial data sources (Cerdá et al., 2009; McDonell, 2007; Mooney et al., 2014). Although these data is freer from the variability and subjectivity of subjective perceptions of disorder (Kubrin, 2008), however, this information is “often collected for administrative purposes, may not fully capture the construct of research interest, and may be collected at a spatial resolution that is not optimal for research purposes” (Mooney et al., 2014, p. 626-627). A second, and widely used, approach is based on resident's perceptions of their neighborhood physical or social characteristics. A number of limitations have been noted, however, regarding this approach, including “same source bias” (e.g., same source reporting perceived neighborhood disorder and related outcomes), confusion with other psychological constructs (e.g., fear of crime), or the influence of stereotypes and neighborhood prejudices (e.g., racial, ethnic, or socioeconomic composition) in perceptions of disorder (Caughy, O’Campo, & Patterson, 2001; Duncan & Raudenbush, 1999; Gómez, Johnson, Selva, & Sallis, 2004; Mooney et al., 2014; Sampson, 2009; Sampson & Raudenbush, 1999, 2004; Schaefer-McDaniel, Caughy, O’ Campo, & Gearey, 2010). Finally, a third approach, that aims to overcome the above limitations, emphasizes the importance of using direct and systematic observations of neighborhood characteristics by trained researchers (Franzini, Caughy, Nettles, & O’Campo, 2008; McDonell, 2007; McDonell & Waters, 2011; Neil, Parke, & McDowell, 2001; Raudenbush & Sampson, 1999; Reiss, 1971; Sampson & Raudenbush, 1999). This approach aims to obtain objective measures of neighborhood conditions, to capture a wide range of factors, which are not always available otherwise, and to allow its replication in other contexts (Caughy et al., 2001; Cohen et al., 2000; Franzini et al., 2008; McDonell & Waters, 2011; Sampson & Raudenbush, 2004; Taylor, 2001).

The Present Study

This study aims to add to this body of research by validating an observational measure of neighborhood disorder in the context of a European city that may differ from the culture and structure of Anglo-Saxon cities where most of this type of measures have been developed and validated (Le Galès & Zagrodzki, 2006; Summers, Cheshire, & Senn, 1999). The availability of a reliable and valid measure of neighborhood disorder in this context may be an important addition to the growing body of literature exploring neighborhood effects (Sampson, 2012). To this end, independent observations by trained raters of neighborhood disorder will be conducted using small-areas of the city as the ecological units (i.e., census block groups, which are the smallest administrative sections of the city available). By using the smallest posible geographical units of the city we expect greater homogeneity of neighborhood characteristics (Ocaña-Riola et al., 2008). Also, by using all census block groups of the city we will obtain greater variability and, as neighborhood characteristics tend to cluster in space, this will provide the possibility to explore the clustering of these characteristics for validation purposes (Gracia et al., 2014).

This study presents data on the development and preliminary validation of an observational instrument to assess neighborhood disorder. The specific objectives of the study are: (1) to test the inter-rater reliability of the scale; (2) to test the factorial validity of the scale using confirmatory factor analysis: we expect a factorial structure reflecting three theoretically a priori factors, physical disorder, social disorder, and physical decay (Sampson & Raudenbush, 2004); (3) to test the reliability of the scale by means of the composite reliability index; and (4) to test the criterion-related validity of the scale also employing structural equations. Drawing from social disorganization theory we expect associations between neighborhood disorder and three neighborhood structural characteristics, central in this theoretical perspective: neighborhood socioeconomic status, immigrant concentration, and residential instability (Caughy et al., 2001; Jones, Pebley, & Sastry, 2011; Kubrin & Weitzer, 2003; McDonell & Waters, 2011; Mooney et al., 2014; Sampson et al., 1997; Sampson & Raudenbush, 1999). Also, for validation purposes, a spatial perspective will be applied. As we expect that disordered neighborhoods will cluster in space, rather than be randomly distributed in the city, spatial correlation analyses will be conducted to test whether neighborhood disorder shows a significant spatial pattern (Bruinsma, Pauwels, Weerman, & Bernasco, 2013; Gracia et al., 2015; Quick, 2013; Veysey & Messner, 1999).

Method

Sample

This study was conducted in the city of Valencia, the third largest city of Spain. As proxy of neighborhood units we used census block groups that were the smallest administrative unit of the city available. Census block groups can be defined as walkable areas within a few number of city blocks and as they are smaller than census tracks, they are particularly appropriate for neighborhood studies (Gracia et al., 2014; Sampson & Raudenbush, 2004). Observations by trained raters were conducted in each of the 552 census block groups in which the city is divided. The total population for these census block groups was 736,580 inhabitants (2013 data), with an average of 1,334 inhabitants per census block group (ranging from 630 to 2,845).

Measures

Neighborhood disorder observation scale. A neighborhood disorder scale was initially constructed with a total of 20 items based on three dimensions of neighborhood disorder proposed by Sampson and Raudenbush (Sampson & Raudenbush, 2004). Thus, the scale included three theoretically motivated subscales measuring physical disorder, social disorder, and physical decay. Physical disorder was defined by 8 items: cigarettes in the street, trash in the street, empty bottles in the street, graffiti, abandoned cars, used condoms and syringes in the street, and political or protest message graffiti. Social disorder was defined by 7 items: adults or young people loitering, people drinking alcohol in public, gangs, public intoxication, adults fighting or arguing, selling drugs, and street prostitution. Physical decay was defined by 5 items: vacant or abandoned houses, abandoned commercial buildings, vandalized and run-down buildings, deteriorated residential units and deteriorated recreation places. The observations are rated on a 5-point response scale (from 0 = no presence, to 4 = highly present). Two trained raters walked each census block group in order to complete the observational scale. All observations were made during business hours.

Criterion-related validity measures. Drawing from social disorganization theory, to test criterion-related validity we will explore relationships between neighborhood disorder and three neighborhood characteristics measured at census block group level: neighborhood socioeconomic status, immigrant concentration, and residential instability. The City of Valencia Statistics Office provided these data for each census block group. Neigbhorhood socioeconomic status was measured with an indicator created through factor analysis (this factor included educational level, property value, percentage of high-end cars, and financial and commercial activities). Immigrant concentration was the percentage of immigrant population in each census block, and residential instability was the proportion of the population who had moved into or out of each census block group during the previous year (rate per 1,000 inhabitants).

Statistical Analysis

To measure inter-rater reliability, two pairs of trained undergraduate students walked a random sample of the census block groups. They observed a subset of 15% of them aproximately (N = 86). Inter-rater reliability scores were computed per each of the three scales by calculating intraclass correlation coefficients (ICC) due to the quantitative nature of data. This analysis was performed with SPSS 22 for Windows.

Several competing confirmatory factor analyses (CFA) were specified, estimated and tested in Mplus 7.3. According to the ordinal nature of the data and its non-normality, WLSMV (weighted least squares mean and variance corrected) estimation was used, the one recommended in the literature (Finney & DiStefano, 2006). Several criteria were used to assess goodness-of-fit: (a) the chi-square statistic, (b) the comparative fit index (CFI), and (c) the root mean squared error of approximation (RMSEA). A model with a CFI of .95 or larger and a RMSEA of .08 or lower would be indicative of very good fit between the hypothesized model and the data (Hu & Bentler, 1999). Nevertheless, overall fit must be accompanied by a careful diagnosis of the analytical fit (parameter estimates) in the model in order to not blindly use the aforementioned thresholds (Kline, 2011). For model comparison, a modeling approach that uses practical fit indices to determine the overall adequacy of a fitted model has been used as recommended by Cheung and Rensvold (2002) or Little (1997). From this point of view, if a parsimonious model evidences adequate levels of practical fit, then it is preferred over the more complex model. Usually, CFI differences (ΔCFI) are used to evaluate measurement invariance. CFI differences lower than .01 (Cheung & Rensvold, 2002) or .05 (Little, 1997) are usually employed as cut-off criteria.

Additionally, internal consistency of the dimensions in the scale has been estimated with the composite reliability index (CRI). Although Cronbach's coefficient alpha is the most widely used estimator of internal consistency, it has been criticized as being only completely appropriate with essentially tau-equivalent items (and tests) and also by being a lower bound for the true reliability (Raykov, 2004). More explicitly, a tau-equivalent test assumes all items measure the same latent variable, on the same scale, with the same degree of precision, with all true scores being equal (Graham, 2006). When tau-equivalence does not hold, alpha will over- or under-estimate (more often the latter) the population value. An alternative to the coefficient alpha is the CRI, which is usually calculated using estimates from confirmatory factor analyses (Graham, 2006). Accordingly, the more adequate CRI was employed.

Criterion-related validity was established by correlating neighborhood disorder factors with other neighborhood constructs theorically linked in the literature (Gracia, 2014; Gracia et al., 2015; Kubrin & Weitzer, 2003; Sampson & Raudenbush, 1999; Sampson et al., 1997; Shaw & McKay, 1942). This correlation was obtained within the context of a structural equation modeling in order to prevent as much as possible the correlation attenuation due to measurement error. Specifically, neighborhood disorder was correlated to neighborhood socioeconomical status, immigrant concentration, and residential instability (see Measures section).

To test criterion-related validity we also used a spatial methodology approach. To assess spatial autocorrelation, we computed Moran's I (Moran, 1950) per each of the three subscales or factors, considering as the observation the midpoint of each of the census block groups. We expected a significant spatial distribution of neighborhood disorder, rather than a random distribution, because we expect that disorder, as other neighborhood characteristics, will show a tendency towards spatial clustering (Bruinsma et al., 2013; DiMaggio, 2015; Quick, 2013; Veysey & Messner, 1999).

Results

Inter-Rater Reliability

Intraclass correlation coefficients were computed to assess inter-rater agreement for the three subscales (see Odgers, Caspi, Bates, Sampson, & Moffit, 2012, for a similar approach). Intraclass correlations ranged from .25 to .71 (see Table 1). Landis and Koch's (1977) criteria were used to interpret results regarding inter-rater agreements: < .20 slight, .21 – .40 fair, .41 – .60 moderate, .61 – .80 substantial, and .81 – 1 almost perfect agreement. Our results indicated fair to substantial levels of agreement between raters (Landis & Koch, 1977). Social disorder obtained the lowest value and physical disorder and physical decay showed similar results.

Table 1.

Inter-rater Agreement. Intra-class Correlations Coefficients (ICC)

Scales	No. of items	M (SD)	ICC1	ICC2
Physical disorder	8	5.97 (3.54)	.55***	.71***
Social disorder	7	0.57 (0.79)	.25***	.40***
Physical decay	5	2.64 (2.82)	.46***	.63***

Note. ICC₁ = index of reliability for a single rater.

ICC₂ = index of reliability for multiple raters averaged together.

* < .05, ** < .01, *** < .001

Confirmatory Factor Analysis

Three a priori competing models were specified. The theoretical model that supports the content validity of the scale a priori hypothesizes three dimensions: physical disorder, social disorder, and physical decay. Indicators were developed to tap these three theoretical dimensions. However, there was doubt about whether two of these dimensions could be too overlapped to have discriminant validity: physical disorder and physical decay. Accordingly, another a priori model was specified with two dimensions: social disorder and all the indicators of physical disorder and physical decay specified to load onto a single dimension. Finally, the most parsimonious latent structure, a one-factor model underlying all the indicators, was also specified. Goodness-of-fit indices for these three a priori models are shown in Table 2. Model fit for the three models was extremely poor, and none of them could be retained as a good approximation to the observed data. Nevertheless, the three-factor model showed a relative better fit compared to the other two models: ΔCFI = .06, compared to the one-factor model and ΔCFI = .04, compared to the two-factor model. Taking this information into account, plus the fact that the theoretical model that supports the scale was three-dimensional, this model was retained for further psychometric scrutiny.

Table 2.

Goodness of Fit Indices for the Tested Models

	χ^2	df	p	CFI	RMSEA	90% CI
One-factor model	1471.75	152	< .001	.643	.12	.12 - .13
Two-factor model	1407.56	151	< .001	.660	.12	.11 - .12
Three-factor model	1272.59	70	< .001	.700	.11	.11 - .12
Modified three-factor model	278.61	87	< .001	.940	.06	.05 - .07

A careful look at the factor loadings, together with the lack of variability in some indicators, allows to remove some of them. Those removed lacked variability and/or had poor consistency with their dimension. The final version of the questionnaire was presumed to measure three factors (physical disorder, social disorder, and physical decay) with 5, 6, and 4 indicators each (see Appendix). This depurated version of the original scale was tested and its goodness-of-fit indices are shown in Table 2. There was a huge improvement in model fit, and it can be said that the model seems to adequately represent the observed data. Factor loadings are shown in Figure 1. They were all statistically significant (p < .01) and, in general, pretty large. These results are indicative of good analytical fit.

Figure 1.

Standardized factor loadings and correlations for the confirmatory factor analyses of the Neighborhood Disorder Observational Scale.

Note. All coefficients statistically significant (p < .01); Intoxicat = intoxication; Dealing = drug dealing; Vacant = vacant houses; Vandal. = vandalized buildings; Residen. = residential deterioration; Deter. = deteriorated facilities.

Internal Consistency

Reliability (internal consistency) estimates were calculated for each dimension or factor in the scale. The calculated reliability estimates were composite reliability indices (CRI), as already mentioned in the method section. All the CRI were large, indicating a good reliability for all the dimensions. Specifically, CRI for physical disorder was .83, .82 was the estimate for social disorder, and .82 the internal consistency estimated for physical decay.

Criterion-Related Validity

Finally, evidence of criterion-related validity was also found. Three criteria were used (neighborhood socioeconomical status, immigrant concentration, and residential instability). The relationship between the criteria and the latent variables were calculated at the latent level, not the observed one. A first structural model included the measurement model found to fit well to the data, plus the first criterion (neighborhood socioeconomical status). Again, this structural model fitted the data well, χ²(99) = 339.78, p < .001; CFI = .93, RMSEA = .066 [.059 - .074]. The correlations between neighborhood socioeconomic status and the criteria and the factors were: ρ = -.13, p < .01 with physical disorder; ρ = .04, p > .05 with social disorder; and ρ = -.29, p < .01 with physical decay. With respect to immigrant concentration, the structural model also adequately fitted the data, χ²(99) = 309.38, p < .001; CFI = .93, RMSEA = .062 [.054 - .070]. The correlations among the criterion and the factors were: ρ = .44, p > .05 with physical disorder; ρ = .13, p < .05 with social disorder; and ρ = .21, p < .01 with physical decay. Finally, a third structural model was specified and tested to relate the three factors with residential instability. Again the model fitted the data well, χ²(99) = 333.54, p < .001; CFI = .92, RMSEA = .066 [.058 - .073], and its correlations with the factors were: ρ = .05, p > .05 with physical disorder; ρ = .15, p < .05 with social disorder; and ρ = .22, p < .01 with physical decay.

Also, for criterion-related validity purposes, tests for spatial correlation were conducted for the three dimensions. Results showed spatial autocorrelation in the three scales as all Moran's I values were positive and significant (p < .001). Moran's I values for the three measures were .20 for social disorder, .39 for physical decay, and .49 for physical disorder, indicating a stronger spatial pattern for physical disorder and physical decay than for social disorder. These results indicate a positive non-random distribution of all types of neighborhood disorder in the city (i.e., rather than being randomly distributed across the city, they tend to cluster in space).

Discussion

In this paper we describe the development and the psychometric properties of a preliminary validation of an observational scale assessing neighborhood disorder. This scale was implemented in the city of Valencia (Spain), using independent observations of all census block groups of the city conducted by trained raters. In general, results have shown that this scale is a psychometrically sound and valid instrument to assess three neighborhood disorder dimensions: physical disorder, social disorder, and physical decay.

Results regarding inter-rater reliability showed fair to substantial levels of agreement (Landis & Koch, 1977), with stronger agreements for physical disorder and physical decay factors, and lower inter-rater reliability for social disorder. Although these results are slightly lower than in other studies (e.g., Caughy et al., 2001; Franzini et al., 2008; Jones et al., 2011), they are, however, comparable to others (e.g., Mooney et al., 2014; Wei et al., 2005). As to why social disorder showed lower levels of agreement, one possibility is that social disorder cues such as fights or public intoxication tend to be less stable over time (e.g., depending on the time of the day) than other physical features of neighborhoods, like those indicating physical decay and disorder, that are more temporally stable (Jones et al., 2011).

With respect to the factor structure of the observational scale, our analyses aimed to validate a three theoretically-based neighborhood disorder dimensions (Sampson & Raudenbush, 2004). To this end, several competing confirmatory factor analyses were first estimated and discarded, yielding a final and depurated version of the scale with good analytical fit. This final scale supported a three-factor model measuring physical disorder, social disorder, and physical decay, as theorized by Sampson and Raudenbush (2004).

This final three-factor model was obtained after removing some items for their low variability or poor consistency with their dimension. Although these items are usually present in other scales measuring neighborhood disorder, however, indicators such as abandoned cars, used condoms and syringes (physical disorder), prostitution (social disorder), or abandoned commercial buildings (physical decay) were discarded in the final model. The presence of these items was very low and, in the case of syringes, there was no presence at all in any of the census block groups observed. Although the time of the observation may have influenced the low presence of some of these indicators (e.g., prostitution), another posible explanation is that, given the characteristics of cities like Valencia, with high density in a relative small area (Le Galès & Zagrodzki, 2006; Summers et al., 1999), some of these indicators could be more present in the outskirts of the city, and therefore outside of the boundaries where the observations were made. This also suggests that differences between the present scale and others developed elsewhere may reflect context-specific features of the cities. Given that these results may be context-dependent, we do not favour the uncritical use of the short-version of the originally proposed scale destilled for this particular study. On the contrary, careful theoretical considerations previous to the use of this scale in other cities should consider whether some (or all) of the items removed could have enough variability and importance as to be included in the instrument. Clearly, a subsequent depuration of the scale according to its psychometric properties is always possible. On the other hand, results for internal consistency of the final three-factor scale by confirmatory analyses also supported its reliability, with CRI values between .82 and .83 for the three scales.

Two different criterion-related validity tests of this observational measure of neighborhood disorder were conducted. For the first one, and drawing from social disorganization theory, we explored associations with three criteria tapping neighborhood characteristics central to this theoretical perspective: neighborhood socioeconomic status, immigrant concentration, and residential instability, masured at the census block group level. As expected, correlations between measures of neighborhood disorder and these structural characteristics of neighborhoods were mostly in the expected direction (Kubrin & Weitzer, 2003; Sampson & Raudenbush, 1999; Sampson et al., 1997), although with stronger associations for physical decay. For example, high levels of physical decay were significantly associated with lower neighborhood socioeconomic status, higher rates of immigrant concentrarion, and higher residential instability. Social disorder was positively related to levels of immigrant concentration and residential instability, but its association with neighborhood socioeconomic status was not statistically significant. Physical disorder was also negatively related to neighborhood socioeconomic status; however, associations with immigrant concentration and residential instability did not reach significance. These results partly support previous research where significant associations between disorder and a number of neighborhood characteristics were also found, especially those regarding the relationship between physical disorder and decay and neighborhood socioeconomic indicators (Caughy et al., 2001; Jones et al., 2011; McDonell & Waters, 2011; Mooney et al., 2014; Sampson & Raudenbush, 1999). Also, some of our results support studies that fail to find a significant relationship between physical disorder and residential instability (Sampson & Raudenbush, 1999), or between physical disorder and immigrant concentration (Jones et al., 2011). It is interesting to note that physical decay was associated with all neighborhood characteristics (socieconomic status, immigrant concentration, and residential instability), suggesting that differentiating between physical disorder and physical decay is an important theoretical distinction that may provide a more detailed analysis when assessing neighborhood disorder and exploring its relationships with different outcomes and processess (Sampson, 2009; Sampson & Raudenbush, 2004).

As we expected that disordered neighborhoods would tend to cluster together in space (Bruinsma et al., 2013; DiMaggio, 2015; Quick, 2013; Veysey & Messner, 1999), for the second criterion-related validity test we used a spatial analytical approach to assess the spatial distribution of the different types of neighborhood disorder across all census block groups observed. Results showed that the three types of disorder (physical disorder, social disorder, and physical decay) were spatially clustered, confirming that they were not randomly distributed in the city. This reflects the existence of different areas of the cities where neighborhood disorder tends to concentrate and shows that this neighborhood risk factor tend to cluster in space. These results support the idea that different manifestations of neighborhood disorder, as other characteristics of the cities, are not randomly distributed in space. As illustrated by a growing body of literature linking neighborhood disorder with a wide array of outcomes, including crime, violence, or health, the spatially patterned nature of this risk factor makes more likely that related outcomes will also be spatially patterned (Cunradi, Mair, Ponicki, & Remer, 2011; Diez-Roux & Mair, 2010; Freisthler et al., 2007; Gracia et al., 2015; Kawachi & Berkman, 2003; Law, Quick, & Chan, 2014; O’Campo et al., 2015; Sampson, 2012).

Finally, this study has both strengths and limitations. Among the strenghts, the use of independent observations of neighborhood conditions, rather than residents’ subjective perceptions, allows to overcome some of the limitations noted in the literature regarding this later approach (Caughy et al., 2001; Mooney et al., 2014; Sampson & Raudenbush, 1999; Schaefer-McDaniel et al., 2010). Relatedly, for the observation of neighborhood disorder at the level of ecological units (rather than the personal level), we also use a high-resolution approach. We used the smallest administrative units available (i.e., census block groups) that allow greater homogeneity and precision than lower resolutions such as census tracks or postal codes, commonly used in other neighborhood studies (Beyer et al., 2015; Bursik, Grasmick, & Chamlin, 1990; Kaufman, Dole, Savitz, & Herring, 2003; Campo, Xue, Wang, & Caughy, 1997). By using small-area units, we also reduced potential ecological bias, as this resolution is closer to the individual level (Gracia et al., 2015; Lawson, 2006, Ocaña-Riola et al., 2008). We used census block groups in our study, which substantially reduced this potential bias. Finally, we used all census block groups of the city, rather than selecting only a sample of them, which provided greater variablity and the possibility to explore potential significant spatial patterns in the distribution of neighborhood disorder across the city (Caughy et al., 2001; Mooney et al., 2014). In this regard, the use of spatial methods to complement the criterion-related validity of our observational scale is an important addition to the existing literature, as neighborhoods, from this perspective, are not treated as independent units (Gorman, Gruenewald, & Waller, 2013; Mooney et al., 2014; Morenoff, Sampson, & Raudenbush, 2001). Although widely used in epidemiological studies (Lawson, 2009), and despite its advantages, this methodological approach is still uncommon in neighborhood studies, and future research would clearly benefit from incorporating a spatial perspective (Cunradi et al., 2011; Gracia et al., 2014, 2015; Law & Quick, 2013; Law et al., 2014; Sparks, 2011). As for limitations, as noted above, some measures of neighborhood disorder may have been affected by the time of the day they were observed. Our observations were limited to business hours, and the same places may have shown different characteristics at night (Caughy et al., 2001). Future research would benefit from including different observations during the day, and revisiting the same areas during night hours. Also, other neighborhood indicators such as trash in the streets may be present only at specific moments, and repeated obervations of the same area would be advisable, although clearly more costly (Wei et al., 2005). In this regard, recently new technologies, such as virtual environments, or Google Street View, provide powerful and easy accesible tools that may help to enhance neighborhood research (Odgers et al., 2012; Toet & van Schaik, 2012).

In conclusion, this paper provides evidence of a reliable and valid observational measure to assess neighborhood disorder. Adequate measures to assess neighborhood characteristics are important research and intervention tools, as they are key to better understanding neighborhood proceses, as well as to evaluate related outcomes and monitor changes after grass-roots efforts or oficial initiatives to reduce neighborhood inequalities.

Conflict of Interest

The authors of this article declare no conflict of interest.

Financial Support

This research was supported by the Spanish Ministerio de Economía y Competitividad (PSI2014-54561-P). Miriam Marco was supported by the FPU program of the Spanish Ministerio de Educación, Cultura y Deporte (FPU2013/00164).