Skip to main content

Why are only some children with autism spectrum disorder misclassified by the social communication questionnaire? An empirical investigation of individual differences in sensitivity and specificity in a clinic-referred sample

Abstract

Background

The Social Communication Questionnaire (SCQ) is a checklist for autism spectrum disorder (ASD) commonly used in research and clinical practice. While the original validation study suggested that the SCQ was an accurate ASD screener with satisfactory sensitivity and specificity, subsequent studies have yielded mixed results, with some revealing low sensitivity, low specificity, and low utility in some settings.

Method

The present study examined the psychometric properties of the SCQ as well as the individual difference characteristics of 187 individuals with and without autism spectrum disorder (ASD) who were misclassified or accurately classified by the SCQ in a clinic-referred sample.

Results

The SCQ showed suboptimal sensitivity and specificity, regardless of age and sex. Compared to true positives, individuals classified as false positives displayed greater externalizing and internalizing problems, whereas individuals classified as false negatives displayed better social communication and adaptive skills.

Conclusions

The findings suggest that non-autistic developmental and behavioral individual difference characteristics may explain high rates of misclassification using the SCQ. Clinicians and researchers could consider using the SCQ in combination with other tools for young children with internalizing and externalizing symptoms and other more complex clinical presentations.

Heterogeneity in symptom presentation makes accurate screening and assessing for autism spectrum disorder (ASD) particularly challenging. While a variety of measures have been developed to identify children at risk for ASD, a commonly used tool in research and clinical practice is the Social Communication Questionnaire (SCQ, formerly named Autism Screening Questionnaire) [1]. The SCQ was developed to be a reliable and valid screener for ASD [2]. It has been widely used in the clinical setting for triage and referral decision-making, as well as in research settings to serve as a screening tool for inclusion or exclusion criteria. Studies examining the SCQ have yielded mixed results, with some suggesting low sensitivity, low specificity, and low utility within certain clinical settings due to its unsatisfactory psychometric properties [3,4,5,6,7,8,9]. Given the recent literature suggesting that the SCQ may not be uniformly useful as a screening tool for ASD, research is needed to better understand factors that might be affecting SCQ’s sensitivity and specificity.

In a recent meta-analysis that systematically assessed the accuracy of the SCQ over the last 15 years, the authors concluded that variability in sampling methods impacted the accuracy of the SCQ for ASD screening [10]. Indeed, the SCQ had significantly higher sensitivity and specificity in samples that were population reference samples compared to clinical samples, community samples, and convenience samples, and the effect size was the biggest when comparing population samples to clinical samples [11]. Moreover, a recent study of a clinic-referred sample of verbally fluent children and adolescents found that the SCQ performed no better than chance in correctly classifying children with ASD and those with non-spectrum diagnoses [3]. The poor performance of the SCQ in accurately classifying ASD among clinical samples further suggested that individual characteristics may influence variability in the SCQ’s effectiveness as a screening tool [12, 13].

An emerging literature suggests that individual variables that may explain variability in SCQ effectiveness include the demographic factors of age and sex. Several studies reveal lower accuracy for the suggested SCQ cut-offs at younger ages. Maintaining a cut-off of 15, the SCQ has demonstrated satisfactory results in school-age populations [5, 14]; however, the sensitivity and specificity hover between 0.50 to 0.75 for younger populations [4, 5, 7, 10]. For children under 8 years old, Corsello et al. [5] also reported low sensitivity and specificity for the published cut-off of 15 and suggested lowering the SCQ cut-off score to ≥ 11 (for age under 5) or ≥ 12 (for age 5–7) to achieve better sensitivity. Subsequent studies confirmed a cut-off score of 11 maximizes sensitivity and specificity (a better balance between these two critical indices) in young children [13, 15,16,17]. Another study found that the recommended cut-off of 15 had accurate sensitivity and specificity for adolescents and adults (ages 13–21), but a cut-off of 11 worked best for children ages 4 to 12 years [11].

Sex differences have also been found on assessment measures for ASD [18, 19], although only a few studies have examined sex differences in SCQ accuracy. Evans and colleagues found that the SCQ performed adequately in boys (sensitivity 0.77; specificity 0.66) and well in girls (sensitivity 1; specificity 0.71) among 272 school-age children [20]. However, in a multisite case-control study of young children with either ASD or non-spectrum developmental disabilities, males scored significantly higher than females on the SCQ in both groups [15]. Several studies did not compare sensitivity and specificity directly but found similar SCQ mean total scores for males and females [2, 3, 5, 13]. Overall, extant research examining the impact of age and sex on SCQ accuracy has produced conflicting results, and work is needed to understand whether these demographic factors impact the effectiveness of SCQ as a screening tool.

In addition to demographic characteristics, individual differences in clinical presentations may also influence variability in the SCQ results [12, 13]. Previous studies have continued to find poor sensitivity–specificity balance in distinguishing between ASD and other developmental disorders among children [21, 22]. Suren and colleagues [23] found that sensitivity of the SCQ was higher for children with ASD who had not developed phrase speech and were significantly delayed in their cognitive development. Similarly, other studies have revealed satisfactory sensitivity (ranging from 0.88 to 1.00) but poor specificity (ranging from 0.17 to 0.43) in distinguishing between individuals with intellectual disabilities with and without ASD [24, 25]. Multiple studies have found a negative correlation between SCQ total scores and adaptive skills and intelligence quotient (IQ) [7, 9, 26, 27]. Findings suggest that parent report on the SCQ is related to individual differences in children’s cognitive ability, language ability, and adaptive functioning.

Internalizing and externalizing psychopathology are also candidate factors for explaining variation in SCQ effectiveness. Studies examining the clinical utility of the SCQ in samples of children with a variety of neurodevelopmental and mental health diagnoses generally have found lower sensitivity and specificity than in the original validation study [3, 7, 26, 27]. For example, using a heterogenous sample, Hollocks et al. [3] found generally adequate sensitivity when using the SCQ cut-offs ranging from 13 to 20, but specificity below 0.40 for all cut-offs (the recommended cut-off of 15 had a sensitivity of 0.84 and specificity of 0.13). Ung et al. [26] also reported suboptimal sensitivity (0.70) and specificity (0.67) at the recommended cut-off of 15, and a cut-off of 11 improved sensitivity to 0.82, at a cost to specificity (0.37). Moreover, studies comparing children with ASD and attention-deficit hyperactivity disorder (ADHD) have found adequate sensitivity and specificity of the SCQ [28, 29], while one study found the optimal cut-off of the SCQ was lower than the recommended cut-off of 15 for differentiating ASD from ADHD [28]. These findings suggested that the SCQ scores are highly influenced by non-autism developmental and behavioral factors, which further raise questions as to whether sensitivity and specificity of the SCQ are adequate for screening purposes.

Taken together, individual differences in clinical presentation, such as cognitive ability, language ability, comorbidities, and the presence of challenging behaviors seem to be impacting the effectiveness of the SCQ. Other factors including sample referral source, sample heterogeneity, and parental understanding of ASD may also impact the performance of the SCQ. Studies to date have identified individual difference factors described above as broad categories of features that can influence the accuracy of SCQ, but more research is needed on the specific behavioral and developmental features that impact accuracy, especially those that cross diagnostic categories.

Present study

The current study aims to overcome the limitations of prior research by focusing on a heterogeneous group of children from a clinic-referred sample and examining whether individual difference characteristics differentiate those who are misclassified by the SCQ. The first aim of the present study adds to the literature on SCQ accuracy by estimating its sensitivity and specificity in a large clinical-referred sample. Based on prior literature, we hypothesize that the SCQ will show lower sensitivity and specificity than that presented in the original validation study. The second aim is to examine the individual characteristics of children misclassified by the SCQ. We hypothesize that the presence of more comorbid conditions, lower cognitive ability, greater autism severity, greater internalizing and externalizing symptoms, and lower adaptive functioning may explain why the sensitivity and specificity of the SCQ are poor in clinic-referred samples. The results of the study provide valuable information when considering using the SCQ in a clinical or research setting, especially for a population who are seeking their first-time ASD diagnosis or assessment.

Methods

Participants

Participants included individuals who received diagnostic neurodevelopmental evaluations at an autism specialty clinic, referred by neontologists, pediatricians, geneticists, pediatric neuropsychologists, psychiatrists, as well as other professionals who work closely with the families (i.e., speech pathologists, occupational therapists). Clinical diagnoses were made by licensed psychologists with extensive experience in the assessment and treatment of ASD and common co-occurring conditions (5–15 years of clinical experience). Licensed psychologists integrated multiple sources of information, including medical history, educational records, caregiver reports, neuropsychological assessments, and comprehensive interviews regarding autistic characteristics to make diagnostic decisions based on DSM-5 criteria. The SCQ was collected as part of the diagnostic evaluation but was not the sole factor considered when making a diagnosis. The methods were approved by the Institutional Review Board at the University of Minnesota.

From 2016 to 2019, data from 793 consecutive evaluations were entered into a de-identified clinical database. To examine the current study hypotheses, only initial evaluation data (N = 519) were included in the analysis. Data of individuals who were younger than 4 years old at their initial evaluations (N = 142) were excluded from analyses. Individuals who did not complete or only partially completed the SCQ (N = 162) were also excluded. Finally, only participants who had completed all the measures used in the present study (described below) were included. Ultimately, initial diagnostic evaluations from 187 children and adolescents were included in the present study.

Among all individuals, 133 (94 males and 39 females) were clinically diagnosed with ASD, and 54 (34 males and 20 females) were identified as non-spectrum with other neurodevelopmental challenges. Within the non-spectrum sample, 32 received a primary diagnosis of ADHD (59.3%), 10 were diagnosed with a language disorder (18.5%), 5 had behavioral disorders (9.3%; e.g., oppositional defiant disorder, conduct disorder, disruptive behavioral disorder), 27 had anxiety disorders (50.0%), 7 had mood disorders (13.0%), and 8 had other genetic and/or physical disabilities (14.8%; e.g., fragile X syndrome, Williams syndrome, or mild cerebral palsy).

Measures

Demographic characteristics were collected through a clinical intake questionnaire and clinical interviews with the primary caregivers. All participants were given the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2), and caregivers were administered the Autism Diagnostic Interview-Revised (ADI-R). Cognitive and language skills were assessed using clinically appropriate measures depending on the child's age and function level. Scores on individual sub-tests are standardized against age-specific norms and then grouped to produce separate measures of verbal and non-verbal IQ, with the former encompassing those tests most related to verbal skills and the latter being more independent of verbal skills. Full Scale IQ (FSIQ) is the composite of these verbal and non-verbal skills. All individuals also completed the following measures.

Social Communication Questionnaire, Lifetime Version (SCQ)

The SCQ [1] is a 40-item questionnaire that measures the symptomatology associated with ASD (e.g., certain communitive behaviors, language uses, and stereotyped behaviors) focusing on the behaviors that are rare in non-affected individuals, based on an established diagnostic interview, the Autism Diagnostic Interview-Revised (ADI-R) [30]. The Lifetime SCQ version asks respondents to focus on characteristics of the individual at age 4 to 5 years for developmentally influenced behaviors, or at any point in their lifetime for behaviors that are atypical at any age (e.g., repetitive motor movements), while the Current SCQ version focuses on characteristics present within the previous 3 months [1]. Total scores range from 0 to 39, with higher scores reflecting the presence of more symptoms.

Behavioral Assessment System for Children, Parent Rating Scale (BASC-PRS)

The BASC-PRS [31] is a parent-report questionnaire using a multi-dimensional approach to evaluate behaviors and adaptive skills for children ages 2 years, 6 months to 21 years. This sample completed the BASC-PRS, Third Edition, which generates four composite scales (Externalizing Problems, Internalizing Problems, Behavioral Symptoms Index, and Adaptive Skills) and 14 Primary Scales (Hyperactivity, Aggression, Conduct Problems, Anxiety, Depression, Somatization, Atypicality, Withdrawal, Attention Problems, Adaptability, Social Skills, Leadership, Activities of Daily Living, and Functional Communication). For behavioral scales, higher T scores indicate greater difficulties, whereas for adaptive scales, lower T scores represent greater challenges [31].

Vineland Adaptive Behavior Scale (VABS)

The VABS [32, 33] is a standardized clinical assessment tool that utilizes a semi-structured interview to measure adaptive behaviors and skills for individuals with developmental challenges. For this sample, the third edition of the VABS, Comprehensive Interview Form, was used to assess participants’ adaptive skills. The VABS consists of three subscales, including Communication (receptive, expressive, written), Socialization (interpersonal relationships, play and leisure, coping skills), and Daily Living skills (person, domestic, community), yielding an overall composite score of adaptive skills (Adaptive Behavior Composite). The VABS also provides an indirect measure of gross and fine motor skills, yielding a Motor skills domain [32, 33].

Autism Diagnostic Observation Schedule, Second Edition (ADOS-2)

The ADOS-2 [34] is a standardized, semi-structured observational assessment used to assess language and communication, reciprocal social interaction, imagination/creativity, as well as stereotyped behaviors and restricted interests to inform diagnosis of ASD. The ADOS-2 is organized into five modules based on the individual’s expressive language level (and, in some cases, chronological age), ranging from preverbal to verbally fluent. The diagnostic algorithm provides separate total scores for the Social Affect (SA) and Restricted and Repetitive Behavior (RRB) domains, as well as a cut-off for the sum of the two domains to provide instrument classifications of autism, autism spectrum, or non-spectrum. The Calibrated Severity Score (CSS) is a standardized version of ADOS-2 raw total scores aimed to minimize the impact of factors such as age, language, and cognitive ability. The ADOS-2 CSS has been suggested as a measure of symptom severity independent of these developmental factors [35].

Analytic approach

Data collected through the comprehensive evaluations were entered, coded, and checked for errors and logic using a standardized procedure. Subsequently, the entered data were transferred to SAS 9.4 (SAS Institute Inc., Cary, NC, USA) to perform range checking and internal consistency checking. Based on the clinical diagnosis of ASD and suggested cut-off score of the SCQ (summed score of 15 or above) [1], individuals were separated into four groups to test our hypotheses: true positives (TP; have a clinical diagnosis of ASD and the SCQ score is above 15; N = 82), false negatives (FN; have a clinical diagnosis of ASD but the SCQ score is below 15; N = 51), false positives (FP; does not have a clinical diagnosis of ASD but the SCQ score is above 15; N = 22), and true negatives (TN; does not have a clinical diagnosis of ASD and the SCQ score is below 15; N = 32). Demographic homogeneity of the groups was assessed using chi-square tests and post-hoc analyses (Fisher’s exact test) for discrete variables. Continuous variables, such as sum or mean score of cognitive measures, symptomatologic items, as well as emotional, behavioral, and adaptive measures, were compared between groups by using independent t tests and post-hoc analyses (Bonferroni correction). To determine the performance of the SCQ as a screener for ASD in this clinical-referred sample, the sensitivity (i.e., number of true positive/[true positive + false negative]) and specificity (i.e., number of true negative/[true negative + false positive]) were calculated at multiple cut-off points. In addition, the area under the curve (AUC; the area under the receiver operating characteristic (ROC) curve) is considered to be a metric of fit between the true positive rate (sensitivity) and the false positive rate (1—specificity). AUC values can range from 0.5 to 1.0, with estimates closer to 1 indicating greater accuracy. Hanley and McNeil [36] suggested that a screener is a failure when an AUC is between 0.5 and 0.6, poor between 0.6 and 0.7, fair between 0.7 and 0.8, acceptable between 0.8 and 0.9, and perfect between 0.9 and 1.0.

Results

Discriminative validity

When applying the suggested cut-off score of ≥ 15 for a classification of ASD versus non-spectrum to all the samples, the AUC was 0.605, with a sensitivity of 0.677 and specificity of 0.593. When adjusting the cut-off score of the SCQ (e.g., cut-off ≥ 11 to ≥ 20), the AUC continued to reveal poor accuracy (range 0.581–0.634) with a better sensitivity when using a lower cut-off score (e.g., cut-off ≥ 11, sensitivity = 0.857, specificity = 0.315; Table 1) and a greater specificity when using a higher cut-off score (e.g., cut-off ≥ 20, sensitivity = 0.406, specificity = 0.759; Table 1).

Table 1 Prediction of ASD status based on Social Communication Questionnaire (SCQ)

When examining by sex, using the suggested cut-off score of ≥ 15, the SCQ achieved acceptable sensitivity among females with poor accuracy and specificity (N = 51; AUC = 0.686, sensitivity = 0.800, specificity = 0.571; Table 1), whereas AUC, sensitivity, and specificity remained poor in males (N = 136; AUC = 0.631, sensitivity = 0.675, specificity = 0.586). When adjusting the cut-off score of the SCQ (e.g., cut-off ≥ 11 to ≥ 20), the AUC among males (range 0.539–0.631) and females (AUC range 0.579–0.691) continued to reveal poor accuracy (Table 1).

To examine the discriminative validity of the SCQ with respect to age, the sample was separated into three age groups: age 4 to 7 years (N = 84), age 8 to 11 (N = 58), and age 12 and above (age range from 12.07 to 18.21; N = 45) based on previous literature (Corsello et al. 2007). In the youngest group, the AUC, sensitivity, and specificity were low when using the suggested cut-off score (cut-off ≥ 15; AUC = 0.587, sensitivity = 0.698, specificity = 0.476; Table 1), but the sensitivity improved when adjusting the cut-off to ≥ 11 (cut-off ≥ 11; AUC = 0.627, sensitivity = 0.873, specificity = 0.381; Table 1). For the older age groups, the 8 to 11 group had similar accuracy, sensitivity, and specificity to the whole group (cut-off ≥ 15; AUC = 0.638, sensitivity = 0.639, specificity = 0.636; Table 1), and age 12 and above yielded a fair AUC with better specificity (cut-off ≥ 15; AUC = 0.702, sensitivity = 0.676, specificity = 0.727; Table 1). All of the results were lower than reported in the original studies [1, 2].

Comparison between groups

False negatives (FN) vs. true negatives (TN)

Compared to the FN, TN had more individuals who received ADHD, anxiety disorders, and depressive disorders as their final diagnoses, and more individuals with two or more comorbid diagnoses (Table 2). Regarding autism symptomatology, FN exhibited more restricted and repetitive behaviors than TN based on clinician observation (ADOS RRB CSS; Table 3, Fig. 1A), but caregivers’ reports (BASC Withdrawal and Atypicality) did not reveal significant differences among the two groups (Table 3, Fig. 1B). The adaptive skills assessed by the caregiver-reported BASC and VABS revealed similar results among FN and TN, suggesting similar adaptive skills among those misclassified and non-spectrum individuals (Table 3, Fig. 1C and D). Moreover, FN reportedly demonstrated better adaptability (i.e., the ability to adapt readily to changes in the environment) than TN on the BASC (Table 3, Fig. 1C). No significant differences on cognitive functioning, internalizing problems, or externalizing issues between groups (Table 3).

Table 2 Demographics for accurately and inaccurately classified cases
Table 3 Comparisons for accurately and inaccurately classified cases
Fig. 1
figure 1

Group comparisons on autism symptoms severity, emotional and behavioral functioning, and adaptive skills

False negatives (FN) vs. true positives (TP)

FN demonstrated similar comorbidity, cognitive functioning, and autism symptomatology with TP based on direct neuropsychological measures and clinicians’ observation on the ADOS-2 (Tables 2 and 3, Fig. 1A). Caregivers also reported similar internalizing and externalizing problems on the BASC among FN and TP (Table 3). Compared to TP, FN displayed less atypical behaviors and social withdrawal based on caregivers’ observation on the BASC (Table 3, Fig. 1B). FN also exhibited better adaptive skills then TP on the BASC and VABS (Table 3, Fig. 1C, D).

False positives (FP) vs. true positives (TP)

Compared to TP, FP had more individuals who were diagnosed with behavioral disorders, but fewer individuals received diagnosis of intellectual disability (Table 2). Consistent with their final diagnoses, TP displayed higher ADOS CSS than FP on SA, RRB, and total comparison score of ADOS (Table 3, Fig. 1A). However, caregivers of FP reported greater externalizing and internalizing problems than TP (Table 3). Specifically, caregivers of FP reported clinically significant scores on domains of hyperactivity, aggression, and conduct problems, and their scores on aggression and conduct problems were significantly higher than TP (Table 3, Fig. 1B). Caregivers of FP also reported greater internalizing problems, especially anxiety, which were significantly higher than TP group (Table 3, Fig. 1B). In general, caregivers of FP reported similar adaptive skills with TP across domains on the BASC and the VABS (Table 3, Fig. 1C, D), though FP reportedly displayed better daily living skills than TP on the VABS, resulted in greater overall Adaptive Behavioral Composite than TP (Table 3, Fig. 1D).

False positives (FP) vs. false negatives (FN)

Compared to FN, FP had more individuals who received ADHD and behavioral disorders as their final diagnoses among this sample, and more individuals with two or more diagnoses (Table 2). Consistent with their final diagnoses by the licensed psychologists, FN displayed higher ADOS CSS than FP on SA, RRB, and total comparison score of ADOS (Table 3, Fig. 1A). When examining internalizing and externalizing symptoms on the BASC, caregivers of FP reported greater behavioral issues than FN, including aggression and conduct problems (Table 3, Fig. 1B). The FP group also exhibited more atypical behaviors than FN (Table 3, Fig. 1B). In terms of adaptive skills, caregivers of FN and FP generally reported similar adaptive skills on the BASC and VABS (Table 3, Fig. 1C, D), with an exception that FN reportedly demonstrated better adaptability than FP based on the BASC (Table 3, Fig. 1C).

Discussion

The present study examined the behavioral characteristics of children who were misclassified by the SCQ in a clinic-referred sample. Among the 187 participants in the present study, only 65% of individuals were accurately classified, revealing low sensitivity (0.68) and specificity (0.59). Individuals with ASD who were accurately classified on the SCQ (true positives) had lower adaptive skills and higher ADOS-2 scores than other groups, including false negatives. True positives did not differ from false negatives (or other groups) in terms of internalizing or externalizing behaviors. Individuals who were falsely classified as ASD (false positives) displayed more externalizing (e.g., aggression and conduct problems) and internalizing psychopathology (e.g., anxiety) and were more likely to be diagnosed with behavioral disorders (e.g., oppositional defiant disorder, conduct disorder) compared to the other groups. Regarding adaptive skills, while the non-spectrum group (true negatives and false positives) in general demonstrated higher adaptive skills than the ASD group (true positives and false negatives), the false negative group had a similar level of adaptive skills (e.g., communication, daily living skills, social skills, leadership, and functional communication) to the non-spectrum group.

Our results make an important contribution to knowledge about the limitations of classification properties of the SCQ in discriminating autism from other conditions in a referral population who seek their first diagnosis. Individuals who were falsely classified as ASD exhibited more externalizing (e.g., aggression and conduct problems) and internalizing issues (e.g., anxiety) compared to other groups. Moreover, they had comparable social impairments and atypical behaviors compared to children in the true positive group. It is likely that there may be a subset of children with more severe behavioral problems who also show early and continued social impairment, secondary to difficulty controlling impulses and regulating emotions. As a result, the caregivers may have endorsed numerous items on the SCQ to reflect the social challenges and general difficulties their children faced. Alternatively, another possibility is that parents are identifying atypical presentations in early development but are unable to clearly differentiate the types of difficulties they are observing on a yes/no questionnaire, thus reporting several social impairments that may better reflect current behavioral challenges. Since these behavioral challenges are not specifically differentiated and measured on the SCQ, the reporters may not have adequate places to voice their concerns, resulting in elevated scores. It is important to note that anxiety and depression in young children can be difficult for parents to differentiate from autistic behaviors, as internalizing symptoms may present as social withdrawal, irritability, rigidity, preference for routine and order, and repetitive behaviors, which are also symptoms of ASD [37, 38]. In fact, there is a known challenge of differentiating autistic symptomology from developmental and behavioral presentations, even when utilizing a standardized diagnostic measure such as the ADOS by trained clinicians [39]. Using the SCQ in combination with other measures for internalizing and externalizing psychopathology is recommended [40,41,42,43].

Another notable finding was that the false negative group appeared to display fewer developmental and adaptive impairments compared to the true positive group, suggesting that the accuracy of the SCQ may be highly influenced by developmental factors. Children in the false negative group clearly exhibited ASD symptoms, and their ADOS-2 scores did not differ from those of the true positive group, but their presentations may have been subtler than those accurately captured with ASD by the SCQ or may be masked by their better-developed adaptive skills. Therefore, their caregivers were not able to reflect their observations on the SCQ. Our results corroborate previous findings that the SCQ may miss individuals with less overall impairment [5]. Moreover, as a parent-report questionnaire, the SCQ highly relies on the reporter’s knowledge, experiences, observation, and understanding of autistic characteristics. Previous studies [44, 45] suggested that if a parent already has a prior understanding of ASD or knows their child has ASD, they are more likely to differentiate core autism symptoms from other challenging behaviors on questionnaires or endorse ASD symptoms to a greater degree, revealing the reporters’ knowledge influences how they respond to the screening questions. Parental concerns about their children’s development have also been suggested to influence parent-report instrument performance [46]. It is possible that the SCQ could vary according to parental understanding of ASD and level of concern. Future research is needed to determine the optimal ways to screen for ASD in children where parents have minimal concerns about their children’s development.

Previous studies utilizing clinic-referred samples have suggested different cut-off scores according to age and purpose to increase accuracy [5, 10, 14]. Consistent with the literature, the current findings revealed suboptimal sensitivity and specificity of the SCQ, lower than those reported in the original study [2] as well as subsequent studies [5, 14]. However, adjusting cut-off scores based on age and sex did not improve the accuracy of the SCQ to a satisfactory level or impact the specificity of the measure on this sample. Our findings suggest that age and sex may not be the critical factors that impact the accuracy and usefulness of the SCQ among clinic-referred samples. Given our results suggesting suboptimal sensitivity and specificity of the SCQ, it is important to highlight for clinicians and researchers that the SCQ has limitations in accurately identifying children who will go on to receive an ASD diagnosis, particularly for children with complex presentations and other co-occurring mental health and behavioral diagnoses. Therefore, the SCQ used in isolation seems to have limited utility in assisting with the clinical triaging process or serving as a screening tool for inclusion or exclusion criteria in the research setting. This is problematic, given that there is a severe shortage of professionals who are trained to provide specialized evaluations for ASD, leaving specialty ASD clinics with long waiting lists, barriers to intervention access, and increased parental stress [47,48,49]. Tools that are validated in accurately classifying young children are critically necessary [50,51,52].

Several limitations of the present study should be addressed in future research. Although we had a relatively large sample, especially for a clinic-referred population that underwent gold-standard measures for ASD diagnosis, our sample sizes were made smaller when stratifying by age, sex, and SCQ status. In addition, selection bias could potentially limit the generalizability of our results. We were unable to explore additional family factors that may contribute to the sensitivity and specificity of the SCQ (e.g., primary language, race/ethnicity, and educational level) due to the limited inclusion of diverse families within our sample and inconsistent demographic information provided by families during the clinical intake process. It is also important to note that the SCQ score was available for the clinicians at the time of making a clinical diagnosis, and the clinical impression and diagnostic classification may not be independent. Further, the comorbidity of this clinic-referred sample was relatively high across the groups. It suggested that this is a complex sample, which might be contributing to the modest classification accuracy of the SCQ. The results may not generalize to other clinically referred samples with different clinical features. Future work should replicate our results in larger samples and explore additional factors that may contribute to individual differences related to the sensitivity and specificity of the SCQ.

In summary, the present study examined the sensitivity and specificity of the SCQ by assessing a clinical sample of children and adolescents with and without ASD. This study further examined the individual differences in sensitivity and specificity by describing the characteristics of children misclassified using the SCQ. These findings suggested that the SCQ may be less useful for young children with internalizing and externalizing symptoms and other more complex clinical presentations, highlighting the importance of taking an individual difference and multi-informant approach for ASD assessment. Clinically, these results underscore the necessity of comprehensive evaluations for children, particularly for internalizing and externalizing psychopathology in addition to autistic symptoms, to promote optimal diagnostic and intervention outcomes. In research settings, our results suggest that solely relying on the SCQ for inclusion criteria may be risky, especially for families who are less familiar with ASD. Future studies will be needed to determine if SCQ performs best in population-based screening (e.g., school-based screening) or clinically referred samples, as well as whether parental awareness of ASD impacts the reporting pattern on the SCQ, further influencing the sensitivity and specificity of the SCQ within certain populations.

Conclusions

Our findings make an important contribution to the literature on ASD measurement by specifying the behavioral characteristics of children misclassified by the SCQ. Specifically, individuals who were falsely classified as ASD by the SCQ exhibited more externalizing (e.g., aggression and conduct problems) and internalizing issues (e.g., anxiety), and they were more likely to be diagnosed with behavioral disorders (e.g., oppositional defiant disorder and conduct disorder) compared to other groups. The false negative group appeared to display fewer developmental and adaptive impairments compared to the true positive group. These findings suggest that the accuracy of the SCQ is highly influenced by developmental and behavioral factors, and the SCQ may be less useful for young children with internalizing and externalizing symptoms and other more complex clinical presentations. Clinicians and researchers could consider using the SCQ in combination with other assessment tools for internalizing and externalizing psychopathology.

Availability of data and materials

The datasets and unpublished materials used and/or analyzed in this study are available from the corresponding author on reasonable request.

Abbreviations

ASD:

Autism spectrum disorder

IQ:

Intelligence quotient

ADHD:

Attention-deficit hyperactivity disorder

TP:

True positives

FN:

False negatives

FP:

False positives

TN:

True negatives

ADOS-2:

Autism Diagnostic Observation Schedule, Second Edition

ADI-R:

Autism Diagnostic Interview, Revised

FSIQ:

Full Scale Intelligence Quotient

BASC-PRS:

Behavioral Assessment System for Children, Parent Rating Scale

VABS:

Vineland Adaptive Behavior Scale

AUC:

Area under the receiver operating characteristic curve

ROC:

Receiver operating characteristic

RRB:

Restricted and Repetitive Behavior

CSS:

Calibrated Severity Scores

References

  1. Rutter M, Bailey A, Lord C. The social communication questionnaire. Los Angeles: Western Psychological Services; 2003.

    Google Scholar 

  2. Berument SK, Rutter M, Lord C, Pickles A, Bailey A. Autism screening questionnaire: diagnostic validity. Br J Psychiatry. 1999;175:444–51. https://doi.org/10.1192/bjp.175.5.444.

    Article  CAS  PubMed  Google Scholar 

  3. Hollocks MJ, Casson R, White C, Dobson J, Beazley P, Humphrey A. Brief report: An evaluation of the social communication questionnaire as a screening tool for autism spectrum disorder in young people referred to child & adolescent mental health services. J Autism Dev Disord. 2019;49(6):2618–23. https://doi.org/10.1007/s10803-019-03982-6.

    Article  PubMed  Google Scholar 

  4. Allen CW, Silove N, Williams K, Hutchins P. Validity of the social communication questionnaire in assessing risk of autism in preschool children with developmental problems. J Autism Dev Disord. 2007;37(7):1272–8. https://doi.org/10.1007/s10803-006-0279-7.

    Article  CAS  PubMed  Google Scholar 

  5. Corsello C, Hus V, Pickles A, Risi S, Cook EH Jr, Leventhal BL, et al. Between a ROC and a hard place: decision making and making decisions about using the SCQ. J Child Psychol Psychiatry. 2007;48(9):932–40. https://doi.org/10.1111/j.1469-7610.2007.01762.x.

    Article  PubMed  Google Scholar 

  6. Eaves LC, Wingert H, Ho HH. Screening for autism: agreement with diagnosis. Autism. 2006;10(3):229–42. https://doi.org/10.1177/1362361306063288.

    Article  PubMed  Google Scholar 

  7. Eaves LC, Wingert HD, Ho HH, Mickelson EC. Screening for autism spectrum disorders with the social communication questionnaire. J Dev Behav Pediatr. 2006;27(2 Suppl):S95–103. https://doi.org/10.1097/00004703-200604002-00007.

    Article  PubMed  Google Scholar 

  8. Norris M, Lecavalier L. Screening accuracy of Level 2 autism spectrum disorder rating scales. A review of selected instruments. Autism. 2010;14(4):263–84. https://doi.org/10.1177/1362361309348071.

    Article  PubMed  Google Scholar 

  9. Snow AV, Lecavalier L. Sensitivity and specificity of the Modified Checklist for Autism in Toddlers and the Social Communication Questionnaire in preschoolers suspected of having pervasive developmental disorders. Autism. 2008;12(6):627–44. https://doi.org/10.1177/1362361308097116.

    Article  PubMed  Google Scholar 

  10. Chesnut SR, Wei T, Barnard-Brak L, Richman DM. A meta-analysis of the social communication questionnaire: screening for autism spectrum disorder. Autism. 2017;21(8):920–8. https://doi.org/10.1177/1362361316660065.

    Article  PubMed  Google Scholar 

  11. Barnard-Brak L, Brewer A, Chesnut S, Richman D, Schaeffer AM. The sensitivity and specificity of the social communication questionnaire for autism spectrum with respect to age. Autism Res. 2016;9(8):838–45. https://doi.org/10.1002/aur.1584.

    Article  PubMed  Google Scholar 

  12. Miranda A, Berenguer C, Rosello B, Baixauli I. Relationships between the social communication questionnaire and pragmatic language, socialization skills, and behavioral problems in children with autism spectrum disorders. Appl Neuropsychol Child. 2020;9(2):141–52. https://doi.org/10.1080/21622965.2018.1550403.

    Article  PubMed  Google Scholar 

  13. Moody EJ, Reyes N, Ledbetter C, Wiggins L, DiGuiseppi C, Alexander A, et al. Screening for autism with the SRS and SCQ: variations across demographic, developmental and behavioral factors in preschool children. J Autism Dev Disord. 2017;47(11):3550–61. https://doi.org/10.1007/s10803-017-3255-5.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Chandler S, Charman T, Baird G, Simonoff E, Loucas T, Meldrum D, et al. Validation of the social communication questionnaire in a population cohort of children with autism spectrum disorders. J Am Acad Child Adolesc Psychiatry. 2007;46(10):1324–32. https://doi.org/10.1097/chi.0b013e31812f7d8d.

    Article  PubMed  Google Scholar 

  15. Rosenberg SA, Moody EJ, Lee LC, DiGuiseppi C, Windham GC, Wiggins LD, et al. Influence of family demographic factors on social communication questionnaire scores. Autism Res. 2018;11(5):695–706. https://doi.org/10.1002/aur.1935.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Wiggins LD, Bakeman R, Adamson L, Robins DL. The Utility of the Social Communication Questionnaire in screening for autism in children referred for early intervention. Focus Autism Other Dev Disabil. 2007;22(1):33–8.

    Article  Google Scholar 

  17. Wiggins LD, Reynolds A, Rice CE, Moody EJ, Bernal P, Blaskey L, et al. Using standardized diagnostic instruments to classify children with autism in the study to explore early development. J Autism Dev Disord. 2015;45(5):1271–80. https://doi.org/10.1007/s10803-014-2287-3.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Frazier TW, Georgiades S, Bishop SL, Hardan AY. Behavioral and cognitive characteristics of females and males with autism in the Simons Simplex Collection. J Am Acad Child Adolesc Psychiatry. 2014;53(3):329-40.e1-3. https://doi.org/10.1016/j.jaac.2013.12.004.

    Article  PubMed  Google Scholar 

  19. Kaat AJ, Shui AM, Ghods SS, Farmer CA, Esler AN, Thurm A, et al. Sex differences in scores on standardized measures of autism symptoms: a multisite integrative data analysis. J Child Psychol Psychiatry. 2020. https://doi.org/10.1111/jcpp.13242.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Evans SC, Boan AD, Bradley C, Carpenter LA. Sex/gender differences in screening for autism spectrum disorder: implications for evidence-based assessment. J Clin Child Adolesc Psychol. 2019;48(6):840–54. https://doi.org/10.1080/15374416.2018.1437734.

    Article  PubMed  Google Scholar 

  21. Ghaziuddin M, Welch K, Mohiuddin S, Lagrou R, Ghaziuddin N. Utility of the Social and Communication Questionnaire in the differentiation of autism from ADHD. J Dev Phys Disabil. 2010;22(4):359–66. https://doi.org/10.1007/s10882-010-9199-8.

    Article  Google Scholar 

  22. Marvin AR, Marvin DJ, Lipkin PH, Law JK. Analysis of Social Communication Questionnaire (SCQ) screening for children less than age 4. Curr Dev Disord Rep. 2017;4(4):137–44. https://doi.org/10.1007/s40474-017-0122-1.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Suren P, Saasen-Havdahl A, Bresnahan M, Hirtz D, Hornig M, Lord C, et al. Sensitivity and specificity of early screening for autism. BJPsych Open. 2019;5(3):e41. https://doi.org/10.1192/bjo.2019.34.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Brooks WT, Benson BA. The validity of the social communication questionnaire in adults with intellectual disability. Res Autism Spectr Disord. 2013;7(2):247–55.

    Article  Google Scholar 

  25. Sappok T, Diefenbacher A, Gaul I, Bolte S. Validity of the social communication questionnaire in adults with intellectual disabilities and suspected autism spectrum disorder. Am J Intellect Dev Disabil. 2015;120(3):203–14. https://doi.org/10.1352/1944-7558-120.3.203.

    Article  PubMed  Google Scholar 

  26. Ung D, Johnco C, McBride NM, Howie F, Scalli L, Storch EA. Optimizing the screening of autism spectrum disorders in outpatient clinics: an examination of the Social Communication Questionnaire-Lifetime. Res Autism Spectr Disord. 2016;27:21–8.

    Article  Google Scholar 

  27. Witwer AN, Lecavalier L. Autism screening tools: an evaluation of the social communication questionnaire and the developmental behaviour checklist-autism screening algorithm. J Intellect Dev Disabil. 2007;32(3):179–87. https://doi.org/10.1080/13668250701604776.

    Article  PubMed  Google Scholar 

  28. Mouti A, Dryer R, Kohn M. Differentiating autism spectrum disorder from ADHD using the social communication questionnaire. J Atten Disord. 2019;23(8):828–37. https://doi.org/10.1177/1087054718781945.

    Article  PubMed  Google Scholar 

  29. Schwenck C, Freitag CM. Differentiation between attention-deficit/hyperactivity disorder and autism spectrum disorder by the social communication questionnaire. Atten Defic Hyperact Disord. 2014;6(3):221–9. https://doi.org/10.1007/s12402-014-0147-9.

    Article  PubMed  Google Scholar 

  30. Lord C, Rutter M, Le Couteur A. Autism Diagnostic Interview-Revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord. 1994;24(5):659–85. https://doi.org/10.1007/BF02172145.

    Article  CAS  PubMed  Google Scholar 

  31. Reynolds CR, Kamphaus RW. Behavior assessment system for children (3rd ed.). Bloomington: Pearson; 2015.

    Google Scholar 

  32. Sparrow SS, Cicchetti DV, Balla AD. Vineland adaptive behavior scales, second edition (Vineland-II). Circle Pines: American Guidance Service; 2005.

    Google Scholar 

  33. Sparrow SS, Cicchetti DV, Saulnier CA. Vineland adaptive behavior scales, third edition (Vineland-3). San Antonio: Pearson; 2016.

    Google Scholar 

  34. Lord C, Rutter M, DiLavore PC, Risi S, Gotham K, Bishop S. Autism diagnostic observation schedule, second edition (ADOS-2). Torrance: Western Psychological Services; 2012.

    Google Scholar 

  35. Gotham K, Pickles A, Lord C. Standardizing ADOS scores for a measure of severity in autism spectrum disorders. J Autism Dev Disord. 2009;39(5):693–705. https://doi.org/10.1007/s10803-008-0674-3.

    Article  PubMed  Google Scholar 

  36. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36.

    Article  CAS  PubMed  Google Scholar 

  37. Ben-Sasson A, Cermak SA, Orsmond GI, Tager-Flusberg H, Kadlec MB, Carter AS. Sensory clusters of toddlers with autism spectrum disorders: differences in affective symptoms. J Child Psychol Psychiatry. 2008;49(8):817–25. https://doi.org/10.1111/j.1469-7610.2008.01899.x.

    Article  CAS  PubMed  Google Scholar 

  38. Adams D, Young K, Simpson K, Keen D. Parent descriptions of the presentation and management of anxiousness in children on the autism spectrum. Autism. 2019;23(4):980–92. https://doi.org/10.1177/1362361318794031.

    Article  PubMed  Google Scholar 

  39. Molloy CA, Murray DS, Akers R, Mitchell T, Manning-Courtney P. Use of the Autism Diagnostic Observation Schedule (ADOS) in a clinical setting. Autism. 2011;15(2):143–62. https://doi.org/10.1177/1362361310379241.

    Article  PubMed  Google Scholar 

  40. Heavner K, Newschaffer C, Hertz-Picciotto I, Bennett D, Burstyn I. Quantifying the potential impact of measurement error in an investigation of autism spectrum disorder (ASD). J Epidemiol Community Health. 2014;68(5):438–45. https://doi.org/10.1136/jech-2013-202982.

    Article  PubMed  Google Scholar 

  41. Hossain MM, Khan N, Sultana A, Ma P, McKyer ELJ, Ahmed HU, et al. Prevalence of comorbid psychiatric disorders among people with autism spectrum disorder: an umbrella review of systematic reviews and meta-analyses. Psychiatry Res. 2020;287:112922. https://doi.org/10.1016/j.psychres.2020.112922.

    Article  PubMed  Google Scholar 

  42. Johnson S, Hollis C, Hennessy E, Kochhar P, Wolke D, Marlow N. Screening for autism in preterm children: diagnostic utility of the Social Communication Questionnaire. Arch Dis Child. 2011;96(1):73–7. https://doi.org/10.1136/adc.2010.194795.

    Article  PubMed  Google Scholar 

  43. Lord C, Charman T, Havdahl A, Carbone P, Anagnostou E, Boyd B, et al. The Lancet Commission on the future of care and clinical research in autism. Lancet. 2022;399(10321):271–334.

    Article  PubMed  Google Scholar 

  44. Miller L, Burke J, Robins D, Fein D. Diagnosing autism spectrum disorders in children with low mental age. J Autism Dev Disord. 2019;49(3):3550–61.

    Article  Google Scholar 

  45. Christopher K, Bishop S, Carpenter LA, Warren Z, Kanne S. The implications of parent-reported emotional and behavioral problems on the modified checklist for autism in toddlers. J Autism Dev Disord. 2021;51:884–91.

    Article  PubMed  Google Scholar 

  46. Havdahl KA, Bishop SL, Suren P, Oyen AS, Lord C, Pickles A, et al. The influence of parental concern on the utility of autism diagnostic instruments. Autism Res. 2017;10(10):1672–86. https://doi.org/10.1002/aur.1817.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Martinez M, Thomas KC, Williams CS, Christian R, Crais E, Pretzel R, et al. Family experiences with the diagnosis of autism spectrum disorder: system barriers and facilitators of efficient diagnosis. J Autism Dev Disord. 2018;48(7):2368–78. https://doi.org/10.1007/s10803-018-3493-1.

    Article  CAS  PubMed  Google Scholar 

  48. Mazurek MO, Curran A, Burnette C, Sohl K. ECHO autism STAT: accelerating early access to autism diagnosis. J Autism Dev Disord. 2019;49(1):127–37. https://doi.org/10.1007/s10803-018-3696-5.

    Article  PubMed  Google Scholar 

  49. Osborne LA, Reed P. Parents’ perceptions of communication with professionals during the diagnosis of autism. Autism. 2008;12(3):309–24. https://doi.org/10.1177/1362361307089517.

    Article  PubMed  Google Scholar 

  50. Robins DL, Adamson LB, Barton M, Connell JE Jr, Dumont-Mathieu T, Dworkin PH, et al. Universal autism screening for toddlers: recommendations at odds. J Autism Dev Disord. 2016;46(5):1880–2. https://doi.org/10.1007/s10803-016-2697-5.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Siu AL, Force USPST, Bibbins-Domingo K, Grossman DC, Baumann LC, Davidson KW, et al. Screening for autism spectrum disorder in young children: US preventive services task force recommendation statement. JAMA. 2016;315(7):691–6. https://doi.org/10.1001/jama.2016.0018.

    Article  CAS  PubMed  Google Scholar 

  52. Towle PO, Patrick PA. Autism spectrum disorder screening instruments for very young children: a systematic review. Autism Res Treat. 2016;2016:4624829. https://doi.org/10.1155/2016/4624829.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We extend our heartfelt appreciation to the participating families whose time and effort made this study possible. Their contribution has been invaluable, and their willingness to participate is essential to advancing research in this field.

Funding

This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program (NSF-GRF 00074041) to M. Altschuler. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

Authors

Contributions

C.M.L and M.R.A. completed the first draft of the manuscript. C.M.L performed data analyses. All authors contributed to conceptualization and revisions of the manuscript.

Corresponding author

Correspondence to Chimei M. Lee.

Ethics declarations

Ethics approval and consent to participate

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from parents of all individual participants included in the study, and all children provided their assent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, C.M., Altschuler, M.R., Esler, A.N. et al. Why are only some children with autism spectrum disorder misclassified by the social communication questionnaire? An empirical investigation of individual differences in sensitivity and specificity in a clinic-referred sample. J Neurodevelop Disord 15, 28 (2023). https://doi.org/10.1186/s11689-023-09497-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s11689-023-09497-7

Keywords