Endophenotype trait domains for advancing gene discovery in autism spectrum disorder

Abstract

Autism spectrum disorder (ASD) is associated with a diverse range of etiological processes, including both genetic and non-genetic causes. For a plurality of individuals with ASD, it is likely that the primary causes involve multiple common inherited variants that individually account for only small levels of variation in phenotypic outcomes. This genetic landscape creates a major challenge for detecting small but important pathogenic effects associated with ASD. To address similar challenges, separate fields of medicine have identified endophenotypes, or discrete, quantitative traits that reflect genetic likelihood for a particular clinical condition and leveraged the study of these traits to map polygenic mechanisms and advance more personalized therapeutic strategies for complex diseases. Endophenotypes represent a distinct class of biomarkers useful for understanding genetic contributions to psychiatric and developmental disorders because they are embedded within the causal chain between genotype and clinical phenotype, and they are more proximal to the action of the gene(s) than behavioral traits. Despite their demonstrated power for guiding new understanding of complex genetic structures of clinical conditions, few endophenotypes associated with ASD have been identified and integrated into family genetic studies. In this review, we argue that advancing knowledge of the complex pathogenic processes that contribute to ASD can be accelerated by refocusing attention toward identifying endophenotypic traits reflective of inherited mechanisms. This pivot requires renewed emphasis on study designs with measurement of familial co-variation including infant sibling studies, family trio and quad designs, and analysis of monozygotic and dizygotic twin concordance for select trait dimensions. We also emphasize that clarification of endophenotypic traits necessarily will involve integration of transdiagnostic approaches as candidate traits likely reflect liability for multiple clinical conditions and often are agnostic to diagnostic boundaries. Multiple candidate endophenotypes associated with ASD likelihood are described, and we propose a new focus on the analysis of “endophenotype trait domains” (ETDs), or traits measured across multiple levels (e.g., molecular, cellular, neural system, neuropsychological) along the causal pathway from genes to behavior. To inform our central argument for research efforts toward ETD discovery, we first provide a brief review of the concept of endophenotypes and their application to psychiatry. Next, we highlight key criteria for determining the value of candidate endophenotypes, including unique considerations for the study of ASD. Descriptions of different study designs for assessing endophenotypes in ASD research then are offered, including analysis of how select patterns of results may help prioritize candidate traits in future research. We also present multiple candidate ETDs that collectively cover a breadth of clinical phenomena associated with ASD, including social, language/communication, cognitive control, and sensorimotor processes. These ETDs are described because they represent promising targets for gene discovery related to clinical autistic traits, and they serve as models for analysis of separate candidate domains that may inform understanding of inherited etiological processes associated with ASD as well as overlapping neurodevelopmental disorders.

Background

Autism spectrum disorder (ASD) is both clinically and etiologically diverse. Rare inherited and de novo pathogenic variants each have been repeatedly implicated and account for up to 20% of cases [1, 2]. For a plurality of individuals with ASD, however, it is believed that the primary causes include gene-gene and gene-environment interactions that involve multiple common inherited variants and non-linear complex genetics [3, 4]. Consistent with this hypothesis, > 1000 different genes show associations with ASD, and the majority of variants implicated each confer only small effects [5]. These findings suggest that many different pathogenic processes contribute to ASD, and that additive or multiplicative genetic effects play prominent roles in the development of autism. This polygenic landscape also suggests significant etiological heterogeneity among autistic individuals, indicating studies of large numbers of individuals will be necessary to help parse the many distinct causal pathways involved. Large, multi-site research networks and data sharing consortia have been leveraged to establish a growing number of candidate genes and greater understanding of their downstream molecular consequences in relation to ASD [6,7,8,9]. Despite these efforts, diagnostic yield from genetic testing in ASD remains low and etiological processes for most autistic individuals remain unexplained [10, 11]. Additional approaches are needed to increase resolution for detecting small but significant genetic effects and advance more individualized identification and therapeutic strategies.

One approach to gene discovery that has proven valuable in separate fields of medicine is the identification of condition-related biological traits that can be used to decompose complex clinical phenotypes into less genetically complex trait structures. Endophenotypes, or discrete, quantitative traits that reflect genetic likelihood for a particular clinical condition, have been used to identify polygenic mechanisms of complex conditions and advance more personalized therapeutic strategies for heart disease [12, 13], obesity [14, 15], diabetes [16], and osteoporosis [17, 18]. Endophenotypes represent a distinct class of biomarkers useful for understanding genetic contributions to clinical entities because they are (A) embedded within the causal chain between genotype and clinical phenotype, (B) closer to the action of the gene(s) than the constellation of clinical phenotypes that define a diagnosis, and (C) quantitative and therefore capable of showing greater sensitivity to additive causal processes than categorical outcomes, such as affectation status. Despite their demonstrated power for guiding new understanding of complex genetic structures of clinical conditions, few endophenotypes associated with ASD have been identified and integrated into family genetic studies.

In this paper, we argue that understanding the complex pathogenic processes that contribute to ASD can be accelerated by refocusing attention toward identifying endophenotypic traits reflective of inherited mechanisms. This pivot requires renewed emphasis on study designs integrating measurement of familial co-variation including infant sibling studies, family trio and quad designs, and analysis of monozygotic and dizygotic twin concordance for select trait dimensions. Given that most, if not all behavioral traits associated with ASD also are implicated in separate behaviorally defined disorders (e.g., repetitive sensorimotor mannerisms are common in intellectual and developmental disability; difficulties with modulating eye contact during interaction also have been demonstrated in multiple anxiety disorders) and show wide variation in neurotypical development, it is likely that endophenotypic traits associated with ASD will cut across diagnostic boundaries, and transdiagnostic designs will be critical.

Multiple candidate endophenotypes associated with ASD likelihood are described herein, though the intent is not to provide a comprehensive review (for more systematic reviews, see [19, 20]), but instead to focus on promising targets for accelerating progress in understanding etiological processes associated with traits involved in ASD. Toward this goal, we propose a new focus on the analysis of “endophenotype trait domains” (ETDs), or traits measured across multiple levels along the causal pathway from genes to behavior. To elucidate ETDs, “dense-phenotyping” approaches will be integral to establishing within-individual associations between traits across molecular, cellular, circuit, system, and behavioral levels, as has been done in separate fields of medicine and areas of psychiatry (see Fig. 1). For example, the Bipolar and Schizophrenia Network for Intermediate Phenotypes (BSNIP) is an on-going, multi-site, transdiagnostic study focused on identifying quantitative traits, measured across multiple levels of analysis, that are associated with psychosis and co-segregate in patients and their first-degree relatives. Studying > 1000 patients affected by psychiatric disorders associated with psychosis (e.g., schizophrenia, bipolar disorder), this network has leveraged broad phenotyping, including measurement of sensory, motor, behavioral, and psychiatric traits, and dense phenotyping strategies, utilizing multiple measures of target domains including genetic, immunological, electrophysiological, oculomotor, functional and structural MR imaging, cognitive, and clinical assays, to derive data-driven “biotypes”, or biologically distinct subgroups of patients [21]. Initial principal components analysis and k-means clustering were used to first derive latent factors across behavioral, cognitive, and brain levels, and then to identify clusters of more biologically homogeneous subtypes. Subsequent work has shown that these biotypes differ on separate external validation characteristics, including brain morphometry, and that affected and unaffected family members co-segregate into similar biotypes implicating high levels of familiality. Similar approaches have been advanced in separate fields of medicine, but no known efforts such as these have been conducted to understand biotypes based on ETDs for neurodevelopmental conditions, including autism.

Fig. 1

Levels of analysis for mapping etiological pathways associated with behavioral and clinical traits. This schematic shows different layers or functional units of analysis that can be evaluated to clarify linkages between genotype and clinical phenotype. Endophenotypic traits closer to the level of genotype are expected to be more closely associated with inherited variation given their relatively simpler genetic architecture compared to behavioral traits, as evidenced above by the reduced complexity (i.e., number of genes or letters) at the higher levels (e.g., cells, circuits/networks). Multiple levels of analysis are depicted, though separate intermediate levels are not included for ease of presentation (e.g., proteomic). Based on this model, analysis of traits closer to genotypes will provide greater sensitivity to inherited variations than assessments of behavior or complex clusters of clinical symptoms. Analysis of traits across multiple levels, or establishment of endophenotypic trait domains (ETDs), also is proposed to offer unique opportunities for understanding etiological pathways contributing to discrete traits associated with ASD. Important environmental and developmental factors also are proposed to modify trait associations across levels and over time

The concept of ETDs is drawn from the NIMH Research Domain Criteria (RDoC) that focuses on the analysis of domains of behavior from molecular to self-report levels [22], but extends this idea to focus on traits that represent key intermediary processes between genetic causes and select behavioral or clinical dimensions. Here, we emphasize the importance of quantitative trait outcomes rather than categorical diagnoses. This emphasis is critical in the context of considerable evidence that ASD, similar to other psychiatric disorders defined behaviorally by DSM, represent heterogeneous clusters of individuals showing both shared and distinct quantitative deviations from typical or normative profiles of functioning rather than any discovered entity grounded in nature or biology [23]. By mapping endophenotypes across multiple levels and across the full range of neurodiversity, ETDs may provide important insights into pathways of inherited traits that contribute to clinical vulnerabilities and maladaptive developmental functioning. Illustrative examples of how this concept can be applied to understand mechanistic pathways associated with ASD are provided.

To inform our central argument for research efforts toward ETD discovery, we first provide a brief review of the concept of endophenotypes and their application to psychiatry. Next, we highlight key criteria for determining the value of candidate endophenotypes, including unique considerations for the study of ASD and neurodevelopmental disorders more broadly. Descriptions of different study designs for assessing endophenotypes in ASD research then are offered, including analysis of how select patterns of results may help prioritize candidate traits in future research. We then present multiple candidate ETDs that collectively cover a breadth of clinical phenomena associated with ASD, including social, language/communication, cognitive control, and sensorimotor processes. These ETDs are described because they represent promising targets for gene discovery related to clinical traits associated with ASD, and they serve as models for analysis of separate candidate domains that may inform understanding of inherited etiological processes contributing to ASD or associated neurodevelopmental traits.

Endophenotypes in psychiatry and their application to ASD

The endophenotype concept was first introduced in insect biology to describe “microscopic and internal traits” that contrast “exophenotypes”, or behaviors that are directly observable [24]. Gottesman and Shields [25] initially brought the concept to psychiatry asserting that endophenotypes could provide greater sensitivity to inherited risk factors for psychopathology because they are “a measurable trait that is not observable by the unaided eye… and that lies more proximal to the underlying genetics of a disorder than the clinical phenotype” [26]. Endophenotypes thus represent a unique class of biomarkers that is necessarily influenced by the genetic factors that confer susceptibility to a particular clinical condition [27, 28]. Consistent with this broader definition, Gould and Gottesman [28] proposed key criteria for evaluating the viability of biomarkers as endophenotypic traits. Briefly, the authors indicated that a candidate endophenotype should be (1) associated with condition-specific traits as demonstrated by its presence in affected individuals and covariation with primary trait dimensions in the population, (2) associated with clinical severity within affected individuals, (3) familial, meaning it shows decreasing levels of severity from affected individuals to unaffected family members to unaffected population controls, (4) heritable, and (5) reliably measured and reproducible [26, 29]. Original definitions of endophenotypes also indicated that they should be “state-independent and detectable regardless of whether an individual is acutely ill or in remission”, though recent amendments to these criteria have been proposed to include traits that are observed “prior to the manifestation of features that define a given condition/disorder” that “may or may not persist through development” [30]. Evidence suggests that the familial traits most predictive of autistic outcomes may in fact be deviations previously considered associated features (e.g., motor control features, attention impairments), rather than (or in addition to) core social-communication or repetitive behavior features that appear to show less predictive value early in ontogeny [31]. Moreover, Mous and colleagues [32] found that attention-deficit/hyperactivity disorder (ADHD) and motor coordination traits in both siblings with family history of autism with ASD (FH⁺) and without ASD (FH⁻), strongly predicted ASD trait severity of their autistic sibling and categorical recurrence of ASD within participants’ families (i.e., accounting for ~ 50% of variance). These results suggest that genetic variation underlying non-specific neurodevelopmental traits may represent background ASD susceptibilities that are inherited and non-specific and may confer additive genetic risk alongside variants conferring specific ASD liability (e.g., BAP and subclinical ASD features in parents). These non-specific traits may be detectable earlier than other ASD-related behaviors and serve as critical early targets for intervention. Together, these results suggest that expanding our endophenotype definition to account for a broader range of traits and their developmental variance is crucial.

Several factors have impeded progress in identifying endophenotypic traits useful for gene discovery in ASD. First, studies aimed at understanding the genetics of ASD are inherently constrained by their reliance on categorical definitions that do not have a strong grounding in biology. The diagnostic standards developed by DSM task forces and the gold-standard assessment tools used to inform ASD classifications each primarily were intended to guide reliable, differential diagnoses based on behavioral observation. Historical beliefs that behaviorally defined categories would structure research that could eventually illuminate distinct pathogenic processes for select disorders have not been borne out, likely reflecting the high level of clinical overlap across diagnostic categories and the profound heterogeneity within categories. This realization was explicitly acknowledged by the American Psychological Association prior to publication of DSM-5 when they indicated “historical aspiration of achieving diagnostic homogeneity by progressive subtyping within disorder categories no longer is sensible” ([33], p. 12). It is not surprising then that separate behaviorally defined neurodevelopmental disorder categories (e.g., ASD, ADHD, and obsessive-compulsive disorder, or OCD) each show considerable etiological heterogeneity and overlapping genetic backgrounds [34,35,36]. Case-control designs that rely on categorical determinations for participant selection also do not capture the full range of trait variation useful for understanding additive genetic effects as demonstrated by population studies showing that defining and associated characteristics of ASD each are normally distributed within the non-autistic population and overlap considerably among autistic and non-autistic individuals [37, 38]. These data indicate transdiagnostic or population-based designs consistent with broader efforts in psychiatry (e.g., the Research Domain Criteria structure of NIMH; Hierarchical Taxonomy of Psychopathology consortium) should be emphasized in the pursuit of identifying familial genetic factors contributing to traits associated with ASD [39].

Translation of the endophenotype concept to ASD also is made difficult by the developmental variance seen in both clinical and biological traits across the life span. Clinical signs of ASD emerge within the first years of life (or perhaps earlier) and evolve in non-linear ways, suggesting endophenotypic traits may differ quantitatively as a function of the age or developmental level at which individuals are studied. In support of this hypothesis, multiple studies have shown that some behavioral differences seen in young siblings of autistic individuals relative to non-autistic peers are not predictive of eventual affectation status [40,41,42,43,44,45,46,47,48]. These findings implicate developmental compensations that mask trait expression at later ages among unaffected first-degree relatives [49]. Further, evidence of “normalization” of key behavioral deficits in later childhood or adulthood among autistic individuals and their unaffected relatives [50,51,52,53] suggests that mapping growth trajectories of endophenotypic traits will be critical for establishing trait markers sensitive to gene variation and changes in expression over time.

Additional support for a developmental perspective comes from studies showing characteristics reflecting ASD likelihood in infancy are not direct phenocopies of the defining features measured in children or adults implicating qualitative differences in trait expression. For example, recent studies of sibling concordance have indicated that trait dimensions often considered co-occurring conditions may present as the earliest indicators of ASD for some individuals (e.g., motor coordination and behavioral control impairments [32]). These data highlight the importance of analyzing developmental traits beyond core ASD features using a “broad” phenotyping strategy in which diverse traits and their neurodevelopmental and molecular substrates are analyzed. This strategy also demands multivariate analytic approaches in which interactions between different trait domains can be examined. Co-occurring specifiers introduced in DSM 5 (e.g., accompanying intellectual or language impairment) offer a useful set of traits that should be considered as they each individually may represent a top-down starting point for identifying biological processes that collectively contribute to neurodevelopmental disorders including ASD (e.g., accompanying language or intellectual impairment). The question of why these associated traits may co-segregate with core autistic traits in some but not other individuals may be best answered by clarifying the etiologic pathways that underpin discrete traits rather than the more complex (and variable) constellation of clinical behaviors.

Sex differences in both etiological processes and clinical outcomes for autistic individuals also are important considerations in the study of autism genetics. ASD disproportionately impacts males relative to females, and there is considerable evidence that males and females with ASD differ clinically [54, 55], at the level of structural and functional brain development [56,57,58], and genetically [54, 59]. Infant sibling studies have shown that at least some early indicators of ASD hold predictive power for males but not for females [60], suggesting that candidate traits useful for mapping individual pathophysiologies may differ as a function of sex. In contrast, Burrows et al. [61] recently utilized a data-driven approach to derive behavioral factors and map their early childhood trajectories among infant siblings. Results identified similar sex ratios in infants identified in a “high concern” cluster based on both social-communication and restricted behavior dimensions. Similarly, findings form the Baby Siblings Research Consortium suggest that out of ~1800 toddlers, sex differences in cognitive performance and repetitive behaviors are observed across siblings with a family history of autism (FH) and children with no family history of autism (nFH), suggesting that some early emerging sex differences in cognitive and behavioral development do not appear to be ASD-specific but instead reflect sex-dependent variation in developmental processes that may or may not be altered in ASD [62]. These findings highlight the critical need to adjust for sex-specific biases across trait development for establishing new endophenotypes predictive of autistic trait expression. Inclusion of autistic females in ASD research has been increasingly emphasized in recent years, though this subpopulation remains understudied and still constitutes only a small minority of individuals included in ASD research. There is a strong need for systematic comparisons of trait variation in males and females in the search for ETDs.

Factors affecting the sensitivity of candidate endophenotypes to genotypes of interest

The genetic landscape of many heritable diseases is defined by complex, non-linear, and polygenic architectures that will not map cleanly onto models developed to predict categorical outcomes (e.g., affectation status) that include highly complex clinical pictures and a diverse range of individuals. While monogenic syndromes show tight genotype-phenotype relationships, complex conditions such as ASD involve high levels of polygenicity, environmental influences, and stochastic events that collectively contribute to diverse behavioral and developmental variation. Defining models that are sensitive to the complex polygenic processes associated with ASD will require identification of dimensional traits that co-vary with additive and non-linear likelihood elements across the affected and unaffected population [63]. By prioritizing dimensional traits sensitive to the full range of expression of candidate genes, power can be maximized for detecting important effects of low penetrant genes that may be accounting for important variation in autistic traits.

The extent to which resolution for identifying pathogenic mechanisms is increased depends on multiple features of candidate traits, including their proximity to the action of the gene. Endophenotypic traits constitute “bridges” linking molecular, cellular, and system-level mechanisms and clinical dimensions. Their associations with genetic variation will be stronger if their relative location on these “bridges” is closer to genetic origins. In principle, endophenotypic traits closer to molecular processes that scaffold brain development will show greater power for informing mechanistic models than more downstream traits that vary as a function of epigenetic processes, environmental factors, and stochastic influences. For example, while the most common endophenotypes studied in psychiatry tend to come from cognitive psychology and take a behavior first, “top down” approach to ETD discovery, cognitive traits often have highly complex genetic architectures themselves, suggesting that their added value relative to measurement of behavior may be limited. It also has been argued that some candidate endophenotypes, including electrophysiological traits, also have shown little power for advancing gene discovery in psychiatric conditions because they have such complex genetic architectures themselves, and thus their study will require very large samples (i.e., tens of thousands) [64]. In contrast, assays of molecular structure or function including transcript- or blood-based traits may offer increased power as they likely will be more directly influenced by the genotype of interest. Mapping ETDs beginning with more molecular traits, or using a “bottom-up” approach, may provide important traction for understanding simpler genetic structures related to discrete trait outcomes. The relationships between molecular traits and the signs and symptoms of ASD often are unclear; however, suggesting that integration of endophenotypes across multiple units of analysis (e.g., transcript, brain function, cognitive processing) will be integral to mapping causal pathways.

Endophenotypic traits also will provide greater power for elucidating genotypes of interest if they are highly translatable across species and model systems. Analyses of preclinical model systems allow for more direct interrogation of cellular and molecular processes and can therefore facilitate more detailed descriptions of the pathways linking genetic substrates and behavioral traits. This consideration is especially important in studies of ASD given that defining symptoms are complex and not easily translatable across primate and more primitive species. For example, translation of the social behavioral difficulties experienced by autistic individuals to model systems is difficult given the complexity of these behaviors and limited conservation of social brain network functions across species. Backwards translation of traits that may be more ontogenetically primitive, including sensory and motor processes, as well as traits measured similarly across species, such as brain structural and functional connectivity, may provide critical insights into the genetic, molecular, and cellular bases of autistic traits. This hypothesis has been supported by studies of “endophenotype ranking values (ERV)” applied to traits associated with separate psychiatric diagnoses [27, 65]. The ERV provides a quantitative ranking system based on estimates of standardized genetic covariance of candidate traits with clinical outcomes. ERV studies have indicated that endophenotypes closer to gene action and translatable across species may show the greatest potential to guide a more mechanistic understanding of complex behavioral dimensions [27, 66].

Study designs and considerations for identifying endophenotypes

Endophenotypic traits are distinct from other biomarker classes because they represent inherited, additive effects related to development of select traits or disease processes. Based on this premise, candidate traits should show high levels of familiality, meaning they both co-segregate within families and track with clinical traits across affected and unaffected family members. To assess the familiality of candidate traits, multiple different family study designs may be leveraged. Integration of findings across these separate designs is important given that each approach is characterized by unique sets of strengths and limitations.

Family studies represent a broad class of methodologies aimed at determining the extent to which discrete traits differ between unaffected relatives and population controls and co-vary across related family members. The underlying assumption of family study designs aimed at identifying candidate endophenotypes is that traits with strong patterns of inheritance should be more similar across first-degree relatives compared to distant relatives and unrelated population controls. According to Gottesman and Gould’s [28] criteria, strong candidate endophenotypes are evidenced by profiles in which the greatest deviations are seen in individuals with a particular condition relative to population controls, but family members show intermediate levels of deviation from controls in the same direction as seen for affected individuals. Both family trio (biological mother, biological father, and autistic offspring) and quad studies (biological parents, affected, and unaffected siblings) will be important for identifying endophenotypes associated with ASD, and initial family studies already have documented cognitive [67], behavioral [68], and brain differences [69] that covary across affected individuals and their biological parents.

Infant sibling designs represent an important family study approach useful for clarifying endophenotypic traits associated with early development in ASD. Based on the high heritability of ASD and findings that ASD recurrence among siblings (13–20% [70, 71]) is considerably higher than base rates in the general population (1–2% [72]), analysis of infant siblings of previously diagnosed children provides important power for identifying both diagnostic predictors and traits associated with ASD that track in family members. Beginning with Bryson et al.’s [73] initial report documenting social and attentional differences in 12-, but not 6-month-old infants with familial history of ASD who later were diagnosed with ASD, infant sibling designs have identified multiple early emerging markers that can be identified years before the age at which children typically are diagnosed [73, 74]. These studies also have identified candidate traits that track in affected (FH⁺) and unaffected (FH⁻) siblings, suggesting that they may reflect inherited genetic influences associated with ASD. Traits that are deviant in both FH⁺ and FH⁻ infant siblings relative to same-age nFH population control infants reflect an important class of familial biomarkers that can guide new knowledge of inherited genetic substructures related to liability for autistic traits. Findings that distinguish FH siblings and nFH peers, regardless of diagnostic status, may reflect familial genetics while stepwise patterns of effects (e.g., FH⁺  > FH⁻  > nFH) may indicate key familial traits that confer susceptibility but only lead to ASD when sufficiently severe, through interaction with separate liabilities, or when exacerbated in the context of developmental or stochastic effects. Familial traits detectable in infancy also may be particularly important for defining endophenotypes because they are less likely to be impacted by compensatory behavioral or brain processes in unaffected relatives. For example, behavioral differences in unaffected first-degree relatives of autistic individuals may not be as severe or detectable in later childhood or adulthood due to compensatory processes used to mask perceived challenges, or atypical maturational trajectories that converge with neurotypical patterns later in development [75, 76].

Additional examples of the power of family study designs for advancing gene discovery related to neurodevelopmental disorders can be found in adult psychiatry. For example, as described above, BSNIP takes a unique approach to family genetic research by developing hypothesis-driven and data-driven endophenotypes associated with psychosis that cut across diagnostic boundaries, deriving multi-level endophenotype factors that separate biologically separable clusters of patients, or “biotypes” [77, 78]. These biotypes do not align with diagnostic categories of DSM but are characterized by greater homogeneity of neural features and higher levels of intra-familial trait aggregation than DSM diagnostic categories [78]. Analyses of these biotypes in family genetic studies offer significant promise for gene discovery, and greater power than case-control studies using classification strategies that have less grounding in biology. Analogous approaches have yet to be leveraged in ASD, but such strategies could greatly increase power for detecting the additive, complex genetic substructures that contribute to autistic traits for the majority of affected individuals.

While family studies are central in the search for endophenotypic traits useful for defining etiological mechanisms associated with familial ASD (as opposed to ASD caused by de novo variants or environmental factors), their limitations also should be considered in the context of research aimed at understanding heritable influences. Family study methods provide only necessary, but not sufficient evidence that select traits are inherited. More specifically, family study designs are not capable of quantifying the impacts of shared environments and are thus not able to directly inform estimates of heritability. Family designs that focus on traits already shown to be highly heritable or less likely to be strongly influenced by social modeling or other environmental influences (e.g., social determinants of health such as socio-economic status or educational quality) may provide greater leverage in advancing endophenotype discovery.

Trait heritability is best demonstrated through twin studies examining concordance in monozygotic (MZ) and dizygotic (DZ) twins raised in the same family environments. Based on the assumption that MZ twins share approximately double the proportion of genetic material as DZ twins, greater similarity (i.e., concordance) between MZ relative to DZ pairs on select traits serves as an index of variation attributable to genetic inheritance rather than shared environment. Large-scale twin studies offer rich information on the heritability of individual traits that may directly guide analyses of candidate endophenotypes for clinical conditions or dimensions and guide interpretation of studies examining autistic individuals and their unaffected family members. For example, basic attentional and sensorimotor traits implicated in ASD each show high levels of heritability based on population twin studies and thus offer strong candidates for endophenotype discovery (e.g., [79, 80]). Importantly though, a series of twin studies examining eye-tracking data also has documented high levels of heritability in more complex processes including visual exploration during processing of social [80] and non-social scenes, as well as executive functions such as visual disengagement [81] and behavioral response inhibition [82]. Each of these behaviors has been implicated in ASD through either case-control or family study designs, suggesting that they may represent promising endophenotypes sensitive to inherited variation. Studies focused on the familial co-segregation of strongly heritable traits in ASD optimize the strengths of both family and twin designs for endophenotype discovery [79,80,81,82].

Twin studies have proven invaluable for developing new knowledge on the heritability of ASD and autistic traits. For example, twin studies consistently have indicated that ASD represents perhaps the most heritable behaviorally defined condition identified in the DSM [83,84,85]. They also have highlighted several key considerations in the analysis of trait inheritance patterns associated with ASD. First, heritability estimates show significant variation across studies (37–90%; [86,87,88,89]) due to multiple features, including the complexity of genotype-environment interactions, and the strong influence of measurement differences on the classification of individuals. For example, Colvert et al. [88] found that additive genetic effects explained 76–95% of diagnostic covariance when classifications were based on the Developmental and Well-Being Assessment and Autism Diagnostic Observation Schedule, but only 56% of covariance when classification was based on the Autism Diagnostic Interview–Revised (ADI-R). Second, evidence that trait concordance may be substantially lower in autistic relative to non-autistic twins implicates greater sensitivity to stochastic influences during development. Conducting the first known quantitative analysis of twin-twin severity of autistic traits, Castelbaum and Constantino [90] documented high levels of diagnostic concordance (96%) but low levels of autistic trait concordance across three different samples of autistic MZ twins (R² < 0.1). Trait concordance was high in non-autistic population control MZ twins (R² = 0.6) indicating that trait heritability estimates from non-autistic twins may not be directly translatable to ASD populations, and that environmental influences on trait outcomes and development may be profound.

In summary, family and twin studies each provide critical, complementary information for establishing endophenotypic traits associated with ASD. While family designs, including trio, quad, and infant sibling methods, each can yield important new insights into traits that “run in families”, they are not able to directly index heritability of candidate traits. Instead, heritability estimates from twin studies are important for understanding inheritance patterns and parsing environmental and genetic contributions to trait outcomes. Twin studies also are limited by difficulties ascertaining sufficient samples of autistic twins and evidence that heritability estimates from non-autistic twins may not be directly applicable to autistic twins. Large-scale twin studies that characterize carefully selected continuously distributed traits in the general population represent a promising future direction. More specifically, focus on analyses of early emerging heritable traits less influenced by stochastic or environmental processes will be important for identifying endophenotypic traits sensitive to inherited genetic processes.

Candidate endophenotypes and trait domains associated with ASD

The majority of traits examined in family studies of ASD have been measured at the behavioral level. It is important to note that endophenotypes originally were differentiated from behavioral traits based on their “internal” and “unobservable” qualities allowing them to be objectively measured and suggesting they lie more proximal to gene action [26, 91, 92]. Based on this premise, it has been argued that familial behavioral traits should be considered separately as intermediate phenotypes [92]. This distinction is important in the context of findings that behavioral and cognitive dimensions studied as familial traits in ASD also have highly complex genetic architectures, suggesting their added value in advancing knowledge of pathogenic processes relative to clinical symptoms may be limited [27, 92, 93].

Despite these caveats, we propose that familial behavioral traits are important to integrate in family genetic studies because they can be leveraged to focus the search for associated “internal” traits closer to genetic substructures. Studies of core autistic traits represent important examples of how understanding familial behavioral phenotypes may guide the search for endophenotypic traits. For example, considerable evidence shows that first-degree relatives of autistic individuals present mild traits qualitatively similar to those found in their autistic family members, justifying the definition of a “broader autism phenotype” (BAP) characterized by social aloofness, pragmatic communication difficulties, and a rigid personality style [94, 95]). Notably, a stepwise pattern of BAP trait loading has emerged for multiplex, simplex, and adoptive parents of autistic children [96, 97] indicating that trait burden reflects an aggregation of ASD-related genetic liability, and that dosage of genetic influences is greater in families with recurring ASD [98, 99]. Consistent with this hypothesis, Lyall et al. [100] documented that child ASD risk is increased by 85% when both parents show elevated autistic traits. Collectively, these findings provide strong support that analysis of BAP traits across family members holds significant power for determining familial ASD processes and informing genetic models [94, 95, 98,99,100].

In principle, identifying endophenotypic traits associated with BAP behaviors and those features closer to molecular and neurodevelopmental mechanisms contributing to ASD will increase resolution for detecting etiological pathways. This process of mapping ETDs by linking behavioral traits back to their genetic origins aims to define the full chains of causality contributing to traits that may collectively constitute ASD. To illustrate how this approach may enhance resolution for detecting genetic substructures, we integrate findings from case-control and family designs assessing the neurodevelopmental substrates of multiple behavioral traits associated with ASD. Our list of candidate ETDs is not intended to be exhaustive. Instead, we focus on discrete behaviors that are highly quantitative, have been studied across multiple levels, and also have shown promise for clarifying etiological processes based on studies of their heritability, familiality in ASD, underlying biology, and potential for back-translation to model systems (see Table 1).

Table 1 Summary of evidence for candidate ASD-related endophenotypic trait domains (ETDs) across multiple levels of analysis. ETDs described here represent some of the most promising targets for ASD research on endophenotypes based on accumulating evidence showing targeted phenotypes are quantitative, translational, present in unaffected relatives of autistic individuals, and measurable across multiple units of analysis. This table is not intended to be a comprehensive list of candidate ETDs associated with ASD but represents perhaps the most promising ETDs based on existing data

Social gaze

Social impairments have served as the pathognomic features of ASD since Kanner’s original case studies [154]. They include a broad class of behavioral issues ranging from differences in basic attentional allocation to social information to more complex cognitive difficulties in the processing of social cues and development of social relationships (e.g., [101, 155]). The complex nature of many social behaviors that are impacted in ASD highlights challenges in establishing their underlying biology and therefore identifying endophenotypic traits. Quantitative studies of social attention have served to deconstruct complex behaviors into simpler cognitive processes that offer more potential for back-translation to preclinical models and may represent simpler genetic substructures than complex social phenotypes. For example, social gaze toward faces represents a highly quantitative biomarker that appears to show familial patterns. Importantly, eye tracking of social gaze can be done in studies of wide ranges of ages as demonstrated by multiple studies documenting non-linear changes in social gaze as early as the first weeks of life through adulthood (e.g., [101]). Leveraging similar measurement approaches, foundational work from Jones and Klin indicated that infants later diagnosed with ASD show declining rates of gaze shifts toward the eye regions of others between 2 and 6 months resulting in reduced overall attention to faces relative to neurotypical peers [102]. These findings are consistent with separate studies documenting reduced attention to faces in autistic adolescents and adults [103, 104]. Reduced eye gaze is strongly associated with the severity of autistic traits suggesting that this highly quantitative trait may represent a key endophenotype related to liability for autistic traits or ASD affectation.

Family studies provide support for social gaze as a useful endophenotypic trait associated with ASD. Studying non-autistic twins, Constantino et al. [79] demonstrated high levels of heritability for infant gaze to eye and mouth regions. MZ twins also showed high concordance of the duration and direction of saccadic eye movements when viewing social scenes, consistent with separate studies documenting high levels of heritability for social gaze in more complex social environments [80]. Social gaze differences also appear to be strongly familial in ASD [80]. Atypical social gaze is seen in parents [105] and siblings of autistic individuals [51], and parents with BAP characteristics show decreased gaze toward social stimuli in complex scenes relative to population controls and parents of autistic individuals who do not show BAP features [105, 106]. These findings suggest that atypical social gaze is associated with core autism behavioral traits in unaffected family members and may serve as a quantitative link between social traits of ASD and their inherited biological underpinnings.

Quantitative EEG/ERP studies of social gaze in ASD offer insights into the neural substrates of social gaze phenotypes. The N170 peaks 150–200 ms after stimulus onset and is sensitive to the visual presentation of social stimuli including faces [156]. During viewing of faces, the N170 is thought to index early structural encoding of facial features and face categorization [157, 158]. In non-autistic individuals, the latency of the N170 ERP is shorter to faces than to houses or other objects [120] suggesting increased automaticity of percept processing. The N170 to faces also typically is right lateralized implicating hemispheric specialization for processing facial information [159]. Delayed N170 latencies have been consistently documented in autistic individuals compared to non-autistic controls (e.g., [120, 121]), and they appear to be associated with reduced memory for faces [160]. Both autistic individuals and their unaffected siblings also demonstrate a reduced difference in amplitudes between inverted and upright faces compared to controls suggesting reduced specialization of neural processing of facial information that is familial [122]. Parents [123] and infant siblings of autistic individuals show diminished right lateralization of N170 amplitudes associated with more severe social-communication impairments and sensory symptoms further indicating that developmental specialization of select social brain networks represents a familial neural endophenotype associated with core autistic characteristics [124].

While a myriad of MRI studies have identified structural and functional brain alterations [104, 161, 162] associated with atypical social gaze in ASD implicating prefrontal and temporo-parietal circuits as well as amygdala nuclei (for reviews, see [163, 164]), only a small number of MRI studies have examined neural correlates of social gaze in unaffected family members. During face viewing, parents of autistic individuals irrespective of BAP status demonstrated increased functional activation of the fusiform face area and amygdala [126] suggesting that increased activation in these regions associated with social gaze processing reflects ASD-familial status; however, parents with the BAP showed more severe hyper-activation of right fusiform gyrus than parents without the BAP and controls. Consistent with this findings, both autistic individuals and their unaffected siblings show reduced frontal (i.e., left superior frontal gyrus; right middle, left posterior, left dorsomedial PFC) and temporal (i.e., temporal pole, fusiform face area) activation during face viewing relative to controls [125] indicating alterations of neurodevelopmental processes supporting the functional specialization of brain networks involved in facial processing represent strong candidate endophenotypes for elucidating familial and perhaps inherited trait processes.

Multiple separate preclinical studies also have been conducted to understand cellular, physiological, and molecular genetic mechanisms of different social traits associated with ASD. Current challenges in back-translation of ASD-related social differences include limits to mimicking complex social behaviors in model systems such as the rodent models that often are used in genetic knock-out (KO) studies. Despite these challenges, studies of non-human primates have provided some insight into cellular and molecular processes associated with social gaze behavior. Chen and colleagues found that non-human primates with TALEN-edited MECP2 mutations exhibited a preference for social over nonsocial stimuli and looked to conspecific faces exhibiting aggressive and submissive expressions for shorter durations than neutral expressions [138]. Based on findings implicating maternal immune system activation in autistic liability, non-human primate maternal immune activation models have also been studied to understand basic physiology of social gaze and relationships with autistic traits. During viewing of facial images of conspecifics, offspring of maternal immune activation models showed multiple gaze differences relative to control animals, including longer latencies to fixate on the eyes, fewer fixations directed at the eyes, and less fixation time on the eyes [139]. Studies of non-human primates defining cellular and synaptic processes supporting social gaze also may be integrated to clarify cellular and molecular mechanisms of social gaze differences in ASD and their genetic underpinnings. Non-human primate studies have documented selective firing of lateral interparietal neurons during social gaze and gaze shift behavior [165] and increased firing of amygdalar cells during viewing of eyes and direct eye contact [166].

In contrast to non-human primate studies, mouse genetic studies allow for interrogation of gene-specific effects and downstream molecular and electrophysiological traits. Yet, they are limited in the extent to which behavioral phenotypes can be translated to inform understanding of complex social traits in humans given limited homology of brain and behavior between humans and rodents. For example, common behavioral assays for rodent models of social traits in ASD include nose-to-nose sniffing, pushing or crawling, time spent in a chamber with (or at a partition next to) another mouse versus alone, partner preference, and social transmission of food preference [167]. Most of these behaviors do not closely mirror naturalistic social behavior in humans; however, social approach, withdrawal, and recognition are cross-species social behaviors that may be more easily translatable from rodent models to humans. Multiple mouse models of autism have documented reduced social approach relative to wild-type (WT) mice, suggesting multiple gene variants associated with ASD may contribute directly to diminished interest in or increased resistance to social interaction [140, 141]. Findings that the selective GABA_B enantiomer R-Baclofen reverses deficits in social approach in BTBR mice [142] and mice with 16p11.2 microdeletions [143] suggest that atypical GABA signaling may underpin social approach differences in some autistic individuals, though R-Baclofen has shown variable effects on social difficulties in autistic individuals. Integration of these preclinical studies therefore offers important traction for understanding molecular and cellular processes that may contribute to social impairment in autistic individuals, though these studies also highlight the significant challenges in translating model system findings regarding the pathophysiology of social difficulties to humans.

Together, data across behavioral, cognitive, and brain system levels suggest that social gaze traits and their neural substrates comprise a promising ETD linked to the pathognomonic features of ASD. Separate social behavioral traits associated with ASD also warrant further study. For example, atypical functional activations of fusiform gyrus, superior temporal sulcus, middle frontal gyrus, and amygdala during biological motion processing have been identified in both autistic individuals and their unaffected siblings, including compensatory activations seen specifically in unaffected siblings [49]. The extent to which biological motion processing differences and associated alterations in functional brain development relate to differences in social gaze or represent a unique ETD in ASD remains unclear. Histological analyses of cellular features and gene expression in targeted brain regions of autistic individuals and their family members also will be important for determining the molecular substrates of familial social gaze differences.

Language and communication

Autistic individuals show a range of language and communication challenges including reduced receptive and expressive language abilities as well as difficulties in pragmatic forms of communication [168]. Differences in the suprasegmental aspects of speech, including prosody (e.g., intonation, rate, and rhythm of speech) also are common. Atypical intonation, rate, and rhythm of speech appear to be familial, suggesting they represent promising targets for establishing endophenotypes useful for understanding genetic pathways associated with autistic traits and liability [20, 168, 169].

Meta-analytic data indicate that autistic individuals exhibit higher mean pitch, greater pitch variability, and longer voice duration relative to controls [107]. These features have been highly reliable for distinguishing autistic from non-autistic individuals [107, 108]. Differences in suprasegmental components of speech among autistic individuals are also evident in tonal languages which use pitch to convey not only pragmatic information, but also word meaning [170]. For example, Cantonese speaking autistic children and adults show less robust prosodic encoding compared to controls, suggesting that difficulties processing suprasegmental features of speech in ASD persist into adulthood despite years of experience with a tonal language [170].

Measuring non-speech vocalizations also is important for understanding language and communication development across the lifespan in autism. For example, infant siblings of autistic children show differences in vocal properties that emerge early in ontogeny, including atypical prosody during crying in infancy [42, 46]. At 6 months, pain cries from FH infants are higher and more variable in pitch than those of nFH peers [46]. Similarly, Esposito and colleagues found that cries of 15-month-old FH toddlers showed higher frequencies [42] and were shorter in duration than nFH toddlers [42, 109]. Importantly, FH⁺ infants had higher fundamental frequencies than FH⁻ infants, who had higher fundamental frequencies than nFH infants (FH⁺ > FH⁻ > nFH) [42]. Parents of autistic children show some overlap with their offspring in suprasegmental speech, including greater variability in frequency driven specifically by mothers showing high levels of BAP features [108]. These findings suggest differences in suprasegmental aspects of speech that may represent endophenotypic traits in a select subset of families and therefore may serve as important targets for parsing etiological heterogeneity [171, 172]. Large-scale quantitative family studies assessing interactions of maternal autistic traits and suprasegmental aspects of speech hold promise for understanding inherited genetic processes contributing to ASD in select clusters of families.

Suprasegmental features of speech represent important targets for studies of endophenotypes because relevant measures also can be translated to neuroimaging environments and across species. Studies of neurophysiological processes associated with speech differences in ASD have provided important information on candidate endophenotypic traits. The frequency following response (FFR) is an early auditory-evoked potential which marks how sounds at the frequency of natural speech and their harmonics are encoded in relevant cortical pathways [173,174,175]. The degree to which frequency of the stimulus is reflected in the FFR indexes the integrity of the auditory pathway for encoding early features of acoustic stimuli [175]. Importantly, FFR responses can be leveraged across the lifespan to understand neural processing of sound, especially since this neurophysiological measurement is a customary practice in newborn hearing screening. Autistic individuals show less stable FFR responses to speech sounds relative to non-autistic peers suggesting greater levels of variability in processing auditory speech information [127]. Differences in the neural processing of sounds may underlie difficulties monitoring, adjusting, or matching aspects of speech like tone, rate, and rhythm, which may in turn affect higher-order features of language like prosody and pragmatics. Additionally, during a task of vocal production with dynamic auditory feedback, autistic individuals and their parents showed reduced auditory P1 amplitudes relative to age-matched controls, reflecting less robust detection of changes in pitch during auditory feedback [128]. Diminished ability for feedforward vocal control in contexts requiring audio-vocal integration may be a mechanism of prosodic differences in ASD and may also be reflective of disrupted neuromotor control of speech.

Back-translation of language and communication traits into preclinical models represents a major challenge given the unique language and communication abilities of humans and non-human primates. Still, analysis of suprasegmental speech offers some opportunity for translational research that may be informative for understanding genetic and molecular substrates of impairments in ASD. For example, differences in the acoustic properties of pup vocalizations can be measured in rodent species and appear to be selectively disrupted by KO of multiple ASD-related genes. More specifically, higher peak frequencies and reduced modulatory abilities of vocalizations have been demonstrated in mouse models of ASD [144], including higher peak frequencies and reduced modulatory abilities in SHANK3 KO, mirroring results found in autistic individuals [145]. Separate non-human primate models of ASD may be leveraged to characterize qualities of vocalizations associated with ASD to help bridge pre-clinical rodent findings to brain system, neuropsychological, and behavioral level communication traits associated with ASD.

Language-related endophenotypes separate from suprasegmental speech characteristics also exist (e.g., pragmatic communication features, expressive vocabulary). Fewer studies have examined the familiality and neural substrates of these distinct language and communication traits owing to multiple considerations, including their relative complexity (e.g., pragmatic communication) and limited suitability for imaging environments sensitive to oromotor movements (e.g., functional MRI). Studies of language-related endophenotypes also are hindered by difficulties assessing the full range of language impairments in ASD, including both non-speaking and fluent individuals. Assessment of neurophysiology may be particularly challenging with autistic individuals who show severe impairments in processing or expressing language, as these individuals may not process instructions or task demands. Similarly, individuals with severe expressive language disability may show limited ability to respond to task demands, or rates of echolalia that may make it difficult to obtain adequate naturalistic language samples [176]. The growing use of augmentative and alternative communication (AAC) strategies may also introduce questions of comparability between expressive language in written and spoken modalities. These challenges have contributed to a dearth of studies including the full range of language traits in studies of ASD, confounding progress in determining etiological processes. There is a critical need to develop strategies to integrate individuals who are minimally- or non-speaking and, where possible and appropriate, utilize compound tasks that involve both direct measurement of language and simultaneous tracking of biobehavioral indices (e.g., imaging, neurophysiology, eye tracking, oromotor movements).

Cognitive control

Executive functions represent a diverse range of cognitive abilities supporting the regulation of thought and action, including the abilities to inhibit dominant responses, update working memory representations, and flexibly shift behavior or cognitive strategies in response to changes in environmental demands. Consistent with their diversity, the genetic substrates of executive abilities are complex. Executive functions show high levels of heritability involving a common genetic “factor” as evidenced in the analysis of twin concordance (e.g., [177]). While these data implicate common, likely polygenic processes contributing to high levels of heritability for executive functions, different executive abilities also appear to be modified by separable genetic influences that contribute to diverse estimates of heritability for individual functions [177, 178]. These results are consistent with studies documenting modest covariation of different executive abilities and the overarching conceptualization that executive processing is a multi-dimensional set of cognitive operations [179, 180]. The genetic structures of executive abilities also appear to show complex interactions with general cognitive abilities, showing both covariation and high levels of heritability that are distinct from genetic influences on IQ, processing speed, and visuospatial abilities [177, 178]. These findings collectively indicate that while executive abilities involve multiple diverse functions, heritability is high and trait inheritance likely reflects common pathways that also can be modified by separable genetic influences that shape individual executive outcomes.

Despite findings that executive functions are among the most heritable psychological traits [177], and that they are consistently disrupted in autistic individuals [181], their familiality in ASD seldom has been examined. Behavioral rigidity, perhaps the most common trait characteristic of family members of autistic individuals [97], appears to involve alterations of executive processing, including reduced abilities to flexibly shift cognitive strategies and to inhibit prepotent behavioral responses [67]. For example, prior work by our group and others has demonstrated that reduced cognitive flexibility and behavioral response inhibition in autistic individuals each are associated with more severe clinically rated RRBs, including a strong need for sameness in the environment and in routines (i.e., insistence on sameness) and compulsions [110,111,112]. These findings together implicate cognitive inflexibility and reduced behavioral response inhibition as important cognitive trait markers associated with behavioral rigidity.

Similar cognitive traits have been implicated in unaffected family members. Reduced behavioral flexibility, characterized by a strong insistence on sameness and intense preoccupations, appears to co-segregate in families of autistic probands [182,183,184]. Studying a probabilistic reversal learning test in which individuals must shift a response selection away from a previously reinforced item after reinforcement contingencies have changed, Schmitt et al. [67] documented that unaffected parents of autistic individuals show an increased rate of regressing back to previously reinforced stimuli, or “regressive errors”, similar to findings from a previous study of autistic individuals performing the same reversal learning task [110]. These findings were consistent with prior studies documenting reduced cognitive flexibility on tests of set shifting in autistic individuals and unaffected first-degree relatives [113,114,115] but also extended this work to show that increased rates of regressive errors were specific to parents showing high levels of BAP features and their autistic children [67]. These results suggest that cognitive control traits are familial and that they may contribute to or interact with core autistic traits to increase ASD likelihood of affectation. These findings are particularly promising for defining multi-factorial endophenotypes (BAP features and cognitive control impairments) that may co-segregate and be useful for establishing more homogeneous subgroups, or biotypes, for family genetic research. Results also may help explain prior findings that some unaffected family members show relatively intact cognitive flexibility as subsets of families with an autistic family member but without parental BAP traits may have relatively preserved cognitive flexibility [185, 186]. Analyses of inhibitory control abilities among this same sample of autistic individuals and their parents showed increased error rates in parents and autistic individuals relative to age-matched controls, though deficits appeared to be independent of parental BAP status [67]. Together, these findings suggest that cognitive flexibility and inhibitory control impairments each may represent important familial endophenotypes associated with RRBs in probands and behavioral rigidity in parents, and that their analysis could help parse etiologic heterogeneity by identifying more cognitively homogeneous subsets of families.

Tests of cognitive flexibility (probabilistic reversal learning) and inhibitory control (stop signal and antisaccade) constitute important measures for ETD studies because each can be adapted to neuroimaging environments to examine underlying brain functions (e.g., [129, 187, 188]). Reversal learning is supported by neural systems including middle frontal gyrus, posterior parietal cortex, striatum, and midbrain nuclei [189]. Using fMRI, D’Cruz et al. [129] documented reduced activation in both prefrontal cortex and ventral striatum in autistic individuals relative to controls during trials in which participants needed to shift their response set. Reduced activation in frontal cortex implicates problems in decision-making and response planning, while atypical activation of ventral striatum suggests limited processing of reinforcement cues, as has been demonstrated more broadly in ASD [190, 191]. These processes, and their integration, are essential for flexible behavior. Alterations in these systems may therefore contribute to a rigid adherence to preferred behavioral patterns in autistic individuals.

Cognitive/behavioral flexibility also is a promising target for studies of ETDs in ASD because of the potential for back-translation to model systems. Probabilistic reversal learning tests in particular have proven to be highly translational strategies for studying neurophysiologic, cellular, and molecular mechanisms of cognitive/behavioral rigidity across species. For example, using a probabilistic reversal learning paradigm similar to that described above in clinical studies of autistic individuals, Ragozzino and colleagues have conducted a series of studies documenting elevated rates of errors in multiple mouse models of ASD [e.g., BTBR mice, mice reared in conditions of high maternal stress or maternal exposure to a selective serotonin reuptake inhibitor (SSRI)] [146, 147]. They also have demonstrated rescue of reversal learning deficits in mouse models with administration of an adenosine A_2a receptor agonist and a 5HT_2a receptor antagonist in dorsomedial striatum implicating selective receptor targets for drug development as well as gene expression studies focused on cognitive/behavioral rigidity in ASD [146,147,148,149].

Studies of inhibitory control show similar potential for translation across behavioral and imaging environments in ASD studies and for back-translation in lower-order species. Using tests of antisaccades, or the ability to consistently suppress prepotent oculomotor responses, multiple fMRI studies have detailed neural networks supporting top-down control of reactive behaviors, including frontal and parietal eye fields, anterior cingulate cortex, middle frontal gyrus, and dorsal striatum [187]. Increased rates of antisaccade errors have been repeatedly documented in autistic individuals [130, 131, 187, 192, 193]. and their unaffected first-degree relatives [111], and shown to be associated with reduced activations across frontal and parietal eye fields [132] that are important for generating motor plans [194]. Atypical activation of anterior cingulate also has been observed during antisaccades in ASD, suggesting deficits in response monitoring processes central to modifying behavior in response to external contingencies [130, 131]. Antisaccade tests frequently have been examined in non-human primates to determine the cellular processes that support inhibitory control (e.g., [195, 196]), and separate tests of behavioral response inhibition have been developed to examine these processes in rodent models (e.g., [197]). Parallel human and model system studies similar to those examining cognitive flexibility described above may help elucidate distinct genetic and molecular processes contributing to familial deficits of inhibitory control in ASD.

Sensorimotor control

Sensorimotor functions frequently are impaired in ASD affecting a range of behaviors and effector systems including oculomotor, vestibulo-motor, and skeletomotor processes [116, 198, 199]. Sensorimotor impairments in ASD also emerge early in development, perhaps earlier than primary social-communication and cognitive/behavioral traits [200,201,202,203,204,205,206,207]. Sensorimotor behaviors represent highly promising targets for endophenotype research because they can be decomposed into quantitative motor control processes subserved by discrete neural systems. For example, while a broad range of sensorimotor behaviors have been shown to be affected in ASD (e.g., [198, 199, 208]), converging evidence indicates a reduced ability to integrate multi-sensory feedback to guide ongoing behaviors resulting in increased variability and regularity of motor output [116, 209,210,211]. Brain systems supporting sensory feedback control during motor behavior have been well-delineated via non-human primate, human lesion, and basic neuroimaging studies and include temporo-parietal and occipital circuits involved in sensory processing and integration, premotor and primary motor cortices involved in planning and executing motor commands sent to the periphery, and cerebellar circuits involved in modulating motor commands to motor cortex based on sensory feedback error information relayed via cortical-pontine circuits (e.g., [212,213,214]). Cortical and cerebellar circuits involved in sensory feedback control of motor behavior each have been implicated in ASD via anatomical MRI studies [215] and post-mortem brain studies [216,217,218,219]. These results highlight the strong promise of investigating sensorimotor behavioral dysfunctions across motor control, brain system, and cellular levels for mapping ETD pathways associated with ASD.

Multiple sensorimotor behaviors, including sensory feedback guided precision manual motor behaviors and sensorimotor sequence learning, each show high levels of heritability in twin studies [220]. Analyses of the neural substrates of these sensorimotor behaviors also show strong inherited genetic contributions. More specifically, structural features of unimodal (sensory/motor) cortical regions show reduced variability and increased heritability relative to structural characteristics of heteromodal association networks involved in more complex cognitive operations [221]. These findings together provide evidence that the study of sensorimotor traits and their neurodevelopmental substrates may offer significant leverage for identifying ETDs associated with inherited autistic traits [220,221,222].

The hypothesis that sensorimotor traits represent promising candidate endophenotypes is supported by studies showing that sensorimotor impairments in ASD also may manifest in first-degree family members. Examining multiple oculomotor behaviors, we previously documented that impairments seen in autistic individuals also were present in their unaffected biological parents and siblings, including reduced accuracies of saccadic eye movements, reduced smooth pursuit eye movement velocity during closed-loop phases, lateralized reductions of smooth pursuit eye movement velocity during the open-loop phase (i.e., the initial period of smooth pursuit that precedes the availability of sensory feedback input due to afferent delays), and atypical lateralization of procedural learning of saccadic eye movements [119]. Each of these oculomotor differences parallels findings from prior studies of autistic probands [223,224,225,226] suggesting oculomotor abnormalities may track in families of autistic individuals. Studies of infant siblings of autistic children also highlight strong familial patterns consistent with the hypothesis that different sensorimotor abilities may represent promising endophenotypes. Infant sibling studies using standardized tests of fine and gross motor developmental abilities have indicated that motor skills are predictive of language outcomes [62, 201, 227] and autistic traits in FH infants [205, 228,229,230]. A stepwise pattern has been documented in which sensorimotor impairments are more severe in FH⁺ siblings relative to FH⁻ siblings who show differences compared to controls [62, 205]. It should be noted that infant sibling studies of sensorimotor behaviors also have yielded inconsistent results owing to multiple factors, including variance related to the abilities tested, diversity of the measures used to test early sensorimotor development, and developmental heterogeneity across the samples studied. Studies using behavioral coding or wearable sensors suggest that more consistent patterns of familiality may be detectable with quantitative measures relative to standardized behavioral assessments that aggregate categorical items (e.g., Mullen Scales of Early Learning). For example, stepwise patterns were seen in infant siblings (FH⁺  > FH⁻  > nFH; [117]), and during postural control, familial patterns were evident that varied according to whether FH siblings were diagnosed with ASD or language delay (FH⁺  < FH⁻ with language delay < FH⁻ without language delay = controls; [118]). Together, these findings offer strong support that separate sensorimotor behaviors may serve as endophenotypic traits representing early emerging inherited factors associated with autistic traits and related developmental issues (e.g., language delay). Separate sensorimotor behaviors do not show familial patterns (e.g., quantity of movement [231]) indicating that only a circumscribed set of sensorimotor behaviors will offer power as endophenotypic traits indexing heritable pathways associated with ASD.

The multiple familial oculomotor differences identified in studies of autistic individuals and first-degree relatives each implicate separate neurophysiological processes. The accuracy of saccadic eye movements, or rapid, ballistic shifts in eye gaze, is guided by frontal and parietal eye fields and modulated on a trial-to-trial basis by cerebellar-brainstem circuits that act to reactively adjust output precision according to error feedback information [232]. Smooth pursuit eye movements include both closed- and open-loop phases separated by the extent to which sensory feedback is available to guide output precision. Reduced accuracy of closed-loop pursuit suggests atypical processing of error feedback information in cerebellum as well as prefrontal cortex and frontal eye field circuits involved in planning motor behavior. Consistent with these findings, multiple studies have documented atypical activation of premotor and motor cortex during motor behavior [133] and cerebellar-cortical functional connectivity during visuomotor behavior [133, 134] in autistic individuals. These sensorimotor data implicate familiality of atypical neural functioning within systems supporting motor planning, sensory feedback guided motor behavior. Findings of reduced lateralized dominance during open-loop smooth pursuit eye movements in autistic individuals and their first-degree relatives [119, 233] also are consistent with findings of atypical functional lateralization of motor circuits in autistic individuals during rest [135], reduced lateralized hand dominance [135], and results showing reduced lateralization of ERP N170 components during facial processing [234], to suggest reduced hemispheric specialization of functional brain networks may be a familial endophenotypic trait affecting multiple developmental abilities in autism. Analyses of how different sensorimotor endophenotypes co-vary across family members are needed to determine whether they represent separate ETDs associated with distinct genetic substructures [119, 133, 135, 232,233,234].

Sensorimotor behaviors also offer important advantages in the search for endophenotypes based on their translational nature. Motor control systems are largely preserved across different species allowing for direct back-translation. Oculomotor systems have been studied extensively in non-human primates advancing resolution of the circuits and cellular processes contributing to distinct behaviors. For example, Takagi et al. [136] documented that ablation of posterior cerebellar vermis impairs feedback mechanisms supporting saccade accuracy. Recovery of function is seen in primates, though persistent deficits modulating accuracy across repeated events/trials are evident, highlighting a critical role in error feedback correction that also is seen in autistic individuals implicating cerebellar modulation of brainstem circuits [137]. Subsequent analyses show that olivary climbing fibers synapsing with cerebellar Purkinje cells are necessary for feedback learning and may act as a crucial circuit for interrogation of ASD-related sensorimotor impairment [235].

Multiple mouse genetic models of ASD also show sensorimotor deficits, including deficits of coordination and increases in movement variability offering important leads for clarifying molecular mechanisms associated with sensorimotor and perhaps downstream behavioral differences in ASD [150, 236]. Another mouse model of ASD via valproic acid exposure produced selective cell loss in motor cortex and cerebellum, with a greater degree of cell loss in males than females. In addition, both male and female mice exhibited motor deficits that were associated with Purkinje cell loss in crus I of cerebellum [151]. Findings that rescue of cerebellar Purkinje cell function through the mTOR pathway also indicate that understanding of the molecular, cellular, and neurophysiological bases of sensorimotor impairments may guide new therapeutic strategies targeting familial risk mechanisms of ASD [152, 153].

Summary

Multiple promising endophenotypes associated with key behavioral traits in ASD, including social, communication/language, cognitive, and sensorimotor behaviors, have been identified. Their analysis across multiple levels holds promise for identifying ETDs that may elucidate pathways of ASD inheritance. Analysis of ETDs is aided by focus on discrete, quantitative, and heritable traits that also are capable of being readily back-translated to model systems for interrogating cellular and molecular processes. The ETDs described above are not intended to be exhaustive, but instead serve as important targets for dense phenotyping family studies as has been done for separate psychiatric traits and clinical populations (e.g., [237,238,239]). Analyses of separate behaviors implicated in ASD (e.g., attention, sleep disturbances), and ETDs within the same individuals and across species will be imperative for mapping causal chains related to separate clinical traits in ASD.

Bottom-up approaches for parsing etiological heterogeneity in ASD

The ETDs described above each were developed based on descriptions of behavioral traits associated with ASD. While we propose that “top-down” mapping of ETDs starting with behavioral traits will accelerate progress in understanding inherited pathways, it also is likely that significant advances will be generated using a “bottom-up” approach in which traits with simpler genetic structures serve as the starting point for connecting genes to behavior. This approach involves developing ETDs that leverage known genetic/molecular targets, blood-based, neurophysiological, or cellular traits to understand neurobiological processes contributing to select behavioral outcomes [240, 241]. Multiple candidates have been implicated based on their associations with autistic traits and ability to provide mechanistic insights, including increased extra-axial fluid [242,243,244], atypical melatonin [245], and immune system dysfunctions [246]. Studies are needed to determine the familiality of these traits and the extent to which they may offer power for more direct linkage with pathogenic variants.

Analysis of serotonergic (5HT) function represents an illustrative example of how an endophenotype-first approach may provide traction for gene discovery in ASD. Elevated whole-blood 5HT (WB5HT), or hyperserotonemia, represents perhaps the most replicated biomarker associated with ASD. It is evident in more than 25% of autistic individuals, tracks in unaffected family members [247, 248], and is highly heritable [249]. Despite WB5HT being a more molecular trait than the behaviors described above, mechanisms of hyperserotonemia remain complex and not well understood. It is possible that hyperserotonemia in ASD reflects atypicalities in synthesis within the intestines, abnormal uptake into platelet, or differences in 5HT release related to receptor expression or function [249,250,251]. One candidate mechanism that has been well studied is 5HT_2a receptor function in platelet aggregation. The vast majority (99%) of blood 5HT is stored in platelets [251], and 5HT_2a receptor enhances platelet aggregation [252] and is correlated with WB5HT levels [251, 253]. While findings regarding 5HT_2a receptors in platelets have not been consistent in ASD studies [254, 255], hyperserotonemic first-degree relatives of autistic children show lower densities of 5HT₂ receptors in platelets in comparison to normoserotonemic relatives [251]. Further, PET and SPECT studies have indicated that individuals with ASD and parents in multiplex families show lower 5HT₂ density and binding in cortex that may be negatively correlated with platelet 5HT [253]. While associations between 5HT_2a binding, WB5HT, and discrete behavioral traits in ASD have been largely inconsistent, these results suggest that WB5HT and 5HT_2a receptor density and function could serve as important candidate traits for taking an endophenotype-first approach to characterize genetic substructures in subsets of families, and is highly heritable [249]. Despite WB5HT being a more molecular trait than the behaviors described above, mechanisms of hyperserotonemia remain complex and not well understood. It is possible that hyperserotonemia in ASD reflects atypicalities in synthesis within the intestines, abnormal uptake into platelet, or differences in 5HT release related to receptor expression or function [249,250,251]. One candidate mechanism that has been well studied is 5HT_2a receptor function in platelet aggregation. The vast majority (~ 99%) of blood 5HT is stored in platelets [251], and 5HT_2a receptor enhances platelet aggregation [252] and is correlated with WB5HT levels [251, 253]. While findings regarding 5HT_2a receptors in platelets have not been consistent in ASD studies [254, 255], hyperserotonemic first-degree relatives of autistic children show lower densities of 5HT_2a5HT₂ receptors in platelets in comparison to normoserotonemic relatives [251]. Further, PET and SPECT studies have indicated that individuals with ASD and parents in multiplex families (i.e., families with more than one autistic child) show lower 5HT_2a5HT₂ density and binding in cortex that may be negatively correlated with platelet 5HT [253]. While associations between 5HT_2a binding, WB5HT, and discrete behavioral traits in ASD have been largely inconsistent, these results suggest that WB5HT and 5HT_2a receptor density and function could serve as important candidate traits for mapping ETDs to characterize genetic substructures of select traits in subsets of families.

Increased total brain volume is perhaps the most replicated finding from MRI studies of autistic individuals [256,257,258]. Brain enlargement also has been documented down to infancy [259, 260] with altered growth trajectories observed across the lifespan [256] involving multiple brain regions and subcortical structures and both gray and white matter [257, 260,261,262,263]. Importantly, head circumference, an index of total brain volume, has been found to be heritable [19]. Further, first-degree family members of individuals with ASD and macrocephaly show increased head circumference relative to family members of individuals with ASD who are not macrocephalic [240, 264], suggesting that co-segregation of autistic traits and macrocephaly may represent separate inherited polygenic backgrounds or biotypes. While increased head circumference has been a consistent finding, associations with clinical and behavioral traits have been inconsistent as studies have documented relationships with social-communication severity, restricted, repetitive behaviors, uneven cognitive development, and language outcomes [265,266,267,268], though these results have varied as a function of the age of individuals studied, their severity, clinical outcome measures, and brain regions studied [19, 269]. These “inconsistencies” are not surprising given the wide range of mechanisms that could underpin generalized brain overgrowth, and studies focused on select subregions or subcortical structures have found strong and somewhat consistent relationships with different clinical behaviors (e.g., [267, 268, 270]). Developmental variations also appear to strongly impact the nature and direction of these relationships [161].

Further, separate mechanisms may contribute to brain enlargement, and the extent to which they may be related to overlapping or distinct phenotypic patterns remains unclear. For example, germline mutations in PTEN, a tumor-suppressing gene, have been linked to conditions of tissue overgrowth in Cowden and Bannayan-Riley Ruvalcaba syndromes [271] and macrocephaly in some individuals with ASD. More specifically, 10–20% of autistic individuals with macrocephaly show PTEN mutations [271,272,273], and macrocephaly also is seen in non-autistic family members of these probands, but only selectively in family members who also have PTEN [272]. These findings are important in potentially understanding downstream molecular signaling mechanisms (e.g., mTOR) that may contribute to risk for autistic traits and macrocephaly, and for guiding imaging and neuropsychological studies to map ETDs useful for indexing etiological processes associated with autistic traits in populations with macrocephaly. Importantly, associations between macrocephaly, autistic traits, and PTEN mutations also inform powerful approaches for interrogating select trait pathways in model systems. For example, happloinsufficient mice with PTEN mutations also show brain overgrowth and some social and behavioral traits associated with ASD [274]. Similarly, selective deletion of PTEN in the hippocampus and layers III and V of cerebral cortex lead to macrocephaly in mice as well as atypical social behaviors, sensory hyperreactivity, and neuronal hypertrophy [275]. It remains unclear if molecular signaling and neurodevelopmental pathways impacted by PTEN mutations may overlap with those affecting some autistic individuals with macrocephaly but without known genetic causes. Studies of PTEN mutations provide a promising illustration of how endophenotype research may benefit from studies examining relationships between the many known genetic variants associated with autistic traits and candidate endophenotypes. Knowledge of the functional significance of known genetic risk variants also may be leveraged to generate new endophenotypes.

Transdiagnostic considerations

While the focus of this review has been on ETDs associated with ASD, the endophenotype approach derives power from measuring traits that are continuously distributed in the population and agnostic to diagnostic status or categorical distinctions. This conceptual shift is critical in the context of repeated findings that the vast majority of variants associated with ASD also are associated with multiple separate psychiatric and neurodevelopmental disorders (e.g., [34, 276]). In support of this proposal, multiple studies have shown that copy number variants (CNVs) associated with ASD and other neuropsychiatric disorders (e.g., schizophrenia, ADHD) also converge on common patterns of functional and structural brain differences. Moreau et al. [277] identified signatures of functional connectivity differences associated with both 16p11.2 and 22q11.2 CNVs that both transcended clinical diagnosis (ASD, ADHD, schizophrenia) and also were seen in select cases without known associated variants. Studies of brain morphometry have shown similar overlap across different CNVs and subsets of individuals without known genetic causes for their diagnosis (i.e., idiopathic cases), suggesting that understanding of the neurodevelopmental processes affected by pathogenic CNVs may inform understanding of broader subgroups of neurodevelopmental traits [278]. Importantly, because these CNVs and associated clinical traits are present across diagnostic categories, they strongly suggest that transdiagnostic strategies will be crucial for understanding pathogenicity.

Historically, endophenotype criteria have stipulated that ETDs must be disorder-specific, but research in psychiatry has yielded strong evidence of phenotypic overlap at multiple levels of analysis. Population studies of non-clinical samples have indicated that, in many instances, self-endorsed clinical symptoms show similar levels of heterogeneity within diagnostic categories as between diagnostic categories [279]. Consistent with this hypothesis, the vast majority of (if not all) traits associated with ASD also are implicated in separate developmental conditions (e.g., atypical social gaze is seen in non-clinical populations as well as social phobia and schizophrenia; repetitive behaviors are seen in non-clinical populations, intellectual developmental disability and OCD). ETD discovery will be most powerful if focused on the full range of trait variation, including clinical and subclinical variation [280]. By measuring the full range of endophenotypic trait variation, we will have more power to identify associations with dimensional behavioral traits and neurodivergent trajectories without the constraints of categorical designations that do not appear to be directly linked to discrete biologies.

Summary and conclusions

It is our position that understanding the complex pathogenic processes associated with ASD can be accelerated by shifting away from traditional case–control designs for gene discovery and instead prioritizing transdiagnostic studies that integrate dimensional endophenotypes for participant stratification. One method for leveraging candidate endophenotypes to understand mechanisms of ASD inheritance is developing ETDs, or associated endophenotypes that cut across functional units of analysis. Similar approaches (e.g., see above descriptions of the BSNIP studies) have proven powerful for identifying more biologically homogeneous subgroups of individuals, but have not yet been leveraged to understand traits associated with neurodevelopmental disorders including ASD. The quest for ETDs involved in ASD should focus on traits that are discrete, quantitative, familial, closer to gene action than behavioral traits, and translational. The dimensional and condition-agnostic nature of ETDs may accelerate gene discovery and advance understanding of heritable, polygenic processes contributing to neurodevelopmental impairments.