Skip to main content
Download PDF
Download ePub

Inter-rater reliability of subthreshold psychotic symptoms in individuals with 22q11.2 deletion syndrome

Journal of Neurodevelopmental Disorders volume 13, Article number: 23 (2021) Cite this article

Abstract

Background

Pathways leading to psychosis in 22q11.2 deletion syndrome (22q11.2DS) have been the focus of intensive research during the last two decades. One of the common clinical risk factors for the evolution of psychosis in 22q11.2DS is the presence of positive and negative subthreshold psychotic symptoms. The gold standard for measuring subthreshold symptoms is the Structured Interview for Prodromal Syndromes (SIPS) and its accompanying Scale of Prodromal Symptoms (SOPS) ratings. Although the scale has been used by many centers studying 22q11.2DS, the inter-site reliability of the scale in this population has never been established.

Methods

In the present study, experienced clinical assessors from three large international centers studying 22q11.2DS independently rated video recordings of 18 adolescents and young adults with 22q11.2DS.

Results

The intraclass correlations coefficients (ICCs) among three raters for the SOPS total scores, as well as for the positive, negative, and disorganization subscale scores, were good-to-excellent (ICCs range 0.73–0.93). The raters were also able to reliably determine the subjects’ subthreshold syndrome status (ICC = 0.71). The reliability of individual items was good-to-excellent for all items, ranging from 0.61 for motor disturbances [G3] to 0.95 for bizarre thinking.

Conclusions

Our results show that trained clinicians can reliably screen for subthreshold psychotic symptoms in individuals with 22q11.2DS. To increase assessment reliability, we suggest specific clarifications and simplifications to the standard SIPS interview for future studies.

Background

The 22q11.2 deletion syndrome (22q11.2DS), also known as DiGeorge or velocardiofacial (VCFS) syndrome, is among the most common microdeletion genetic disorder characterized by a variety of medical, cognitive, and psychiatric manifestations [1, 2]. One of the most frequent co-occurring psychiatric disorders in 22q11.2DS is schizophrenia-like psychotic disorder [3, 4]. In comparison to a lifetime prevalence of ~ 3% in the general population [5], up to 40% of persons with 22q11.2DS develop psychotic disorders [4].

Schizophrenia is a neurodevelopmental disorder with a prodromal phase of milder symptoms, including disturbances in perception thought processes, with yet intact reality testing. These symptoms are considered “attenuated positive symptoms.” The Structured Interview for Prodromal Symptoms (SIPS) was developed to objectively quantify the attenuated psychotic symptoms that are subthreshold and present in the clinical high risk (CHR) state [6]. The conversion rate from CHR to schizophrenia is about 15–20% per year [7]. Because psychotic disorders are prevalent in 22q11.2DS, research efforts are devoted to identifying early signs and predictors of psychosis in this population, leading to the study of attenuated psychotic symptoms in 22q11.2DS [8,9,10,11,12,13,14,15,16,17].

Weisman et al. [17] analyzed data on subthreshold psychosis in 760 individuals, ages 6 to 55 years, with 22q11.2DS from 10 centers. The highest rate of subthreshold psychotic symptoms was found in adolescence and young adulthood, with about one-third of adolescents and young adults with 22q11.2DS meeting criteria for positive and negative/disorganized subthreshold symptoms. The annual conversion rate to psychotic disorder for individuals with 22q11.2DS and positive subthreshold symptoms, as reported in longitudinal studies, is between 9% and 15% [14, 18, 19]. Accumulating evidence suggests that negative subthreshold symptoms in 22q11.2DS are clinically significant, as they are associated with the severity of neurocognitive deficits [17, 19] and with the presence of anxiety disorders and attention deficit/hyperactivity disorder (ADHD) [19]. Furthermore, the severity of some negative symptoms has been shown to be a significant predictor of transition to psychosis [20].

The above-mentioned studies used the Structured Interview for Prodromal Syndromes (SIPS) and the Scale of Prodromal Symptoms (SOPS) to measure subthreshold symptoms. Semi-structured diagnostic interviews like the SIPS are the gold standard instruments for evaluating attenuated psychotic symptoms in the general population [21, 22]. Several studies have supported its inter-rater reliability and predictive validity in typical help-seeking persons [23,24,25,26]. Yet, to date, no study has established the inter-rater reliability of this commonly used assessment instrument in 22q11.2DS. Lack of reliability in using the SIPS in this population might partially explain the high variability in the rates of subthreshold symptoms of psychosis, as determined by the SIPS/SOPS, among 22q11.2DS cohorts. The rates of positive subthreshold psychotic symptoms varied among studies from 20 to 56% (see [15] Table 1). Of note are the complexities of assessing subthreshold psychotic symptoms in individuals with neurodevelopmental disorders such as 22q11.2DS, which often co-occur with cognitive disabilities and medical issues. Taken together, it seems essential to verify the inter-rater reliability of the SIPS in 22q11.2DS.

Table 1 Demographic and clinical characteristics of the study sample
Full size table

The overarching aim of the present study was to establish cross-site reliability for the SIPS interview and SOPS scoring in individuals with 22q11.2DS. For this purpose, clinical assessors from three international centers with experience both with the SIPS interview and with 22q11.2DS independently rated video recordings of adolescents and young adults with 22q11.2DS. Intraclass correlation coefficients (ICCs) among three raters were assessed.

Method

Participants

The demographic and clinical characteristics of the study sample are outlined in Table 1. Videotaped SIPS interviews of 18 individuals with 22q11.2DS were conducted among 22q11.2DS patients recruited from three established international centers—Children’s Hospital of Philadelphia (CHOP) and University of Pennsylvania (Penn), University of California Los Angeles (UCLA), and Sheba Medical Center in Tel-Aviv. The individuals selected for recording were consecutive presentations to the participating sites with good recording quality. They are representative of youth with the syndrome who can assent/consent and engage in research. All study participants were Caucasian. Inclusion criteria included genetically confirmed diagnosis of 22q11.2DS by fluorescent in situ hybridization, chromosomal microarray, and/or multiplex ligation probe amplification [27]; verbal IQ > 70; and willingness of participants (and their parents in cases of minors) to have the interview video-recorded and shared with collaborators for scoring. All participants (and their parents, if involving minors) received a full explanation about the study and provided their written informed assent/consent to participate in the study. This study was approved by the Institutional Review Boards of Sheba Medical Center, Penn, and UCLA.

Cognitive and psychiatric assessments

Cognitive assessment was performed using the Wechsler Intelligence Scale for Children, 3rd edition (WISC III) [28] for participants aged ≤ 17 years or the Wechsler Adult Intelligence Scale, 3rd edition (WAIS III) [29] for participants aged > 17.

Psychiatric evaluation for children and adolescents was carried out using a modified version of the Kiddie Schedule for Affective Disorders and Schizophrenia for School Age Children-Present and Lifetime Version (K-SAD-PL) [30]. Adults (age > 18) were also evaluated by the Structured Clinical Interview for DSM-IV Diagnoses (SCID) as previously described [17].

Assessment of subthreshold psychotic symptoms

Assessment of subthreshold psychotic symptoms was conducted using the SIPS interview and its accompanied scoring—the SOPS [31]. The SOPS is composed of 19 items, each representing 1 out of 4 domains—subscales of attenuated psychotic symptoms: positive (5 items), negative (6 items), disorganization (4 items), and general (4 items) symptoms subscales [31]). The scale specifies severity scoring accompanied by anchoring criteria for each item, ranging from 0 (absent) to 6 (severe and psychotic/extreme.

Eighteen consecutive 22q11.2DS individuals agreeing to be video-recorded were enrolled to the study. Participants were interviewed by skilled masters level and experienced interviewers who underwent training and achieved certification at their respective institution in SIPS administration as part of ongoing collaborative studies. All interviews were in English except for one patient who was recruited from Israel. That interview was translated to English (by REG) to enable the rating by the US clinicians. The interviews’ video recordings were uploaded to a secure website and were downloaded and scored, and these scorings were used for calculating the inter-rater reliabilities. The video recordings were accompanied by a short clinical summary, which included brief participant background information—age, education, brief medical history, living situation, and cognitive functioning. Each recording was independently rated by three senior doctoral level clinicians, one from each center, who were not familiar with the SIPS ratings of the participants. These raters are experienced in standardized psychiatric assessment of individuals with 22q11.2DS—a child psychiatrist (DG) and two clinical psychologists (CEB and MEC). SOPS scorings of each interview were entered in the REDCap database independently by each faculty rater. Importantly, the rater was not familiar with the rated participants prior to rating the interview. Monthly telephone meetings were held with the participation of the three assessors and REG. At all telephone meetings the independent scorings of each item were discussed. Items without consensus in scoring were discussed to reach a consensus, which was determined to serve as the gold standard scoring in future usage of the videos for training purposes.

Based on each rater’s independent scores of the individual SOPS items, and in line with earlier work [25], summary measures were derived for further analyses: (1) total SOPS score was calculated by summing the 19 items ratings; (2) total score for each SOPS subscale was calculated by summing relevant item scores per subscale (positive, negative, disorganization, and general).

The presence of subthreshold psychotic symptoms was determined by summing up item ratings of positive, negative, and disorganization subscales. Four categories of subthreshold symptoms were calculated as previously described [17, 32]: (1) positive subthreshold symptoms—at least one positive symptom rated 3–5 (3, moderate; 4, moderately severe; 5, severe but not psychotic); (2) negative/disorganized subthreshold symptoms—at least two or more negative/disorganized symptoms rated 3–6, in the absence of positive symptoms; (3) positive + negative/disorganized subthreshold symptoms—at least one positive symptom rated 3–5 and at least two negative/disorganized symptoms rated 3–6; and (4) Acute positive subthreshold symptoms—at least one positive symptom with a rating of 6.

Statistical analyses

All analyses were conducted using SPSS and the irrNA package [33] in R [34]. SOPS’ endpoint measures were analyzed for inter-rater reliability by Intra class Correlations Coefficients (ICCs) [two-way random model; single measurements form; absolute agreement type]. ICCs were estimated using the Brueckl [35] method, which handles missing data without imputation or data loss. Reliability levels were interpreted according to the following ICC guidelines [35, 36] representing different levels of agreement among judges: < 0.4—low; 0.40–0.59—fair; 0.60–0.74—good; and ≥ 0.75—excellent. ICCs were calculated on a max of 14 subjects (100% valid cases). Finally, to explore the extent to which our results are influenced by the relatively low symptom levels in this sample, the above analyses were repeated after splitting the sample into clinical high risk (CHR; many symptoms) and non-CHR (few or no symptoms).

Results

Distribution of subthreshold psychotic symptoms based on consensus scores

SOPS’ gold standard ratings for the 18 interviews were used to analyze the prevalence rates of subthreshold psychotic symptoms in the current sample. Thirty-nine percent of the sample (n = 7) had no subthreshold psychotic symptoms, 33% had negative/disorganized symptoms (n = 6), 17% (n = 3) positive and positive + negative/disorganized subthreshold symptoms were present in two subjects (11%).

Inter-site reliability of the SOPS and its subscales

ICCs of the inter-rater reliability analyses conducted on the total scores of the SOPS and its four subscales are shown in Table 2. Results revealed excellent agreement among raters with respect to the overall SOPS and all subscales [ICCs: 0.73–0.96].

Table 2 Interclass correlation coefficients (ICCs) of the total SOPS scale and subscales
Full size table

Inter-site reliability of the SOPS’ individual items

The inter-rater agreement for each item of the SOPS is shown in Table 3. For 17 out of the 19 SOPS items, there was excellent agreement among raters [ICCs 0.76–0.95]. Two items had good agreement [ICCs: 0.61–0.65]. For finer-grained comparisons, Supplementary Table S2 shows the mean, minimum, and maximum rating of each rater for each item. Note that several items had less than 100% valid data. The interviews were conducted with complex patients that commonly manifest short attention span and restlessness and therefore for a few patients the last questions of the SIPS interview had to be shortened. In these cases, the raters decided that the information in the video is insufficient to provide a valid score and these items were left as “missing.” Furthermore, it is important to note that for the “positive,” “negative”, and “disorganized” items the percent of valid cases was between 93 and 100%. The lower rates were for the “general” items, which are of minor importance in the SIPS interview as they are not part of the definition of clinical high risk state.

Table 3 Intraclass correlation coefficients (ICCs) of the SOPS’ 19 individual items
Full size table

Inter-site reliability of the SOPS in assessing subthreshold symptoms status

Finally, to further test whether the raters were able to distinguish reliably between subthreshold symptoms and non-clinically significant symptoms, ICC was computed on the combined total scores of three SOPS subscales: positive, negative, and disorganized symptoms. As seen in Fig. 1, agreement among raters was found to be in the good range for determining negative/disorganized subthreshold symptoms, positive subthreshold symptoms, positive and negative/disorganized subthreshold symptoms, acute positive subthreshold symptoms, or as not psychosis-prone ICC = 0.71 [95% CI 0.47–0.89].

Fig. 1
figure 1

Inter-rater agreement between three sites regarding the subthreshold symptoms status of each subject. Npro: non-prodromal; Neg/Dis: negative/disorganized subthreshold symptoms; Pos: positive subthreshold symptoms; Pos + Neg/Dis: positive and negative/disorganized subthreshold symptoms; AcPos: acute positive subthreshold symptoms. Note that a 1-point difference in one item can change the status of the patient from one category to the other. In case of subject#9, there was a 1-point difference in the score of one positive symptom of the SIPS—one rater gave a score of 6 “acute positive” and the other two gave a score of ≤ 5 within the positive (subthreshold) range; and 1-point difference in the negative/disorganized symptoms—one rater gave a score of ≥ 3 for only one negative/disorganized symptom and two raters had a score of ≥ 3 only for one negative/disorganized symptom

Full size image

Comparison of inter-site reliability in CHR and non-CHR

Supplementary Table S1 shows the same results as in Table 3 but split by CHR status (11 CHR compared to 7 non-CHR). Of the 14 items comparable across samples, six showed higher ICCs in the non-CHR sample. Item N3 showed the largest difference (0.33 in non-CHR, compared to 0.93 in CHR), while item G2 showed the largest difference in the opposite direction (0.92 in non-CHR, compared to 0.75 in CHR).

Discussion

To our knowledge, this is the first study to test SIPS/SOPS inter-rater reliability in 22q11.2DS. The few SIPS reliability studies have been conducted only by the developers of the tool and only in US clinical high risk, help-seeking individuals, without a neurogenetic syndrome [25, 26]. Thus, our study is also the first inter-rater study of individuals with developmental disabilities. To date, there is no reliability study assessing subthreshold psychotic symptoms not only for 22q11.2DS but also for any neurogenetic syndrome. In this study, international experts from three large academic centers studying 22q11.2DS were able to obtain overall good inter-rater reliability. These efforts can advance the development of guidelines and gold standard assessment training videos, which can be useful for other researchers using the SIPS in evaluating 22q11.2DS individuals. Our results and the accompanying guidelines are relevant also for researchers studying individuals with other genetic syndromes associated with increased risk for psychosis such as Prader-Willi syndrome and 16p11.2 duplication syndrome [36]. Establishing reliability for measuring subthreshold symptoms can benefit treatment studies since the tool can be used to obtain outcome measures to determine efficacy of intervention. Importantly, information on reliability will help contextualize studies in neurogenetic populations with those carried out in community samples.

We found excellent agreement among raters (ICCs > 0.75) for total SOPS’ scores and for all subscales and good reliability (ICC = 0.71, Fig. 1) in distinguishing between the various subthreshold syndromes. These results are in line with those demonstrated by Miller et al. [25] in clinical high-risk, help-seeking non-22q11.2DS population. While our sample was relatively small it was almost 30% larger (n = 18) than that of Miller et al. [25] (n = 14). Inter-rater reliability was at least in the “good” range for all SOPS items, where 17 of those 19 were in the “excellent” range. Of note, ICC values are inherently sensitive to the level of variability among participants’ scores (i.e., higher between-subjects variability improves inter-rater reliability coefficients) [37].

A limitation of our study was that we did not employ the traditional two-step process used in some reliability studies. At the first step, similarly to what we did in our study, after raters score interviews independently, they discuss differences in ratings to improve consensus and reach a set of agreed criteria. In the second step, not done in our study, these criteria are used by a new, other group of raters that score interviews without any discussion, thereby avoiding self-biases. Thus, additional study is recommended with a new sample of 22q11.2DS assessments and other raters.

Our data were acquired from three sites, but 11 out of the 18 subjects were from one site (UCLA). Each research group likely has some site-specific practices in its method of interviewing individuals with 22q11.DS, which can in turn influence the scoring. Thus, the over-representation of subjects from one site might somewhat bias our results. Future studies should expand the number of participants and include participants that are well-distributed across a variety of sites.

Notably, in this study IQ > 70 was an inclusion criterion and the conclusions may not extend to individuals with cognitive disability. Based on our experience, we have suggestions for administering the SIPS and assessing symptoms in individuals with 22q11.2DS, consisting of (1) clarification/simplification of interview questions, (2) elaboration of qualifiers with examples, and (3) use of numerical scales to assist the participant in rating his/her symptoms.

For example, in assessing grandiose ideas (P3), the interview includes the questions “Have you ever behaved without regard to painful consequences? For example, do you ever go on excessive spending sprees?” We recommend stating the question as it is written, but following the question up with clarifications / simplifications of language consisting of “In other words, have you ever behaved without thinking about the bad things that could happen because of what you are doing? For example, have you ever spent a lot of money on things you can’t afford?”

In assessing suspiciousness/persecutory ideas, the interview begins with the question: “Do you ever feel that people around you are thinking about you in a negative way” and follows the question up with qualifiers that assess onset-duration, degree of distress, degree of interference with life, and degree of conviction/meaning. For individuals with cognitive disabilities, we recommend that the interviewer elaborate on and concretize these qualifiers and provide numerical scales for some of the probes. For example, in probing for when symptoms began, we suggest providing a specific timepoint or life event anchor that would be relevant to the participant (e.g., “Did this start before or after you began middle school?”). In probing for degree of distress, we suggest that the interviewer follow up the interview question “What is this experience like for you? Does it bother you” with “On a scale from 0 to 10, with 10 being the most upsetting; how upsetting would you rate this?” Visual representations of such a scale can also be used, which permit the participant to point to the rating with which s/he agrees. We have modified the SIPS to reflect these and other guidelines in an effort to optimize its administration for individuals with cognitive disability. (Our full protocol of suggested modifications for individuals with cognitive disability can be obtained from the authors).

Conclusions

To summarize, given the high rates of subthreshold psychotic symptoms in 22q11.2DS individuals, it is of utmost importance to validate the SIPS in the 22q11.2DS population. Our preliminary results with three international assessors reviewing SIPS video assessments of 18 22q11.2DS individuals suggest that good inter-reliability can be achieved for most SOPS items.

Availability of data and materials

The datasets used and/or analyzed for the present paper can be made available upon a reasonable request to the corresponding author.

Abbreviations

22q11.2DS:

22q11.2 deletion syndrome

ADHD:

Attention deficit/hyperactivity disorder

ASD:

Autism spectrum disorder

CHOP:

Children’s Hospital of Philadelphia

FSIQ:

Full-Scale IQ

GAF:

Global Assessment of Functioning

ICCs:

Intraclass correlations coefficients

K-SAD-PL:

The Kiddie Schedule for Affective Disorders and Schizophrenia for School Age Children-Present and Lifetime Version

SCID:

The Structured Clinical Interview for DSM-IV Diagnoses

SIPS:

Structured Interview for Prodromal Syndromes

SNRI:

Selective norepinephrine reuptake inhibitor

SOPS:

Scale of Prodromal Symptoms

SSRI:

Selective serotonin reuptake inhibitor

UCLA:

University of California Los Angeles

VCFS:

Velocardiofacial syndrome

WAIS III:

Wechsler Adult Intelligence Scale, 3rd edition

WISC III:

Wechsler Intelligence Scale for Children, 3rd edition

References

  1. Shprintzen R. Velo-cardio-facial syndrome: a distinctive behavioral phenotype. Ment. Retard. Dev. Disabil. Res. Rev. 2000;6:142–7. https://doi.org/10.1002/1098-2779(2000)6:2%3C142::AID-MRDD9%3E3.0.CO.

    Article  CAS  PubMed  Google Scholar 

  2. Swillen A, Vogels A, Devriendt K, Fryns JP. Chromosome 22q11 deletion syndrome: update and review of the clinical features, cognitive-behavioral spectrum, and psychiatric complications. Am. J. Med. Genet. 2000;97:128–35. https://doi.org/10.1002/1096-8628(200022)97:2%3C128::AID-AJMG4%3E3.0.CO.

    Article  CAS  PubMed  Google Scholar 

  3. Green T, Gothelf D, Glaser B, Debbane M, Frisch A, Kotler M, et al. Psychiatric disorders and intellectual functioning throughout development in velocardiofacial (22q11.2 deletion) syndrome. J. Am. Acad. Child Adolesc. Psychiatry. 2009;48(11):1060–8 Available from: https://linkinghub.elsevier.com/retrieve/pii/S0890856709602537.

    Article  Google Scholar 

  4. Schneider M, Debbané M, Bassett AS, Chow EWC, Fung WLA, van den Bree MBM, et al. Psychiatric disorders from childhood to adulthood in 22q11.2 deletion syndrome: results from the international consortium on brain and behavior in 22q11.2 deletion syndrome. Am. J. Psychiatry. 2014;171(6):627–39. https://doi.org/10.1176/appi.ajp.2013.13070864.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Perälä J, Suvisaari J, Saarni S, Kuoppasalmi K. Lifetime prevalence of psychotic and bipolar I disorders in general population. Arch. Gen. Psychiatry. 2007;64(1):19–28 Available from: https://jamanetwork.com/journals/jamapsychiatry/fullarticle/209973. https://doi.org/10.1001/archpsyc.64.1.19.

    Article  PubMed  Google Scholar 

  6. Yung AR, Yuen HP, Berger G, Francey S, Hung TC, Nelson B, et al. Declining transition rate in ultra high risk (prodromal) services: dilution or reduction of risk? Schizophr Bull. 2007;33:673–81 8.

    Article  Google Scholar 

  7. Ruhrmann S, Schultze-Lutter F, Klosterkötter J. Probably at-risk, but certainly ill: advocating the introduction of a psychosis spectrum disorder in DSM-V. Schizophr Res. 2010;120(1-3):23–37. https://doi.org/10.1016/j.schres.2010.03.015.

    Article  PubMed  Google Scholar 

  8. Antshel KM, Fremont W, Ramanathan S, Kates WR. Predicting cognition and psychosis in young adults with 22q11.2 deletion syndrome. Schizophr. Bull. 2016;43(4):833–42 Available from: https://academic.oup.com/schizophreniabulletin/article/43/4/833/2327380.

    Google Scholar 

  9. Armando M, Girardi P, Vicari S, Menghini D, Digilio MC, Pontillo M, et al. Adolescents at ultra-high risk for psychosis with and without 22q11 deletion syndrome: a comparison of prodromal psychotic symptoms and general functioning. Schizophr. Res. 2012;139:151–6. https://doi.org/10.1016/j.schres.2012.04.020.

    Article  PubMed  Google Scholar 

  10. Kates WR, Mariano MA, Antshel KM, Chandra S, Gamble H, Giordano M, et al. Trajectories of psychiatric diagnoses and medication usage in youth with 22q11.2 deletion syndrome: a 9-year longitudinal study. Psychol. Med. 2018;49(11):1914–22. https://doi.org/10.1017/S0033291718002696.

    Article  PubMed  Google Scholar 

  11. Schneider M, Schaer M, Mutlu AK, Menghetti S, Glaser B, Debbané M, et al. Clinical and cognitive risk factors for psychotic symptoms in 22q11.2 deletion syndrome: a transversal and longitudinal approach. Eur. Child Adolesc. Psychiatry. 2014;23(6):425–36. https://doi.org/10.1007/s00787-013-0469-8.

    Article  PubMed  Google Scholar 

  12. Shapiro DI, Cubells JF, Ousley OY, Rockers K, Walker EF. Prodromal symptoms in adolescents with 22q11.2 deletion syndrome and schizotypal personality disorder. Schizophr. Res. 2011;129(1):20–8. https://doi.org/10.1016/j.schres.2011.03.030.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Stoddard J, Niendam T, Hendren R, Carter C, Simon TJ. Attenuated positive symptoms of psychosis in adolescents with chromosome 22q11.2 deletion syndrome. Schizophr. Res. 2010;(1-3):118, 118–21. https://doi.org/10.1016/j.schres.2009.12.011.

  14. Tang SX, Moore TM, Calkins ME, Yi JJ, McDonald-McGinn DM, Zackai EH, et al. Emergent, remitted and persistent psychosis-spectrum symptoms in 22q11.2 deletion syndrome. Transl. Psychiatry. 2017;7(7):e1180. https://doi.org/10.1038/tp.2017.157.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Mekori-Domachevsky E, Guri Y, Yi J, Weisman O, Calkins ME, Tang SX, et al. Negative subthreshold psychotic symptoms distinguish 22q11.2 deletion syndrome from other neurodevelopmental disorders: a two-site study. Schizophr. Res. 2017;188:42–9. https://doi.org/10.1016/j.schres.2016.12.023.

    Article  PubMed  Google Scholar 

  16. Tang SX, Moore TM, Calkins ME, Yi JJ, Savitt A, Kohler CG, et al. The psychosis spectrum in 22q11.2 deletion syndrome is comparable to that of nondeleted youths. Biol. Psychiatry. 2017;82(1):17–25. https://doi.org/10.1016/j.biopsych.2016.08.034.

    Article  CAS  PubMed  Google Scholar 

  17. Weisman O, Guri Y, Gur RE, McDonald-McGinn DM, Calkins ME, Tang SX, et al. Subthreshold psychosis in 22q11.2 deletion syndrome: multisite naturalistic study. Schizophr. Bull. 2017;43(5):1079–89. https://doi.org/10.1093/schbul/sbx005.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Schneider M, Armando M, Pontillo M, Vicari S, Debbané M, Schultze-Lutter F, et al. Ultra high risk status and transition to psychosis in 22q11.2 deletion syndrome. World Psychiatry. 2016;15(3):259–65 Available form: https://onlinelibrary.wiley.com/doi/pdf/10.1002/wps.20347.

    Article  Google Scholar 

  19. Weinberger R, Weisman O, Guri Y, Harel T, Weizman A, Gothelf D. The interaction between neurocognitive functioning, subthreshold psychotic symptoms and pharmacotherapy in 22q11.2 deletion syndrome: a longitudinal comparative study. Eur. Psychiatry. 2018;48(1):20–6. https://doi.org/10.1016/j.eurpsy.2017.10.010.

    Article  CAS  PubMed  Google Scholar 

  20. Schneider M, Armando M, Schultze-Lutter F, Pontillo M, Vicari S, Debbané M, et al. Prevalence, course and psychosis-predictive value of negative symptoms in 22q11.2 deletion syndrome. Schizophr Res. 2019;206:386–93. https://doi.org/10.1016/j.schres.2018.10.014.

    Article  PubMed  Google Scholar 

  21. McGlashan TH, Miller TJ, Woods SW, Hoffman RE, Davidson L. Instrument for the assessment of prodromal symptoms and states BT-early intervention in psychotic disorders. In: Miller T, Mednick SA, McGlashan TH, Libiger J, Johannessen JO, editors. . Dordrecht: Springer Netherlands; 2001. p. 135–49. https://doi.org/10.1007/978-94-010-0892-1_7.

    Chapter  Google Scholar 

  22. Miller TJ, McGlashan TH, Woods SW, Stein K, Driesen N, Corcoran CM, et al. Symptom assessment in schizophrenic prodromal states. Psychiatr. 1999;70:273–87 Available form: https://link.springer.com/content/pdf/10.1023/A:1022034115078.pdf.

    CAS  Google Scholar 

  23. Comparelli A, Savoja V, Kotzalidis GD, Woods SW, Mosticoni S, Vassallo F, et al. Factor-structure of the Italian version of the Scale of Prodromal Symptoms (SOPS): a comparison with the English version. Epidemiol. Psychiatr. Sci. 2011;20(1):45–54. https://doi.org/10.1017/S2045796011000114.

    Article  CAS  PubMed  Google Scholar 

  24. Fusar-Poli P, Cappucciati M, Rutigliano G, Lee TY, Beverly Q, Bonoldi I, et al. Towards a standard psychometric diagnostic interview for subjects at ultra high risk of psychosis: CAARMS versus SIPS. Psychiatry J. 2016:1–11. https://doi.org/10.1155/2016/7146341.

  25. Miller TJ, McGlashan TH, Rosen JL, Cadenhead K, Ventura J, McFarlane W. Prodromal assessment with the structured interview for prodromal syndromes and the scale of prodromal symptoms. Schizophr. Bull. 2003;29(4):703–15 Available form: https://academic.oup.com/schizophreniabulletin/article/29/4/703/1887836.

    Article  Google Scholar 

  26. Miller TJ, McGlashan TH, Rosen JL, Somjee L, Markovich PJ, Stein K, et al. Prospective diagnosis of the initial prodrome for schizophrenia based on the structured interview for prodromal syndromes: preliminary evidence of interrater reliability and predictive validity. Am. J. Psychiatry. 2002;159:863–5. https://doi.org/10.1176/appi.ajp.159.5.863.

    Article  PubMed  Google Scholar 

  27. Michaelovsky E, Frisch A, Carmel M, Patya M, Zarchi O, Green T, Basel-Vanagaite L, Weizman A, Gothelf D.Genotype-phenotype correlation in 22q11.2 deletion syndrome. BMC Med. Genet. 2012;13(1):122 https://doi.org/10.1186/1471-2350-13-122

  28. Wechsler D. Wechsler Intelligence Scale for Children. 3rd ed. San Antonio, TX: Psychological Corporation; 1991.

    Google Scholar 

  29. Wechsler D. WAIS-III Administration and Scoring Manual. San Antonio, TX: Psychol Corp; 1997.

    Google Scholar 

  30. Kaufman J, Birmaher B, Brent D, Rao UMA, Flynn C, Moreci P, et al. Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (K-SADS-PL): Initial Reliability and Validity Data. J. Am. Acad. Child Adolesc.Psychiatry. 1997;36:980–8. https://doi.org/10.1097/00004583-199707000-00021.

    Article  CAS  PubMed  Google Scholar 

  31. McGlashan TH, Woods S. Walsh B. The psychosis-risk syndrome: Handbook for Diagnosis and Follow-Up. Oxford University Press, New York; 2010.

  32. Tang SX, Yi JJ, Moore TM, Calkins ME, Kohler CG, Whinna DA, et al. Subthreshold psychotic symptoms in 22q11.2 deletion syndrome. J. Am. Acad. Child Adolesc. Psychiatry. 2014;53(9):991–1000. https://doi.org/10.1016/j.jaac.2014.05.009.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Brueckl M, Heuer F.irrNA: coefficients of interrater reliability - generalized for randomly incomplete datasets. R package version 0.1.4; 2018. Available form: https://CRAN.R-project.org/package=irrNA

  34. R Core Team R. A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria; 2018. Available form: https://www.R-project.org/.

  35. Brueckl M. Statistische Verfahren zur Ermittlung der Urteileruebereinstimmung. In: Altersbedingte Veraenderungen der Stimme und Sprechweise von Frauen. Berlin: Logos; 2011. p. 88–103.

    Google Scholar 

  36. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol. Assess. 1994;6:284–90. https://doi.org/10.1037/1040-3590.6.4.284.

    Article  Google Scholar 

  37. Portney LG, Watkins MP. Foundations of clinical research : applications to practice, 3rd editio. ed. Upper Saddle River, N.J: Pearson/Prentice Hall; 2015.

Download references

Acknowledgements

The authors thank the research teams in the institutions collaborating on this project and the research participants and their families.

Funding

The study reported in this publication was supported by the Binational Science Foundation, grant number 2017369; The National Institute of Mental Health of the National Institutes of Health grants U01MH101722, U01MH101719, U01MH119738, and RO1 MH085953 (CEB). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Author notes

  1. Tyler M. Moore, Deby Salzer and Doron Gothelf contributed equally to this work.

Authors and Affiliations

  1. Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA

    Tyler M. Moore, Monica E. Calkins, Robert Sean Gallagher & Raquel E. Gur

  2. Lifespan Brain Institute of Penn Medicine and the Children’s Hospital of Philadelphia (CHOP), Philadelphia, USA

    Tyler M. Moore, Monica E. Calkins, Robert Sean Gallagher & Raquel E. Gur

  3. The Behavioral Neurogenetics Center, Edmond and Lily Safra Children’s Hospital, Sheba Medical Center, Tel Hashomer, Israel

    Deby Salzer, Dafna Sofrin Frumer, Ronnie Weinberger & Doron Gothelf

  4. Semel Institute for Neuroscience and Human Behavior, University of California, Los Angeles, USA

    Carrie E. Bearden

  5. Department of Psychiatry and Behavioral Sciences, State University of New York at Upstate Medical University, Syracuse, NY, USA

    Wendy R. Kates

  6. 22q and You Center, CHOP, Philadelphia, USA

    Leila Kushan & Donna M. McDonald-McGinn

  7. Department of Child and Adolescent Psychiatry and Behavioral Sciences, CHOP, Philadelphia, USA

    Raquel E. Gur

  8. Sackler Faculty of Medicine and the Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel

    Doron Gothelf

Authors
  1. Tyler M. Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deby Salzer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carrie E. Bearden
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Monica E. Calkins
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wendy R. Kates
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Leila Kushan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Robert Sean Gallagher
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Dafna Sofrin Frumer
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ronnie Weinberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Donna M. McDonald-McGinn
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Raquel E. Gur
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Doron Gothelf
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TMM, DS, CEB, MEC, WRK, LK, RSG, DSF, RW, DMM, REG, and DG were all responsible for the initial conceptualization of the study. DG, CEB, WRK, CMM, and REG oversaw data collection. DS, TMM, LK, RSG, DSF, and RW participated in the data collection. DG, CEB, and MEC rated the standardized psychiatric assessment. DS, TMM, REG, and DG were responsible for data management and the statistical analyses. All authors participated in writing and editing the manuscript. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Doron Gothelf.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Institutional Review Board at each of the three participating universities of the authors. Informed written consent was obtained from either the participant or the parent/legal guardian prior to participation and assent was obtained from each participant as appropriate. The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and international committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file1: Table S1.

Intraclass Correlation Coefficients (ICCs) of the SOPS’ 19 individual items. Table S2. Mean Itemwise and Overall Mean SOPS Ratings, by Rater.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moore, T.M., Salzer, D., Bearden, C.E. et al. Inter-rater reliability of subthreshold psychotic symptoms in individuals with 22q11.2 deletion syndrome. J Neurodevelop Disord 13, 23 (2021). https://doi.org/10.1186/s11689-021-09372-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s11689-021-09372-3

Keywords

Journal of Neurodevelopmental Disorders

ISSN: 1866-1955

Contact us