Inter-rater reliability of subthreshold psychotic symptoms in individuals with 22q11.2 deletion syndrome
Journal of Neurodevelopmental Disorders volume 13, Article number: 23 (2021)
Pathways leading to psychosis in 22q11.2 deletion syndrome (22q11.2DS) have been the focus of intensive research during the last two decades. One of the common clinical risk factors for the evolution of psychosis in 22q11.2DS is the presence of positive and negative subthreshold psychotic symptoms. The gold standard for measuring subthreshold symptoms is the Structured Interview for Prodromal Syndromes (SIPS) and its accompanying Scale of Prodromal Symptoms (SOPS) ratings. Although the scale has been used by many centers studying 22q11.2DS, the inter-site reliability of the scale in this population has never been established.
In the present study, experienced clinical assessors from three large international centers studying 22q11.2DS independently rated video recordings of 18 adolescents and young adults with 22q11.2DS.
The intraclass correlations coefficients (ICCs) among three raters for the SOPS total scores, as well as for the positive, negative, and disorganization subscale scores, were good-to-excellent (ICCs range 0.73–0.93). The raters were also able to reliably determine the subjects’ subthreshold syndrome status (ICC = 0.71). The reliability of individual items was good-to-excellent for all items, ranging from 0.61 for motor disturbances [G3] to 0.95 for bizarre thinking.
Our results show that trained clinicians can reliably screen for subthreshold psychotic symptoms in individuals with 22q11.2DS. To increase assessment reliability, we suggest specific clarifications and simplifications to the standard SIPS interview for future studies.
The 22q11.2 deletion syndrome (22q11.2DS), also known as DiGeorge or velocardiofacial (VCFS) syndrome, is among the most common microdeletion genetic disorder characterized by a variety of medical, cognitive, and psychiatric manifestations [1, 2]. One of the most frequent co-occurring psychiatric disorders in 22q11.2DS is schizophrenia-like psychotic disorder [3, 4]. In comparison to a lifetime prevalence of ~ 3% in the general population , up to 40% of persons with 22q11.2DS develop psychotic disorders .
Schizophrenia is a neurodevelopmental disorder with a prodromal phase of milder symptoms, including disturbances in perception thought processes, with yet intact reality testing. These symptoms are considered “attenuated positive symptoms.” The Structured Interview for Prodromal Symptoms (SIPS) was developed to objectively quantify the attenuated psychotic symptoms that are subthreshold and present in the clinical high risk (CHR) state . The conversion rate from CHR to schizophrenia is about 15–20% per year . Because psychotic disorders are prevalent in 22q11.2DS, research efforts are devoted to identifying early signs and predictors of psychosis in this population, leading to the study of attenuated psychotic symptoms in 22q11.2DS [8,9,10,11,12,13,14,15,16,17].
Weisman et al.  analyzed data on subthreshold psychosis in 760 individuals, ages 6 to 55 years, with 22q11.2DS from 10 centers. The highest rate of subthreshold psychotic symptoms was found in adolescence and young adulthood, with about one-third of adolescents and young adults with 22q11.2DS meeting criteria for positive and negative/disorganized subthreshold symptoms. The annual conversion rate to psychotic disorder for individuals with 22q11.2DS and positive subthreshold symptoms, as reported in longitudinal studies, is between 9% and 15% [14, 18, 19]. Accumulating evidence suggests that negative subthreshold symptoms in 22q11.2DS are clinically significant, as they are associated with the severity of neurocognitive deficits [17, 19] and with the presence of anxiety disorders and attention deficit/hyperactivity disorder (ADHD) . Furthermore, the severity of some negative symptoms has been shown to be a significant predictor of transition to psychosis .
The above-mentioned studies used the Structured Interview for Prodromal Syndromes (SIPS) and the Scale of Prodromal Symptoms (SOPS) to measure subthreshold symptoms. Semi-structured diagnostic interviews like the SIPS are the gold standard instruments for evaluating attenuated psychotic symptoms in the general population [21, 22]. Several studies have supported its inter-rater reliability and predictive validity in typical help-seeking persons [23,24,25,26]. Yet, to date, no study has established the inter-rater reliability of this commonly used assessment instrument in 22q11.2DS. Lack of reliability in using the SIPS in this population might partially explain the high variability in the rates of subthreshold symptoms of psychosis, as determined by the SIPS/SOPS, among 22q11.2DS cohorts. The rates of positive subthreshold psychotic symptoms varied among studies from 20 to 56% (see  Table 1). Of note are the complexities of assessing subthreshold psychotic symptoms in individuals with neurodevelopmental disorders such as 22q11.2DS, which often co-occur with cognitive disabilities and medical issues. Taken together, it seems essential to verify the inter-rater reliability of the SIPS in 22q11.2DS.
The overarching aim of the present study was to establish cross-site reliability for the SIPS interview and SOPS scoring in individuals with 22q11.2DS. For this purpose, clinical assessors from three international centers with experience both with the SIPS interview and with 22q11.2DS independently rated video recordings of adolescents and young adults with 22q11.2DS. Intraclass correlation coefficients (ICCs) among three raters were assessed.
The demographic and clinical characteristics of the study sample are outlined in Table 1. Videotaped SIPS interviews of 18 individuals with 22q11.2DS were conducted among 22q11.2DS patients recruited from three established international centers—Children’s Hospital of Philadelphia (CHOP) and University of Pennsylvania (Penn), University of California Los Angeles (UCLA), and Sheba Medical Center in Tel-Aviv. The individuals selected for recording were consecutive presentations to the participating sites with good recording quality. They are representative of youth with the syndrome who can assent/consent and engage in research. All study participants were Caucasian. Inclusion criteria included genetically confirmed diagnosis of 22q11.2DS by fluorescent in situ hybridization, chromosomal microarray, and/or multiplex ligation probe amplification ; verbal IQ > 70; and willingness of participants (and their parents in cases of minors) to have the interview video-recorded and shared with collaborators for scoring. All participants (and their parents, if involving minors) received a full explanation about the study and provided their written informed assent/consent to participate in the study. This study was approved by the Institutional Review Boards of Sheba Medical Center, Penn, and UCLA.
Cognitive and psychiatric assessments
Cognitive assessment was performed using the Wechsler Intelligence Scale for Children, 3rd edition (WISC III)  for participants aged ≤ 17 years or the Wechsler Adult Intelligence Scale, 3rd edition (WAIS III)  for participants aged > 17.
Psychiatric evaluation for children and adolescents was carried out using a modified version of the Kiddie Schedule for Affective Disorders and Schizophrenia for School Age Children-Present and Lifetime Version (K-SAD-PL) . Adults (age > 18) were also evaluated by the Structured Clinical Interview for DSM-IV Diagnoses (SCID) as previously described .
Assessment of subthreshold psychotic symptoms
Assessment of subthreshold psychotic symptoms was conducted using the SIPS interview and its accompanied scoring—the SOPS . The SOPS is composed of 19 items, each representing 1 out of 4 domains—subscales of attenuated psychotic symptoms: positive (5 items), negative (6 items), disorganization (4 items), and general (4 items) symptoms subscales ). The scale specifies severity scoring accompanied by anchoring criteria for each item, ranging from 0 (absent) to 6 (severe and psychotic/extreme.
Eighteen consecutive 22q11.2DS individuals agreeing to be video-recorded were enrolled to the study. Participants were interviewed by skilled masters level and experienced interviewers who underwent training and achieved certification at their respective institution in SIPS administration as part of ongoing collaborative studies. All interviews were in English except for one patient who was recruited from Israel. That interview was translated to English (by REG) to enable the rating by the US clinicians. The interviews’ video recordings were uploaded to a secure website and were downloaded and scored, and these scorings were used for calculating the inter-rater reliabilities. The video recordings were accompanied by a short clinical summary, which included brief participant background information—age, education, brief medical history, living situation, and cognitive functioning. Each recording was independently rated by three senior doctoral level clinicians, one from each center, who were not familiar with the SIPS ratings of the participants. These raters are experienced in standardized psychiatric assessment of individuals with 22q11.2DS—a child psychiatrist (DG) and two clinical psychologists (CEB and MEC). SOPS scorings of each interview were entered in the REDCap database independently by each faculty rater. Importantly, the rater was not familiar with the rated participants prior to rating the interview. Monthly telephone meetings were held with the participation of the three assessors and REG. At all telephone meetings the independent scorings of each item were discussed. Items without consensus in scoring were discussed to reach a consensus, which was determined to serve as the gold standard scoring in future usage of the videos for training purposes.
Based on each rater’s independent scores of the individual SOPS items, and in line with earlier work , summary measures were derived for further analyses: (1) total SOPS score was calculated by summing the 19 items ratings; (2) total score for each SOPS subscale was calculated by summing relevant item scores per subscale (positive, negative, disorganization, and general).
The presence of subthreshold psychotic symptoms was determined by summing up item ratings of positive, negative, and disorganization subscales. Four categories of subthreshold symptoms were calculated as previously described [17, 32]: (1) positive subthreshold symptoms—at least one positive symptom rated 3–5 (3, moderate; 4, moderately severe; 5, severe but not psychotic); (2) negative/disorganized subthreshold symptoms—at least two or more negative/disorganized symptoms rated 3–6, in the absence of positive symptoms; (3) positive + negative/disorganized subthreshold symptoms—at least one positive symptom rated 3–5 and at least two negative/disorganized symptoms rated 3–6; and (4) Acute positive subthreshold symptoms—at least one positive symptom with a rating of 6.
All analyses were conducted using SPSS and the irrNA package  in R . SOPS’ endpoint measures were analyzed for inter-rater reliability by Intra class Correlations Coefficients (ICCs) [two-way random model; single measurements form; absolute agreement type]. ICCs were estimated using the Brueckl  method, which handles missing data without imputation or data loss. Reliability levels were interpreted according to the following ICC guidelines [35, 36] representing different levels of agreement among judges: < 0.4—low; 0.40–0.59—fair; 0.60–0.74—good; and ≥ 0.75—excellent. ICCs were calculated on a max of 14 subjects (100% valid cases). Finally, to explore the extent to which our results are influenced by the relatively low symptom levels in this sample, the above analyses were repeated after splitting the sample into clinical high risk (CHR; many symptoms) and non-CHR (few or no symptoms).
Distribution of subthreshold psychotic symptoms based on consensus scores
SOPS’ gold standard ratings for the 18 interviews were used to analyze the prevalence rates of subthreshold psychotic symptoms in the current sample. Thirty-nine percent of the sample (n = 7) had no subthreshold psychotic symptoms, 33% had negative/disorganized symptoms (n = 6), 17% (n = 3) positive and positive + negative/disorganized subthreshold symptoms were present in two subjects (11%).
Inter-site reliability of the SOPS and its subscales
ICCs of the inter-rater reliability analyses conducted on the total scores of the SOPS and its four subscales are shown in Table 2. Results revealed excellent agreement among raters with respect to the overall SOPS and all subscales [ICCs: 0.73–0.96].
Inter-site reliability of the SOPS’ individual items
The inter-rater agreement for each item of the SOPS is shown in Table 3. For 17 out of the 19 SOPS items, there was excellent agreement among raters [ICCs 0.76–0.95]. Two items had good agreement [ICCs: 0.61–0.65]. For finer-grained comparisons, Supplementary Table S2 shows the mean, minimum, and maximum rating of each rater for each item. Note that several items had less than 100% valid data. The interviews were conducted with complex patients that commonly manifest short attention span and restlessness and therefore for a few patients the last questions of the SIPS interview had to be shortened. In these cases, the raters decided that the information in the video is insufficient to provide a valid score and these items were left as “missing.” Furthermore, it is important to note that for the “positive,” “negative”, and “disorganized” items the percent of valid cases was between 93 and 100%. The lower rates were for the “general” items, which are of minor importance in the SIPS interview as they are not part of the definition of clinical high risk state.
Inter-site reliability of the SOPS in assessing subthreshold symptoms status
Finally, to further test whether the raters were able to distinguish reliably between subthreshold symptoms and non-clinically significant symptoms, ICC was computed on the combined total scores of three SOPS subscales: positive, negative, and disorganized symptoms. As seen in Fig. 1, agreement among raters was found to be in the good range for determining negative/disorganized subthreshold symptoms, positive subthreshold symptoms, positive and negative/disorganized subthreshold symptoms, acute positive subthreshold symptoms, or as not psychosis-prone ICC = 0.71 [95% CI 0.47–0.89].
Comparison of inter-site reliability in CHR and non-CHR
Supplementary Table S1 shows the same results as in Table 3 but split by CHR status (11 CHR compared to 7 non-CHR). Of the 14 items comparable across samples, six showed higher ICCs in the non-CHR sample. Item N3 showed the largest difference (0.33 in non-CHR, compared to 0.93 in CHR), while item G2 showed the largest difference in the opposite direction (0.92 in non-CHR, compared to 0.75 in CHR).
To our knowledge, this is the first study to test SIPS/SOPS inter-rater reliability in 22q11.2DS. The few SIPS reliability studies have been conducted only by the developers of the tool and only in US clinical high risk, help-seeking individuals, without a neurogenetic syndrome [25, 26]. Thus, our study is also the first inter-rater study of individuals with developmental disabilities. To date, there is no reliability study assessing subthreshold psychotic symptoms not only for 22q11.2DS but also for any neurogenetic syndrome. In this study, international experts from three large academic centers studying 22q11.2DS were able to obtain overall good inter-rater reliability. These efforts can advance the development of guidelines and gold standard assessment training videos, which can be useful for other researchers using the SIPS in evaluating 22q11.2DS individuals. Our results and the accompanying guidelines are relevant also for researchers studying individuals with other genetic syndromes associated with increased risk for psychosis such as Prader-Willi syndrome and 16p11.2 duplication syndrome . Establishing reliability for measuring subthreshold symptoms can benefit treatment studies since the tool can be used to obtain outcome measures to determine efficacy of intervention. Importantly, information on reliability will help contextualize studies in neurogenetic populations with those carried out in community samples.
We found excellent agreement among raters (ICCs > 0.75) for total SOPS’ scores and for all subscales and good reliability (ICC = 0.71, Fig. 1) in distinguishing between the various subthreshold syndromes. These results are in line with those demonstrated by Miller et al.  in clinical high-risk, help-seeking non-22q11.2DS population. While our sample was relatively small it was almost 30% larger (n = 18) than that of Miller et al.  (n = 14). Inter-rater reliability was at least in the “good” range for all SOPS items, where 17 of those 19 were in the “excellent” range. Of note, ICC values are inherently sensitive to the level of variability among participants’ scores (i.e., higher between-subjects variability improves inter-rater reliability coefficients) .
A limitation of our study was that we did not employ the traditional two-step process used in some reliability studies. At the first step, similarly to what we did in our study, after raters score interviews independently, they discuss differences in ratings to improve consensus and reach a set of agreed criteria. In the second step, not done in our study, these criteria are used by a new, other group of raters that score interviews without any discussion, thereby avoiding self-biases. Thus, additional study is recommended with a new sample of 22q11.2DS assessments and other raters.
Our data were acquired from three sites, but 11 out of the 18 subjects were from one site (UCLA). Each research group likely has some site-specific practices in its method of interviewing individuals with 22q11.DS, which can in turn influence the scoring. Thus, the over-representation of subjects from one site might somewhat bias our results. Future studies should expand the number of participants and include participants that are well-distributed across a variety of sites.
Notably, in this study IQ > 70 was an inclusion criterion and the conclusions may not extend to individuals with cognitive disability. Based on our experience, we have suggestions for administering the SIPS and assessing symptoms in individuals with 22q11.2DS, consisting of (1) clarification/simplification of interview questions, (2) elaboration of qualifiers with examples, and (3) use of numerical scales to assist the participant in rating his/her symptoms.
For example, in assessing grandiose ideas (P3), the interview includes the questions “Have you ever behaved without regard to painful consequences? For example, do you ever go on excessive spending sprees?” We recommend stating the question as it is written, but following the question up with clarifications / simplifications of language consisting of “In other words, have you ever behaved without thinking about the bad things that could happen because of what you are doing? For example, have you ever spent a lot of money on things you can’t afford?”
In assessing suspiciousness/persecutory ideas, the interview begins with the question: “Do you ever feel that people around you are thinking about you in a negative way” and follows the question up with qualifiers that assess onset-duration, degree of distress, degree of interference with life, and degree of conviction/meaning. For individuals with cognitive disabilities, we recommend that the interviewer elaborate on and concretize these qualifiers and provide numerical scales for some of the probes. For example, in probing for when symptoms began, we suggest providing a specific timepoint or life event anchor that would be relevant to the participant (e.g., “Did this start before or after you began middle school?”). In probing for degree of distress, we suggest that the interviewer follow up the interview question “What is this experience like for you? Does it bother you” with “On a scale from 0 to 10, with 10 being the most upsetting; how upsetting would you rate this?” Visual representations of such a scale can also be used, which permit the participant to point to the rating with which s/he agrees. We have modified the SIPS to reflect these and other guidelines in an effort to optimize its administration for individuals with cognitive disability. (Our full protocol of suggested modifications for individuals with cognitive disability can be obtained from the authors).
To summarize, given the high rates of subthreshold psychotic symptoms in 22q11.2DS individuals, it is of utmost importance to validate the SIPS in the 22q11.2DS population. Our preliminary results with three international assessors reviewing SIPS video assessments of 18 22q11.2DS individuals suggest that good inter-reliability can be achieved for most SOPS items.
Availability of data and materials
The datasets used and/or analyzed for the present paper can be made available upon a reasonable request to the corresponding author.
22q11.2 deletion syndrome
Attention deficit/hyperactivity disorder
Autism spectrum disorder
Children’s Hospital of Philadelphia
Global Assessment of Functioning
Intraclass correlations coefficients
The Kiddie Schedule for Affective Disorders and Schizophrenia for School Age Children-Present and Lifetime Version
The Structured Clinical Interview for DSM-IV Diagnoses
Structured Interview for Prodromal Syndromes
Selective norepinephrine reuptake inhibitor
Scale of Prodromal Symptoms
Selective serotonin reuptake inhibitor
University of California Los Angeles
- WAIS III:
Wechsler Adult Intelligence Scale, 3rd edition
- WISC III:
Wechsler Intelligence Scale for Children, 3rd edition
Shprintzen R. Velo-cardio-facial syndrome: a distinctive behavioral phenotype. Ment. Retard. Dev. Disabil. Res. Rev. 2000;6:142–7. https://doi.org/10.1002/1098-2779(2000)6:2%3C142::AID-MRDD9%3E3.0.CO.
Swillen A, Vogels A, Devriendt K, Fryns JP. Chromosome 22q11 deletion syndrome: update and review of the clinical features, cognitive-behavioral spectrum, and psychiatric complications. Am. J. Med. Genet. 2000;97:128–35. https://doi.org/10.1002/1096-8628(200022)97:2%3C128::AID-AJMG4%3E3.0.CO.
Green T, Gothelf D, Glaser B, Debbane M, Frisch A, Kotler M, et al. Psychiatric disorders and intellectual functioning throughout development in velocardiofacial (22q11.2 deletion) syndrome. J. Am. Acad. Child Adolesc. Psychiatry. 2009;48(11):1060–8 Available from: https://linkinghub.elsevier.com/retrieve/pii/S0890856709602537.
Schneider M, Debbané M, Bassett AS, Chow EWC, Fung WLA, van den Bree MBM, et al. Psychiatric disorders from childhood to adulthood in 22q11.2 deletion syndrome: results from the international consortium on brain and behavior in 22q11.2 deletion syndrome. Am. J. Psychiatry. 2014;171(6):627–39. https://doi.org/10.1176/appi.ajp.2013.13070864.
Perälä J, Suvisaari J, Saarni S, Kuoppasalmi K. Lifetime prevalence of psychotic and bipolar I disorders in general population. Arch. Gen. Psychiatry. 2007;64(1):19–28 Available from: https://jamanetwork.com/journals/jamapsychiatry/fullarticle/209973. https://doi.org/10.1001/archpsyc.64.1.19.
Yung AR, Yuen HP, Berger G, Francey S, Hung TC, Nelson B, et al. Declining transition rate in ultra high risk (prodromal) services: dilution or reduction of risk? Schizophr Bull. 2007;33:673–81 8.
Ruhrmann S, Schultze-Lutter F, Klosterkötter J. Probably at-risk, but certainly ill: advocating the introduction of a psychosis spectrum disorder in DSM-V. Schizophr Res. 2010;120(1-3):23–37. https://doi.org/10.1016/j.schres.2010.03.015.
Antshel KM, Fremont W, Ramanathan S, Kates WR. Predicting cognition and psychosis in young adults with 22q11.2 deletion syndrome. Schizophr. Bull. 2016;43(4):833–42 Available from: https://academic.oup.com/schizophreniabulletin/article/43/4/833/2327380.
Armando M, Girardi P, Vicari S, Menghini D, Digilio MC, Pontillo M, et al. Adolescents at ultra-high risk for psychosis with and without 22q11 deletion syndrome: a comparison of prodromal psychotic symptoms and general functioning. Schizophr. Res. 2012;139:151–6. https://doi.org/10.1016/j.schres.2012.04.020.
Kates WR, Mariano MA, Antshel KM, Chandra S, Gamble H, Giordano M, et al. Trajectories of psychiatric diagnoses and medication usage in youth with 22q11.2 deletion syndrome: a 9-year longitudinal study. Psychol. Med. 2018;49(11):1914–22. https://doi.org/10.1017/S0033291718002696.
Schneider M, Schaer M, Mutlu AK, Menghetti S, Glaser B, Debbané M, et al. Clinical and cognitive risk factors for psychotic symptoms in 22q11.2 deletion syndrome: a transversal and longitudinal approach. Eur. Child Adolesc. Psychiatry. 2014;23(6):425–36. https://doi.org/10.1007/s00787-013-0469-8.
Shapiro DI, Cubells JF, Ousley OY, Rockers K, Walker EF. Prodromal symptoms in adolescents with 22q11.2 deletion syndrome and schizotypal personality disorder. Schizophr. Res. 2011;129(1):20–8. https://doi.org/10.1016/j.schres.2011.03.030.
Stoddard J, Niendam T, Hendren R, Carter C, Simon TJ. Attenuated positive symptoms of psychosis in adolescents with chromosome 22q11.2 deletion syndrome. Schizophr. Res. 2010;(1-3):118, 118–21. https://doi.org/10.1016/j.schres.2009.12.011.
Tang SX, Moore TM, Calkins ME, Yi JJ, McDonald-McGinn DM, Zackai EH, et al. Emergent, remitted and persistent psychosis-spectrum symptoms in 22q11.2 deletion syndrome. Transl. Psychiatry. 2017;7(7):e1180. https://doi.org/10.1038/tp.2017.157.
Mekori-Domachevsky E, Guri Y, Yi J, Weisman O, Calkins ME, Tang SX, et al. Negative subthreshold psychotic symptoms distinguish 22q11.2 deletion syndrome from other neurodevelopmental disorders: a two-site study. Schizophr. Res. 2017;188:42–9. https://doi.org/10.1016/j.schres.2016.12.023.
Tang SX, Moore TM, Calkins ME, Yi JJ, Savitt A, Kohler CG, et al. The psychosis spectrum in 22q11.2 deletion syndrome is comparable to that of nondeleted youths. Biol. Psychiatry. 2017;82(1):17–25. https://doi.org/10.1016/j.biopsych.2016.08.034.
Weisman O, Guri Y, Gur RE, McDonald-McGinn DM, Calkins ME, Tang SX, et al. Subthreshold psychosis in 22q11.2 deletion syndrome: multisite naturalistic study. Schizophr. Bull. 2017;43(5):1079–89. https://doi.org/10.1093/schbul/sbx005.
Schneider M, Armando M, Pontillo M, Vicari S, Debbané M, Schultze-Lutter F, et al. Ultra high risk status and transition to psychosis in 22q11.2 deletion syndrome. World Psychiatry. 2016;15(3):259–65 Available form: https://onlinelibrary.wiley.com/doi/pdf/10.1002/wps.20347.
Weinberger R, Weisman O, Guri Y, Harel T, Weizman A, Gothelf D. The interaction between neurocognitive functioning, subthreshold psychotic symptoms and pharmacotherapy in 22q11.2 deletion syndrome: a longitudinal comparative study. Eur. Psychiatry. 2018;48(1):20–6. https://doi.org/10.1016/j.eurpsy.2017.10.010.
Schneider M, Armando M, Schultze-Lutter F, Pontillo M, Vicari S, Debbané M, et al. Prevalence, course and psychosis-predictive value of negative symptoms in 22q11.2 deletion syndrome. Schizophr Res. 2019;206:386–93. https://doi.org/10.1016/j.schres.2018.10.014.
McGlashan TH, Miller TJ, Woods SW, Hoffman RE, Davidson L. Instrument for the assessment of prodromal symptoms and states BT-early intervention in psychotic disorders. In: Miller T, Mednick SA, McGlashan TH, Libiger J, Johannessen JO, editors. . Dordrecht: Springer Netherlands; 2001. p. 135–49. https://doi.org/10.1007/978-94-010-0892-1_7.
Miller TJ, McGlashan TH, Woods SW, Stein K, Driesen N, Corcoran CM, et al. Symptom assessment in schizophrenic prodromal states. Psychiatr. 1999;70:273–87 Available form: https://link.springer.com/content/pdf/10.1023/A:1022034115078.pdf.
Comparelli A, Savoja V, Kotzalidis GD, Woods SW, Mosticoni S, Vassallo F, et al. Factor-structure of the Italian version of the Scale of Prodromal Symptoms (SOPS): a comparison with the English version. Epidemiol. Psychiatr. Sci. 2011;20(1):45–54. https://doi.org/10.1017/S2045796011000114.
Fusar-Poli P, Cappucciati M, Rutigliano G, Lee TY, Beverly Q, Bonoldi I, et al. Towards a standard psychometric diagnostic interview for subjects at ultra high risk of psychosis: CAARMS versus SIPS. Psychiatry J. 2016:1–11. https://doi.org/10.1155/2016/7146341.
Miller TJ, McGlashan TH, Rosen JL, Cadenhead K, Ventura J, McFarlane W. Prodromal assessment with the structured interview for prodromal syndromes and the scale of prodromal symptoms. Schizophr. Bull. 2003;29(4):703–15 Available form: https://academic.oup.com/schizophreniabulletin/article/29/4/703/1887836.
Miller TJ, McGlashan TH, Rosen JL, Somjee L, Markovich PJ, Stein K, et al. Prospective diagnosis of the initial prodrome for schizophrenia based on the structured interview for prodromal syndromes: preliminary evidence of interrater reliability and predictive validity. Am. J. Psychiatry. 2002;159:863–5. https://doi.org/10.1176/appi.ajp.159.5.863.
Michaelovsky E, Frisch A, Carmel M, Patya M, Zarchi O, Green T, Basel-Vanagaite L, Weizman A, Gothelf D.Genotype-phenotype correlation in 22q11.2 deletion syndrome. BMC Med. Genet. 2012;13(1):122 https://doi.org/10.1186/1471-2350-13-122
Wechsler D. Wechsler Intelligence Scale for Children. 3rd ed. San Antonio, TX: Psychological Corporation; 1991.
Wechsler D. WAIS-III Administration and Scoring Manual. San Antonio, TX: Psychol Corp; 1997.
Kaufman J, Birmaher B, Brent D, Rao UMA, Flynn C, Moreci P, et al. Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (K-SADS-PL): Initial Reliability and Validity Data. J. Am. Acad. Child Adolesc.Psychiatry. 1997;36:980–8. https://doi.org/10.1097/00004583-199707000-00021.
McGlashan TH, Woods S. Walsh B. The psychosis-risk syndrome: Handbook for Diagnosis and Follow-Up. Oxford University Press, New York; 2010.
Tang SX, Yi JJ, Moore TM, Calkins ME, Kohler CG, Whinna DA, et al. Subthreshold psychotic symptoms in 22q11.2 deletion syndrome. J. Am. Acad. Child Adolesc. Psychiatry. 2014;53(9):991–1000. https://doi.org/10.1016/j.jaac.2014.05.009.
Brueckl M, Heuer F.irrNA: coefficients of interrater reliability - generalized for randomly incomplete datasets. R package version 0.1.4; 2018. Available form: https://CRAN.R-project.org/package=irrNA
R Core Team R. A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria; 2018. Available form: https://www.R-project.org/.
Brueckl M. Statistische Verfahren zur Ermittlung der Urteileruebereinstimmung. In: Altersbedingte Veraenderungen der Stimme und Sprechweise von Frauen. Berlin: Logos; 2011. p. 88–103.
Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol. Assess. 1994;6:284–90. https://doi.org/10.1037/1040-35220.127.116.114.
Portney LG, Watkins MP. Foundations of clinical research : applications to practice, 3rd editio. ed. Upper Saddle River, N.J: Pearson/Prentice Hall; 2015.
The authors thank the research teams in the institutions collaborating on this project and the research participants and their families.
The study reported in this publication was supported by the Binational Science Foundation, grant number 2017369; The National Institute of Mental Health of the National Institutes of Health grants U01MH101722, U01MH101719, U01MH119738, and RO1 MH085953 (CEB). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Ethics approval and consent to participate
This study was approved by the Institutional Review Board at each of the three participating universities of the authors. Informed written consent was obtained from either the participant or the parent/legal guardian prior to participation and assent was obtained from each participant as appropriate. The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and international committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Moore, T.M., Salzer, D., Bearden, C.E. et al. Inter-rater reliability of subthreshold psychotic symptoms in individuals with 22q11.2 deletion syndrome. J Neurodevelop Disord 13, 23 (2021). https://doi.org/10.1186/s11689-021-09372-3