The Dyslexia Marker Test for Children: Development and Validation of a New Test

The Dyslexia Marker Test for Children: Development and Validation of a New Test Trude Nergård-Nilssen, PhD1 and Oddgeir Friborg, PhD1 Abstract Assessment for Effective Intervention 1 –11 © Hammill Institute on Disabilities 2021 Article reuse guidelines: DOI: 10.1177/ This article describes the development and psychometric properties of a new Dyslexia Marker Test for Children (Dysmate-C). The test was designed to identify Norwegian students who need special instructional attention. The computerized test includes measures of letter knowledge, phoneme awareness, rapid automatized naming, working memory, decoding, and spelling skills. Data were collected data from a sample of more than 1,100 students. Item response theory (IRT) was used for the psychometric evaluation, and principal component analysis for checking uni-dimensionality. IRT was further used to select and remove items, which significantly shortened the test battery without sacrificing reliability or discriminating ability. Cronbach’s alphas ranged between .84 and .95. Validity was established by examining how well the Dysmate-C identified students already diagnosed with dyslexia. Logistic regression and receiver operating characteristic (ROC) curve analyses indicated good to excellent accuracy in separating children with dyslexia from typical children (area under curve [AUC] = .92). The Dysmate-C meets the standards for reliability and validity. The use of regression-based norms, voice over instructions, easy scoring procedures, accurate timing, and automatic computation of scores, make the test a useful tool. It may be used in as part screening procedure, and as part of a diagnostic assessment. Limitations and practical implications are discussed. Keywords dyslexia, assessment, computerized, IRT Dyslexia represents a difficulty in learning to decode print. Individuals with dyslexia often have problems because they do not master the critical factors underlying decoding and spelling, which, in turn, often hamper reading comprehen sion. To this date, many Norwegian children with dyslexia are missed due to the lack of norm-referenced assessment tools. A recent report reveals that 51% of the affected indi viduals are not identified or diagnosed until they enter high school (Solem, 2021). The purpose of this study was to develop a dyslexia marker test for Norwegian students and to examine its psychometric properties. A considerable body of research shows that problems with orthography (Georgiou et al., 2021) and phonology (Snowling & Melby-Lervåg, 2016) are the major proximal causal risk markers for dyslexia. In alphabetic languages, learning to read starts with learning the mapping between phonemes (the smallest units of speech distinguishing one word from another) and graphemes (one letter, or a group of letters, that represent a speech sound, or phoneme). Difficulties with the ability to attend to, discriminate, and manipulate sounds in words are highly likely to lead to dif ficulties with mapping speech and print, or rather, mapping phonology and orthography. Family risk studies demonstrate that phonological problems are present long before formal reading instruction begins (Thompson et al., 2015). It is widely reported however that phonological prob lems are neither necessary nor sufficient to account for dys lexia. Other known underlying risk factors include problems with learning letters (Torppa et al., 2016), rapid word retrieval (Parrila et al., 2020), and with working memory (Peng & Fuchs, 2014). The research literature shows that when they operate together with decoding and spelling problems, the risk markers accumulate toward a threshold for a diagnosis (Snowling et al., 2020). In a longitudinal study, Catts et al. (2017) found that children with a phono logical awareness deficit in kindergarten were five times more likely to have dyslexia in second grade than children without such a deficit. This risk ratio substantially increased with the addition of deficits in both oral language and rapid naming. However, some of the children with heightened 1UiT The Arctic University of Norway, Tromso, Norway Corresponding Author: Trude Nergård-Nilssen, Department of Education, UiT the Arctic University of Norway, Universitetsvegen 39, Tromsø 9019, Norway. Email: 2 Assessment for Effective Intervention 00(0) risk were later found to be adequate readers, and Catts et al. (2017) argue for a multifactorial model of dyslexia which also includes protective factors that offset the impact of phonological and other cognitive-linguistic deficits. These ideas and similar findings are encapsulated in the multiple deficit model (MDM) originally proposed by Pennington (2006). The two fundamental tenets of the MDM is that multiple predictors contribute probabilistically to neurode velopmental disorders (e.g., dyslexia) and that shared risk factors contribute to comorbidity. McGrath et al. (2020) highlight that the clinical and diagnostic implications of the MDM are that no single cognitive deficit can be used to rule in or out dyslexia at the individual level and that the dimen sional and probabilistic nature of dyslexia (and other disor ders) preclude clear mappings of cognitive profiles to the diagnosis. In line with these ideas, assessments should instead focus on the defining symptoms of dyslexia and should therefore include brief assessments of reading and their proximal skills (Snowling & Hulme, 2021). The components underpinning reading performance and dyslexia appear universal. For example, Landerl et al. (2013) found that phoneme awareness and rapid automa tized naming were strong concurrent predictors of develop mental dyslexia across six European languages. A logistic regression analysis revealed however that more participants were classified correctly when the orthography was more complex. Similarly, Reis et al. (2020) report in their meta analysis that orthographic transparency has a significant effect on the manifestation of dyslexia, with dyslexia symp toms being less marked and weaker in transparent com pared to intermediate and opaque orthographies. Numerous studies furthermore show that in transparent languages, in which every grapheme roughly corresponds to one pho neme, reading accuracy hits the ceiling soon after formal reading instruction begins (Torppa et al., 2016). Nevertheless—although growth of reading skills is faster and follow a different trajectory in more regular orthogra phies than in English—phoneme awareness, letter-sound knowledge, and rapid automatized naming measured at the onset of literacy instruction are similarly important as pre dictors of variations in growth rate across languages (Caravolas et al., 2019). The Norwegian orthography, in which context this study took place, has consistent graph eme-phoneme correspondences (feed-forward consistency) but less consistent phoneme-grapheme correspondences (feedback consistency). Consequently, spelling accuracy is a bigger obstacle than reading accuracy to young readers and individuals with dyslexia, and similarly, reading speed appears to be a bigger obstacle than reading accuracy (Nergård-Nilssen & Hulme, 2014). The test reported here was designed to address these and other characteristic fea tures of the Norwegian phonology and orthography by including a time-limited word decoding test and a spelling test that measures orthographic knowledge. An increasing number of studies report high stability into adulthood and that weaknesses in phoneme awareness, rapid naming, and working memory are strong and residual correlates of dyslexia (Nergård-Nilssen & Hulme, 2014). At the same time, there is a growing body of literature affirm ing the value of providing early reading intervention to struggling readers. For example, Mathes et al. (2005) and Lovett et al. (2017) report that children who received inter vention in first and second grade, made gains almost twice that of children receiving the same intervention in third grade and that the early intervention child continued to out perform the late intervention group. Miciak and Fletcher (2020) highlight that when risk for dyslexia is identified before Grade 3, the percentage of children who do not respond to explicit core and supplemental reading instruc tion are reduced to 2%–5%. It is thus critical to have valid tests for identifying this group of children available so that intervention can be provided to prevent or ameliorate read ing disorders. Assessment in all its forms—including screening, diag nostic testing, and monitoring—play a key role in any suc cessful intervention. Screening can provide an indication of which children are “at risk” and would benefit from further support. A diagnostic assessment, on the contrary, can pro vide a clear indication of a child’s strengths and weaknesses and specify which skills should be targeted within an inter vention. It also gives a picture of the severity of the child’s difficulties and to what extent support needs to be adapted. This study presents a norm-referenced test that is named The Dyslexia Marker Test for Children (acronym: Dysmate-C). The construction of the Dysmate-C was devel oped within the framework of the MDM. The defining markers are operationalized and construed as liabilities for dyslexia and include—in addition to decoding and spell ing—letter knowledge, the ability to manipulate speech sounds (phoneme awareness), and the ability to name com mon symbols at speed (referred to as rapid automatized naming, or RAN). These markers are identified in cross linguistic studies (Caravolas et al., 2019), in individual studies (Thompson et al., 2015), and in meta-analyses (Snowling & Melby-Lervåg, 2016). The Dysmate-C test was designed to identify children at risk for dyslexia and who thus need special instructional attention. Ideally, the psychometric properties of any novel test are established by comparing how the results of the new test agree with the “true” outcome. In this study, some of our experimental instruments could not be validated against established measures of the same constructs. In the absence of a Norwegian “gold standard,” we instead examined how well the Dysmate-C could identify children that were already diagnosed with dyslexia, and thus how well test outcomes would reflect our a priori expectations of poor performance in this group. Nergård Nilssen and Friborg 3 In summary, the main objective of this study was to develop a dyslexia marker test that can identify students who need special instructional attention. Another main objective was to establish its psychometric properties. A subordinate objective was to examine if gender, as a covari ate, explains the ability (or trait score) on any of the tests. The research literature gives no reason to expect reading achievement or dyslexia to vary by gender. (Snowling & Hulme, 2021) point out that differences in the reported sex ratio between studies are likely related to measurement issues or to sampling bias. As the Dysmate-C is a new mea surement, however, we wanted to rule out that it favors either gender. Method Sample In Norway, children start school the calendar year they turn 6 years. Elementary school includes Year 1–7, and students transfer to secondary/high school the calendar year they turn 13. This study includes two samples: The first includes unscreened primary school children of which presence of reading disorders thus is unknown (n > 1,000). The second includes children that had been diagnosed with dyslexia prior to this study (n = 50). Boys and girls were roughly equally represented in both samples. Table S1 shows sam ple sizes reported for each subtest and grade level. All par ticipants were recruited from 22 different schools across Norway. Students who were diagnosed with dyslexia were allocated to “the validation sample.” Unfortunately, we do not have access to the diagnostic assessment that led to their diagnosis. As a rule, however, the school refers a student to the local educational-psychological service when dyslexia is suspected, that is, when a student shows signs of labori ous reading of longer texts, or performs poorly on the national reading test in grade 5 or grade 8. The psychologist then typically carries out a standardized test that measures reading speed of connected text, reading comprehension, and listening comprehension, as well as reading-related skills. The diagnosis is established if certain criteria are ful filled, for example, if orthographic word recognition is poor, or if listening comprehension is much better than reading comprehension. The Dysmate-C Test In this section, we will describe the original tests as they were administered to the children in the normative study. The Dysmate-C test battery is computerized and is pre sented on a laptop or tablet computer, with one available at each testing site. All instructions, practice items and correc tive feedback are provided by voice-over. Feedback is pro vided in the practice sessions but not during the actual tests. Simple textual instructions and illustrations appear on the screen prior to each test to remind the child of the task requirements. The assessor is on hand to answer any addi tional questions during testing and to score oral responses. Responses are scored by the keyboard “A” (correct), “S” (wrong) or “D” (no response) keys, respectively, or alterna tively by the left mouse button on the corresponding tab on the screen. The program automatically records scorings, and the timed tasks are regulated, and response time auto matically recorded by a built-in timer. When assessor has entered the student’s age and grade level, the test battery is composed accordingly. For example, the spelling test is not administered to Year-1 and -2 students, whereas the Letter Knowledge and Phoneme Isolation (PI) tests is considered too easy and thus not administered from Year 3 onwards. All except two tests (i.e., the letter knowledge and the spell ing test) are either time-limited or speeded measures. For the youngest students, the session typically takes 10–15 minutes, whereas the session from Year 3 and above typi cally takes 20–30 minutes, including the spelling test. The Letter Knowledge Test. There are 29 letters in the Nor wegian alphabet. The child was asked to give the sounds and names of all letters. The letters were presented in ran dom order to avoid use of rote learning from alphabet songs. If only the letter name was given, the child was prompted to provide the sound, and vice versa. Two points were awarded if the child produced a correct response for both the name and the sound, whereas one point was rewarded if only name or sound was produced correctly. This test was admin istered to students in grade 1 and 2, and to the validation sample. Phoneme Isolation. To evaluate phoneme awareness, we asked the child to identify specific phonemes in words. In this test, the child was presented with sets of four illustra tions and was asked to point at the object on the screen that either started or ended with a given sound (e.g., /s/). Four practice items introduced the test to familiarize the child to point at an object that “begins with” or “ends with” a sound. If no response was given within 10 seconds, the child was automatically presented with the next set and target sound. The score here was the number of correctly identified target words, with a maximum score of 16. Phoneme Deletion. To measure phoneme awareness in stu dents in Grade 3–7 further, we developed a phoneme dele tion test. Here, the child was asked to produce the word that remained when a particular sound was omitted (e.g., /b/ in brød [bread], where rød [red] is the correct answer). Prac tice items introduced the test to familiarize the child with omitting the first, the last sound, or the middle sound, respectively. If no response was given within 10 s, the child was presented with the next word and target sound. The