# Research Report: Conversation-Based Assessment in Education

## Executive Summary

Conversation-based assessment encompasses a broad family of approaches that position dialogue — between teacher and student, among peers, or between learner and intelligent system — as both the medium and the instrument of educational evaluation. Its theoretical roots extend to Lev Vygotsky's sociocultural learning theory and Mikhail Bakhtin's dialogism, which together argue that knowledge is co-constructed through language rather than individually possessed and that authentic assessment must therefore engage learners in live communicative acts. These foundations have generated a rich set of applied frameworks — dynamic assessment, assessment-as-dialogue, dialogic formative feedback, Socratic seminar evaluation — each translating sociocultural principles into practice across K–12 and higher education contexts.

The empirical evidence on effectiveness is promising but uneven. Direct comparative studies remain scarce, and randomized controlled trials with standardized effect sizes are rare in the conversational assessment literature specifically. The available evidence points to several consistent findings: conversational and oral assessment reduces achievement-gap variability relative to written formats, face-to-face dialogic feedback produces larger learning gains than electronic feedback (SMD = 0.92 versus 0.63 in medical education), and combining oral argumentation with structured writing instruction produces significant gains particularly for structurally disadvantaged students. Equity evidence is especially compelling for English Language Learners, for whom written standardized assessments carry documented construct-irrelevant variance because they measure language proficiency alongside — and often instead of — the targeted content knowledge.

The emergence of AI-powered conversational assessment represents the field's most dynamic frontier. Named systems span legacy automated essay scoring (ETS's e-rater), adaptive language proficiency testing (Duolingo English Test), multi-agent LLM architectures for Socratic dialogue assessment (AIED 2025), and explainable spoken language assessment pipelines. These systems achieve technically competitive performance against human rater benchmarks in controlled tasks but show significant limitations for uncontrolled spontaneous speech, face algorithmic bias concerns documented across racial, gender, and socioeconomic dimensions, and are now subject to binding regulatory obligations under the EU Artificial Intelligence Act (2024), which classifies educational AI as high-risk.

Psychometric validity is the central contested terrain. Oral and dialogic assessment can achieve acceptable inter-rater reliability when analytical rubrics are combined with structured rater training — one study found a G-coefficient of 0.864 for a single well-trained rater — but the measurement literature identifies irreducible tensions between the standardization required for high-stakes summative use and the contextual responsiveness that gives dialogic assessment its pedagogical distinctiveness. The emerging consensus favors hybrid designs in which dialogic methods function formatively and are embedded in summative processes with transparent criteria and trained human adjudication, rather than displaced entirely by automated systems.

---

## Theoretical Foundations: Sociocultural Learning Theory and Dialogic Education

### Vygotsky's Zone of Proximal Development

The intellectual scaffolding for conversation-based assessment rests most fundamentally on Lev Vygotsky's concept of the Zone of Proximal Development (ZPD), defined as "the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance or in collaboration with more capable peers" [The zone of proximal development in Vygotsky's analysis of learning](https://blogs.ubc.ca/vygotsky/files/2013/11/chaiklin.zpd_.pdf) [11]. This formulation makes an explicit epistemological argument against assessment practices that measure only what learners can do independently: such measures reveal only functions that have already fully developed, systematically obscuring the maturing functions that are educationally most significant.

Vygotsky's research demonstrated that the size of a child's ZPD — the gap between independent and assisted performance — was more predictive of future school success than IQ scores [11]. This finding carries direct implications for assessment design: if collaborative performance in dialogue with a more competent partner reveals more about learning potential than solo test performance, then assessment must create conditions for that collaboration. The ZPD is neither a property of the learner alone nor of the environment alone; it is "a property of the interaction between the two" [Vygotsky's Zone of Proximal Development: Instructional Implications](https://files.eric.ed.gov/fulltext/EJ1081990.pdf) [12], which means only conversational, dialogic assessment instruments can access it directly.

Vygotsky emphasized that psychological tools — pre-eminently language — are the primary mediational means through which higher psychological functions develop. Learning is therefore inherently social and cultural before it becomes individual and internal: what a learner can accomplish "one day with assistance, s/he is able to do tomorrow alone" [12]. This developmental sequence — from intermental (social) to intramental (individual) functioning — means that an assessment interaction is not merely a measurement event but a developmental event in its own right. Properly designed, a conversation-based assessment can simultaneously diagnose current functioning, scaffold emerging capacities, and contribute to learning.

Proper application of ZPD theory in assessment requires the technical concept of imitation. Vygotsky's imitation is "not a mindless copying of actions" but collaborative problem-solving that reveals maturing functions "inadequate for independent performance" [11]. Assessment that employs guided dialogue — prompting, hinting, extending — enables examiners to distinguish between what a student can perform spontaneously and what they can achieve with calibrated support, providing what Vygotsky termed "true diagnosis" that "provides an explanation, prediction, and scientific basis for practical prescription" [11].

### Bakhtin's Dialogism and Heteroglossia

Parallel to Vygotsky's developmental psychology, Mikhail Bakhtin's literary and linguistic theory provides the discourse-theoretical foundations for dialogic assessment. Where Vygotsky focuses on the psychological mechanisms of assisted learning, Bakhtin illuminates the semiotic character of the language through which that assistance operates. Language, on Bakhtin's account, is "inherently social and contextual in nature" — not a neutral system for transmitting pre-formed meanings but "a contested space where meaning is constantly negotiated and constructed through interaction" [Dialogic learning](https://en.wikipedia.org/wiki/Dialogic_learning) [20].

Bakhtin's concept of heteroglossia — the presence of multiple voices, perspectives, and discourse types within any communicative act — has particular implications for classroom assessment [20]. When students from diverse backgrounds enter dialogue, their unique Discourses and cultural models create a fundamentally multi-voiced environment. Monologic assessment — the single correct answer evaluated against a fixed rubric — suppresses this heteroglossia, measuring performance on a normative standard that may disadvantage learners whose linguistic histories and cultural scripts differ from those in which the assessment was designed [A Dialogic Pedagogy](https://ices.library.ubc.ca/index.php/criticaled/article/download/182238/182310/) [21]. Dialogic assessment, by contrast, creates "contact zones" where heteroglot voices interact and meaning is co-constructed, enabling teachers to evaluate not merely product but process — the quality and character of the reasoning that students bring to collaborative sense-making [21].

Bakhtin's insight that "meaning is realized only in the process of active, responsive understanding" [21] directly motivates dialogue-based evaluation. An answer to a closed question in isolation may not represent genuine understanding at all; understanding reveals itself in how a learner responds to challenge, extends an idea, recognizes counter-evidence, or recasts a claim in a new register. Assessment instruments that create conditions for these moves — teacher conferencing, Socratic seminars, peer debate — are measuring something closer to what education aims to develop than written tests of recalled propositions can capture.

### Extending the Foundation: Mercer, Alexander, and Wells

Neil Mercer, Emeritus Professor of Education at the University of Cambridge and Director of Oracy@Cambridge, has produced the most empirically grounded extension of Vygotskian dialogic principles into classroom assessment [Neil Mercer: Faculty of Education](https://www.educ.cam.ac.uk/people/staff/mercer/) [38]. His most significant methodological contribution is the **Thinking Together** teaching approach, developed with Lyn Dawes, Karen Littleton, and Rupert Wegerif, which "has been demonstrated to improve children's communication, learning, and reasoning skills" and whose outcomes "were incorporated into the National Strategies for primary and secondary education and, more recently, the Teaching and Learning Toolkit of the Educational Endowment Foundation" [38].

Central to Mercer's framework is the concept of **exploratory talk** — a specific register of classroom dialogue in which "speakers consider one another's viewpoints in a critically constructive manner. If those viewpoints are challenged, justifications and alternative perspectives must be suggested" [Students' use of exploratory talk to exercise critical thinking](https://pmc.ncbi.nlm.nih.gov/articles/PMC7471859/) [39]. In a study of 125 students aged 11–12 implementing Mercer's Thinking Together approach in Hong Kong primary English classrooms, researchers found that students "engaged in explicit reasoning and used 'exploratory talk' as a dialogic tool in exercising critical thinking," producing 244 instances of reasoning words indicating explicit justification, and demonstrating that students "asked questions, challenged ideas, provided evidence-based explanations, and co-constructed shared understanding" [39]. Mercer's related concept of **interthinking** — "engaging collaboratively in a manner that went beyond any individual student's own reasoning" [Feeding forward: A case study of dialogic assessment](https://my.chartered.college/impact_article/feeding-forward-a-case-study-of-dialogic-assessment/) [42] — captures what dialogic assessment uniquely can access: the quality of collaborative reasoning that no individual written artifact can produce or document.

Robin Alexander's dialogic teaching framework develops a complementary account at the level of pedagogical design. Since the early 2000s, Alexander has "developed and progressively refined a framework for explicating dialogic teaching and supporting teachers who wish to enhance their practice" [Dialogic Teaching — Robin Alexander](https://robinalexander.org.uk/dialogic-teaching/) [40] comprising five core principles — collectivity, reciprocity, cumulation, support, and purposefulness — and four basic repertoires (organizational settings, talk for everyday life, teaching talk, and learning talk) with 61 planning and review indicators [DIALOGIC TEACHING IN BRIEF](https://coleridgeprimary.org/wp-content/uploads/2019/11/Dialogc-teaching-in-brief-170622.pdf) [41].

Alexander's framework specifically contrasts dialogic teaching with the dominant "recitation or IRE (initiation-response-evaluation) mode of teaching, which centres on closed questions, recall answers and minimal feedback and is the Anglo-American and possibly international default, remains strongly resistant to change, despite evidence that it is essentially wasteful of talk's true cognitive and educational potential" [41]. The fundamental criterion of quality is cognitive: "What counts is the extent to which instruction requires students to think, not just to report someone else's thinking," and "if an answer does not give rise to a new question from itself, it falls out of the dialogue" [41]. The international evidence base shows that "students who had experienced dialogic teaching broadly defined 'performed better on standardised tests than those in control groups, retained their learned knowledge for longer, and more effectively transferred their knowledge and understanding from one subject to another" [41]. A large-scale EEF-funded trial involving 5,000 students across 78 schools found that students in intervention classrooms achieved approximately two months additional progress in English, mathematics, and science after only 20 weeks — positioning Alexander's work among only 17 of 190 EEF-evaluated programmes identified as "promising interventions" [40].

Gordon Wells provides the third major extension of this tradition through his work on **dialogic inquiry**. His foundational text *Dialogic Inquiry*, published by Cambridge University Press (1999), "emphasizes co-construction of knowledge between more and less mature participants through joint activity and semiotic mediation" [Dialogic Inquiry — Cambridge University Press](https://www.cambridge.org/core/books/dialogic-inquiry/C64C8553C45813842441DFDEEE338C68) [55] and has accumulated 1,385 citations, testifying to its influence. Wells grounds his methodology in the **Developing Inquiring Communities in Education Project (DICEP)**, which "demonstrated how collaborative dialogue between teachers and researchers can improve classroom practices," with findings showing that "dialogic inquiry significantly increases student engagement through discussion-based learning and empowers teachers as active participants in research rather than passive subjects" [Dialogic Inquiry as Collaborative Action Research](https://www.academia.edu/28844382/Dialogic_Inquiry_as_Collaborative_Action_Research) [54]. Wells emphasizes "sustained participation and authentic dialogue are essential for understanding classroom dynamics, moving beyond limitations of traditional longitudinal studies" [54], and demonstrates measurable shifts toward more dialogic classroom interaction patterns when teachers engage in collaborative inquiry practices.

---

## Established Methodological Frameworks

### Dynamic Assessment

Dynamic Assessment (DA) represents the most fully theorized assessment framework derived from Vygotsky's ZPD. DA "departs from the traditional distinction between formative and summative assessment, as it understands teaching to be an inherent part of all assessment regardless of purpose or context" [Dynamic Assessment — Springer](https://link.springer.com/rwe/10.1007/978-3-319-02261-1_18) [56]. Where static, psychometrically-oriented assessment reveals only what learners have already achieved, DA specifically probes for maturing, partially-developed capacities by having the examiner intervene during the assessment itself.

The operational mechanism is mediation. When learners encounter difficulties, "the examiner (referred to as a mediator in this approach) intervenes and provides clues and hints, starting from the most implicit and gradually moving towards more explicit ones, and if necessary, ending with the provision of the correct answer and some explanations" [How to implement dynamic assessment to enhance L2 development](https://files.eric.ed.gov/fulltext/EJ1324122.pdf) [57]. This graduated scaffolding reveals not only current ability but developmental responsiveness — the extent to which a learner can extend their functioning with support, and which forms of assistance prove most generative. DA pursues three goals: "revealing current knowledge, illuminating developmental processes and emerging abilities, and guiding learners beyond their current level" [57].

Two primary organizational formats exist. Interventionist DA uses standardized, predetermined hint sequences suitable for larger groups; Interactionist DA provides individualized, dialogic mediation tailored to each learner's needs and is better suited for small classes or one-to-one settings [56]. Both can be administered as "cake" (mediation embedded within the assessment) or "sandwich" (mediation provided separately between pre- and post-tests) [57]. In second language acquisition contexts, "L2 DA studies have generally been pursued in collaboration with classroom teachers, emphasizing dialogic interaction in one-to-one or small group settings" [56], with applications spanning reading, listening, speaking, and writing assessment. For speaking specifically, "individualized mediation reveals true grammatical knowledge through guided dialogue" [57] that standardized tests cannot access.

Empirical evidence supports DA's claims. A study of six Iranian EFL students implementing a three-stage DA procedure (pre-test, mediation sessions, post-test) demonstrated that "when reading strategies are mediated to the participants appropriately, they helped improve the learners' reading skills effectively," and critically revealed that two students with identical pre-test scores showed different responsiveness to mediation, "revealing different learning potentials that static assessment could not detect" [Dynamic Assessment — CIBTech](https://www.cibtech.org/sp.ed/jls/2015/02/380-JLS-S2-390-SARA-DYNAMIC.pdf) [9]. A doctoral study by Natalie Hasson developing a DA task for children with specific language impairment reported 88% inter-rater reliability and demonstrated that "children making little progress in their ongoing therapy were shown to derive most benefit from the modified intervention" based on DA-derived diagnostic information — findings that "Speech-language therapists found useful for altering intervention strategies" [Hasson, Natalie.pdf — City Research Online](https://openaccess.city.ac.uk/id/eprint/1119/1/Hasson%2C_Natalie.pdf) [10].

### Assessment-as-Dialogue and Dialogic Assessment

Beyond DA's formal apparatus, the assessment-as-dialogue framework positions ongoing conversational exchange as integral to — not separate from — learning assessment. A case study from St Helen's School documents this approach through a reading journal method in Year 7 English literature. The method addressed a fundamental problem: "feedback often becomes a one-way monologue with misaligned teacher-student perceptions, causing students to view grades as judgments on past work disconnected from future learning" [42]. By combining "words, pictures, and diagrams to capture classroom discussion, expanding assessment to include oracy, argumentation, and student voice," the approach created a "shared dialogic space" for interthinking [42].

The pedagogical justification for assessment-as-dialogue rests on a tripartite reconceptualization. Assessment embedded in dialogue serves simultaneously as assessment *of* learning (documenting what students know), assessment *for* learning (providing information that guides future instruction), and assessment *as* learning (the dialogic interaction itself constitutes a learning event) [42]. The concept of **feeding forward** — dialogic exchange that shapes future rather than past learning — illustrates this third function: when students in the St Helen's study questioned gender exclusion in *Northern Lights*, the teacher connected it to historical context (female undergraduates at Girton College), "setting groundwork for Key Stage 5 feminist analysis" [42]. Individual student cases demonstrate inclusive benefit across ability levels, with both higher-ability students (verbal CAT score 121) and lower-ability students (CAT score 97) demonstrating transformation through collaborative discourse [42].

### Formative Assessment in Sociocultural Perspective

Formative assessment — defined as "all those activities undertaken by teachers and/or by their students [that] provide information to be used as feedback to modify the teaching and learning activities in which they are engaged" [Understanding Formative Assessment — WestEd](https://www2.wested.org/www-static/online_pubs/resource1307.pdf) [53] — is not itself a conversational methodology, but its theoretical grounding in Vygotskian sociocultural principles makes it a natural ally of dialogue-based practices. The sociocultural constructivist perspective underpinning formative assessment holds that "teachers and students share responsibility for learning, with students developing metacognitive skills for self-regulation" [53], and that "language is integral to learning and assessment; students' linguistic familiarity, cultural backgrounds, and communication norms significantly affect assessment interpretation" [53].

A critical observation is that cultural validity in formative assessment "is achieved when an assessment takes into consideration students' sociocultural backgrounds, including their cultural worldviews, their life contexts and values, the kinds of home and school experiences they have had (i.e., the foundation of their prior knowledge), their language preferences and proficiency, and their ways of using language to communicate and learn" [53]. Formative assessment's flexibility and tailored, ongoing character make it particularly suited to dialogue: "there can be no prescription for what a single instance of formative assessment should look like. Any instructional activity that allows teachers to uncover the way students think about what is being taught and that can be used to promote improvements in students' learning can serve a formative purpose" [53].

---

## Empirical Evidence: Effectiveness of Conversational and Oral Assessment

### Comparative Studies and Learning Outcomes

Direct comparative studies pitting conversational or oral assessment against written formats with quantitative outcome measures remain relatively scarce, and meta-analyses providing pooled effect sizes for this specific comparison do not yet exist in the published literature. The evidence base consists primarily of individual studies, systematic reviews of related practices, and indirect evidence from feedback and dialogic teaching research.

The most methodologically rigorous direct comparison is a UC San Diego pilot study using random assignment of 61 undergraduate electrical engineering students to early or late oral assessment conditions [Using Oral Assessments to Improve Student Learning Gains](https://peer.asee.org/using-oral-assessments-to-improve-student-learning-gains.pdf) [15]. Students receiving early oral assessment (OA1, n=30) experienced a smaller grade decline from midterm to final (8.7-point drop) compared to those receiving late oral assessment (OA2, n=31, 10.9-point drop). More significantly for equity purposes, grade variability (standard deviation) decreased substantially for the early oral assessment group, from 29.1% to 20.6%, compared with a smaller reduction for the late group (27.1% to 22.8%), "suggesting oral assessments may reduce achievement gaps by promoting more equitable learning" [15]. Approximately 70–78% of students reported increased confidence and improved understanding following oral assessment, and oral assessment stress levels were lower than for written examinations [15]. Students specifically highlighted the value of "personalized and specific feedback from the teaching staff" and that assessment required them to "focus on explaining the reasoning behind my work rather than just solving the problems mindlessly" [15].

A German quasi-experimental intervention study with switching-replication design across six schools examined the combined impact of debate training and Self-Regulated Strategy Development (SRSD) writing instruction on argumentative writing skills among 357 ninth-grade students in higher and lower academic tracks [Merging Oral and Written Argumentation](https://www.mdpi.com/2227-7102/15/11/1471) [31]. Both debate (oral argumentation) and SRSD interventions "significantly improved text quality in post-tests," with "gains especially visible for lower-track students" — an equity finding of particular significance in a German educational system where lower-track students typically have one year less experience with pro-and-con argumentation than their higher-track peers [31]. The study concludes that "debating and SRSD are complementary: SRSD provides a structured framework for self-regulated writing, while debating provides the authentic argumentation practice that motivates and contextualizes these strategies" [31].

In college algebra for STEM students, a study of oral examinations as an alternative assessment method found "no significant difference in summative assessment performance between the two groups" in quantitative terms, but qualitative data highlighted "immediate feedback, enhanced critical thinking, and holistic evaluation of student capabilities" as distinctive benefits [Effects of Oral Exams on Entry-Level STEM Mathematics Students](https://docs.lib.purdue.edu/ijtlhe/vol36/iss1/7/) [47]. The absence of significant quantitative differences is consistent with a pattern seen across the literature: oral and written formats often produce comparable or modestly superior summative scores while producing larger and more durable differences in metacognitive engagement, preparation strategy, and learning process.

### Evidence from Dialogic Feedback and Formative Assessment Research

A systematic review and meta-analysis of 26 randomized controlled trials examining feedback in undergraduate medical education provides the field's most rigorous quantitative evidence on conversational feedback effectiveness [Systematic Review and Meta-Analysis of RCTs on Feedback in Medical Education](https://pmc.ncbi.nlm.nih.gov/articles/PMC8651958/) [28]. The overall effect of feedback versus no or unstructured feedback was substantial (SMD = 0.80, 95% CI: 0.56–1.04, p < 0.001), with 24 of 26 studies (92%) showing significantly better results in feedback groups [28]. Critically for conversational assessment, sub-analyses demonstrated that non-electronic feedback — delivered through direct instructor-student interaction — produced higher effect sizes (SMD = 0.92) than electronic methods (SMD = 0.63). The authors attribute this to "direct contact between feedback provider and recipient. This allows the provider to observe the student's behavior in terms of receptiveness through the verbal and non-verbal communication which takes place in this relationship" [28]. Timing also significantly influenced effectiveness: immediate feedback produced substantially stronger effects (SMD = 1.16) than delayed feedback (SMD = 0.40) [28].

A systematic review of 23 empirical studies on dialogic feedback in English Language Teaching published between 2010 and 2025 found consistent positive outcomes across multiple dimensions: "improvements in writing proficiency, enhanced feedback literacy, increased learner engagement and reflection, and the development of learner autonomy and self-regulation" [Dialogic feedback in English Language Teaching](https://files.eric.ed.gov/fulltext/EJ1483229.pdf) [5]. The review identifies six modalities of dialogic feedback (face-to-face, written, technology-mediated, peer, supervisory, and teacher professional development), each with distinctive advantages. Notably, it identifies significant evidence gaps including "the scarcity of longitudinal studies tracking long-term retention," minimal investigation of speaking and listening (as opposed to writing), and geographic underrepresentation outside Asian EFL contexts [5]. A 2025 scoping review of 40 sources on dialogic formative feedback in higher education similarly found "promise for self-regulated learning" but identified gaps on "academic achievement outcomes" [A scoping review of dialogic formative feedback practices in higher education](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2025.1696703/full) [6].

A case study of dialogic feed-forward — teacher-student dialogue about work quality prior to final submission — tracked four second-year undergraduate geography students through a 12-week unit involving essay drafts, 30-minute one-to-one meetings, and revision before final submission [Dialogic Feed-Forward in Assessment](https://files.eric.ed.gov/fulltext/EJ1367808.pdf) [4]. The research found that "dialogic feed-forward can act as a pivotal moment in learning, where students reflect on their work, judge their standards against criteria, and co-create positive actions for improvement" [4]. Implementation challenges were also documented: the approach "is resource-intensive in terms of time," shows "ceiling effects with high-achieving students reluctant to change work," and "fails to engage disengaged students who avoided meetings" [4]. These limitations are important cautions for scaling dialogic assessment without adequate institutional support.

### Equity Considerations: English Language Learners and Underserved Populations

The equity case for conversational over written standardized assessment is particularly well-supported for English Language Learners (ELLs). A practitioner-researcher capstone examining English-only standardized testing for ELL students in grades 3–12 documents a critical observation: "ELLs who could successfully explain their thinking during math activities, contribute meaningfully to science experiments with lab partners, and demonstrate comprehension through interactive reading activities would receive test scores that grossly underrepresented their actual academic abilities and growth" [Beyond Language Barriers](https://digitalcommons.hamline.edu/cgi/viewcontent.cgi?article=2121&context=hse_cp) [49]. This divergence between oral demonstration and written test performance reflects a documented psychometric problem: English-only written tests "measure language proficiency rather than the targeted content knowledge" for ELL populations [49].

The reliability disparities are statistically significant. Abedi (2004), cited in [49], documented alpha reliability coefficients of 0.603 for ELLs compared to 0.808 for native English speakers — "rendering the data statistically unsound for comparing school effectiveness" [49]. A key timeline issue compounds this validity threat: "while it takes 3–5 years to develop conversational English proficiency, academic English proficiency — the language required for standardized tests — takes significantly longer at 4–7 years" [49], meaning ELL students assessed by written standardized tests in grades 3–6 are necessarily performing in a linguistic register they have not yet acquired.

Educational Testing Service guidelines for ELL assessment confirm and formalize these concerns. "Because almost all assessments measure language proficiency to some degree, ELLs may receive lower scores on content area assessments administered in English than they would if they took the same tests in a language in which they were proficient" — a phenomenon termed "construct-irrelevant variance" [Guidelines for the Assessment of English Language Learners — ETS](https://www.ets.org/pdfs/about/ell-guidelines.pdf) [50]. The ETS framework recommends six areas of attention to minimize this variance: explicit construct definition (including whether English proficiency is part of the target construct), use of accessible language in item development, involvement of ELL specialists in external review, one-on-one interview-based item tryouts specifically with ELLs, rubrics that separate construct from language in scoring, and statistical evaluation of accommodations versus modifications [50]. The implicit suggestion is that conversational, interactive assessment — which can separate content knowledge from standardized English production — is precisely the format that addresses these validity threats most directly.

For high school ELL students with disabilities — a population facing "doubly challenging" circumstances from the intersection of language learning needs and disabilities — a systematic review of reading interventions finds that structured oral interventions (fluency through repeated reading with feedback, comprehension strategies delivered interactively) "reported largely positive effects on improving students' reading outcomes" [Reading Interventions to Support English Learners with Disabilities](https://www.mdpi.com/2227-7102/15/2/223) [30]. However, the review identifies a significant research gap, with only seven studies meeting rigorous inclusion criteria, calling for interventions specifically designed for this population.

---

## AI-Powered Conversational Assessment: Systems, Architecture, and Validation

### Natural Language Processing and Automated Scoring Systems

Automated scoring systems grounded in natural language processing represent the longest-standing AI technology in educational assessment. ETS's **e-rater** is a widely studied instance for writing assessment. The system analyzes three primary dimensions — structural features (syntax variety, subordination), organizational features (rhetorical cues, transitions), and content (prompt-specific vocabulary) — using linear regression to select 8–12 of 50–60 extractable features for scoring [Comparing the Validity of Automated and Human Essay Scoring — ETS](https://www.ets.org/Media/Research/pdf/RR-00-10.pdf) [43]. Validation studies showed e-rater achieved 48% exact agreement with human raters and 94% within-one-point agreement [43]. When combined with human raters, "automated and human scores together showed minimal validity loss compared to dual human scoring," though e-rater produced less variable score distributions than humans and showed "somewhat weaker" correlations with external validity indicators than human raters achieved [43].

The **Duolingo English Test (DET)** represents a more recent adaptive architecture for language proficiency assessment. The test delivers five item formats plus extended-response speaking and writing tasks from a bank of 25,000+ items, using computer-adaptive testing (CAT) powered by machine learning [Analysis of the Scoring and Reliability for the Duolingo English Test](https://dy8n3onijof8f.cloudfront.net/media/resources/standards/scoring.pdf) [44]. Scoring methods are format-specific: C-test items use weighted-average logistic regression; dictation and elicited imitation use logistic classifiers trained on human judgments; speaking and writing use automated models evaluating grammatical accuracy, complexity, lexical sophistication, task relevance, length, and acoustic features [44]. Results are reported on a 10–160 scale aligned to CEFR levels with four subscores (Literacy, Conversation, Comprehension, Production). Reliability is strong: test-retest coefficients range from 0.79–0.90, internal consistency from 0.95–0.96, substantially exceeding the 0.80 threshold for high-stakes testing [44].

For spoken language proficiency specifically, **Natural Language-based Assessment (NLA)** represents an emerging zero-shot paradigm using large language models. Researchers applied Qwen 2.5 72B (with ASR-generated transcriptions via Whisper) to evaluate 10 CEFR-aligned proficiency dimensions without task-specific fine-tuning [Natural Language-based Assessment of L2 Oral Proficiency using LLMs](https://arxiv.org/html/2507.10200v1) [18]. Results on the Speak & Improve Corpus 2025 showed NLA achieves Pearson correlation of 0.761–0.771 with human ratings — consistently surpassing a fine-tuned BERT baseline (0.727 PCC) and matching a speech LLM trained only on read-aloud data, while not outperforming a speech LLM fine-tuned on matched spontaneous data (0.821 PCC) [18]. Task-dependent feature weighting emerged: fluency dominates short responses while grammatical accuracy and sociolinguistic appropriateness dominate longer-form tasks; vocabulary features consistently receive lower weights than discourse-level abilities [18].

A three-stage automated scoring pipeline for children's narrative assessment (SLATE 2025) combines ASR hypothesis refinement, rubric-aligned scoring, and natural language feedback generation, applying it to L2 children's narrative retellings [Leveraging ASR and LLMs for Automated Scoring and Feedback in L2 Children's Narratives](https://www.isca-archive.org/slate_2025/shankar25_slate.pdf) [19]. An **explainable spoken language assessment system** for Dutch integration exams extracts 21 features covering word use, grammar, fluency, coherence, and pronunciation. Using ordinary least squares regression on the Speak & Improve Corpus, it achieves RMSE of 0.414, outperforming the challenge baseline (0.440) and matching complex closed-track models, with 99.3% of predictions within one CEFR level of true scores [Towards Explainable Automatic Spoken Language Assessment](https://www.isca-archive.org/slate_2025/marchal25_slate.pdf) [61]. A critical validation insight: "not all features predict scores in the expected direction. The number of grammatical errors positively predicts score, suggesting that a higher number of grammatical errors yields a higher score... Such counter-intuitive relationships can only be detected in an explainable model" [61], demonstrating that black-box automated scoring can embed construct-invalid patterns that transparent systems expose.

### Intelligent Tutoring Systems with Dialogue Components

Intelligent tutoring systems (ITSs) represent the historical backbone of AI-mediated educational dialogue. Research demonstrates that "well-designed ITSs can be as effective as human tutoring" [Insights from Intelligent Tutoring Systems and the Learning Sciences](https://arxiv.org/html/2405.04645v2) [22]. ITSs comprise four core components: a domain model (expert knowledge), student model (learner's cognitive state and progress), tutor model (pedagogical strategies), and interface model (user interaction). Writing Pal — described as "the only ITS for writing developed to date" — provides instructional video modules, game-based practice, and essay feedback through formative and summative channels [The Future of Intelligent Tutoring Systems for Writing](https://link.springer.com/chapter/10.1007/978-3-031-36033-6_23) [59]. Digital writing support systems include Criterion (NLP-based feedback on essay traits), Research Writing Tutor, Peerceptiv (peer review), and HARRY (narrative writing tutor with dialogue-based prompting) [59].

**Ruffle&Riley**, an LLM-based conversational tutoring system, addresses the prohibitive authoring costs that have historically limited ITS adoption by automatically generating tutoring scripts from lesson text using GPT-4, then orchestrating free-form conversations through two agents — Ruffle (student) and Riley (professor) — in a learning-by-teaching format [Ruffle&Riley: Towards Authorship-Agnostic, Personalized Learning Conversations](https://arxiv.org/pdf/2404.17460) [58]. Two online user studies (N=200 total) compared Ruffle&Riley to reading-only controls and simpler QA chatbots. The studies found no significant learning outcome differences across conditions on standard comprehension measures, but Ruffle&Riley users reported significantly higher ratings for understanding, remembering, helpfulness, and enjoyment [58]. Interaction analysis identified four distinct usage patterns, with "conversation-focused users (group III) achieve the highest learning gains," and positive correlations emerged between learning outcomes and both word count in explanations and total learning time [58]. The authors emphasize that these findings suggest the quality of engagement within the conversational format — not the format itself — is the active ingredient.

### LLM-Based Multi-Agent Conversational Assessment

The most architecturally sophisticated recent development is multi-agent LLM systems designed specifically for assessment through dialogue. An AIED 2025 paper presents a five-agent architecture for conversation-based assessments (CBAs) within an Evidence-Centered Design framework [An LLM-Enhanced Multi-Agent Architecture for Conversation-Based Assessment](https://xinyinghou.org/assets/publications/papers/aied2025.pdf) [51]. The system coordinates: an Expert Agent that initiates and guides Socratic questioning; a Peer Agent that maintains engagement when students struggle; a Formative Assessor that classifies student responses in real time into seven speech act categories (correct, partial correct, incorrect, metacommunicative, metacognitive, irrelevant, other); a Summative Assessor that generates Toulmin argumentation diagrams linking student evidence to assessment constructs; and a non-LLM Watcher orchestrating the workflow [51].

Evaluated with 37 secondary-level students assessing science inquiry skills using GPT-3.5, the Formative Assessor achieved 52.2% agreement with human raters on speech act classification, with most disagreements in nuanced distinctions between correct and partial correct responses [51]. The Summative Assessor successfully generated appropriate Toulmin diagrams for all students [51]. The authors identify a key limitation: the current "turn-level assessment evaluates each Q-A turn in isolation rather than considering multiple answer parts together," and commit to future work on expectation-misconception tailored dialogue [51]. They characterize these as "first steps show[ing] promise for a multi-agent computing system for CBA that aligns with ECD, working toward an adaptive, engaging, reliable, and valid test-less assessment experience" [51].

A 2025 EMNLP survey proposes a task-centric taxonomy of LLM agents in education, categorizing them into Teaching Assistance Agents (classroom simulation, feedback generation, curriculum design) and Student Support Agents (adaptive learning, knowledge tracing, error detection) [LLM Agents for Education: Advances and Applications](https://aclanthology.org/2025.findings-emnlp.743.pdf) [52]. Six core capabilities are identified: memory management for contextual understanding, tool use for external resources, planning, personalization, explainability, and multi-agent communication [52]. The survey notes that "realizing their full potential will require both technical rigor and thoughtful system design" and recommends moving beyond task accuracy to practical metrics including "learning gains, user trust, and engagement," and developing "culturally adaptive multilingual agents, pedagogically-aware frameworks, and cost-effective deployment models" [52].

A comparative study of AI-powered conversation bots for L2 speaking development (Nature, 2025) evaluated Mondly with 60 undergraduate IELTS preparation students using a pre/post quasi-experimental design [Investigating the role of AI-powered conversation bots in enhancing L2 oral proficiency](https://www.nature.com/articles/s41599-025-05550-z) [16]. The experimental group showed significant improvements in speaking skills and substantial foreign language speaking anxiety reduction, with qualitative data reporting "enhanced fluency, pronunciation, grammar, motivation, and confidence" [16]. The research frames AI chatbots as virtual "More Knowledgeable Others" within Vygotsky's ZPD, providing "judgment-free environments where learners can engage in authentic conversational scenarios without fear of social repercussions" [16]. An evaluation of EAP Talk, an AI-assisted speaking assessment tool examined with 64 undergraduates, found the tool "performs well in the controlled task but less so in the uncontrolled one," and that "oral peer feedback markedly improves speaking skills through effective human-computer collaboration" [Score accuracy, perceived validity, and oral peer feedback in AI-powered peer assessment](https://www.sciencedirect.com/science/article/pii/S1475158525000360) [17] — underscoring that spontaneous speech remains more challenging for AI evaluation than scripted or controlled responses.

### Validity, Accuracy, and Fairness in Automated Systems

Algorithmic bias in AI assessment systems is a documented and growing concern. Systematic review of generative AI in education identifies that "facial-expression recognition and automated grading reproduced racial, gender, and socioeconomic biases, while emotion recognition systems produced unreliable results that could harm marginalized groups" [Ethical and Regulatory Challenges of Generative AI in Higher Education](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2025.1565938/full) [46]. Three core ethical challenges characterize current AI deployment in education: data privacy risks, algorithmic biases that perpetuate educational inequalities, and loss of cognitive autonomy [46]. Research on LLM-based assessment also identifies hallucination as a critical validity threat: "The 'hallucination' phenomenon, in which LLMs generate plausible but incorrect or nonsensical information, poses a significant challenge to their reliability in educational contexts" [52].

Many-facet Rasch modeling and generalizability analyses of LLMs as writing raters reveal context-dependent performance: "LLMs align more with humans in ranking tasks while differ greatly in scoring tasks. Humans excel in holistic scoring scenarios but struggle with complex analytical rubrics where LLMs demonstrate advantages" [Evaluating Large Language Models as Raters in Large-Scale Writing Assessments](https://www.sciencedirect.com/science/article/pii/S2666920X25001213) [32]. Both human and LLM raters exhibit "scoring biases including conservative scoring and halo effects" [32]. A score concordance study between IELTS Academic and TOEFL iBT — the two dominant high-stakes English proficiency assessments — involving 937 verified test-takers provides convergent validity data: score correlations between tests ranged from .68 (Writing) to .85 (Overall/Total), notably stronger than the prior 2010 study [A Score Concordance Study Between IELTS Academic and TOEFL iBT](https://ielts.org/cdn/ielts-research-reports/aligning-scores-of-language-proficiency-test-ikeda-et-al-2025.pdf) [60]. The authors stress that "concordance results represent estimates only and should not serve as the sole basis for setting comparable test score requirements" [60], a caution applicable to AI-automated scores generally.

### Regulatory and Ethical Frameworks

The EU Artificial Intelligence Act (adopted 2024) has introduced binding regulatory obligations for educational AI. The Act classifies educational AI as "high-risk" and introduces four decisive mechanisms: mandatory transparency (requiring providers to disclose data sources and model logic with legal penalties for non-compliance), stronger accountability through quality-management systems and risk assessments, enforceable fairness (prohibiting discriminatory models), and prohibition of emotion-detection systems ("Any system designed to infer student emotions from facial, vocal or biometric cues is classified as 'unacceptable risk' and is banned in EU educational settings") [The EU AI Act: Implications for Ethical AI in Education](https://swisscyberinstitute.com/research/eu-ai-act-implications-ethical-ai-education/) [45]. The Act "establishes education as a special-protection domain, similar to health care or critical infrastructure" and signals a shift "from 'trust me' to 'show me and prove it'" [45]. Implementation challenges persist for smaller institutions lacking resources for continuous audits, and ambiguity around text-based sentiment analysis remains [45].

Beyond the EU, a 2025 Frontiers in Education systematic review recommends that "effective regulation of GenAI in education relies on the pillars of cooperative governance, strategic institutional adoption, and ongoing assessment to ensure its responsible and impactful integration," specifically recommending "teacher training, hybrid models where AI complements rather than replaces human instruction, and continuous governance evaluation" [46]. Implementation case studies — including Stanford's AI-supported personalized tutoring and the University of Toronto's human oversight in assessment — demonstrate viable approaches that preserve human judgment at critical decision points [46].

---

## Formative Assessment Through Dialogue

### Teacher-Student Conferencing and Dialogic Feedback Modalities

The systematic review of dialogic feedback in ELT [5] identifies six distinct implementation modalities, each with a characteristic pedagogical profile. Face-to-face dialogic feedback enables "real-time synchronous interaction and immediate clarification" in which the teacher can observe receptiveness through verbal and nonverbal cues, adjust feedback content and phrasing in response to the learner's reactions, and pursue productive threads of inquiry [5]. This modality aligns most closely with the theoretical ideal of Vygotskian mediation: it is responsive to the individual's ZPD in real time, cannot be pre-scripted, and requires the teacher to develop sophisticated improvisational capacities alongside content expertise.

Written dialogic feedback — delivered through margin comments, portfolio dialogues, or annotated drafts — supports "reflective, asynchronous engagement with iterative revision cycles" [5]. Technology-mediated dialogic feedback using platforms such as Google Docs enables collaborative annotations and synchronous or asynchronous exchanges that may expand participation for students who are reticent in face-to-face settings [5]. Peer dialogic feedback — structured exchanges in which students provide and receive critique from each other — constitutes a distinct modality with its own dynamics, leveraging "learner-to-learner collaborative meaning-making" [5]. Each modality presents distinctive advantages and limitations, with face-to-face dialogue offering immediacy and interpersonal depth at the cost of scale and documentation.

The scoping review of dialogic formative feedback in higher education [6] found that the most common purposes documented in the literature were "promoting self-regulated learning and cognitive growth" (n=27 of 40 sources) and "achieving shared understanding of assessment criteria" (n=15), with group-based feedback settings most common (n=16). These findings suggest that the field has developed a relatively coherent understanding of why dialogic feedback is valuable pedagogically (self-regulation, metacognition, criterion understanding) but that the evidence on academic achievement outcomes specifically remains underdeveloped — a gap the review explicitly flags [6].

### The Resource-Intensity Problem

A consistent finding across the formative dialogic assessment literature is implementation resource intensity. The dialogic feed-forward study [4] documents that teacher-student one-to-one meetings in a 12-week unit constitute a substantial time investment, and the approach "fails to engage disengaged students who avoided meetings." The formative assessment literature broadly [53] acknowledges that formative assessment's flexibility requires teachers to possess sophisticated knowledge of student cognition, cultural backgrounds, and linguistic proficiency to reason accurately from evidence — capacities that require sustained professional development to develop. Inquiry-based learning assessment in K–12 contexts confirms these institutional challenges: "Factors such as class size, institutional alignment, and teacher education programs influence the implementation of inquiry-based learning, highlighting the need for professional development and a strong support network" [The Importance of K-12 Teachers' Approaches to Assessment](https://dspace.library.uvic.ca/bitstreams/028ff00f-5d0d-4e6a-82d2-3ad1bc3ccb25/download) [27].

---

## Socratic Assessment: Principles, Rubrics, and Evidence

### Design Principles and Implementation

Socratic seminar assessment operationalizes the dialogic teaching tradition's core commitments in structured, facilitated whole-class dialogue. The design principle is that collaborative inquiry through questioning — building, extending, challenging, and revising claims — constitutes both the learning activity and the assessment opportunity. Socratic questioning functions as a scaffold for the kind of reasoning that dialogic theorists identify as educationally central: hypothesis generation, evidence evaluation, perspective-taking, and logical extension.

The OSPI (Washington State) Socratic Seminar Guidelines and Rubric [Socratic Seminar Guidelines and Rubric — OSPI](https://ospi.k12.wa.us/sites/default/files/2023-10/socraticseminarguidelinesandrubric.docx) [1] provides a formal framework for Socratic seminar assessment, though the document format limits direct content extraction. The Northwest Association for Biomedical Research (NWABR) rubric [Socratic Seminar Rubric — NWABR](https://www.nwabr.org/sites/default/files/RubricsAssess.pdf) [2] employs trait-based assessment on a five-point scale covering: analysis and reasoning (text reference, logical organization, insightful connections); discussion skills (perspective recognition, paraphrasing, building on contributions); vocabulary and evidence use; and ethical reasoning and civility. The rubric specifies that at the highest level, "student's own thinking becomes more complex and thorough with added perspectives" and "Clearly references text to support reasoning. Demonstrates thoughtful consideration of the topic. Provides relevant and insightful comments, makes new connections. Demonstrates exceptionally logical and organized thinking" [2].

Both rubrics address the multidimensional character of Socratic seminar performance: oral communication quality is inseparable from content quality, and reasoning depth is assessed through its dialogic expression rather than in isolation. Notably, the research base uncovered no published validation or norming studies for either Socratic seminar assessment instrument — these rubrics appear to be practitioner-developed without accompanying psychometric validation in the peer-reviewed literature, representing a significant gap in the evidence base for this widely-used assessment form.

### Evidence of Learning Impact

Direct empirical evidence on Socratic assessment's impact on learning outcomes is embedded within Alexander's dialogic teaching research. The EEF trial involving 5,000 students demonstrated that "those whose teachers had received the dialogic teaching intervention made on average two months additional progress in tests in English, mathematics and science compared with their control group peers" after 20 weeks [40]. While this trial tested a broader dialogic teaching framework rather than Socratic seminar specifically, the questioning and dialogue practices central to Socratic methods are integral to Alexander's approach. The St Helen's School case study [42] illustrates Socratic questioning in practice — teachers posing etymological and cross-disciplinary questions that generate authentic inquiry — but as a qualitative case study, it does not generate quantitative effect estimates.

The AI-powered Socratic dialogue system from AIED 2025 [51], while primarily an assessment architecture, provides indirect evidence that structured Socratic questioning within an ECD framework can elicit and classify student reasoning in ways that align with human assessment expectations. The Expert Agent's Socratic questioning approach produced "cohesion analysis results indicating that a standardized score of cohesion (0–100) aligned with student speech acts as expected" [51], suggesting that the dialogic structure of Socratic questioning generates scoreable evidence of reasoning.

---

## Psychometric Validity and Measurement Challenges

### Reliability Coefficients in Oral and Dialogic Assessment

Oral and dialogic assessment has historically been viewed with skepticism by psychometricians concerned about inter-rater reliability, but recent evidence suggests that well-designed protocols can achieve acceptable reliability coefficients. In a study of EFL oral presentations using an analytical rubric, intraclass correlation coefficients showed single-measure ICC of .575 and average-measure ICC of .730 (p = .001), interpreted as "moderate to good inter-rater reliability" [Examining Rater Reliability When Using an Analytical Rubric for Oral Assessment](https://files.eric.ed.gov/fulltext/EJ1470720.pdf) [13]. Critically, generalizability theory analysis demonstrated that "a single well-trained rater achieved acceptable reliability (G-coefficient: 0.864)" when using a detailed analytical rubric [13]. Variance analysis revealed that "students accounted for 37.5% of total variance while rater variance alone was negligible (0.2%)," indicating that with proper training, student performance rather than rater idiosyncracy drives assessment variation [13].

In a multiphased rater training study for a fidelity assessment tool using Cohen's kappa and ICC, researchers achieved κ = 0.881 for occurrence/non-occurrence judgments (almost perfect agreement), ICC = 0.999 for talk analysis (excellent), and ICC = 0.867 for independent capacity rating (good) [Achieving Inter-Rater Agreement and Inter-Rater Reliability](https://pmc.ncbi.nlm.nih.gov/articles/PMC12033844/) [36]. These results were attributed to three critical success factors: "comprehensive multifaceted rater training incorporating didactic, practical, and applied modalities; integration of real-life client contexts and ambiguous cases in training materials; and precisely defined scoring criteria based on observable, quantifiable qualities" [36].

Cohen's kappa interpretation guidelines categorize values of 0.61–0.80 as substantial and 0.81–1.00 as almost perfect agreement [Interrater Reliability: The Kappa Statistic](https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/) [33]. However, for high-stakes assessment, some methodologists argue that "when kappa values are below 0.60, the confidence intervals are sufficiently wide that one can surmise that about half the data may be incorrect" — a demanding standard that highlights the distance between acceptable reliability for research purposes and what high-stakes summative use would require [33].

Generalizability theory provides the most powerful framework for oral assessment research because it "can concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results" [Generalizability Theory: A Practical Guide](https://www.sciencedirect.com/science/article/abs/pii/S0022440513001106) [14] — particularly valuable given the multiple facets (rater, task, occasion, setting) that contribute to variability in oral performance assessment.

### Rater Training and Consistency

The rater training literature identifies four core strategies applicable to oral and dialogic assessment: Rater Error Training (familiarizing raters with common biases including central tendency, halo effect, leniency, primacy/recency effects, contrast effect, similar-to-me bias, and stereotype bias); Performance Dimension Training (using behavioral examples to clarify criteria); Frame-of-Reference training (iterative practice with group discussion to calibrate raters — producing "the largest improvements in rating accuracy, with moderate to large effects"); and Behavioral Observation Training (developing skill in detecting key behavioral triggers) [Rater Training to Support High-Stakes Simulation-Based Assessments](https://pmc.ncbi.nlm.nih.gov/articles/PMC3646087/) [3]. Crucially, "rater training should be considered an ongoing activity. As with any other skill, training raters requires monitoring and skill maintenance through practice" [3] rather than a one-time workshop.

Analytical rubrics play a synergistic role with rater training. The EFL oral assessment study found that "detailed analytical rubrics and cooperation sessions between raters" together were crucial for improving reliability, and thematic analysis of rater reasoning processes "identified key themes influencing ratings: delivery (voice control, fluency, body language), content (structure, transitions, technical explanation), and visuals (clarity, professionalism)" [13] — concrete dimensions that analytical rubrics must explicitly address to achieve consistent scoring.

### The Standardization Debate

The measurement literature identifies a fundamental tension: the contextual responsiveness that makes dialogic assessment educationally distinctive is structurally in tension with the standardization that high-stakes summative assessment requires. A systematic review of oral assessments in higher education confirms that "the validity, reliability and capacity of the oral assessment to reduce academic integrity breaches were dependent on whether it has been designed, scaffolded, and implemented well" [The Validity, Reliability, Academic Integrity and Integration of Oral Assessments](http://www.iier.org.au/iier34/nallaya.pdf) [35]. Key validity concerns include "student anxiety (the most significant reported variable), disadvantages for English-as-additional-language (EAL) students, and personality-based performance differences favoring extroverts" [35] — factors that systematically introduce construct-irrelevant variance. Yet the review simultaneously found that "most students performed better in oral assessments than they did in written assessments and that this assessment inspired them to engage in more thorough preparation" [35], suggesting that the construct being measured — genuine understanding — may be better captured orally despite the reliability challenges.

The National Academy of Education's comprehensive analysis of large-scale assessment comparability frames the standardization issue in its broadest form: "comparisons are most defensible when the same assessment is given under substantively the same conditions to similar student samples at the same point in time. The legitimacy of comparisons thus becomes less certain as the assessment, the assessment conditions, student samples, and the time of administration diverge" [Comparability of Large-Scale Educational Assessments](https://naeducation.org/wp-content/uploads/2020/06/Comparability-of-Large-Scale-Educational-Assessments.pdf) [34]. For oral and dialogic assessment, where conditions vary by definition (different examiners, different dialogue trajectories, different prompting sequences), this principle creates a fundamental tension that no amount of rubric refinement fully resolves [34].

An international Delphi panel of experts on teacher evaluation reliability expressed skepticism about checklist-based solutions, finding consensus that "current competency frameworks focus too much on observable behaviours, neglecting more complex and essential aspects of teaching, such as professional dispositions and the teacher's impact on student engagement and learning" [Reliability and Consistency in Judging New Teacher Practices](https://soc-for-ed-studies.org.uk/wp-content/uploads/2024/11/Reliability-and-Consistency-in-Judging-New-Teacher-Practices_Final_edited.pdf) [37]. The panel explicitly rejected AI-driven assessment as a solution, "emphasizing judgement's inherent human dimension" [37], and proposed a Duplexity Model that "balances complementary tensions (standardization vs. contextualization, fairness vs. complexity)" through "standardized tools, shared criteria, co-designed frameworks, and continuous feedback loops" [37].

### Construct Validity and Score Comparability

Score comparability in oral assessment is affected by multiple sources of construct-irrelevant variance. The National Academy of Education analysis identifies language proficiency as a particularly potent threat in cross-population comparisons: "language that is hard for some students to understand can hinder comparability on tests not intended to measure language skills" [34]. For oral assessments in particular, pronunciation, accent, and nonstandard dialect features may systematically disadvantage certain demographic groups even when content knowledge is equivalent [35].

An important positive finding on construct validity is that the EFL oral assessment study found student performance accounted for 37.5% of total variance while rater variance was negligible (0.2%) with trained raters [13], suggesting that the construct of interest (student speaking ability) can dominate the variance structure when confounds are controlled. The medical education meta-analysis [28] provides additional construct evidence: the fact that face-to-face feedback consistently outperforms electronic feedback (SMD = 0.92 versus 0.63) and immediate feedback outperforms delayed feedback (SMD = 1.16 versus 0.40) is consistent with a construct interpretation of dialogic assessment as genuinely measuring and developing different competencies than standardized written formats.

---

## Conclusions

Conversation-based assessment encompasses a theoretically coherent and empirically developing field with strong roots in Vygotskian sociocultural theory and Bakhtinian dialogism, diverse methodological implementations from dynamic assessment to Socratic seminars, and an emerging AI-powered technology layer that is simultaneously expanding capability and introducing new validity and equity risks.

Several actionable conclusions emerge from the synthesis. First, the strongest theoretical and empirical case for conversation-based assessment is equity-based: for English Language Learners, students with disabilities, and other populations whose performance on standardized written tests is systematically contaminated by construct-irrelevant linguistic and cultural factors, oral and dialogic formats offer more valid access to the constructs education aims to assess. The reliability coefficient disparity (0.603 for ELLs versus 0.808 for native English speakers) in written standardized assessment [49], combined with the UC San Diego evidence that oral assessment reduces grade variability [15], makes the equity argument empirically grounded rather than merely intuitive.

Second, the effectiveness evidence, while insufficiently powered for definitive conclusions, consistently points in a positive direction. The medical education meta-analysis showing SMD = 0.92 for face-to-face versus 0.63 for electronic feedback [28], Alexander's EEF trial showing two months additional progress [40], the quasi-experimental evidence of benefits for structurally disadvantaged students from oral argumentation training [31], and the consistent positive findings across 23 ELT dialogic feedback studies [5] together constitute a cumulative pattern of support that justifies continued investment in conversational assessment practices.

Third, psychometric concerns about oral and dialogic assessment are real but tractable. The G-coefficient of 0.864 for a single well-trained rater using an analytical rubric [13], and the near-perfect ICC values achieved with multiphased training protocols [36], demonstrate that acceptable reliability is achievable for low- to medium-stakes purposes. High-stakes summative use remains contested and demands hybrid designs with trained human adjudication, shared criteria developed with stakeholder input, and ongoing rater calibration processes.

Fourth, AI-powered conversational assessment is technically feasible and advancing rapidly, with LLM-based systems achieving competitive correlation with human raters in constrained tasks (Pearson correlation 0.761–0.771 [18]), but falling short for spontaneous speech, facing documented bias risks, and now operating within a binding EU regulatory framework that classifies educational AI as high-risk. Responsible deployment requires transparency in training data and model logic, bias auditing across demographic groups, and human oversight at high-stakes decision points — conditions that most current commercial systems do not yet fully satisfy.

Fifth, the field urgently needs more rigorous primary research: randomized controlled trials directly comparing oral/conversational and written assessment formats with pre-registered outcome measures including effect sizes, longitudinal designs tracking knowledge retention and transfer, equity studies specifically examining outcomes for ELL and disability populations across assessment formats, and validation studies for widely-used Socratic seminar rubrics. The synthesis of dialogic theory with psychometric methodology — developing reliable and valid instruments that preserve the dialogic responsiveness that makes conversation educationally powerful — remains the field's central methodological challenge.

## Sources

[1] [[DOC] Socratic Seminar Guidelines and Rubric - OSPI](https://ospi.k12.wa.us/sites/default/files/2023-10/socraticseminarguidelinesandrubric.docx)
[2] [[PDF] 113 Socratic Seminar Rubric - NWABR.ORG](https://www.nwabr.org/sites/default/files/RubricsAssess.pdf)
[3] [Rater Training to Support High-Stakes Simulation-Based Assessments](https://pmc.ncbi.nlm.nih.gov/articles/PMC3646087/)
[4] [Dialogic Feed-Forward in Assessment: Pivotal to Learning ...](https://files.eric.ed.gov/fulltext/EJ1367808.pdf)
[5] [[PDF] Dialogic feedback in English Language Teaching (ELT) - ERIC](https://files.eric.ed.gov/fulltext/EJ1483229.pdf)
[6] [A scoping review of dialogic formative feedback practices in higher ...](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2025.1696703/full)
[7] [Oral Assessments: Benefits, Drawbacks, and Considerations](https://tlconestoga.ca/oral-assessments-benefits-drawbacks-and-considerations/)
[8] [A systematic scoping review of empirical research on ...](https://www.sciencedirect.com/science/article/pii/S0883035525002265)
[9] [DYNAMIC ASSESSMENT](https://www.cibtech.org/sp.ed/jls/2015/02/380-JLS-S2-390-SARA-DYNAMIC.pdf)
[10] [[PDF] Hasson,_Natalie.pdf - City Research Online](https://openaccess.city.ac.uk/id/eprint/1119/1/Hasson%2C_Natalie.pdf)
[11] [[PDF] The zone of proximal development in Vygotsky's analysis of learning](https://blogs.ubc.ca/vygotsky/files/2013/11/chaiklin.zpd_.pdf)
[12] [Vygotsky's Zone of Proximal Development: Instructional ...](https://files.eric.ed.gov/fulltext/EJ1081990.pdf)
[13] [[PDF] Examining Rater Reliability When Using an Analytical Rubric for Oral](https://files.eric.ed.gov/fulltext/EJ1470720.pdf)
[14] [Generalizability theory: A practical guide to study design ...](https://www.sciencedirect.com/science/article/abs/pii/S0022440513001106)
[15] [[PDF] Using Oral Assessments to Improve Student Learning Gains](https://peer.asee.org/using-oral-assessments-to-improve-student-learning-gains.pdf)
[16] [Investigating the role of AI-powered conversation bots in enhancing ...](https://www.nature.com/articles/s41599-025-05550-z)
[17] [Score accuracy, perceived validity, and oral peer feedback ...](https://www.sciencedirect.com/science/article/pii/S1475158525000360)
[18] [Natural Language-based Assessment of L2 Oral Proficiency using ...](https://arxiv.org/html/2507.10200v1)
[19] [[PDF] Leveraging ASR and LLMs for Automated Scoring and Feedback in ...](https://www.isca-archive.org/slate_2025/shankar25_slate.pdf)
[20] [Dialogic learning - Wikipedia](https://en.wikipedia.org/wiki/Dialogic_learning)
[21] [[PDF] A Dialogic Pedagogy: - UBC Library](https://ices.library.ubc.ca/index.php/criticaled/article/download/182238/182310/)
[22] [Insights from Intelligent Tutoring Systems and the Learning Sciences](https://arxiv.org/html/2405.04645v2)
[23] [A Comprehensive Review of AI-based Intelligent Tutoring Systems](https://arxiv.org/html/2507.18882v1)
[24] [[PDF] Automated test generation to evaluate tool-augmented LLMs as ...](https://aclanthology.org/2024.genbench-1.4.pdf)
[25] [What the Current State of Large Language Models in Education ...](https://arxiv.org/html/2507.02180v1)
[26] [A Quasi-Experimental Study of the Achievement Impacts of a ... - MDPI](https://www.mdpi.com/2227-7102/15/11/1422)
[27] [[PDF] The Importance of K-12 Teachers' Approaches to Assessment in an ...](https://dspace.library.uvic.ca/bitstreams/028ff00f-5d0d-4e6a-82d2-3ad1bc3ccb25/download)
[28] [a Systematic Review and Meta-analysis of Randomized Controlled ...](https://pmc.ncbi.nlm.nih.gov/articles/PMC8651958/)
[29] [Health promotion for oral health in children and adolescents](https://pmc.ncbi.nlm.nih.gov/articles/PMC12257711/)
[30] [Reading Interventions to Support English Learners with Disabilities ...](https://www.mdpi.com/2227-7102/15/2/223)
[31] [Merging Oral and Written Argumentation](https://www.mdpi.com/2227-7102/15/11/1471)
[32] [Evaluating large language models as raters in large-scale writing ...](https://www.sciencedirect.com/science/article/pii/S2666920X25001213)
[33] [Interrater reliability: the kappa statistic - PMC - NIH](https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/)
[34] [[PDF] Comparability of Large-Scale Educational Assessments](https://naeducation.org/wp-content/uploads/2020/06/Comparability-of-Large-Scale-Educational-Assessments.pdf)
[35] [[PDF] The validity, reliability, academic integrity and integration of oral ...](http://www.iier.org.au/iier34/nallaya.pdf)
[36] [Achieving Inter-Rater Agreement and Inter-Rater Reliability to ... - PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC12033844/)
[37] [Reliability-and-Consistency-in-Judging-New-Teacher- ...](https://soc-for-ed-studies.org.uk/wp-content/uploads/2024/11/Reliability-and-Consistency-in-Judging-New-Teacher-Practices_Final_edited.pdf)
[38] [Neil Mercer : Faculty of Education](https://www.educ.cam.ac.uk/people/staff/mercer/)
[39] [Students' use of exploratory talk to exercise critical thinking - PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC7471859/)
[40] [Dialogic Teaching - Robin Alexander](https://robinalexander.org.uk/dialogic-teaching/)
[41] [[PDF] DIALOGIC TEACHING IN BRIEF Robin Alexander](https://coleridgeprimary.org/wp-content/uploads/2019/11/Dialogc-teaching-in-brief-170622.pdf)
[42] [Feeding forward: A case study of dialogic assessment](https://my.chartered.college/impact_article/feeding-forward-a-case-study-of-dialogic-assessment/)
[43] [Comparing the Validity of Automated and Human Essay ...](https://www.ets.org/Media/Research/pdf/RR-00-10.pdf)
[44] [[PDF] Analysis of the Scoring and Reliability for the Duolingo English Test](https://dy8n3onijof8f.cloudfront.net/media/resources/standards/scoring.pdf)
[45] [The EU AI Act: Implications for Ethical AI in Education.](https://swisscyberinstitute.com/research/eu-ai-act-implications-ethical-ai-education/)
[46] [Ethical and regulatory challenges of Generative AI in ...](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2025.1565938/full)
[47] [Effects of Oral Exams on Entry-Level STEM Mathematics Students](https://docs.lib.purdue.edu/ijtlhe/vol36/iss1/7/)
[48] [Why Students Cheat and How Understanding This Can Help ... - PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC10653228/)
[49] [[PDF] Beyond Language Barriers - DigitalCommons@Hamline](https://digitalcommons.hamline.edu/cgi/viewcontent.cgi?article=2121&context=hse_cp)
[50] [[PDF] Guidelines for the Assessment of English Language Learners - ETS](https://www.ets.org/pdfs/about/ell-guidelines.pdf)
[51] [[PDF] An LLM-enhanced Multi-Agent Architecture for Conversation-Based ...](https://xinyinghou.org/assets/publications/papers/aied2025.pdf)
[52] [[PDF] LLM Agents for Education: Advances and Applications](https://aclanthology.org/2025.findings-emnlp.743.pdf)
[53] [[PDF] Understanding Formative Assessment - WestEd](https://www2.wested.org/www-static/online_pubs/resource1307.pdf)
[54] [Dialogic Inquiry as Collaborative Action Research - Academia.edu](https://www.academia.edu/28844382/Dialogic_Inquiry_as_Collaborative_Action_Research)
[55] [Dialogic Inquiry - Cambridge University Press & Assessment](https://www.cambridge.org/core/books/dialogic-inquiry/C64C8553C45813842441DFDEEE338C68)
[56] [Dynamic Assessment | Springer Nature Link](https://link.springer.com/rwe/10.1007/978-3-319-02261-1_18)
[57] [[PDF] How to implement dynamic assessment to enhance l2 development](https://files.eric.ed.gov/fulltext/EJ1324122.pdf)
[58] [[PDF] arXiv:2404.17460v1 [cs.CL] 26 Apr 2024](https://arxiv.org/pdf/2404.17460)
[59] [The Future of Intelligent Tutoring Systems for Writing - Springer Link](https://link.springer.com/chapter/10.1007/978-3-031-36033-6_23)
[60] [[PDF] A score concordance study between IELTS Academic and TOEFL iBT](https://ielts.org/cdn/ielts-research-reports/aligning-scores-of-language-proficiency-test-ikeda-et-al-2025.pdf)
[61] [[PDF] Towards explainable automatic spoken language assessment](https://www.isca-archive.org/slate_2025/marchal25_slate.pdf)
