UPTLC 2024

We will demonstrate a language model-powered tool developed to assess executive summaries in students' technical reporting deliverables for industry-sponsored senior capstone design projects. Our web-based tool uses DistilBERT [1], one of numerous open-source language models that can power AI tools like chatGPT, Google Bard, etc. Our tool automatically scores key "rhetorical moves (RMs)" in executive summaries within a second, offering objective, repeatable, and helpful feedback. Rhetorical moves are organizational patterns writers employ to persuade an audience. Currently, faculty advisers use rubrics to assess the presence of five RMs in the executive summary paragraphs delivered by students. These RMs are expected to be organized one per paragraph, viz., "problem," "research," "uses," "requirements," and "deliverables." Our work is at the intersection of "undergraduate writing" through which students construct and evaluate knowledge in academia [2], "professional engineering communication" to perform and realize professional functions [3,4], and "AI-based assessment of learning [5]."

We will measure the RM-labeling accuracy of the tool through a standard interrater reliability analysis in the future. The potential benefits of our work include quick and real-time formative feedback to students, adaptability of our framework to assess RMs in writing, and the ability to parse and label RMs in large volumes of text in minutes, thereby reducing lead times and for large-scale analysis of writing patterns for interventions and program continuous improvement. Our work can build a bridge between subjective and variation-prone instructor judgment and repeatable objective assessment of professional communication skills.

[1] Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
[2] Hardy, J. A. (2020). Undergraduate writing. In The Routledge Handbook of Corpus Approaches to Discourse Analysis (pp. 235-251). Routledge.
[3] K. Kong, Professional discourse. Cambridge University Press, 2014.
[4] M. Leung, “Engineering discourse,” in The Routledge Handbook of Corpus Approaches to Discourse Analysis. Routledge, 2020, pp. 376 – 393
[5] B. du Boulay, A. Mitrovic, and K. Yacef, Handbook of Artificial Intelligence in Education. Edward Elgar Publishing, 2023.