Inglés (pdf)
Articulo en XML
Referencias del artículo
Traducción automática
Enviar articulo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
1. Introduction
1.1 Background
The advent of novel educational technologies frequently elicits powerful emotions ranging from fear and apprehension to irrepressible joy. However, the possible implications of writing and using Artificial Intelligence (AI) in English as a Second Language (ESL) settings remain unclear. First, learning to write is viewed as a creative process that should resemble real-world tasks. As a result, students should be allowed to use the same technological tools typically found in out-of-class contexts (Oh, 2020). Not only does AI assist writers, but it can also replace them. Artificial intelligence now writes sentences, paragraphs, or even essays in seconds and is virtually error-free regarding grammar and spelling. In addition, many professors are unaware of the possible advantages and disadvantages of using AI as a writing aid. This lack of awareness may hinder students’ writing development. For example, students could utilize writing models and tailored materials. However, they could, intentionally or unintentionally, commit plagiarism by passing off AI-generated text as their own. Finally, students’ writing development may also be hindered by inconsistent marking. Heterogeneous grading across courses or academic levels may confuse students, impact their grades, and diminish their writing motivation.
Currently, the international community has devoted increasing attention to AI and its capabilities. Some of this attention has been directed toward academic writing, often labeling the use of AI as academic dishonesty. However, the use of AI-generated texts in educational settings has received little to no attention, especially in ESL settings where learning to write requires practice, time, and patience. Many professors are unaware of the possible implications of unrestricted access to AI technologies (Martín-Marchante, 2022). In contrast to AI, calibration has received growing attention in the past few years. Evaluators have attempted to increase the reliability of graded activities through better test designs and calibration. However, despite decades of research on calibration, its implementation in the core writing courses of the English and English teaching majors has been less than satisfactory.
The research findings will directly benefit two populations. On the one hand, the university and professors will incorporate calibration training to improve clarity and reliability in writing assessments, thereby adding objectivity to an often subjective evaluation process (Gunnell et al., 2016; Ricker-Pedley, 2011; Sundqvist et al., 2020). As a result, adequate and ongoing writing assessment training benefits both students and the curriculum. On the other hand, students will have access to a fairer, less subjective, and more consistent evaluation throughout their major. This will make students less anxious about writing assessments and establish more precise goals for them to achieve.
The specific objectives of this article are to compare ESL student-created writing to AI-generated writing at a general level and to examine the role of calibration in ESL writing courses at a Costa Rican public university.
2. Literature Review
In recent years, writing instruction and assessment have been the subject of extensive research. However, to the best of the researcher’s knowledge, experiments that compare and contrast ESL students’ writing to AI writing are scarce. This review of the literature is not intended to be exhaustive. Nevertheless, it explores some basic concepts to support the arguments derived from this study.
2.1 Definition and Uses of AI
Artificial intelligence is the simulation of human intelligence in machines. These machines are programmed to think and learn like humans or to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns, learning, and problem-solving tasks (Arora, 2022; Cameron, 2019; Kent, 2022). This can also involve comprehending natural language, recognizing objects and sounds, making decisions, and reproducing information in various forms (Cameron, 2019; Zimmerman, 2018). Artificial intelligence can be subdivided into two main categories: narrow or weak AI. Weak AI is designed to perform a specific task. General or strong AI, on the other hand, can perform any intellectual task that a human can (Cameron, 2019).
As previously stated, narrow or weak AI, also known as “applied AI,” refers to artificial intelligence systems designed to perform a particular task or set of tasks. These systems are trained to perform specific tasks by being fed large volumes of data, which they then utilize to make judgments, perform actions, or accomplish other tasks (Gulson et al., 2022). Examples of narrow AI include virtual assistants like Apple’s Siri and Amazon’s Alexa, which are designed to understand and respond to voice commands. Other examples include self-driving cars, which are designed to navigate roads and avoid obstacles. Narrow AI can effectively complete specific tasks; however, it is limited in its ability to adapt to new situations or perform tasks outside of its specific area of expertise. It is also not self-aware; it does not have consciousness or a sense of self.
Depending on their basic structure, AI can also be subdivided into three different types:
Rule-based AI: It follows predetermined rules to perform a task. For example, a rule-based AI system might be programmed to recognize and classify objects in an image based on specific features (Bourg and Seemann, 2004; Kochmar, 2022; Raynor, 2009).
Machine learning: It allows a system to learn and improve over time without being explicitly programmed. Machine learning algorithms use data to “train” the system to recognize patterns and make decisions based on those patterns (Bernard, 2021; Jones, 2018; Raynor, 2009).
Deep learning: It is a subset of machine learning that involves training artificial neural networks on a large dataset. These networks are inspired by the structure and function of the human brain and are able to learn and make decisions on their own (Jones, 2018; Roberts, 2022).
These types of AI are representative of what AI can currently do. As can be seen, the progression from a rule-based AI that is more dependent on humans to a deep learning stage in which machines can act and use language more like humans should generally raise awareness in society, as well as in academic and educational settings.
2.2 Natural Language Processing
When discussing computer language recognition or production, the user deals with Natural Language Processing (NLP). It is a branch of AI that deals with the interaction between computers and human language and seeks to guarantee a proper interaction among all the parties involved (Kochmar, 2022; McRoy, 2021). The goal of NLP is to enable computers to understand, interpret, and generate human language so that it is both meaningful and useful. Natural Language Processing can be used for many applications, including speech recognition, text-to-speech synthesis, machine translation, sentiment analysis, and text summarization.
Natural Language Processing relies on linguistics, computer science, and machine learning techniques to analyze and understand human language (Raaijmakers, 2022). It involves several steps, including tokenization, which breaks down text into individual words or phrases, and parsing, which analyzes the grammatical structure of the text. Natural Language Processing also involves using algorithms and models to determine the meaning of texts, such as word embeddings or neural networks (Kochmar, 2022; McRoy, 2021).
Recently, NLP has advanced to the point where it can quickly generate original, largely trustworthy content. For example, chatbots are trained on a massive amount of text data and can generate responses to a wide range of topics, including answering questions, generating text, and carrying out conversations. The model is designed to be flexible, so it can be fine-tuned for specific use cases and domains, making it useful for various applications, such as customer service chatbots and personal assistants (Adamopoulou and Moussiades, 2020; Luo et al., 2022). Content generators, or AI-powered writing generators, are computer programs or tools that automatically produce written, visual or multimedia content for websites, advertisements, and other forms of media (Giansiracusa, 2021; Sharples and Pérez y Pérez, 2022). Although both systems are designed to generate responses to a prompt, the role of content generators focuses on producing text or variations of it based on a single prompt, while a chatbot may need to remember a previous input to decide on a possible answer.
One of the critical goals of AI research is to create systems that can learn and adapt on their own rather than being specifically programmed for every task. Artificial intelligence has the potential to revolutionize many industries, including healthcare, finance, transportation, and education (Clark, 2020; Hamdan et al., 2021; Kent, 2022; Lasry and Kobayashi, 2018; Popenici, 2023; Yu and Yu, 2021). However, it also raises ethical concerns, such as the possibility of job loss, the need for appropriate safeguards to prevent unintended consequences, plagiarism, and a lack of human creativity (Holmes and Porayska-Pomsta, 2023; Johnston, 2023; Lasry and Kobayashi, 2018; Roumate, 2023; Srinivasan, 2018; Tzen and Moquet, 2018).
2.3 Writing Assessment
Writing assessment is the process of evaluating and scoring an individual’s written work in terms of its quality, content, style, grammar, and other language skills (Nation, 2009). These categories may vary from evaluation to evaluation, and professors may weigh them differently according to their specific requirements or students’ needs (Brown and Lee, 2015). For some raters, not all errors deserve the same attention (Reid, 2006), and language programs should emphasize what students need to know to communicate effectively in written form (Adler-Kassner and O’Neill, 2010).
Since writing is one of the most challenging skills for students to master (Campbell, 2019; Dunn, 2021; Nosratinia and Razavi, 2016; Tillema, 2012), evaluators should consider several factors when designing writing assessment tasks. For example, according to Coombe et al. (2007) and Hyland (2019), writing assessment tasks should include a well-designed rubric, a prompt, an expected response, and a post-task evaluation where the professor reflects upon the writing task. This task design promotes a fair and orderly valuation of students’ writing. Furthermore, other authors emphasize writing construct dimensions in which students should also be aware of extra-linguistic aspects such as context and purpose, audience awareness, and genre conventions (Sparks et al., 2014). Some of these aspects are not usually attributed to machines or AI, as will be explained later. A third line of thought views writing as a process. From this perspective, writing is often the result of a process of thought, design, and revision that requires special skills that not all speakers develop naturally (Brown and Lee, 2015). This viewpoint stresses behaviors that are typically associated with humans and, to some extent, excludes writing created by AI.
Another factor that deserves special attention is the use of rubrics. Rubrics have become a staple in writing assessments, and their use is widely recommended (Ferris and Hedgcock, 2023; Harmer, 2011; Li, 2022; Shabani and Panahi, 2020). They lower students’ anxiety (Arindra and Ardi, 2020), increase reliability, and avoid bias (Glass, 2005). Rubrics are often classified as analytic or holistic. Analytic rubrics assign markings to separate descriptors, such as grammar, vocabulary, or punctuation. The weight each descriptor receives may be equal or variable. On the other hand, when using holistic marking, raters assign an overall grade to students’ written work (Carr, 2000; Coombe et al., 2007; Nation, 2009; Reid, 2006). Although holistic rubrics are time-saving and straightforward, an analytic rubric is usually preferred since it provides specific feedback to students (Ghalib and Al-Hattami, 2015; Ma, 2022; Peaci̇, 2020).
Finally, machine scoring programs and Intelligent Tutoring Systems have been developed and used by educators over the last few decades (Ericsson and Haswell, 2006). According to McAllister and White (2006), computerized writing assessment has its beginnings in Ellis Page’s Project Essay Grade. In this study (Page, 1966) reported that the results provided by the computer could be used to predict human ratings. Similar results were later obtained from subsequent studies where some human scoring disadvantages, such as slowness and marking inconsistency, were also pointed out (Page and Dieter, 1968). By 1995, computers started to demonstrate a better judgement than their human counterparts (Page and Petersen, 1995). Despite this growing evidence, the academic community, particularly writing instructors and composition experts, has largely remained silent on the topic of machine scoring (Ericsson and Haswell, 2006) and the benefits it may bring over human raters.
2.4 Calibration and reliability
Assessment takes into account various factors to guarantee a fair and dependable evaluation process. These factors include validity, reliability, usefulness, practicality, and transparency (Brown and Abeywickrama, 2019; Coombe et al., 2007; Fulcher, 2010). For the purposes of the present study, only reliability will be explored. Reliability deals with the consistency of test scores (Brown and Abeywickrama, 2019; Coombe et al., 2007), and it can be divided into two subcategories: intra-rater and inter-rater reliability.
Intra-rater reliability refers to the consistency or stability of measurement results obtained by a single rater when assessing a phenomenon multiple times (Scheel et al., 2018). In other words, it measures the degree to which a single rater’s assessments are consistent over time. For example, if the same rater assesses similar groups in different places or over time, neither time nor place should affect the results. Intra-rater reliability deals more with the evaluators and their environment and mental or physical state, among other factors.
On the other hand, the results of two professors rating the same group of students should be consistent. When this does not happen, inter-rater reliability might be problematic. Inter-rater reliability assessment is a method used to evaluate the consistency of ratings or scores assigned by multiple evaluators, such as human raters or algorithms (Gwet, 2014). It measures the degree of agreement between raters on the scores assigned to a set of items or tasks, such as test questions, job performance evaluations, or oral or written productions. Inter-rater reliability assessment aims to ensure that the ratings or scores are consistent and accurately reflect the performance being evaluated, even when different evaluators are involved. Despite its importance in evaluation settings, inter-rater reliability has received little attention (Wilhelm et al., 2018).
Calibration in assessment refers to the process of ensuring that scores or ratings assigned to test items, tasks, or candidates are consistent and accurate across different raters or evaluators (Congdon and McQueen, 2000; Wendler et al., 2019). Although calibration is not one of the cornerstones of evaluation, it seeks to attain some of them, especially during grading. Calibration improves scoring accuracy and the reliability of ratings (Gunnell et al., 2016; Ricker-Pedley, 2011). This process helps to reduce the impact of subjective biases and ensure that ratings or scores are reliable and valid indicators of performance (Gunnell et al., 2016; Sundqvist et al., 2020). Calibration may involve training raters to use consistent standards, providing them with clear scoring guidelines, and regularly evaluating their performance through inter-rater reliability assessments (Ferris and Hedgcock, 2023). Calibration is a multi-phase process often requiring multiple rounds of recalibration to ensure consistency. Additionally, it plays a crucial role in test validation. The goal of test validation is to establish the credibility of test scores by confirming that the test is well-founded (Weir, 2005) and accurately measures what it is designed to assess (Coombe, 2007). Both calibration and test validation are essential for achieving fairness and generating reliable data.
3. Methodology
The following section describes the study’s methodological framework.
3.1 Approach
This study used a quantitative approach since it involves the collection and analysis of data that can be analyzed statistically. Quantitative research uses structured methods such as surveys, tests, or experiments to collect data, which are then usually analyzed through statistical methods (Creswell, 2019; Mackey and Gass, 2016; Mertler, 2019). The results of this type of research are typically presented in the form of statistical models, graphs, or tables.
Its design is quasi-experimental since it bypasses the need for random assignment. Instead of random assignment, quasi-experimental designs generally enable researchers to manage the allotment to the treatment condition. This is typically followed by the application of a specific matching technique to construct a control group that closely aligns with the characteristics of the treatment group (Creswell, 2019). The ultimate objective is to fabricate two groups that, barring the treatment condition, are analogous on all pertinent attributes (Mackey and Gass, 2016; Mertler, 2019).
It also uses an exploratory type of research because this type of research is utilized when the study topic is very novel and there is not much prior information available (Hernández Sampieri et al., 2010). Its primary objective is to familiarize oneself with the phenomenon or concept and gain a better understanding of it (Hernández Sampieri et al., 2010). Exploratory research, in general, is more open and flexible because its purpose is to explore the terrain, discover new variables and problem formulations, and formulate more precise hypotheses.
3.2 Unit of analysis
This study includes eight Costa Rican university students currently taking their last writing course in English or English teaching major at a public university. The researcher visited students’ writing classes to invite them to participate. The participants were selected because they were currently enrolled in an advanced writing course. The final list included eight students who agreed to participate in the study. Participants received monetary compensation of $10 and a small bag of snacks for their collaboration. All participants speak Spanish as their first language. Of the eight students who wrote their paragraph, four (50%) were female and four (50%) were male. No student chose to identify as non-binary or did not respond to the question. The eight students (100%) reported being between the ages of 18 and 24. All students are native Spanish speakers who are studying ESL. Regarding studies, students enrolled in the BA in English (n = 6, 75%) or English teaching (n = 2, 25%). Both majors share the same core language courses, including writing courses.
Raters were Costa Rican university professors who taught writing skills to ESL students at the same public university. All professors had more than ten years of experience teaching writing courses. The researcher sent eleven professors an electronic invitation. The final list of professors included ten raters who agreed to participate in the study; one professor did not reply to the invitation. The professors did not receive any compensation for the collaboration. Furthermore, of the ten professors who collaborated in the grading process, six (60%) were female and four (40%) were male. All professors (100%) work in an ESL BA program and have over ten years of experience teaching English to university students. In terms of their academic pursuits, two professors have a Ph.D. (20%), while the remaining eight professors (80%) have a Master’s degree related to language teaching.
3.3 Data collection
3.3.1 Materials
The materials include written consent, a writing prompt created by the researcher (see Appendix 1), a rubric, and sixteen paragraphs, eight of which were created by students and eight generated by AI. The written consent was sent to students electronically. A checkbox labeled “agree to the terms and conditions” was included to certify voluntary participation in the study. The prompt (see Appendix 1) was created considering criteria such as not requiring specialist background knowledge, being accessible to students, and being clear and unambiguous (Coombe et al., 2007). The rubric was adapted from Booth (n.d.) and then revised by two professors with experience designing and using writing rubrics. The rubric included five criteria (content, organization, sentence structure, mechanics, and vocabulary), a set of numerical and descriptive labels (Masterful, 5 points; Skilled, 4 points; Able, 3 points; Developing, 2 points; Novice, 1 point; and Unacceptable, 0 points), and their corresponding descriptors. The study used sixteen paragraphs to simulate a small composition class. Paragraphs were evenly divided into human-created and AI-generated. Each paragraph was around ten sentences long. All paragraphs were standardized in terms of format (font type, font size, alignment, and spacing). However, they were not modified in any other manner.
3.3.2 Procedure
This study used a language elicitation data collection process. The researcher created a simple yet informative prompt for students to write a paragraph and adapted the rubric to match five general criteria. The rubric did not include aspects modified by the researcher (e.g., format) or not generated by AI (e.g., title). Afterward, the rubric was revised by two professors experienced in writing courses. In the case of the students, each student wrote one paragraph of around ten sentences following the prompt given and in a place with minimal noise and no distractions. For this task, students were given 40 minutes. No student asked for extra time or reached the time limit. Students did not have access to any printed or electronic resources to aid them during their writing. To minimize the researcher’s interference, they typed their paragraph directly on the computer. The AI-written texts were created using copy.ai. At the time of writing, Copy.ai was one of the newest and most reliable writing generators. In addition, Copy.ai is a free AI-powered writing generator, which allowed anyone who use it. Paragraphs created with this application were randomly chosen based on their length and not on content or other characteristics. Therefore, students’ and AI-written paragraphs were uniform and did not bias raters. Eight AI-generated paragraphs were chosen to match the number of human-created paragraphs. All paragraphs were standardized in terms of format (font type, font size, alignment, and spacing) and were based on the prompt provided by the researcher (see Appendix 1). However, they were not modified in any other manner.
The ten professors were asked for their collaboration via email. In this stage, they were able to see a sample paragraph (not included to be evaluated) and the rubric. Once they accepted to participate, they received the paragraphs, rubric, and detailed instructions. They were given two months to grade the paragraphs. Paragraphs were given an alphanumeric ID and randomly organized. Before and during this stage, the researcher asked professors to grade the sixteen paragraphs; however, the inclusion of AI-generated paragraphs was not disclosed. The study was conducted from August 2022 to March 2023.
3.4 Data Processing and Analysis
The specific methods of analysis employed were chosen to best address the research questions posited at the outset of this study. Descriptive statistics were selected since they provide a snapshot of the data at hand through measures of central tendency and measures of variability (or dispersion). Some measures of central tendency included in this paper comprise the mean (average), median (the middle value when data is ordered from lowest to highest), and mode (the most frequently occurring value). These measures give a center point of data distribution. Each paragraph was graded following a five point criteria which included content, organization, grammar, mechanics, and vocabulary. These criteria were selected since they are widely used in composition courses and are familiar to the evaluating professors.
The data set was subjected to computational analysis using Microsoft Excel 2016. The data was derived from professors’ markings on students’ paragraphs. Grades were organized in various data sheets according to the intended analysis (grades for AI-created paragraphs, grades for human-created paragraphs, and sub-category comparisons). As previously mentioned, the analysis included descriptive statistics, where nominal data, percentages, and the standard deviation, among other basic statistics, were performed to compare students’ results using the data analysis tools add-in.
4. Analysis of the Results
The following description presents the study’s results in three distinct sections. The first section includes scores assigned to each paragraph and contrasts the five analytic criteria used to assess the paragraphs. The second section compares the consistency of the raters’ marks.
4.1 Analysis of Participants’ Obtained Scores
In order to examine participants’ writing, professors graded 16 paragraphs. Half of the paragraphs were written by advanced ESL students. The other eight paragraphs were written using Copy.ai (2022), an artificial intelligence. For the purposes of this study, human writers and AI are referred to as “participants.” No paragraph was retained from the analysis, and no artificially generated paragraph was modified. To provide professors with sufficient context, they were instructed to grade the paragraphs using a rubric and to imagine this was a diagnostic test. Table 1 summarizes the main findings of this section.
Table 1 Summary of participants’ paragraph scores: mean and standard deviation, Costa Rica, 2023
| Paragraph Source | Min. | Max. | X̄ | SD |
| Human | 6,22 | 9,09 | 7,56 | 0,79 |
| Artificial Intelligence | 7,28 | 8,01 | 7,61 | 0,21 |
Source: Compiled by the author based on survey responses.
Note.N = 16. Min. = Minimum; Max. = Maximum; X̄ = arithmetic mean; SD = Standard Deviation
As can be seen, AI participants outperformed human participants by a small margin. These results suggest that, at least in short writing samples, AI-generated paragraphs can easily compete with those created by humans. The standard deviation of students’ scores indicates a moderate level of variability in the dataset. Although the scores are not all identical, they are not evenly distributed across the entire 1 to 10 scale, and they tend to cluster somewhat close to the mean. However, AI scores present a lower level of variability. These scores are more similar and are not evenly distributed across the full range of the scale. Finally, humans also obtained the highest and lowest scores of the 16 samples. This may also suggest that AI-generated paragraphs are more standard and share common features, while human abilities vary from individual to individual.
Additionally, some important information can be extracted by comparing participants’ grades. First, only two students outperformed at least one AI sample. Second, only one student scored above 9; however, only one student scored below 7, which is a non-passing grade in this context. Most grades (n = 13) were in the 7 to 8 band, which, as seen before, indicates a low level of dispersion. Table 2 compares human and AI scores from the highest to the lowest.
Table 2 Participants’ average scores: points and grades, Costa Rica, 2023
| Sample | Human | Artificial Intelligence | ||
| Average Points | Average Grade | Average Points | Average Grade | |
| Sample 1 | 22,73 | 9,09 | 20,02 | 8,01 |
| Sample 2 | 19,52 | 7,81 | 19,29 | 7,71 |
| Sample 3 | 19,03 | 7,61 | 19,06 | 7,62 |
| Sample 4 | 18,98 | 7,59 | 19,06 | 7,62 |
| Sample 5 | 18,78 | 7,51 | 18,95 | 7,58 |
| Sample 6 | 18,49 | 7,40 | 18,84 | 7,53 |
| Sample 7 | 18,09 | 7,24 | 18,74 | 7,49 |
| Sample 8 | 15,56 | 6,22 | 18,19 | 7,28 |
Source: Compiled by the author based on survey responses.
Note.Each set of participants contributed eight paragraphs. Numbers depict the average grade from the criteria, where 0 was the minimum number of points, and 25 was the highest. Grades appear on a scale of 0 - 10.
Some data indicates variations in outcomes based on the criteria used. For example, students achieved better results in content and organization. On the other hand, AI exhibited superior performance in grammar, mechanics, and vocabulary. This may imply that when written by humans, content and organization remain more appealing. Usually, content and organization do not follow specific rules; therefore, humans may incorporate more varied elements that appeal more to them. In contrast, elements such as grammar, mechanics, and vocabulary follow stricter rules that can be objectively verified and, therefore, programmed for AI to use. Yet, AI may still use wordy structures or exhibit a restricted range of vocabulary, which may affect professors’ perceptions when grading each paragraph. Table 3 summarizes human and AI criteria scores.
Table 3 Participants’ average scores per criterion, Costa Rica, 2023
| Paragraph Source | Content | Organization | Grammar | Mechanics | Vocabulary |
| Human | 3,76 | 3,88 | 3,76 | 3,84 | 3,66 |
| Artificial Intelligence | 3,64 | 3,54 | 3,92 | 4,12 | 3,78 |
Source: Compiled by the author based on survey responses.
Note.Each criterion presents the average score obtained on a 0 - 5 scale.
As previously stated, human and AI writing share many characteristics and are usually indistinguishable. However, when analyzing written production, specific patterns in dispersion, grade average, and scores per criterion show significant differences.
4.2 Analysis of Raters’ Scores
When examining a particular piece of writing, professors have to take into account and are subject to several variables. Although professors across disciplines share some common traits, they also have unique qualities based on the subject they teach, their personal philosophies, or their teaching methods, among other characteristics. However, professors should also strive toward fairness and objectives when assessing students. In this sense, any institution should strive to train its staff to be consistent and accurate when grading. Taking this into account, this section focused on raters’ average grades. Table 4 summarizes the most important findings of this section.
Table 4 Summary of raters’ average given scores, Costa Rica, 2023
| Source | Min. | Max. | Mode | X̄ | SD |
| Rater 1 | 6,00 | 9.60 | 7.20 | 7.55 | 0.95 |
| Rater 2 | 8.16 | 9.92 | 9.88 | 9.56 | 0.48 |
| Rater 3 | 4.40 | 8.20 | 7.60 | 7.18 | 0.99 |
| Rater 4 | 6,00 | 10,00 | 9.20 | 8.25 | 1.19 |
| Rater 5 | 5.20 | 7.60 | 5.20 | 6.43 | 0.93 |
| Rater 6 | 6,00 | 10,00 | 9.40 | 8.63 | 1.13 |
| Rater 7 | 4.80 | 8.80 | 7.20 | 6.70 | 1.18 |
| Rater 8 | 6.40 | 9.20 | 6.80 | 7.60 | 0.89 |
| Rater 9 | 6.40 | 9.60 | 6.40 | 7.59 | 1.05 |
| Rater 10 | 4.40 | 9.20 | 5.60 | 6.35 | 1.48 |
Source: Compiled by the author based on survey responses.
Note.Each rater graded 16 paragraphs and contributed eight paragraphs. Min. = Minimum; Max. = Maximum; X̄ = arithmetic mean; SD = Standard Deviation
The analysis of these data reveals some important grading behaviors or practices. Except for one professor, all raters graded at least one student with a non-passing mark. However, some professors scored students severely while others were more lenient, considering that a difference of up to two points remains important. Equally important are some differences found in the maximum grades, where the greatest gap surpasses the two points. Several professors consider some participants excellent or outstanding, while others consider them average.
This lack of inter-rater reliability can also be observed when analyzing the mode and the mean. On average, three raters tend to assign non-passing grades to students. Four raters graded students in the range of seven to eight marks. The other three raters graded students’ writing above eight. This includes one case where marks were comparatively high (mode = 9.88, X̄ = 9.56).
In addition, the standard deviation, coupled with the other data, reveals crucial information. For example, Rater 2 has the lowest standard deviation and the high scores. This trend suggests that this professor consistently tends to grade students less severely than other colleagues. In contrast, Rater 5 presents consistent but more severe behavior when grading. Finally, Rater 10 has the highest level of dispersion when grading; however, the given grades tend to score low but acknowledge certain students’ written work.
To exemplify this further, Table 5 presents the nominal results of the four largest grade differences.
Table 5 Summary of raters’ average given scores, Costa Rica, 2023
| Sample | Min. | Max. | Range |
| Sample 1 | 4.44 | 9.88 | 5.48 |
| Sample 2 | 4.44 | 9.72 | 5.32 |
| Sample 3 | 4.48 | 9.72 | 5.24 |
| Sample 4 | 4.8 | 9.56 | 4.76 |
Source: Compiled by the author based on survey responses.
Note.Each rater graded 16 paragraphs. Min. = Minimum; Max. = Maximum; X̄ = arithmetic mean; SD = Standard Deviation
As these data show, depending on the professor, a student may completely fail an assignment or get an almost perfect mark on another. These four samples are the highest, according to the range. In addition, all of them belong to human participants. Considering the range of these grades, this information evidences the need to calibrate professors and offer them the necessary guidance to increase their accuracy and objectiveness when grading students’ writing.
The analysis provided valuable insights into how professors perceive students’ writing and the possible similarities and differences between it and AI writing, shedding light on educational areas that demand greater attention and a more direct approach. These findings also highlight specific aspects of evaluating written works and the need for calibration to provide more accurate and objective feedback when grading writing.
5. Conclusions
Artificial intelligence has gained popularity in recent years, and its capabilities have, for good or bad, already permeated educational settings. The present analysis concludes that paragraphs generated by artificial intelligence performed slightly better than those written by human participants in the study. The results indicate that AI-generated paragraphs can effectively compete with human-created ones in short writing samples. While there was a moderate level of variability in the students’ scores, the scores tended to cluster close to the mean, suggesting a consistent level of performance among the human participants. On the other hand, the AI scores exhibited less variability, indicating a more standardized and consistent performance. Moreover, the AI-generated paragraphs displayed fewer extremes in scores compared to human writing, suggesting a more standardized and predictable outcome.
Furthermore, the analysis revealed differences in scores based on the grading criteria used. Human participants achieved higher scores in content and organization, indicating that human-written paragraphs may possess more varied elements that appeal to readers. In contrast, AI performed better in grammar, mechanics, and vocabulary, which are areas governed by stricter rules. However, AI-generated paragraphs may still exhibit wordy structures and limited vocabulary, potentially influencing professors’ perceptions when grading. These findings suggest that content and organization may be more appealing in human writing, while AI excels in rule-based aspects of writing. This opens up new possibilities for assessing student writing. In line with Nation's (2009) suggestion that multiple markers should evaluate student work to ensure reliability, a hybrid approach combining both human and machine assessments could offer a viable solution for enhancing the quality and reliability of grading and feedback from professors.
The analysis of raters’ scores highlighted variations in grading practices among professors. While some professors graded more leniently, others scored more strictly, resulting in discrepancies in students’ final grades. The results obtained by students vary greatly depending on the professor who evaluated each paragraph. As pointed out by Gunnell et al. (2016) and Ricker-Pedley (2011), this indicates the need for calibration in language courses. Although incorporating human-created writing is crucial, AI-written samples could serve to model what is expected from students according to their proficiency level and guide professors toward a more objective, efficient, and fair grading of students’ writing.
Another important factor to consider is plagiarism (Smith, 2022). Although several techniques can be employed to detect computer-generated writing (Abd-Elaal et al., 2022), professors may be unaware of their use or have difficulty asserting that a piece of writing was completely or partially generated with AI. As this study shows, both sets of participants yielded similar results. Professors do not always have the tools, training, time, or definitive proof to label a piece of writing as plagiarism. Students often present their writing printed, making identifying longer writing pieces more demanding and inefficient. Other researchers (Beyduz, 2023; Salas-Pilco and Yang, 2022; Schiff, 2022; Smith, 2022) have claimed that AI may merge with educational practices and help teach students. However, the results demonstrate that professors may not be ready to cope with this integration or guide students to use AI educationally and ethically.
The analysis concluded that AI-generated paragraphs could compete with human-written paragraphs in short writing samples. While human writing may excel in content and organization, AI outperforms in grammar, mechanics, and vocabulary. Variations in grading practices among professors emphasized the need for calibration and training to enhance consistency and objectivity. In addition, professors tend to grade at slower rates, be influenced by external factors, or even intrinsic factors that may alter scorings (Page and Dieter, 1968). These insights shed light on the evaluation of written works and the importance of addressing specific areas for improvement in writing assessment.
This study has three main limitations that deserve attention. First, the sample size was relatively small. It is impossible to generalize the performance of the entire population based on eight paragraphs. It is also impossible to generalize the potential of AI based on eight samples. However, the number of paragraphs adequately simulates a writing class. In addition, a larger set of paragraphs may also hinder professors from collaborating or dedicating enough time to each sample.
Second, as previously stated, writing includes prewriting strategies, drafting, revising, and revising, among other strategies and activities. Producing a good piece of writing takes time and requires revisiting and improving one’s writing (Cheung, 2016; Murray and Christison, 2011; Sethuraman and Radhakrishnan, 2020). In this study, students had a limited time frame to complete the task. Although no student reached the time limit, they did not have enough time to prewrite or self-edit their work. To minimize this limitation, professors were instructed to consider this a diagnostic or placement test in which students would have similar conditions.
Third, research has suggested that one piece of writing may be insufficient to properly assess a student’s writing ability (Nation, 2009). A more reliable setting would include several markers (at least two or three) and several pieces of writing (at least two or three) (Nation, 2009). Multiple evaluations were not practical for practical reasons. In particular, the contrast with AI-generated texts would not have been possible since, unlike humans, machines do not interact differently and are not susceptible to being influenced by environmental, social, or daily situations. An improvement or decline in these types of paragraphs cannot be measured over short periods of time.
This study should be replicated in other ESL settings or with different types of AI. For example, not all populations write general English but instead focus more on it for academic or specific purposes. In addition, not all AI applications write alike; some might be able to develop cohesion among paragraphs to create essays or more extended pieces of writing. They may also adapt to the user or become more proficient over time. Further research should also be undertaken to determine to what extent the limitations described here may affect the results of the present study.
