Exploring Grammarly's Effectiveness Across L2 Writing Genres: Insights for Language Instruction

Automatic Writing Evaluation (AWE) tools have gained significant attention in the field of second language (L2) writing instruction. These tools, such as Grammarly, offer potential benefits for both learners and instructors by providing instant feedback on various aspects of writing. However, the effectiveness of AWE across different writing genres and error types remains a subject of debate among language educators and researchers. This study aims to explore the use of Grammarly, a popular AWE tool, in the context of L2 writing. By comparing error frequency, error types, and writing complexity across different genres - including university admission test essays, textbook-based descriptive essays, social network site (SNS) posts, and SNS comments - the research seeks to provide insights into how AWE tools perform in various L2 writing contexts. The growing interest in AWE stems from its potential to save time for instructors and provide immediate feedback to learners. Previous studies have shown that AWE can be effective in detecting errors and improving linguistic writing quality. However, questions remain about its ability to address deeper-level, global errors and its impact on students' engagement with language learning strategies.

The study employed a corpus analysis approach to examine four distinct sets of L2 writing: 1. University admission test essays from the Yonsei English Learners' Corpus (YELC) 2. Textbook-based descriptive essays from English majors 3. Social network site (SNS) posts from a Facebook-based language learning program 4. SNS comments from the same Facebook program Each corpus was analyzed using Grammarly's commercial version, which provides detailed performance reports on error types, syntactic complexity, and overall writing scores. The study focused on error types occurring at least 0.3 times per hundred words in at least one of the writing sets. In addition to Grammarly's analysis, the research utilized Lu's (2012) lexical complexity analyzer to identify noun, verb, and modifier variation across the different writing genres. This combination of tools allowed for a comprehensive examination of both grammatical accuracy and lexical sophistication in L2 writing. Statistical analysis, including one-way ANOVA and Bonferroni post hoc tests, was conducted to identify significant differences in error types and writing complexity among the four corpora.

The study revealed several significant findings regarding error types and writing complexity across the different L2 writing genres: 1. Grammar Errors: - Grammar mistakes were the most common across all corpora, with test-taking conditions producing the highest frequency of errors. - Determiner errors, particularly with article usage, were the most frequent grammar mistake across all groups. - Incomplete sentences were more common in SNS writing, especially in comments. 2. Conciseness and Convention Errors: - SNS writing exhibited more conciseness errors (wordiness) compared to test-based and textbook-based writing. - Convention errors (e.g., capitalization and spacing) were present in all writing sets, with Facebook posts showing the least amount. 3. Spelling Errors: - Spelling mistakes were the second most frequent error type across all corpora. - Test-based writing and SNS posts had more spelling errors than textbook-based writing. 4. Punctuation Errors: - Facebook comments had the highest frequency of punctuation errors, particularly with comma usage in clauses. 5. Lexical and Syntactic Complexity: - Textbook-based writing featured longer sentences and words. - SNS-based writing demonstrated a wider variety of nouns, verbs, and modifiers. - Test-based and textbook-based writing showed more complex sentence structures but less lexical variation compared to SNS writing. These findings highlight the varying challenges L2 writers face across different writing contexts and the potential role of AWE tools in addressing these challenges.

Based on the study's findings, several pedagogical implications emerge for L2 writing instruction: 1. Targeted Error Correction: - Instructors should focus on high-frequency errors, such as article usage, especially in test preparation programs. - AWE tools can help students identify and correct these common errors consistently. 2. Vocabulary Development: - Encourage students to use AWE tools to identify overused words and practice using synonyms, particularly in textbook-based writing. - Incorporate SNS-based writing activities to promote broader vocabulary use in authentic contexts. 3. Writing Process Integration: - Implement AWE tools like Grammarly immediately after first drafts to help students recognize areas of concern before instructor review. 4. Balancing Complexity and Clarity: - Help students understand the balance between sentence complexity and clarity, especially in high-stakes writing tasks. - Use SNS writing activities to practice clear, concise communication. 5. Peer Modeling and Collaborative Learning: - Utilize SNS platforms to facilitate peer modeling of good writing practices. - Encourage students to learn from and adapt the language used by more proficient peers. 6. Complementary Feedback Approaches: - Use AWE for addressing local, surface-level errors while reserving instructor feedback for global issues like cohesion and content. 7. Metacognitive Strategies: - Encourage students to keep archives of AWE feedback to reflect on and improve their writing over time. By incorporating these strategies, instructors can leverage the strengths of AWE tools while addressing their limitations, ultimately supporting more effective L2 writing development across various genres and contexts.

This study on the use of Grammarly across different L2 writing genres provides valuable insights into the potential benefits and limitations of Automatic Writing Evaluation (AWE) tools in language learning contexts. The findings demonstrate that while AWE can be effective in identifying and correcting surface-level errors, its performance varies across different writing genres and error types. AWE tools like Grammarly show particular strength in addressing local errors such as grammar, punctuation, and spelling. They can provide consistent, immediate feedback that helps students improve their writing accuracy over time. This is especially valuable in contexts where instructor time is limited or where students need frequent practice and feedback. However, the study also highlights the limitations of AWE. These tools may struggle with more complex aspects of writing, such as coherence, cohesion, and content-related issues. Additionally, the varying error patterns and complexity levels across different writing genres suggest that a one-size-fits-all approach to AWE is not sufficient. Ultimately, the most effective approach to L2 writing instruction appears to be a combination of AWE and human feedback. By using AWE tools to address routine, surface-level errors, instructors can focus their attention on higher-order concerns in student writing. This complementary approach can lead to more comprehensive writing development for L2 learners. As AWE technology continues to evolve, further research will be necessary to understand its impact on different aspects of L2 writing and to develop best practices for its integration into language learning curricula. The goal should be to harness the efficiency and consistency of AWE while maintaining the irreplaceable role of human instruction in developing well-rounded, proficient L2 writers.