EVALUASI KOMPARATIF MANUSIA, CHATGPT, DAN GEMINI DALAM MENGANALISIS POLA KALIMAT PADA TEKS AKADEMIK

Abstract

The increasing interest of high school students in utilizing LLM-based applications (ChatGPT and Gemini) as learning tools needs to be evaluated. Sentence analysis results from ChatGPT and Gemini, which are frequently used by students in their work, require careful review. Therefore, a comparative evaluation of the sentence analysis results from ChatGPT and Gemini is important to determine the reliability of the two LLM applications. This study presents the results of a comparative evaluation of the ChatGPT and Gemini applications in analyzing sentence patterns, based on the analysis of Indonesian linguists as the “gold standard”. A comparative quantitative approach was used in this study. The data used were 139 sentences from academic texts found in the grade XI student textbooks curated by the Ministry of Education, Culture, Research, and Technology. Agreement classification tests and performance tests were conducted between ChatGPT and Gemini. The level of agreement for ChatGPT was classified as almost perfect with a coefficient of 0.843, while Gemini's agreement was classified as moderate with a coefficient of 0.597. The ChatGPT evaluation results showed an accuracy of 89.93%, a macro precision of 91.70%, a macro recall of 87.97%, and a macro F1-score of 89.21%. In ChatGPT, classification errors focused on distinguishing between compound-equivalent and compound-equivalent sentences. Furthermore, the Gemini evaluation results showed an accuracy of 73.38%, a macro precision of 79.77%, a macro recall of 72.42%, and a macro F1-score of 66.99%. In Gemini, classification errors focused on the classification of compound-equivalent sentences. Thus, the analysis by Indonesian linguists, as the gold standard, indicates that ChatGPT's suitability and performance are superior to Gemini's in sentence analysis.