ANALISIS KOMPARATIF EFEKTIVITAS RULE-BASED CHATBOT DAN GENERATIVE AI CHATBOT DALAM MENDUKUNG LAYANAN AKADEMIK BERBASIS SMART CAMPUS

Agung Tantra Sepzulfi; Irvan Haviz

doi:10.260396/ts19qa42

Authors

Agung Tantra Sepzulfi STIKOM Tunas Bangsa Pematang Siantar Author
Irvan Haviz STIKOM Tunas Bangsa Pematang Siantar Author

DOI:

https://doi.org/10.260396/ts19qa42

Abstract

ABSTRAK

Perkembangan kecerdasan buatan (Artificial Intelligence/AI) telah mendorong transformasi layanan akademik di perguruan tinggi menuju paradigma Smart Campus. Namun, implementasi chatbot tunggal sering dihadapkan pada masalah trade-off antara kecepatan respons dan fleksibilitas bahasa. Penelitian ini mengusulkan kebaruan (novelty) berupa perancangan dan analisis Hybrid Chatbot Architecture yang mengintegrasikan Rule-Based Engine (Jaccard Similarity) dan Generative AI Engine (RAG berbasis Gemini API) melalui mekanisme Decision Router. Penelitian menggunakan metode kuantitatif komparatif berbasis eksprimen prototype dengan melibatkan 60 responden mahasiswa aktif yang dipilih menggunakan teknik purposive sampling. Evaluasi kinerja komparatif dilakukan pada lima dimensi utama: akurasi respons, response time, kepuasan pengguna, naturalness komunikasi, dan usability. Hasil penelitian menunjukkan bahwa Generative AI Chatbot memperoleh skor unggul pada dimensi akurasi respons (92.4% vs 83.4%), kepuasan pengguna (mean 4.37 vs 3.82), naturalness komunikasi (mean 4.56 vs 3.41), dan usability System Usability Scale/SUS (80.3 vs 71.5). Sementara Rule-Based Chatbot menunjukkan keunggulan signifikan pada aspek response time (215 ms vs 1.207 ms). Uji Independent Sample T-Test mengonfirmasi perbedaan yang signifikan secara statistik ($p < 0.05$) pada sebagian besar dimensi evaluasi. Penelitian ini menyimpulkan bahwa integrasi arsitektur hybrid yang diusulkan berhasil menjadi solusi optimal untuk Smart Campus di Indonesia dengan merekonsiliasi kecepatan model aturan dan kecerdasan model generatif secara simultan.

Kata Kunci: Hybrid Chatbot Architecture, Generative AI, Smart Campus, Decision Router, Retrieval-Augmented Generation (RAG), User Satisfaction.

ABSTRACT

The development of Artificial Intelligence (AI) has driven the transformation of academic services in higher education towards a Smart Campus paradigm. However, single chatbot implementations often face a trade-off between response speed and language flexibility. This study proposes a novelty by designing and analyzing a Hybrid Chatbot Architecture that integrates a Rule-Based Engine (Jaccard Similarity) and a Generative AI Engine (RAG based on Gemini API) via a Decision Router mechanism. This research employs a comparative quantitative method based on prototype experimentation, involving 60 active student respondents selected using a purposive sampling technique. Comparative performance evaluation was conducted across five main dimensions: response accuracy, response time, user satisfaction, communication naturalness, and usability. The results indicate that the Generative AI Chatbot achieved superior scores in response accuracy (92.4% vs 83.4%), user satisfaction (mean 4.37 vs 3.82), communication naturalness (mean 4.56 vs 3.41), and System Usability Scale (SUS) scores (80.3 vs 71.5). Meanwhile, the Rule-Based Chatbot demonstrated significant advantages in response time (215 ms vs 1,207 ms). The Independent Sample T-Test confirmed statistically significant differences ($p < 0.05$) across most evaluation dimensions. This study concludes that the proposed hybrid architecture successfully serves as the optimal solution for Smart Campus in Indonesia by simultaneously reconciling the speed of the rule-based model and the intelligence of the generative model.

Keywords: Hybrid Chatbot Architecture, Generative AI, Smart Campus, Decision Router, Retrieval-Augmented Generation (RAG), User Satisfaction.

Downloads

Download data is not yet available.

References

Adamopoulou, E., & Moussiades, L. (2022). An overview of chatbot technology. IFIP Advances in Information and Communication Technology, 584, 373-383. https://doi.org/10.1007/978-3-030-49186-4_31

Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114-123.

Brooke, J. (1996). SUS: A 'quick and dirty' usability scale. In P. Jordan, B. Thomas, & B. Weerdmeester (Eds.), Usability Evaluation in Industry (pp. 189-194). Taylor & Francis.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2024). GPT-4 in academic advisory: Opportunities and challenges. Journal of Educational Technology & Society, 27(1), 145-162.

Grand View Research. (2024). Chatbot market size, share & trends analysis report 2024-2030. Grand View Research, Inc.

Hien, H. T., Cuong, P. N., Nam, L. N. H., Nhung, H. L. T. K., & Thang, L. D. (2021). Intelligent assistants in higher education: A chatbot approach for administrators and students. Proceedings of the 9th International Symposium on Information and Communication Technology, 213-219. https://doi.org/10.1145/3287921.3287937

Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28, 973-1018. https://doi.org/10.1007/s10639-022-11177-3

Pereira, J., & Diaz, O. (2022). Using health chatbots for behavior change: A mapping study. Journal of Medical Systems, 43(135), 1-11. https://doi.org/10.1007/s10916-019-1237-1

Ranoliya, B. R., Raghuwanshi, N., & Singh, S. (2023). Chatbot for university related FAQs. Proceedings of IEEE International Conference on Advances in Computing, Communications and Informatics (ICACCI), 1525-1530.

Santoso, B., Purnama, H., & Wijaya, R. (2024). Implementasi chatbot berbasis NLP untuk layanan akademik di perguruan tinggi Indonesia: Studi kasus Universitas Bina Nusantara. Jurnal Teknologi dan Sistem Informasi, 10(2), 87-102. https://doi.org/10.25126/jtiik.2024.10.2.7821