Retrieval-Enhanced Log Question Answering System

doi:10.11871/jfdc.issn.2096-742X.2026.01.006

Abstract

Abstract:

[Objective] In the field of AI for IT Operations (AIOps), log question answering is a critical task that helps support teams and system administrators efficiently locate and resolve system issues. However, the application of large language models to log question answering faces challenges such as discrepancies between training corpora and log content, as well as insufficient accuracy in retrieving the contextual information required for answering questions. This study aims to propose a novel approach to improve the performance and generalization capability of log question answering systems. [Coverage] This article focuses on reviewing the current state of research on log question answering tasks in the AIOps domain, with an emphasis on analyzing the limitations of existing large language models in processing system logs. [Methods] This paper introduces a retrieval-enhanced log question answering system named LogMind. The system employs an iterative feedback mechanism to jointly train the retrieval model and the large language model, while also incorporating a robust training strategy. [Results] Experiments conducted on 16 system log datasets across 6 domains demonstrate that the LogMind framework significantly improves the accuracy of both the retrieval model and the large language model. Additionally, the framework exhibits strong cross-model generalization capabilities. [Limitations] This study primarily evaluates the proposed method in offline scenarios. Further exploration is needed to address real-time performance and scalability in production environments. [Conclusions] The LogMind framework provides a reliable and intelligent solution for log question answering in the AIOps domain, offering critical support for advanced system management. It also presents new perspectives for the research and application of log question answering tasks.

Key words: AIOps, log question answering, log retrieval, large language models, question answering

WU Zhihui,HUANG Shaohan,ZHANG Yifei,QI Jiaxing,XIAO Zhiwen,ZENG Chang,LUAN Zhongzhi. Retrieval-Enhanced Log Question Answering System[J]. Frontiers of Data and Computing, 2026, 8(1): 64-76, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2026.01.006.

Figures/Tables 12

Fig.1

Fig.2

Table 1

Table 2

Fig.3

Table 3

Fig.4

Table 4

Fig.5

Table 5

Fig.6

Fig.7

References 17

[1]	DANG Y, LIN Q, HUANG P. Aiops: real-world challenges and research innovations[C]. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2019: 4-5.
[2]	JOSH A, ADLER S, AGARWAL S, et al. GPT-4 technical report[R]. arXiv preprint arXiv:2303.08774, 2023.
[3]	HU E J, SHEN Y L, WALLIS P, et al. Lora: Low-rank adaptation of large language models[C]. International Conference on Learning Representations (ICLR), 2022, 3.
[4]	OLINER A, GANAPATHI A, XU W. Advances and challenges in log analysis[J]. Communications of the ACM, 2012, 55(2): 55-61.
[5]	JIANG Z, LIU J, CHEN Z, et al. LLMParser: A LLM-based Log Parsing Framework[R]. arXiv preprint arXiv:2310.01796, 2023.
[6]	QI J, HUANG S, LUAN Z, et al. LogGPT: Exploring ChatGPT for Log-based Anomaly Detection[C]. In 2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys). IEEE, 2023: 273-280.
[7]	HUANG S, LIU Y, QI J, et al. GLOSS: Guiding Large Language Models to Answer Questions from System Logs[C]. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2024: 91-101.
[8]	RAM O, LEVINE Y, DALMEDIGOS I, et al. In-Context Retrieval-Augmented Language Models[J]. Transactions of the Association for Computational Linguistics, 2023, 11: 1316-1331. doi: 10.1162/tacl_a_00605
[9]	CHENG D, HUANG S, BI J, et al. Uprise: Universal Prompt Retrieval for Improving Zero-Shot Evaluation[J]. arXiv preprint arXiv:2303.08518, 2023.
[10]	LIU H, LIU J, HUANG S, et al. SE2: Sequential Example Selection for In-Context Learning[J]. arXiv preprint arXiv:2402.13874, 2024.
[11]	REIMERS N, GUREVYCH I. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks[J]. arXiv preprint arXiv:1908.10084, 2019.
[12]	ROBERTSON S, ZARAGOZA H. The Probabilistic Relevance Framework: BM25 and Beyond[J]. Foundations and Trends in Information Retrieval, 2009, 3(4): 333-389. doi: 10.1561/1500000019
[13]	TOVRON H, LAVRIL T, IZACARD G, et al. LLaMA: Open and Efficient Foundation Language Models[J]. arXiv preprint arXiv:2302.13971, 2023.
[14]	TOVRON H, MARTIN L, STONE K, et al. LLaMA 2: Open Foundation and Fine-Tuned Chat Models[J]. arXiv preprint arXiv:2307.09288, 2023.
[15]	GUO D, YANG D, ZHANG H, et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning[J]. arXiv preprint arXiv:2501. 12948, 2025.
[16]	WANG L, YANG N, HUANG X, et al. Text Embeddings by Weakly-Supervised Contrastive Pre-Training[J]. arXiv preprint arXiv:2212. 03533, 2022.
[17]	WANG L, YANG N, WEI F. Learning to Retrieve In-Context Examples for Large Language Models[J]. arXiv preprint arXiv:2307.07164, 2023.

模型超参数	取值	检索模型参数	取值
LoRA Alpha	16	预热步数	100
LoRA Rank	8	优化器	AdamW
LoRA Dropout	0.1	学习率	2e-5
Epoch	3	学习率衰减	Linear
学习率	2e-4	上下文候选数量	10
学习率衰减	Linear	正例数量	1
权重衰减	0.01	负例数量	3
预热比例	0.03	-	-
最大长度	2048	-	-

数据集	时间跨度	问答对
分布式系统（Distributed System，DS）
HDFS	38.7小时	1,619
Hadoop	未知	1,088
Spark	未知	1,576
Zookeeper	26.7天	939
OpenStack	未知	1,619
超级计算机（Super Computer，SC）
BGL	214.7天	1,270
HPC	未知	1,725
Thunderbird	244天	1,528
操作系统（Operation System，OS）
Windows	226.7天	1,389
Linux	263.9天	1,502
Mac	7天	1,433
移动系统（Mobile System，MS）
Android	未知	1,269
HealthApp	10.5天	1,490
服务器应用程序（Server Application，SA）
Apache	263.9天	1,457
OpenSSH	28.4天	1,561
独立软件（Standalone Software，SS）
Proxifier	未知	1,822

	DS	SC	OS	MS	SA	SS
SBERT	19.79	20.35	29.24	16.64	44.45	26.75
LLM-R	30.69	20.5	34.38	27.23	43.43	33.06
LogMind	32.98	21.45	39.51	29.09	45.67	38.21

	No Context	EM25	LLM-R	LogMind
Hadoop	2.96	6.53	12.84	14.74
HPC	1.76	7.72	11.25	11.50
Windows	2.84	6.89	13.22	12.73
Android	2.13	7.36	14.18	15.92
OpenSSH	7.00	11.27	17.26	21.79

Dataset	Deepseek-R1-7B		LogMind
Dataset	EM	Fl	EM	Fl
HDFS	6.53	8.79	15.62	29.15
Hadoop	9.5	17.22	16.36	31.51
Spark	2.33	8.59	8.56	31.15
Zookeeper	5.57	11.64	20.42	34.65
OpenStack	15.49	19.89	23.31	38.45
BGL	2.73	5.69	5.89	11.39
HPC	6.3	9.45	11.14	17.76
Thunderbird	4.27	7.5	19.35	35.2
Windows	9.2	13.64	36.35	52.64
Linux	12.45	19.1	25.43	35.78
Mac	8.08	13.97	20.41	30.12
Android	20	24.8	28.5	37.73
HealthApp	5.42	7.04	14.62	20.45
Apache	12.75	22.83	36.97	51.96
SSH	12.52	20	26.36	39.37
Proxifier	20.35	27.09	21.45	38.21
AVERAGE	9.59	14.83	20.67	33.47