Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques

Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, Lionel Briand

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Log message template identification aims to convert raw logs containing free-formed log messages into structured logs to be processed by automated log-based analysis, such as anomaly detection and model inference. While many techniques have been proposed in the literature, only two recent studies provide a comprehensive evaluation and comparison of the techniques using an established benchmark composed of real-world logs. Nevertheless, we argue that both studies have the following issues: (1) they used different accuracy metrics without comparison between them, (2) some ground-truth (oracle) templates are incorrect, and (3) the accuracy evaluation results do not provide any information regarding incorrectly identified templates. In this paper, we address the above issues by providing three guidelines for assessing the accuracy of log template identification techniques: (1) use appropriate accuracy metrics, (2) perform oracle template correction, and (3) perform analysis of incorrect templates. We then assess the application of such guidelines through a comprehensive evaluation of 14 existing template identification techniques on the established benchmark logs. Results show very different insights than existing studies and in particular a much less optimistic outlook on existing techniques.

Original languageEnglish
Title of host publicationProceedings - 2022 ACM/IEEE 44th International Conference on Software Engineering, ICSE 2022
PublisherIEEE Computer Society
Pages1095-1106
Number of pages12
ISBN (Electronic)9781450392211
DOIs
Publication statusPublished - 2022
Externally publishedYes
Event44th ACM/IEEE International Conference on Software Engineering, ICSE 2022 - Pittsburgh, United States
Duration: 22 May 202227 May 2022

Publication series

NameProceedings - International Conference on Software Engineering
Volume2022-May
ISSN (Print)0270-5257

Conference

Conference44th ACM/IEEE International Conference on Software Engineering, ICSE 2022
Country/TerritoryUnited States
CityPittsburgh
Period22/05/2227/05/22

Keywords

  • logs
  • metrics
  • template identification

Fingerprint

Dive into the research topics of 'Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques'. Together they form a unique fingerprint.

Cite this