Detecting large-scale system problems by mining console logs

Wei XuLing HuangA. FoxD. PattersonMichael I. Jordan

article2009SOSP1,408 citations

Combines source code analysis with machine learning to automatically parse unstructured console logs and detect runtime anomalies in large-scale systems at line-rate without requiring code modifications or manual intervention.

Listen

Modern datacenter services combine hundreds of software components running across thousands of machines. When operational failures occur, operators face massive volumes of interleaved, unstructured textual console logs that are practically impossible to inspect manually. Standard troubleshooting methods, such as simple keyword searches for error labels or rigid rule-based filtering, frequently fail because critical system failures often leave no explicit error message or present misleading warning signals.

The article demonstrates an automated, end-to-end methodology to detect operational problems by mining unstructured console logs. The primary objective is to accurately parse raw text logs without manual modification to source applications and apply unsupervised machine learning to detect system anomalies.

The approach relies on static source code analysis to uncover the implicit structure of console logs, extracting message templates, object identifiers, and system state variables. Using these extracted elements, the system constructs numerical feature vectors: state ratio vectors that monitor aggregate system health over time, and message count vectors that track execution paths tied to specific transactions or files. The system then applies Principal Component Analysis—combined with term-weighting techniques from information retrieval—to separate normal execution patterns from anomalous outliers without requiring labeled training data. The analysis concludes by compiling the anomaly detection results into an easily readable, one-page decision tree for system operators.

Key findings show that this approach achieves over 99.8% parsing accuracy across millions of unstructured messages while handling rare message types. In an evaluation on the Hadoop Distributed File System, the method processed 24 million lines of logs in under three minutes on cloud infrastructure, accurately isolating complex anomalies and discovering an unhandled file deletion bug confirmed by developers. In testing on the Darkstar online game server, the system detected severe performance degradation during resource contention by observing that the ratio of aborted to committed transactions shifted drastically from roughly 1:2000 to 1:2. Furthermore, the decision tree visualization effectively converted high-dimensional mathematical anomaly outputs into plain operational logic.

These results show that organizations can significantly reduce troubleshooting timelines, system downtime, and operational risks without rewriting legacy code or adopting expensive custom monitoring frameworks. The findings also highlight that developer logging practices often misjudge event severity, meaning automated statistical anomaly detection across execution paths provides a much more reliable indicator of system health than developer-assigned log levels.

Organizations operating large-scale distributed systems should consider adopting source-informed log parsing pipelines and machine learning classifiers to automate incident detection. Operators should also implement clear logging standards, such as consistently including unique identifiers in threaded communications, to maximize detection efficacy. Future implementations should explore online stream detection and extracting templates directly from compiled binary files.

The study's primary limitation is its reliance on source code availability for template extraction, which restricts direct application to proprietary, closed-source components. Additionally, unsupervised anomaly detection inherently produces a small number of false positives on rare but normal system routines. Nevertheless, confidence in the methodology remains high across open-source environments given its linear computational scalability and demonstrated success on production-grade distributed architectures.

Intel University of California Berkeley decision trees file systems hadoop console log mining problem detection log parsing

Xu et al (2009).pdf

Paper: Induction of Decision Trees, J. R. Quinlan (1986). Introduces top-down induction of decision trees, providing the fundamental classification and rule-distillation mechanism used by the source paper to produce operator-friendly decision trees for log-based failure detection.
Paper: Pig latin: a not-so-foreign language for data processing, Christopher Olston et al. (2008). Presents Pig Latin and data processing over large-scale systems such as Hadoop, which forms a core part of the distributed log processing environment evaluated in the source paper.
Paper: Algorithms for Mining Distance-Based Outliers in Large Datasets, Edwin M. Knorr et al. (1998). Establishes foundational techniques for mining distance-based outliers and anomalies across large multi-dimensional datasets without requiring prior distribution assumptions.

Paper: Kafka : a Distributed Messaging System for Log Processing, Jay Kreps et al. (2011). Introduces Kafka, a distributed messaging and log architecture designed to ingest and deliver high-throughput system logs for the kind of downstream real-time analysis and anomaly detection pioneered by the source.
Paper: Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection, Bo Zong et al. (2018). Extends unsupervised anomaly detection in complex, high-dimensional system data by using deep autoencoders coupled with Gaussian mixture models.
Paper: Deep Learning for Anomaly Detection: A Survey, Raghavendra Chalapathy et al. (2019). Surveys the evolution of deep learning architectures applied to anomaly detection across large-scale unstructured system sequences and logs.
Paper: Graph Neural Network-Based Anomaly Detection in Multivariate Time Series, Ailin Deng et al. (2021). Advances modern system fault detection and root cause localization by modeling inter-metric relationship graphs alongside anomaly scores.
Paper: Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network, Ya Su et al. (2019). Builds on unsupervised incident detection in server clusters by applying stochastic recurrent neural networks to provide automated failure explanations.

Cover for Detecting large-scale system problems by mining console logs

Abstract

Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.

Topics

decision trees file systems hadoop information retrieval software component source code analysis association rules

Keywords

console log mining problem detection log parsing message template extraction anomaly detection HDFS logs decision trees code analysis

1 Introduction
2 Overview of Approach
2.1 Information buried in textual logs
2.2 Workflow of our approach
2.3 Case study and data collection
3 Log Parsing with Source Code
4 Feature Creation
4.1 State variables and state ratio vectors
4.2 Identifiers and message count vectors
4.3 Implementing feature creation algorithms
5 Anomaly Detection
6 Evaluation and Visualization
6.1 Log parsing accuracy and scalability
6.2 Darkstar experiment results
6.3 Hadoop experiment results
6.4 Visualizing detection results with decision trees
7 Discussion
8 Related Work
9 Conclusions and Future Work
Acknowledgements
References
A Appendix: Extracting message templates from source code

Knowls

Knowl 1 — Source-Code-Augmented Console Log Parsing
model/method
Console log mining relies on transforming unstructured, free-text console logs into structured event types and variable key-value pairs without modifying target systems. Source code is used as the implicit schema for console logs through a two-phase analysis pipeline:
1. Static Source Code Analysis:
  - The abstract syntax tree (AST) of the software source code is generated (e.g., via the Eclipse IDE parser) to identify all method invocations on logger objects (such as log4j or custom wrappers), yielding partial message templates.
  - A toString Table is constructed by analyzing toString() methods across all declared classes, recording variable formatting and string concatenations.
  - A Class Hierarchy Table is constructed. For each non-primitive variable interpolated into a log template, the method recursively resolves its string representation through inherited or subclass toString() methods. If an object belongs to a class with subclasses, a separate message template is generated for each subclass known at compile time. Recursion terminates when all variables are resolved to primitive types or when reaching classes with more than 100 descendants (such as java.lang.Object), where it falls back to a regex wildcard (.*).
  - The static phase outputs complete message templates as regular expressions paired with the names, data types, and source locations of all variables.
2. Runtime Log Parsing:
  - Message templates are indexed into an Apache Lucene reverse index after stripping numeric and special characters to form text queries.
  - For each incoming free-text log message, query tokens retrieve candidate templates ranked by relevance, and the highest-ranked template that matches the log message via regular expression is assigned.
  - Message parsing is executed as a distributed MapReduce job by replicating the Lucene reverse index across mapper nodes and partitioning raw log lines.
Knowl 2 — Message Count Vector Construction via Object Identifiers
algorithm
Message count vectors capture the execution path of individual system entities (such as transactions or data blocks) across distributed nodes by grouping log messages by object identifiers and compiling message type frequencies into fixed-dimensional vectors.
Input: Parsed log dataset $L = \{m_1, m_2, \dots, m_N\}$ with $N$ total messages, set of distinct message types $T = \{t_1, t_2, \dots, t_n\}$
Output: Message count matrix $Y^m \in \mathbb{R}^{m \times n}$
1: Scan $L$ to identify candidate identifier variables $V_{\text{id}}$ satisfying:
    a: Variable is reported in at least $0.2 N$ log lines
    b: Variable has at least $0.02 N$ distinct values across $L$
    c: Variable appears across at least 5 distinct message types in $T$
2: Select identifier variable $v^* \in V_{\text{id}}$ (e.g., transaction ID or block ID)
3: Partition log messages $L$ into $m$ groups $\{G_1, G_2, \dots, G_m\}$ , where all messages in group $G_k$ share the same distinct value of $v^*$
4: for each group $G_k$ from $k = 1$ to $m$ do
5: Initialize count vector $y^{(k)} = [0, 0, \dots, 0] \in \mathbb{R}^n$
6: for each log message $m \in G_k$ do
7: Let $j \in \{1, \dots, n\}$ be the message type index corresponding to $m$
8: $y^{(k)}_j \leftarrow y^{(k)}_j + 1$
9: end for
10: end for
11: Stack count vectors into matrix $Y^m = [y^{(1)}; y^{(2)}; \dots; y^{(m)}] \in \mathbb{R}^{m \times n}$
12: return $Y^m$
The resulting matrix $Y^m$ contains $m$ rows (each representing an object's execution path) and $n$ columns (each representing a message type).
Knowl 3 — Subspace PCA Anomaly Detection on Log Feature Vectors
model/method
Correlations among log messages in a group or time window constrain normal system behavior to a low-dimensional subspace within the original $n$ -dimensional feature space. Anomaly detection is performed via Principal Component Analysis (PCA) subspace separation on a feature matrix $Y \in \mathbb{R}^{m \times n}$ :
1. Subspace Decomposition: PCA selects the first $k$ principal components $P = [v_1, v_2, \dots, v_k] \in \mathbb{R}^{n \times k}$ that capture at least 95% of the total variance in $Y$ . The matrix $P$ spans the $k$ -dimensional normal subspace $S_d$ . The remaining $(n-k)$ principal components span the abnormal subspace $S_a$ .
2. Projection and Residual Metric: For any observation vector $y \in \mathbb{R}^n$ , its projection onto the abnormal subspace $S_a$ is given by: $y_a = (I - P P^T)y$ where $I$ is the $n \times n$ identity matrix and $P P^T$ is the projection matrix onto $S_d$ .
3. Squared Prediction Error (SPE): The magnitude of deviation from the normal subspace is measured by the SPE statistic: $\text{SPE} \equiv \|y_a\|^2 = \|(I - P P^T)y\|^2$
4. Detection Threshold: A vector $y$ is classified as an anomaly if: $\text{SPE} > Q_\alpha$ where $Q_\alpha$ is the threshold computed from the Jackson-Mudholkar $Q$ -statistic at significance level $(1 - \alpha)$ with $\alpha = 0.001$ . The $Q$ -statistic provides a theoretical bound on the false alarm probability.
Knowl 4 — State Ratio Vector Construction for Aggregate System Monitoring
model/method
State ratio vectors capture the aggregate health and concurrency behavior of a system across successive time windows by tracking state variables (enumerated labels reflecting system or transaction status, such as ACTIVE, PREPARING, COMMITTING, ABORTING, or host IDs).
1. State Variable Identification: Variables are automatically selected as state variables if:
  - They appear in at least $0.2 N$ log messages (where $N$ is total log lines).
  - They take a small constant number $D$ of distinct values that does not scale with $N$ for large $N$ .
2. Dynamic Window Sizing: The time window size $\Delta t$ is automatically chosen as the smallest duration such that the state variable appears at least $10 D$ times in at least 80% of all windows. This guarantees statistical significance for state counts while retaining temporal sensitivity to transient performance anomalies.
3. Matrix Representation: For $m$ contiguous time windows and $n = D$ distinct state variable values, the state ratio matrix $Y^s \in \mathbb{R}^{m \times n}$ is constructed such that entry $Y^s_{i,j}$ is the count of occurrences of state value $j$ during time window $i$ .
During normal operation, the ratios among different state variable values remain roughly constant, forming a low-dimensional subspace ( $k \ll n$ ) detectable by PCA.
Knowl 5 — TF-IDF Weighting for Message Count Feature Matrices
equation
To prevent frequent, routine log messages from dominating the Euclidean distance in PCA anomaly detection, Term Frequency-Inverse Document Frequency (TF-IDF) weighting is applied to the message count matrix $Y^m \in \mathbb{R}^{m \times n}$ before subspace analysis.

Each raw count $y_{i,j}$ (the number of occurrences of message type $j$ in object group $i$ ) is transformed into a weighted entry $w_{i,j}$ : $w_{i,j} = y_{i,j} \log\left(\frac{m}{df_j}\right)$ where $m$ is the total number of message groups (unique identifier instances), $y_{i,j}$ is the raw count of message type $j$ in group $i$ , and $df_j$ is the document frequency of message type $j$ (the number of groups in which message type $j$ appears at least once).

Following TF-IDF weighting, each row vector $w_i = [w_{i,1}, w_{i,2}, \dots, w_{i,n}]$ is normalized by its Euclidean norm $\|w_i\|_2$ prior to PCA.
Knowl 6 — Post-Hoc Decision Tree Distillation for Explaining PCA Log Anomalies
model/method
Because PCA projects feature vectors into uninterpretable principal component coordinates, unsupervised PCA anomaly detection is augmented with a post-hoc supervised decision tree to generate human-readable diagnostic rules:
1. Label Generation: Unsupervised PCA labels each feature vector $y_i$ as normal ( $0$ ) or abnormal ( $1$ ) using the threshold condition $\text{SPE}(y_i) > Q_\alpha$ .
2. Surrogate Tree Training: A decision tree classifier (such as CART) is trained using the original, unprojected feature coordinates (e.g., raw message type counts $y_{i,j}$ ) as inputs and the binary PCA anomaly classifications as target labels.
3. Interpretation: The resulting decision tree reflects the underlying logic of the PCA anomaly detector in original domain coordinates. Split nodes define threshold conditions on specific message types (e.g., blockMap updated >= 4 or Received block <= 2), and leaf nodes map to normal or abnormal classifications. This distills high-dimensional anomalies into simple rule sets that resemble operator event-processing rules.

Knowl 7 — Anomaly Detection Accuracy and Evaluation on HDFS Logs

data/table

The log mining method was evaluated on 24,396,061 log lines collected over 48 hours from a 203-node Hadoop Distributed File System (HDFS) cluster running on Amazon EC2 (processing over 200 TB of data). Grouping messages by block_id formed a message count matrix $Y^m$ of size $575,139 \times 29$ . The table compares raw PCA against TF-IDF preprocessed PCA across 680 distinct execution paths labeled manually by HDFS developers.

Anomaly Description	Actual	Raw PCA	TF-IDF PCA
Namenode not updated after deleting block	4297	475	4297
Write exception client give up	3225	3225	3225
Write failed at beginning	2950	2950	2950
Replica immediately deleted	2809	2803	2788
Received block that does not belong to any file	1240	20	1228
Redundant addStoredBlock	953	33	953
Delete a block that no longer exists on data node	724	18	650
Empty packet for block	476	476	476
Receive block exception	89	89	89
Replication Monitor timedout	45	37	45
Other anomalies	108	91	107
Total Anomalies Detected	16916	10217	16808
False Positive Description	Actual	Raw PCA	TF-IDF PCA
Normal background migration	1399	1399	1397
Multiple replica (for task / job desc files)	372	372	349
Unknown reason	26	26	0
Total False Positives	1797	1797	1746

TF-IDF preprocessing increased anomaly detection from 10,217 to 16,808 of the 16,916 true anomalies (99.4% recall). The analysis uncovered a previously hidden HDFS bug where deleting a block during over-replication failed to update the NameNode, causing subsequent deletions to fail. It also suppressed false alarms on benign client disconnection exceptions (Got Exception while serving..., HADOOP-3678).

Knowl 8 — Performance Anomaly Detection and Root-Cause Diagnosis in Project Darkstar
empirical result
The log mining method was evaluated on the Sun Project Darkstar online game server running the DarkMud application with 60 emulated clients on Amazon EC2 for 4,800 seconds. A performance disturbance was injected between $t = 1400$ s and $t = 1800$ s by capping available CPU to 50%.
1. State Ratio Anomaly Detection: The variable state appeared in 456,996 log messages (28% of the 1,640,985 total lines) across 8 distinct values ( $n = 8$ , with $k = 1$ principal component capturing >95% of variance). Using an automatically computed 3-second window, PCA on the state ratio matrix $Y^s$ detected anomalies that coincided with the injected disturbance interval. During the disturbance, the ratio of ABORTING to COMMITTING transactions shifted from a normal baseline of $\approx 1:2000$ to $\approx 1:2$ . This revealed the root cause: under CPU contention, fixed transaction timeouts caused cascade aborts and retries under Darkstar's optimistic concurrency control, amplifying load.
2. Message Count Anomaly Detection: Grouping by transaction_id ( $m = 68,029$ , $n = 18$ message types, $k = 3$ ) flagged 504 abnormal transactions exhibiting missing commit operations or unexpected abort_txn calls. Message count vectors detected recovery queuing effects continuing from $t = 1800$ s to $t = 2000$ s that were missed by aggregate state ratio time windows.
Knowl 9 — Parsing Accuracy and Distributed Scalability on Cloud Clusters
empirical result
Source-code-guided log parsing and feature extraction were evaluated across 22 open-source systems and tested for scalability on Amazon EC2 instances.
1. Parsing Accuracy: The parser achieved 99.879% accuracy on HDFS logs (24,366,425 of 24,396,061 messages parsed successfully; 0.121% failure rate) and 99.998% accuracy on Darkstar logs (1,640,950 of 1,640,985 messages parsed successfully; 0.002% failure rate). Failures were limited to startup/shutdown phases where raw state dumps produced long unformatted strings that overwhelmed reverse index template matching.
2. Execution Scalability: When executed as a MapReduce job across Amazon EC2 high-CPU medium instances, parsing and message count vector generation scaled almost linearly up to 50 nodes. Processing 24.4 million lines of HDFS logs took under 10 minutes on 10 nodes and under 3 minutes on 50 nodes. Beyond 60 nodes, index distribution and job scheduling overheads dominated running time.
Knowl 10 — Limitations of Static Source Code Analysis for Log Parsing
limitation
Extracting log schemas via static analysis of source code ASTs encounters specific structural limitations:
1. Polymorphic Base Types: If log statements log variables declared with high-level abstract types (such as java.lang.Object), resolving candidate subclasses statically becomes intractable. The parser bounds subclass traversal to a maximum of 100 descendants to avoid traversing base language libraries, falling back to a greedy regex wildcard (.*) if unresolved.
2. Dynamic String Operations: Static analysis cannot infer message structures generated through runtime loops, recursion, or dynamic string manipulations, requiring fallback to unparsed wildcards (.*).
3. Undecorated Messages: Log statements that output primitive variables without any static prefix or suffix string literals cannot be assigned a distinct message template and are omitted from analysis.
4. Concurrency Without Identifiers: Systems that log inter-thread or inter-node communications without explicit session IDs or object identifiers (e.g., multi-threaded Cassandra message transfers) prevent object-based message grouping, restricting analysis to time-window grouping.

Coverage note — Omitted the survey statistics of 22 open-source systems (Table 2 LOC and LOL counts) as these are motivational background rather than core analytical contributions.

References

1.A. W. Appel. Modern Compiler Implementation in Java. Cambridge University Press, second edition, 2002.
2.D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007.
3.M. Y. Chen and et al. Path-based failure and evolution management. In Proc. NSDI’04, pages 23–23, San Francisco, California, 2004. USENIX.
4.M. H. DeGroot and M. J. Schervish. Probability and Statistics. Addison-Wesley, 3rd edition, 2002.
5.R. Dunia and S. J. Qin. Multi-dimensional fault diagnosis using a subspace approach. In Proc. ACC, 1997.
6.R. Feldman and J. Sanger. The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data. Cambridge Univ. Press, 12 2006.
7.K. Fisher, D. Walker, K. Q. Zhu, and P. White. From dirt to shovels: fully automatic tool generation from ad hoc data. In Proceedings of ACM POPL ’08, pages 421–434, 2008.
8.R. Fonseca and et al. Xtrace: A pervasive network tracing framework. In In Proc. NSDI, 2007.
9.C. Gulcu. Short introduction to log4j, March 2002. http://logging.apache.org/log4j.
10.S. E. Hansen and E. T. Atkins. Automated system monitoring and notification with Swatch. In Proc. USENIX LISA ’93, pages 145–152, 1993.
11.E. Hatcher and O. Gospodnetic. Lucene in Action. Manning Publications Co., Greenwich, CT, 2004.
12.J. Hellerstein, S. Ma, and C. Perng. Discovering actionable patterns in event data. IBM Sys. Jour, 41(3), 2002.
13.J. E. Jackson and G. S. Mudholkar. Control procedures for residuals associated with principal component analysis. Technometrics, 21(3):341–349, 1979.
14.W. Jiang and et al. Understanding customer problem troubleshooting from storage system logs. In Proceedings of USENIX FAST’09, 2009.
15.I. Jolliffe. Principal Component Analysis. Springer, 2002.
16.A. Lakhina, M. Crovella, and C. Diot. Diagnosing network-wide traffic anomalies. In Proc. ACM SIGCOMM, 2004.
17.C. Lim, N. Singh, and S. Yajnik. A log mining approach to failure analysis of enterprise telephony systems. In Proc. DSN, June 2008.
18.S. Ma and J. L. Hellerstein. Mining partially periodic event patterns with unknown periods. In Proc. IEEE ICDE, Washington, DC, 2001.
19.A. A. Makanju, A. N. Zincir-Heywood, and E. E. Milios. Clustering event logs using iterative partitioning. In Proceedings of KDD ’09, 2009.
20.C. Manning, P. Ragahavan, and et al. Introduction to Information Retrieval. Cambridge University Press, 2008.
21.I. Mierswa, M. Wurst, R. Klinkenberg, M. Scholz, and T. Euler. Yale: Rapid prototyping for complex data mining tasks. In Proc. ACM KDD, New York, NY, 2006.
22.A. Oliner and J. Stearley. What supercomputers say: A study of five system logs. In Proc. IEEE DSN, Washington, DC, 2007.
23.K. Papineni. Why inverse document frequency? In Proc. NAACL ’01:, pages 1–8, Morristown, NJ, 2001. Asso. for Comp. Linguistics.
24.J. E. Prewett. Analyzing cluster log files using logsurfer. In Proc. Annual Conf. on Linux Clusters, 2003.
25.T. Sager, A. Bernstein, M. Pinzger, and C. Kiefer. Detecting similar java classes using tree algorithms. In Proc. ACM MSR ’06, pages 65–71, 2006.
26.G. Salton and C. Buckley. Term weighting approaches in automatic text retrieval. Technical report, Cornell, Ithaca, NY, USA, 1987.
27.J. Stearley. Towards informatic analysis of syslogs. In Proc. IEEE CLUSTER, Washington, DC, 2004.
28.Sun. Project darkstar. www.projectdarkstar.com, 2008.
29.Sun. Solaris Dynamic Tracing Guide, 2008.
30.J. Tan and et al. SALSA: Analyzing logs as StAte machines. In Proc. of WASL ’08, 2008.
31.L. Tan, D. Yuan, G. Krishna, and Y. Zhou. /icomment: bugs or bad comments?/. In Proc. ACM SOSP ’07, New York, NY, 2007. ACM.
32.R. Vaarandi. A data clustering algorithm for mining patterns from event logs. Proc. IPOM, 2003.
33.R. Vaarandi. A breadth-first algorithm for mining frequent patterns from event logs. In INTELLCOMM, volume 3283, pages 293–308. Springer, 2004.
34.I. H. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 2000.
35.K. Yamanishi and Y. Maruyama. Dynamic syslog mining for network failure monitoring. In Proc. ACM KDD, New York, NY, 2005.

Citation

MLA

Xu, W., et al. “Detecting Large-scale System Problems by Mining Console Logs”. Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, 2009, pp. 117–32, https://doi.org/10.1145/1629575.1629587.

APA

Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009). Detecting large-scale system problems by mining console logs. Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, 117–132. https://doi.org/10.1145/1629575.1629587

Chicago

Xu, W., L. Huang, A. Fox, D. Patterson, and M. I. Jordan. 2009. “Detecting Large-scale System Problems by Mining Console Logs”. Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, 117–32. https://doi.org/10.1145/1629575.1629587.

Harvard

Xu, W. et al. (2009) “Detecting large-scale system problems by mining console logs”, Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. ACM, pp. 117–132. Available at: https://doi.org/10.1145/1629575.1629587.

Vancouver

1. Xu W, Huang L, Fox A, Patterson D, Jordan MI (2009) Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. ACM, pp 117–132

BibTeX

@inproceedings{Xu_2009, series={SOSP09}, title={Detecting large-scale system problems by mining console logs}, url={http://dx.doi.org/10.1145/1629575.1629587}, DOI={10.1145/1629575.1629587}, booktitle={Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles}, publisher={ACM}, author={Xu, Wei and Huang, Ling and Fox, Armando and Patterson, David and Jordan, Michael I.}, year={2009}, month=Oct, pages={117–132}, collection={SOSP09} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Detecting large-scale system problems by mining console logs

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Source-Code-Augmented Console Log Parsing

Knowl 2 — Message Count Vector Construction via Object Identifiers

Knowl 3 — Subspace PCA Anomaly Detection on Log Feature Vectors

Knowl 4 — State Ratio Vector Construction for Aggregate System Monitoring

Knowl 5 — TF-IDF Weighting for Message Count Feature Matrices

Knowl 6 — Post-Hoc Decision Tree Distillation for Explaining PCA Log Anomalies

Knowl 7 — Anomaly Detection Accuracy and Evaluation on HDFS Logs

Knowl 8 — Performance Anomaly Detection and Root-Cause Diagnosis in Project Darkstar

Knowl 9 — Parsing Accuracy and Distributed Scalability on Cloud Clusters

Knowl 10 — Limitations of Static Source Code Analysis for Log Parsing

References

Citation

BibTeX

Access the Paper