Scalable Algorithms for Association Mining
Mohammed J. Zaki
Introduces the ECLAT and related lattice-decomposition algorithms for frequent itemset mining, demonstrating an order-of-magnitude speedup over Apriori by combining vertical database layouts with independent memory-bound sublattice search strategies.
Organizations increasingly rely on discovering shared patterns across vast volumes of customer transactions to optimize retail layouts, catalog designs, and diagnostic systems. However, uncovering these frequent associations in massive datasets is computationally expensive and constrained by disk operations. Established methods require repeated scans over the entire database and rely on intricate internal memory structures that cause substantial processing delays when handling millions of records.
The article develops and evaluates scalable data mining algorithms designed to identify frequent item combinations rapidly while minimizing costly database reads. Specifically, it demonstrates how organizing items into mathematical structures called lattices allows the search space to be broken down into independent chunks that can be processed directly in computer memory.
The evaluated approach utilizes a vertical data format where each item tracks its own list of transaction identifiers. By performing simple intersection operations between these lists, the system determines pattern frequencies without building complex indexing trees. The article introduces techniques to decompose the full problem into smaller subproblems using common item prefixes or graph clusters known as maximal cliques, combined with bottom-up, top-down, or hybrid search strategies. Testing was conducted on synthetic retail benchmark databases scaling from 100,000 up to 5 million transactions, assessing performance under varying pattern lengths and transaction volumes.
The experimental findings show that the proposed methods dramatically outperform traditional techniques. The leading algorithm, MaxClique, reduced search candidate operations by factors ranging from 3 to 35 and accelerated execution speeds by a factor of 20 to over 30 compared to standard benchmarks like Apriori, reaching improvements exceeding two orders of magnitude in challenging scenarios. The baseline vertical algorithm, Eclat, routinely outperformed traditional tools by more than an order of magnitude while maintaining an average memory footprint of under 2 percent of total database size. Furthermore, the hybrid search methods successfully discovered long patterns of over 20 items where previous methods failed due to memory exhaustion, all while demonstrating linear scaling as transaction volumes grew.
These results demonstrate that organizations can process enterprise-scale association mining tasks with significantly lower hardware overhead, shorter processing windows, and reduced memory costs. Because the core operations rely on list intersections, these algorithms can be integrated directly into relational database management systems using standard database query operations, reducing the risk and cost of deploying specialized external data structures.
Technical leaders seeking to implement frequent pattern mining should deploy the MaxClique algorithm when long frequent combinations exist and graph density remains manageable, while using Eclat as a robust default when processing very dense datasets or extremely low support thresholds. Systems can be configured with modest memory allocations, as subproblem decomposition keeps working memory requirements exceptionally low.
The primary operational limitation involves dense association graphs—which occur under very large transaction sizes or very low support thresholds—where clique generation introduces computational overhead and overlap. The article provides high confidence in linear scalability across large transaction volumes on benchmark datasets, though organizations operating on unusually dense item relationships should evaluate prefix-based approaches before committing to clique-based partitioning.
- Paper: Fast Algorithms for Mining Association Rules in Large Databases, R. Agrawal et al. (1994). It introduces the foundational Apriori algorithm and level-wise candidate generation framework that the source specifically seeks to accelerate and decompose using subset lattice traversal.
- Paper: Mining association rules between sets of items in large databases, R. Agrawal et al. (1993). It defines the core problem of mining association rules from large transactional databases and establishes the two-phase paradigm of finding frequent itemsets before generating rules.
- Paper: Dynamic itemset counting and implication rules for market basket data, Sergey Brin et al. (1997). It presents dynamic itemset counting as an early advancement over Apriori to reduce database passes, establishing key multi-pass optimization challenges addressed by the source.
- Paper: Data Mining: An Overview from a Database Perspective, Ming-Syan Chen et al. (1996). It provides a comprehensive overview of early data mining and association rule discovery methods from a database perspective, detailing the candidate generation and vertical layout trade-offs analyzed in the source.
- Paper: Mining frequent patterns without candidate generation, Jiawei Han et al. (2000). It takes frequent itemset discovery beyond candidate generation and lattice traversal entirely by introducing the FP-tree and FP-growth algorithm.
- Paper: Mining and summarizing customer reviews, Minqing Hu et al. (2004). It directly applies association rule mining techniques to extract frequent feature terms and summarize opinions from online customer reviews.
