Distributed Apriori Mining on Yelp
Frequent-itemset mining on Hadoop MapReduce, two storage backends
Java Hadoop MapReduce HBase Hive AWS S3 · Feb.–May 2024 · Northeastern CS 6240
Identifying similar restaurants by treating co-reviewed businesses as frequent itemsets, over ~550K Yelp reviews and ~15K businesses.
The pipeline decomposes into candidate generation, support counting, pruning, and iterative frequent-itemset discovery, with Hive external tables preprocessing raw JSON into per-user review baskets.
The part worth keeping: two interchangeable storage backends for cross-pass state. An HBase path materializes each Apriori pass as its own table, loading the previous pass’s frequent itemsets to prune baskets before generating k-combinations, and applying the support threshold at write time. A file-based path writes each pass to HDFS/S3, with a runtime flag switching between local directory traversal and S3 object listing to discover per-pass outputs.