Predictive modelling of MapReduce job performance in cloud environments using machine learning techniques

Mohammed Bergui, Soufiane Hourri, Said Najah, Nikola S. Nikolov

Research output: Contribution to journalArticlepeer-review

Abstract

Within the Hadoop ecosystem, MapReduce stands as a cornerstone for managing, processing, and mining large-scale datasets. Yet, the absence of efficient solutions for precise estimation of job execution times poses a persistent challenge, impacting task allocation and distribution within Hadoop clusters. In this study, we present a comprehensive machine learning approach for predicting the execution time of MapReduce jobs, encompassing data collection, preprocessing, feature engineering, and model evaluation. Leveraging a rich dataset derived from comprehensive Hadoop MapReduce job traces, we explore the intricate relationship between cluster parameters and job performance. Through a comparative analysis of machine learning models, including linear regression, decision tree, random forest, and gradient-boosted regression trees, we identify the random forest model as the most effective, demonstrating superior predictive accuracy and robustness. Our findings underscore the critical role of features such as data size and resource allocation in determining job performance. With this work, we aim to enhance resource management efficiency and enable more effective utilisation of cloud-based Hadoop clusters for large-scale data processing tasks.

Original languageEnglish
Article number98
JournalJournal of Big Data
Volume11
Issue number1
DOIs
Publication statusPublished - Dec 2024

Keywords

  • Big data
  • Hadoop
  • Machine learning
  • MapReduce
  • Performance modelling
  • Runtime prediction

Fingerprint

Dive into the research topics of 'Predictive modelling of MapReduce job performance in cloud environments using machine learning techniques'. Together they form a unique fingerprint.

Cite this