Comparison of Resource Management Frameworks for Processing Big Data

Abstract

Big data analytic solutions are usually deployed on computer-cluster or cloud computing environment. Efficient resource management is a challenging task in a cluster or cloud environment. Big data processing frameworks hide the details of resource management from the end user. In this paper, we have described the traditional Hadoop and state-of-the-art resource management frameworks used for big data processing. For the evaluation of resource management framework we have identified a feature vector comprising resource assignment, scheduling, security, and physical systemlevel resources. Using the feature vector, resource management frameworks have been compared in order to identify strengths and weaknesses of each. Identified weaknesses indicate the future research directions for further improvement of respective framework. It has been observed that YARN framework qualifies for the most features in the feature vector.