Big Data Analysis - Big Data: Related Technologies, Challenges and Future Prospects - page 53

Tabl e 5. 1 Comparison of MPI, MapReduce and Dryad

MPI

MapReduce Dryad

Deployment Computing node and data Computing and data storage Computing and data storage

storage arranged separately arranged at the same node arranged at the same node

(Data should be moved (Computing should (Computing should

computing node) be close to data) be close to data)

Resource - Workqueue(google) Not clear

management/ HOD(Yahoo)

scheduling
Low level

MPI API MapReduce API Dryad API

programming
High level Null Pig, Hive, Jaql, Scope, DryadLINQ

programming
Data storage

The local file system, GFS(google) NTFS,

NFS, HDFS(Hadoop), KFS Cosmos DFS

Amazon S3,

Task User manually partition Automation Automation

partitioning the tasks

Communication Messaging, Remote Files(Local FS, DFS) Files, TCP Pipes,

memory access Shared-memory FIFOs

Fault-tolerant Checkpoint Task re-execute Task re-execute