Distributed Programming for the Cloud - Large Scale and Big Data: Processing and Management - page 6

network card interfaces typically represent major bottlenecks in terms of bandwidth,
performance, and/or throughput. By distributing work across machines, data can be
serviced from multiple disks simultaneously, thus offering an increasingly aggregate
I/O bandwidth, improving performance, and maximizing throughput. In summary, dis-
tributed programs play a critical role in rapidly solving various computing problems and
effectively mitigating resource bottlenecks. This subsequently improves performances,
increases throughput and reduces costs, especially on the cloud.
1.5 MODELS OF DISTRIBUTED PROGRAMS
Distributed programs are run on distributed systems, which consist of networked
computers. The cloud is a special distributed system. In this section, we first define
distributed systems and draw a relationship between clouds and distributed systems.
Second, in an attempt to answer the question of how to program the cloud, we present
two traditional distributed programming models, which can be used for that sake, the
shared-memory and the message-passing programming models. Third, we discuss
the computation models that cloud programs can employ. Specifically, we describe the
synchronous and asynchronous computation models. Fourth, we present the two main
parallelism categories of distributed programs intended for clouds, data parallelism
and graph parallelism . Lastly, we end the discussion with the architectural models
that cloud programs can typically utilize, master/slave and peer-to-peer architectures.
1.5.1 D istributeD s ystems anD the C louD
Networks of computers are ubiquitous. The Internet, high-performance computing
(HPC) clusters, mobile phone, and in-car networks, among others, are common exam-
ples of networked computers. Many networks of computers are deemed as distributed
systems. We define a distributed system as one in which networked computers com-
municate using message passing and/or shared memory and coordinate their actions to
solve a certain problem or offer a specific service. One significant consequence of our
definition pertains to clouds. Specifically, since a cloud is defined as a set of Internet-
based software, platform and infrastructure services offered through a cluster of
networked computers (i.e., a datacenter), it becomes a distributed system. Another con-
sequence of our definition is that distributed programs will be the norm in distributed
systems such as the cloud. In particular, we defined distributed programs in Section 1.1
as a set of sequential programs that run on separate processors at different machines.
Thus, the only way for tasks in distributed programs to interact over a distributed system
is to either send and receive messages explicitly or read and write from/to a shared dis-
tributed memory supported by the underlying distributed system. We next discuss these
two possible ways of enabling distributed tasks to interact over distributed systems.
1.5.2 t raDitional P rogramming m oDels anD
D istributeD a nalytiCs e ngines
A distributed programming model is an abstraction provided to programmers so
that they can translate their algorithms into distributed programs that can execute