Thread-Level Parallelism - Computer Architecture: A Quantitative Approach - page 344

The multiprocessors we examine in this chapter typically range in size from a dual pro-
cessor to dozens of processors and communicate and coordinate through the sharing of
memory. Although sharing through memory implies a shared address space, it does not ne-
cessarily mean there is a single physical memory. Such multiprocessors include both single-
chip systems with multiple cores, known as multicore , and computers consisting of multiple
chips, each of which may be a multicore design.
In addition to true multiprocessors, we will return to the topic of multithreading, a tech-
nique that supports multiple threads executing in an interleaved fashion on a single multiple
issue processor. Many multicore processors also include support for multithreading.
In the next chapter, we consider ultrascale computers built from very large numbers of pro-
cessors, connected with networking technology and often called clusters ; these large-scale sys-
tems are typically used for cloud computing with a model that assumes either massive num-
bers of independent requests or highly parallel, intensive compute tasks. When these clusters
grow to tens of thousands of servers and beyond, we call them warehouse-scale computers .
In addition to the multiprocessors we study here and the warehouse-scaled systems of
the next chapter, there are a range of special large-scale multiprocessor systems, sometimes
called multicomputers , which are less tightly coupled than the multiprocessors examined in this
chapter but more tightly coupled than the warehouse-scale systems of the next. The primary
use for such multicomputers is in high-end scientiic computation. Many other topics, such as
Culler, Singh, and Gupta [1999] , cover such systems in detail. Because of the large and chan-
ging nature of the field of multiprocessing (the just-mentioned Culler et al. reference is over
1000 pages and discusses only multiprocessing!), we have chosen to focus our atention on
what we believe is the most important and general-purpose portions of the computing space.
Appendix I discusses some of the issues that arise in building such computers in the context
of large-scale scientific applications.
Thus, our focus will be on multiprocessors with a small to moderate number of processors (2
to 32). Such designs vastly dominate in terms of both units and dollars. We will pay only slight
atention to the larger-scale multi-processor design space (33 or more processors), primarily
in Appendix I, which covers more aspects of the design of such processors, as well as the be-
havior performance for parallel scientific workloads, a primary class of applications for large-
scale multiprocessors. In large-scale multiprocessors, the interconnection networks are a crit-
ical part of the design; Appendix F focuses on that topic.
Multiprocessor Architecture: Issues And Approach
To take advantage of an MIMD multiprocessor with n processors, we must usually have at
least n threads or processes to execute. The independent threads within a single process are
typically identified by the programmer or created by the operating system (from multiple in-
dependent requests). At the other extreme, a thread may consist of a few tens of iterations of
a loop, generated by a parallel compiler exploiting data parallelism in the loop. Although the
amount of computation assigned to a thread, called the grain size , is important in considering
how to exploit thread-level parallelism efficiently, the important qualitative distinction from
instruction-level parallelism is that thread-level parallelism is identified at a high level by the
software system or programmer and that the threads consist of hundreds to millions of in-
structions that may be executed in parallel.
Threads can also be used to exploit data-level parallelism, although the overhead is likely
to be higher than would be seen with an SIMD processor or with a GPU (see Chapter 4 ). This
overhead means that grain size must be sufficiently large to exploit the parallelism efficiently.