Hardware Reference
In-Depth Information
10.2.5 Flows
Flow is an important I/O mechanism that combines Trove and BMI for
transporting large amounts of data from disk (via Trove) across the network
(via BMI) to the client's memory (via BMI on the client) or vice versa. Flow
works like software Direct Memory Access (DMA), so client and server can
start a complex transfer and then simply allow it to complete. The alter-
native is to repeatedly interact with higher-layer code to start and com-
plete each part of the transfer, which involves considerably more latency.
Flows also utilize a protocol based on MPI's datatypes to implement non-
contiguous data access on the server without constructing a distinct request
for each part of the transfer. This involves using the le's distribution as well
as the request description to manage data flows from each server to the client
eciently.
10.2.6 Job Layer
Each invocation of BMI, Trove, or the flow subsystem is considered a
job. The job layer manages concurrent tasks and interfaces to the high-level
code implemented in the state machines. The basic workflow involves a state
machine running until it issues a job (asynchronously), at which point other
state machines can run until the job completes and restarts the state machine.
The code that implements and facilitates this interaction is the job layer.
10.2.7 Request State Machines
Both the server and the client library utilize state machines to implement
the complex set of steps needed to start, complete, and finish a request/re-
sponse between the client and server. On the server, whenever a request is
received, its opcode is used to map the receiving state machine to the cor-
rect state machine for that request. Thus there is generally one state machine
per request on the server. On the client, when an application calls a system
interface function, it creates a state machine, which in its course will create
one or more requests and then wait for the responses. Thus there is one state
machine per system interface call. On each end there may be a few additional
state machines for common operations and management function. A state ma-
chine is simply a collection of C-language functions, tied together by a set of
data structures, that allows a program to execute one function at a time and
then move on to the next depending on the return value of the function. A
simple language is used to implement the state transition code. Multiple state
machines can be executing concurrently, and the program can move back and
forth between them in a manner similar to coroutines. Each state potentially
issues an asynchronous operation via the job layer, and it will not execute un-
til that task has returned. In this way, the code cleanly handles large amounts
