Hardware Reference
In-Depth Information
FIGURE 8.2: Lustre RPC. [Image courtesy of Intel Corporation.]
the request using unique matchbits, usually the sender's RPC XID. These
matchbits are passed in the RPC request and used by the server effectively as
RDMA descriptors. The client then PUTs the RPC request MB which sends
the request to the server, and waits for all completion events on all MBs. If
successful, all bulk data will now have been transferred under server control
and the reply buffer will contain the RPC reply status.
On startup, Lustre servers attach an initial working set of MEs and as-
sociated MBs to well-known portals corresponding to the services they ex-
port. MEs are set to match incoming PUTs from any Lustre client. They are
matched in the order in which they are attached and each MB is capable of
buffering many incoming RPC requests. A completion event is posted on each
incoming PUT, which is handled by queuing the request on an appropriate
service request queue. This event also signals whether the ME has been de-
tached because its MB is now full. If the number of attached MEs falls below a
low water threshold, additional ME/MBs are allocated and attached to handle
the increased buffering requirement.
Each server maintains a pool of service threads which consume the service
request queues. At rst, only urgent processing is performed|e.g., to deter-
mine whether the client's RPC timeout is inconsistent with current expected
service latency|and requests are queued for processing later. When request
processing proper starts, the service thread handling the request initiates any
required bulk data movement using the RMA descriptors in the request and
then finally PUTs the RPC reply to complete the request.
The number of threads in the service queue determines the number of
RPCs that are processed concurrently. This in turn determines both the num-
ber of concurrent bulk data movements between client and server and the
number of concurrent I/O requests to back-end storage.
