High Throughput Data Movement - Scientific Data Management - page 168

5.2.4 In-Transit Services
In addition to the data movement and low-level data capture interfaces de-
scribed above, applications that require high-throughput adaptive I/O must
also depend on robust transport and specialization services. Such specializa-
tion services are required to perform “in-transit” data inspection and manip-
ulation, including filtering, aggregation, or other types of processing actions
that tune the data output to the current user- or application-specific require-
ments. The use of specialization services jointly with the basic data move-
ment results in attainment of adaptive I/O services, needed to address the
dynamism in application inputs and outputs, computational and communica-
tion loads and operating conditions, and end-user interests. We focus here on
techniques for the autonomic tuning of these transports to provide the user-
level quality of information and specification of utility that next-generation
application data flows require. An example of this is a scientist who is partic-
ularly interested in one type of interatomic bond during a molecular dynamics
simulation and who is, under bandwidth constraints, willing to rely on a spe-
cialized transport service that filters out simulation output not related to
atoms involved in such bonds, or that gives those data outputs higher priority
compared with other outputs. The detection of the bandwidth limitation and
the selection of the appropriate specialization action (i.e., filtering or change
in priority) should happen autonomically, without additional intervention of
the simulation user.

5.2.4.1 Structured Data Transport: EVPath

After data events have been captured through the DataTap implementation of
the ADIOS interface, an event processing architecture is provided in support
of high-performance data streaming in networks with internal processing ca-
pacity. EVPath, the newest incarnation of a publish/subscribe infrastructure
developed over many years, 22 , 23 is designed to allow for easy implementation of
overlay networks with active data processing, routing, and management at all
points within the overlay. In addition, EVPath allows the use of a higher-level
control substrate to enable global overlay creation and management. Domain-
specific control layers allow the management of the overlay to best utilize the
underlying physical resources and provide for overlays that best address the
application needs. For instance, the IFLOW management layer described in
Reference 14 is best suited for large-scale, wide-area, streaming applications,
whereas another version of a control layer is more suitable for a massively
parallel processor (MPP) such as the Cray system, where management com-
ponents on compute nodes have limited ability for interaction with external
control entities.
The basic building block in EVPath is a stone . An overlay path is com-
prised of a number of connected stones. A stone is a lightweight entity that
roughly corresponds to processing points in a dataflow diagram. Stones can