Adaptive Execution of Scientific Workflow Applications on Clouds - Cloud Computing with e-Science Applications - page 83

To support not only these requirements for data and computation but also
the requirements listed in the previous section, we proposed and developed
a workflow middleware whose architecture is described next.
4.5 System Architecture
The requirements presented previously are addressed by software middle-
ware comprising a WfMS augmented with capabilities for data analytics
integrated as a second layer above the WfMS. The overall organization of
the system is depicted in Figure 4.3. It shows the S 3 application architecture,
which consists of the adaptive cloud WfMS, the ETL or preprocessing algo-
rithm, the data analysis algorithm, and the agent-based simulation.
Cloud WfMS system . The cloud WfMS is responsible for workflow
scheduling, big data handling, and dynamic resource scaling on
hybrid clouds. The Cloud WfMS comprises the workflow engine,
task dispatcher, and resource management. The workflow sched-
uling coordinates the execution of tasks, handles communication
between components, implements the scheduling algorithm, and
manages the execution of applications on distributed resources. The
task dispatcher component submits tasks to resources for execu-
tion. The resource management component interacts with the cloud
infrastructure to enable resource allocation.
Preprocessing and data analysis . This component is responsible
for managing preprocessing and data analysis activities that
are required to train the synthetic journey function that gener-
ates the synthetic journey. It tackles the scalability challenge by
dynamically scaling up the number of VM instances; thus, the pre-
processing processes are executed in parallel. Since this is a compu-
tationally intensive task with a long duration and the total number
of origin-station and destination-station pairs is large (composed
of more than 8,000 pairs), VM instances are pooled from a hybrid
cloud where each VM instance processes the travel duration for
each origin-station and destination-station pair.
Agent-based simulation. There are three phases of agent-based simu-
lation: agent creation, attribute definition, and simulation execution.
Our module is able to scale the process of agent-based generation
in orders of magnitude of up to millions of agents. Further in this
chapter, we demonstrate the process for 6.9 million commuter agents,
90 station agents, and 200 train agents. The activities of the process
of simulation execution are (1) time series simulation with 1-second