Remote Storage Resource Management in WS-PGRADE/gUSE - Science Gateways for Distributed Computing Infrastructures - page 76

authentication data. When the con guration of a job has been completed and all

remote les

along with references of the necessary credential information (used to access the
storage) are recorded in the workflow.
WS-PGRADE/gUSE automatically resolves remote

files are assigned to the relevant ports, the URI of the selected remote

file references during

workflow execution and provides the relevant remote files for jobs transparently.

Depending on whether the job is executed via a so-called wrapper script , one of
following two mechanisms ensures fetching inputs and storing outputs:

1. If the job cannot be executed by a wrapper (e.g., on Portable Batch System
clusters), the DCI Bridge downloads the necessary remote

first using the
Data Avenue API. These are then passed to the computing element before job
execution. Once the job completes and the results are passed back to the DCI
Bridge, outputs are uploaded to the speci

les

ed location by the DCI Bridge using

Blacktop services.
2. If it is possible to use a wrapper for job execution (e.g., on clouds), DCI Bridge
merely creates HTTP aliases for the appropriate remote storage resources, and
these alias URLs are delegated along with the wrapper script to the computing
element. The wrapper script

files via the
provided URLs, executes the job, and then uploads the results to the output
URLs. The advantage of this solution over (1) is that it lowers the CPU load of
the DCI Bridge, and also the network traffic, as remote files are transferred once,
directly to the location of use.

first downloads the necessary input

Credentials required to access remote storage resources are not stored directly in
the workflows. Instead, only references to credentials are recorded allowing safe
publication and sharing of workflows

dential
authentication data. On workflow export, only credential references are saved, and
these references need to be speci

— without compromising con

ed. The credential references are then resolved
when the workflow is imported by another user, likely owning different authenti-
cation data.
It is also possible to use robot certi

cates for Data Avenue port types. These

certi cates are credentials de ned with a global-scope within a portal instance.

Robot certi cates are set by workflow developers in advance, and portal users, in

this way, are not required to have individual credentials to access storage or
computing resources (authorization is granted to all portal users automatically).
Parameter-sweep applications are performed in slightly different way for per-
formance reasons. Generator jobs generate a number of

files that will serve as input
for consecutive job(s). As there may be possibly thousands of files created by

generators, these files are zipped first, resulting in an archive which is uploaded as a

single tar.gz file via Data Avenue. Such uploads are performed using a special

“ ”

le
on-the-fly, i.e., the extracted content will be stored on the remote storage resource.
On such uploads Data Avenue returns the number of entries contained by the
archive; this number is used later by the workflow interpreter to determine the
number of jobs to be launched. The workflow interpreter then starts the required

archive attribute; this mode instructs Data Avenue to extract the uploaded