Database Reference
In-Depth Information
• Running ordinary commands on remote machines
• Distributing local data directly among remote machines
• Sending files to remote machines, processing them, and retrieving the results
Get a List of Running AWS EC2 Instances
In this section, we're creating a file named
instances
that will contain one hostname of
a remote machine per line. We're using Amazon Web Services in this section. If you're
using a different cloud computing service, or have your own servers, make sure that
you create an
instances
file yourself.
We can obtain a list of running AWS EC2 instances from the command line using
aws
, the command-line interface to the AWS API (Amazon Web Services, 2014). If
you're not using the Data Science Toolbox, install
awscli
using
pip
(PyPA, 2014) as
follows:
$
pip install awscli
With
aws
, you can virtually do everything you can do with the online AWS Manage‐
ment Console. We just use this tool to obtain a list of running EC2 instances from
AWS, but it can do a lot more. We assume that you know how to launch instances,
either through the online AWS Management Console or through the
aws
command-
line tool.
The command
aws ec2 describe-instances
returns a lot of information about all
your EC2 instances in JSON format (see
the AWS documentation
.
We extract the rel‐
evant fields using
jq
:
$
aws ec2 describe-instances | jq
'.Reservations[].Instances[] | '
\
>
'{public_dns: .PublicDnsName, state: .State.Name}'
{
"state": "running",
"public_dns": "ec2-54-88-122-140.compute-1.amazonaws.com"
}
{
"state": "stopped",
"public_dns": null
}
The possible states of an EC2 instance are
pending
,
running
,
shutting-down
,
termi‐
nated
,
stopping
, and
stopped
. Because we can only distribute our pipeline to running
instances, we filter out the nonrunning instances:
$
aws ec2 describe-instances | jq -r
'.Reservations[].Instances[] | '
\
>
'select(.State.Name=="running") | .PublicDnsName'
> instances
$
cat instances
