Parallel Pipelines - Data Science at the Command Line - page 127

ec2-54-88-122-140.compute-1.amazonaws.com
ec2-54-88-89-208.compute-1.amazonaws.com
(If we would leave out -r , which stands for raw , the hostnames would have been sur‐
rounded by double quotes.) We save the output to instances , so that we can pass this
to parallel later.
As mentioned, parallel employs SSH to connect to the EC2 instances. Add the fol‐
lowing to ~/.ssh/conig , so that SSH knows how to connect to the EC2 instances:
Host *.amazonaws.com
IdentityFile ~/.ssh/MyKeyFile.pem
User ubuntu
Depending on which distribution you're running, your username may be different
than ubuntu .
Running Commands on Remote Machines
The first flavor of distributed processing is to simply run ordinary commands on
remote machines. Let's first double-check that parallel is working by running the
command-line tool hostname to get a list of hosts:
$ parallel --nonall --slf instances hostname
ip-172-31-23-204
ip-172-31-23-205
Here, the --slf option is short for the --sshloginfile option. The --nonall option
instructs parallel to execute the same command on every remote machine in the
instances file without using any parameters. Remember, if you don't have any remote
machines to utilize, you can replace --slf instances with --sshlogin : so that the
command runs on your local machine:
$ parallel --nonall --sshlogin : hostname
data-science-toolbox
Running the same command on every remote machine once only requires one CPU
core per machine. If we wanted to distribute the list of arguments passed in to paral
lel then it could potentially use more than one CPU core. If the number of CPU
cores is not specified explicitly, parallel will try to determine this:
$ seq 2 | parallel --slf instances echo 2>&1 | fold
bash: parallel: command not found
parallel: Warning: Could not figure out number of cpus on ec2-54-88-122-140.comp
ute-1.amazonaws.com (). Using 1.
1
2
In this case, we have parallel installed on one of the two remote machines. We're
getting a warning message indicating that parallel is not found on one of them. As a