Building Analytics Workf lows Using Python and Pandas - Data Just Right: Introduction to Large-Scale Data and Analytics - page 173

example, we take advantage of iPython's direct interface , which makes it easy to apply a
Python function to a collection of objects in parallel. The DirectView object exposes a
map function that works a lot like the standard Python map, applying the same func-
tion to each potential prime number. The difference here is that iPython can farm
out the prime-number checks to any core available rather than trying to run them in
sequence.
To start a new cluster, create an iPython profile to hold the configuration informa-
tion needed for the local test cluster. Simply run iPython's profile create command,
and then use the ipcluster command to start the cluster engines (see Listing 12.8). It's
also very easy to activate local parallel engines using the iPython notebook with controls
under the cluster tab.
To augment our original prime-number script, we will start by importing the
Client class from the IPython.parallel package. This will enable our script to connect to
the individual engines in our running cluster. Because our engines do not have access
to the prime-number-checking functions that we've defined in their namespace, we
use the dview.push command to add them to all engines. Now we pass our prime-
number-reporting function as a parameter in our client's dataview.map method with
our list of potential prime numbers as the other input. iPython's cluster-management
API will handle the complexities of our parallel computation automatically. See
Listing 12.8 for the code.
Listing 12.8 Parallelizing a computationally intensive task
# Create a new profile to hold parallel configuration
> ipython profile create --parallel -profile=testprofile
# Start a cluster with 4 engines using the testprofile configuration
> ipcluster start --n=4 --profile=testprofile
# parallel_prime_finder.py: iPython parallel prime-number finder
from IPython.parallel import Client
import numpy as np
# Create an iPython parallel client instance and DataView
rc = Client()
dview = rc[:]
# Check if an integer is prime
def prime_check(num):
# Check all integers between 2 and num
for test in range(2, int(num**0.5) + 1):
# Divisible by test number?
if num % test == 0:
return False
return True