5. Using Multiple Processors

This section of the tutorial shows all the big work that is needed to paralellize operations in eap.

5.1. Multiprocessing Module

This part of the tutorial shows all the work that is needed to use the multiprocessing module in EAP.

Note

While Python 2.6 is required for the multiprocessing module, the pickling of partial function is possible only since Python 2.7 (or 3.1), earlier version of Python may throw some strange errors when using partial function in the multiprocessing multiprocessing.Pool.map(). This may be avoided by creating local function outside of the toolbox (in Python version 2.6).

Note

The pickling of lambda function is not yet available in Python.

5.1.1. Evaluation

The multiprocessing example shows how to use the multiprocessing module in order to enhance the computing power during the evaluations. First the toolbox contains a method named map(), this method has the same function as the built-in map() function. In order to use the multiprocessing module into the built-in algorithms, the only thing to do is to replace the map operation by a parallel one. Then the difference between the Multiprocessing One Max Example and the Regular One Max Example is the addition of these two lines

# Process Pool of 4 workers
pool = multiprocessing.Pool(processes=4)
tools.register("map", pool.map)

5.1.2. Variation

The paralellization of the variation operators is not directly supported in the algorithms, although it is still possible. What one needs is to create its own algorithm (from one in the algorithm module for example) and change the desired lines in order to use the map() method from the toolbox. This may be achieved for example, for the crossover operation from the eaSimple() algorithm by replacing the crossover part of the algorithms by

parents1 = list()
parents2 = list()
to_replace = list()
for i in range(1, len(offsprings), 2):
    if random.random() < cxpb:
        parents1.append(offsprings[i - 1])
        parents2.append(offsprings[i])
        to_replace.append(i - 1)
        to_replace.append(i)

children = tools.map(tools.mate, (parents1, parents2))

for i, child in zip(to_replace, children):
    del child.fitness.values
    offsprings[i] = child

Since the multiprocessing map does take a single iterable we must bundle/unbundle the parents, respectively by creating a tuple in the tools.map() function of the preceding code example and the following decorator on the crossover function.

def unbundle(func):
    def wrapUnbundle(bundled):
        return func(*bundled)
    return wrapUnbundle

tools.decorate("mate", unbundle)