pymor.parallel.basic

This module contains a base class for implementing WorkerPool.

Module Contents

class pymor.parallel.basic.GenericRemoteObject(pool, remote_id, uid=None)

Bases: pymor.parallel.interface.RemoteObject

Handle to remote data on the workers of a WorkerPool.

See documentation of WorkerPool for usage of these handles in conjunction with apply, scatter_array, scatter_list.

Remote objects can be used as a context manager: when leaving the context, the remote object’s remove method is called to ensure proper cleanup of remote resources.

removed

True, after remove has been called.

class pymor.parallel.basic.WorkerPoolBase

Bases: WorkerPoolDefaultImplementations, pymor.parallel.interface.WorkerPool

Interface for parallel worker pools.

WorkerPools allow to easily parallelize algorithms which involve no or little communication between the workers at runtime. The interface methods give the user simple means to distribute data to workers (push, scatter_array, scatter_list) and execute functions on the distributed data in parallel (apply), collecting the return values from each function call. A single worker can be instructed to execute a function using the WorkerPool.apply_only method. Finally, a parallelized map function is available, which automatically scatters the data among the workers.

All operations are performed synchronously.

Methods

apply	Apply function in parallel on each worker.
apply_only	Apply function on a single worker.
map	Parallel version of the builtin map function.
push	Push a copy of obj to all workers of the pool.

apply(function, *args, **kwargs)

Apply function in parallel on each worker.

This calls function on each worker in parallel, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function

The function to execute on each worker.

args

The positional arguments for function.

kwargs

The keyword arguments for function.

Returns

List of return values of the function executions, ordered by worker number (from 0 to len(pool) - 1).

apply_only(function, worker, *args, **kwargs)

Apply function on a single worker.

This calls function on on the worker with number worker, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function

The function to execute.

worker

The worker on which to execute the function. (Number between 0 and len(pool) - 1.)

args

The positional arguments for function.

kwargs

The keyword arguments for function.

Returns

Return value of the function execution.

map(function, *args, **kwargs)

Parallel version of the builtin map function.

Each positional argument (after function) must be a sequence of same length n. map calls function in parallel on each of these n positional argument combinations, always passing kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function

The function to execute on each worker.

args

The sequences of positional arguments for function.

kwargs

The keyword arguments for function.

Returns

List of return values of the function executions, ordered by the sequence of positional arguments.

push(obj)

Push a copy of obj to all workers of the pool.

A RemoteObject is returned as a handle to the pushed object. This object can be used as a keyword argument to apply, apply_only, map and will then be transparently mapped to the respective copy of the pushed object on the worker.

Immutable objects will be pushed only once. If the same immutable object is pushed a second time, the returned RemoteObject will refer to the already transferred copy. It is therefore safe to use push to ensure that a given immutable object is available on the worker. No unnecessary copies will be created.

Parameters

obj

The object to push to all workers.

Returns

A RemoteObject referring to the pushed data.

class pymor.parallel.basic.WorkerPoolDefaultImplementations

Methods

scatter_array
scatter_list

scatter_array(U, copy=True)

scatter_list(l)