pymor.parallel package¶

Submodules¶

basic module¶

This module contains a base class for implementing WorkerPool.

class pymor.parallel.basic.GenericRemoteObject(pool, remote_id, uid=None)[source]¶

Bases: pymor.parallel.interface.RemoteObject

Methods

Attributes

class pymor.parallel.basic.WorkerPoolBase[source]¶

Bases: pymor.parallel.basic.WorkerPoolDefaultImplementations, pymor.parallel.interface.WorkerPool

Methods

`WorkerPoolBase`	`apply`, `apply_only`, `map`, `push`
`WorkerPoolDefaultImplementations`	`scatter_array`, `scatter_list`
`WorkerPool`	`__len__`
`BasicObject`	`disable_logging`, `enable_logging`

Attributes

BasicObject

logger, logging_disabled, name, uid

apply(function, *args, **kwargs)[source]¶

Apply function in parallel on each worker.

This calls function on each worker in parallel, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute on each worker.
args: The positional arguments for function.
kwargs: The keyword arguments for function.

Returns

List of return values of the function executions, ordered by worker number (from 0 to len(pool) - 1).

apply_only(function, worker, *args, **kwargs)[source]¶

Apply function on a single worker.

This calls function on on the worker with number worker, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute.
worker: The worker on which to execute the function. (Number between 0 and len(pool) - 1.)
args: The positional arguments for function.
kwargs: The keyword arguments for function.

Returns

Return value of the function execution.

map(function, *args, **kwargs)[source]¶

Parallel version of the builtin map function.

Each positional argument (after function) must be a sequence of same length n. map calls function in parallel on each of these n positional argument combinations, always passing kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute on each worker.
args: The sequences of positional arguments for function.
kwargs: The keyword arguments for function.

Returns

List of return values of the function executions, ordered by the sequence of positional arguments.

push(obj)[source]¶

Push a copy of obj to all workers of the pool.

A RemoteObject is returned as a handle to the pushed object. This object can be used as a keyword argument to apply, apply_only, map and will then be transparently mapped to the respective copy of the pushed object on the worker.

Immutable objects will be pushed only once. If the same immutable object is pushed a second time, the returned RemoteObject will refer to the already transferred copy. It is therefore safe to use push to ensure that a given immutable object is available on the worker. No unnecessary copies will be created.

Parameters

obj: The object to push to all workers.

Returns

A RemoteObject referring to the pushed data.

class pymor.parallel.basic.WorkerPoolDefaultImplementations[source]¶

Bases: object

Methods

WorkerPoolDefaultImplementations

scatter_array, scatter_list

default module¶

pymor.parallel.default.new_parallel_pool(ipython_num_engines=None, ipython_profile=None, allow_mpi=True)[source]¶

Creates a new default WorkerPool.

If ipython_num_engines or ipython_profile is provided as an argument or set as a default, an IPythonPool WorkerPool will be created using the given parameters via the ipcluster script.

Otherwise, when allow_mpi is True and an MPI parallel run is detected, an MPIPool WorkerPool will be created.

Otherwise, a sequential run is assumed and pymor.parallel.dummy.dummy_pool is returned.

Defaults

ipython_num_engines, ipython_profile, allow_mpi (see pymor.core.defaults)

dummy module¶

class pymor.parallel.dummy.DummyPool[source]¶

Bases: pymor.parallel.interface.WorkerPool

Methods

`DummyPool`	`apply`, `apply_only`, `map`, `push`, `scatter_array`, `scatter_list`
`BasicObject`	`disable_logging`, `enable_logging`

Attributes

BasicObject

logger, logging_disabled, name, uid

__len__()[source]¶: The number of workers in the pool.

apply(function, *args, **kwargs)[source]¶

Apply function in parallel on each worker.

This calls function on each worker in parallel, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute on each worker.
args: The positional arguments for function.
kwargs: The keyword arguments for function.

Returns

List of return values of the function executions, ordered by worker number (from 0 to len(pool) - 1).

apply_only(function, worker, *args, **kwargs)[source]¶

Apply function on a single worker.

This calls function on on the worker with number worker, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute.
worker: The worker on which to execute the function. (Number between 0 and len(pool) - 1.)
args: The positional arguments for function.
kwargs: The keyword arguments for function.

Returns

Return value of the function execution.

map(function, *args, **kwargs)[source]¶

Parallel version of the builtin map function.

Each positional argument (after function) must be a sequence of same length n. map calls function in parallel on each of these n positional argument combinations, always passing kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute on each worker.
args: The sequences of positional arguments for function.
kwargs: The keyword arguments for function.

Returns

List of return values of the function executions, ordered by the sequence of positional arguments.

push(obj)[source]¶

Push a copy of obj to all workers of the pool.

A RemoteObject is returned as a handle to the pushed object. This object can be used as a keyword argument to apply, apply_only, map and will then be transparently mapped to the respective copy of the pushed object on the worker.

Immutable objects will be pushed only once. If the same immutable object is pushed a second time, the returned RemoteObject will refer to the already transferred copy. It is therefore safe to use push to ensure that a given immutable object is available on the worker. No unnecessary copies will be created.

Parameters

obj: The object to push to all workers.

Returns

A RemoteObject referring to the pushed data.

scatter_array(U, copy=True)[source]¶

Distribute VectorArray evenly among the workers.

On each worker a VectorArray is created holding an (up to rounding) equal amount of vectors of U. The returned RemoteObject therefore refers to different data on each of the workers.

Parameters

U: The VectorArray to distribute.
copy: If False, U will be emptied during distribution of the vectors.

Returns

A RemoteObject referring to the scattered data.

scatter_list(l)[source]¶

Distribute list of objects evenly among the workers.

On each worker a list is created holding an (up to rounding) equal amount of objects of l. The returned RemoteObject therefore refers to different data on each of the workers.

Parameters

l: The list (sequence) of objects to distribute.

Returns

A RemoteObject referring to the scattered data.

class pymor.parallel.dummy.DummyRemoteObject(obj)[source]¶

Bases: pymor.parallel.interface.RemoteObject

Methods

Attributes

interface module¶

class pymor.parallel.interface.RemoteObject[source]¶

Bases: object

Handle to remote data on the workers of a WorkerPool.

See documentation of WorkerPool for usage of these handles in conjunction with apply, scatter_array, scatter_list.

Remote objects can be used as a context manager: when leaving the context, the remote object’s remove method is called to ensure proper cleanup of remote resources.

Methods

Attributes

removed¶: True, after remove has been called.

remove()[source]¶

Remove the remote object from the workers.

Remove the object this handle refers to from all workers. Note that the object will only be destroyed if no other object on the worker holds a reference to that object. Moreover, immutable objects will only be destroyed if remove has been called on all RemoteObjects which refer to the object (see push).

class pymor.parallel.interface.WorkerPool[source]¶

Bases: pymor.core.base.BasicObject

Interface for parallel worker pools.

WorkerPools allow to easily parallelize algorithms which involve no or little communication between the workers at runtime. The interface methods give the user simple means to distribute data to workers (push, scatter_array, scatter_list) and execute functions on the distributed data in parallel (apply), collecting the return values from each function call. A single worker can be instructed to execute a function using the WorkerPool.apply_only method. Finally, a parallelized map function is available, which automatically scatters the data among the workers.

All operations are performed synchronously.

Methods

`WorkerPool`	`apply`, `apply_only`, `map`, `push`, `scatter_array`, `scatter_list`, `__len__`
`BasicObject`	`disable_logging`, `enable_logging`

Attributes

BasicObject

logger, logging_disabled, name, uid

abstract __len__()[source]¶: The number of workers in the pool.

abstract apply(function, *args, **kwargs)[source]¶

Apply function in parallel on each worker.

This calls function on each worker in parallel, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute on each worker.
args: The positional arguments for function.
kwargs: The keyword arguments for function.

Returns

List of return values of the function executions, ordered by worker number (from 0 to len(pool) - 1).

abstract apply_only(function, worker, *args, **kwargs)[source]¶

Apply function on a single worker.

This calls function on on the worker with number worker, passing args as positional and kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute.
worker: The worker on which to execute the function. (Number between 0 and len(pool) - 1.)
args: The positional arguments for function.
kwargs: The keyword arguments for function.

Returns

Return value of the function execution.

abstract map(function, *args, **kwargs)[source]¶

Parallel version of the builtin map function.

Each positional argument (after function) must be a sequence of same length n. map calls function in parallel on each of these n positional argument combinations, always passing kwargs as keyword arguments. Keyword arguments which are RemoteObjects are automatically mapped to the respective object on the worker. Moreover, keyword arguments which are immutable objects that have already been pushed to the workers will not be transmitted again. (Immutable objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)

Parameters

function: The function to execute on each worker.
args: The sequences of positional arguments for function.
kwargs: The keyword arguments for function.

Returns

List of return values of the function executions, ordered by the sequence of positional arguments.

abstract push(obj)[source]¶

Push a copy of obj to all workers of the pool.

A RemoteObject is returned as a handle to the pushed object. This object can be used as a keyword argument to apply, apply_only, map and will then be transparently mapped to the respective copy of the pushed object on the worker.

Immutable objects will be pushed only once. If the same immutable object is pushed a second time, the returned RemoteObject will refer to the already transferred copy. It is therefore safe to use push to ensure that a given immutable object is available on the worker. No unnecessary copies will be created.

Parameters

obj: The object to push to all workers.

Returns

A RemoteObject referring to the pushed data.

abstract scatter_array(U, copy=True)[source]¶

Distribute VectorArray evenly among the workers.

On each worker a VectorArray is created holding an (up to rounding) equal amount of vectors of U. The returned RemoteObject therefore refers to different data on each of the workers.

Parameters

U: The VectorArray to distribute.
copy: If False, U will be emptied during distribution of the vectors.

Returns

A RemoteObject referring to the scattered data.

abstract scatter_list(l)[source]¶

Distribute list of objects evenly among the workers.

On each worker a list is created holding an (up to rounding) equal amount of objects of l. The returned RemoteObject therefore refers to different data on each of the workers.

Parameters

l: The list (sequence) of objects to distribute.

Returns

A RemoteObject referring to the scattered data.

ipython module¶

class pymor.parallel.ipython.IPythonPool(num_engines=None, **kwargs)[source]¶

Bases: pymor.parallel.basic.WorkerPoolBase

WorkerPool based on the IPython parallel computing features.

Parameters

num_engines: Number of IPython engines to use. If None, all available engines are used.
kwargs: Keyword arguments used to instantiate the IPython cluster client.

Methods

`WorkerPoolBase`	`apply`, `apply_only`, `map`, `push`
`WorkerPoolDefaultImplementations`	`scatter_array`, `scatter_list`
`BasicObject`	`disable_logging`, `enable_logging`

Attributes

BasicObject

logger, logging_disabled, name, uid

__len__()[source]¶: The number of workers in the pool.

class pymor.parallel.ipython.RemoteId[source]¶

Bases: int

Methods

int

bit_length, conjugate, from_bytes, to_bytes, __ceil__, __floor__, __new__, __round__, __sizeof__, __trunc__

Attributes

int

denominator, imag, numerator, real

class pymor.parallel.ipython.new_ipcluster_pool(profile=None, cluster_id=None, num_engines=None, ipython_dir=None, min_wait=1, timeout=60)[source]¶

Bases: pymor.core.base.BasicObject

Create a new IPython parallel cluster and connect to it.

This context manager can be used to create an IPythonPool WorkerPool. When entering the context a new IPython cluster is created using the ipcluster script and an IPythonPool is instantiated for the newly created cluster. When leaving the context the cluster is shut down.

Parameters

profile: Passed as --profile parameter to the ipcluster script.
cluster_id: Passed as --cluster-id parameter to the ipcluster script.
nun_engines: Passed as --n parameter to the ipcluster script.
ipython_dir: Passed as --ipython-dir parameter to the ipcluster script.
min_wait: Wait at least this many seconds before trying to connect to the new cluster.
timeout: Wait at most this many seconds for all Ipython cluster engines to become available.

Methods

BasicObject

disable_logging, enable_logging

Attributes

BasicObject

logger, logging_disabled, name, uid

manager module¶

class pymor.parallel.manager.RemoteObjectManager[source]¶

Bases: pymor.core.base.BasicObject

A simple context manager to keep track of RemoteObjects.

When leaving this context, all RemoteObjects that have been managed by this object will be removed.

Methods

`RemoteObjectManager`	`manage`, `remove_objects`
`BasicObject`	`disable_logging`, `enable_logging`

Attributes

BasicObject

logger, logging_disabled, name, uid

manage(remote_object)[source]¶

Add a RemoteObject to the list of managed objects.

Parameters

remote_object: The object to add to the list.

Returns

remote_object

remove_objects()[source]¶: Call remove for all managed objects.

mpi module¶

class pymor.parallel.mpi.MPIPool[source]¶

Bases: pymor.parallel.basic.WorkerPoolBase

WorkerPool based pyMOR’s MPI event loop.

Methods

`WorkerPoolBase`	`apply`, `apply_only`, `map`, `push`
`WorkerPoolDefaultImplementations`	`scatter_array`, `scatter_list`
`BasicObject`	`disable_logging`, `enable_logging`

Attributes

BasicObject

logger, logging_disabled, name, uid

__len__()[source]¶: The number of workers in the pool.