pymor.parallel.basic
¶
This module contains a base class for implementing WorkerPool.
Module Contents¶
Classes¶
Interface for parallel worker pools. |
|
Handle to remote data on the workers of a |
Functions¶
- class pymor.parallel.basic.WorkerPoolBase[source]¶
Bases:
WorkerPoolDefaultImplementations
,pymor.parallel.interface.WorkerPool
Interface for parallel worker pools.
WorkerPools
allow to easily parallelize algorithms which involve no or little communication between the workers at runtime. The interface methods give the user simple means to distribute data to workers (push
,scatter_array
,scatter_list
) and execute functions on the distributed data in parallel (apply
), collecting the return values from each function call. A single worker can be instructed to execute a function using theWorkerPool.apply_only
method. Finally, a parallelizedmap
function is available, which automatically scatters the data among the workers.All operations are performed synchronously.
- push(self, obj)[source]¶
Push a copy of
obj
to all workers of the pool.A
RemoteObject
is returned as a handle to the pushed object. This object can be used as a keyword argument toapply
,apply_only
,map
and will then be transparently mapped to the respective copy of the pushed object on the worker.Immutable
objects will be pushed only once. If the sameimmutable
object is pushed a second time, the returnedRemoteObject
will refer to the already transferred copy. It is therefore safe to usepush
to ensure that a givenimmutable
object is available on the worker. No unnecessary copies will be created.Parameters
- obj
The object to push to all workers.
Returns
A
RemoteObject
referring to the pushed data.
- apply(self, function, *args, **kwargs)[source]¶
Apply function in parallel on each worker.
This calls
function
on each worker in parallel, passingargs
as positional andkwargs
as keyword arguments. Keyword arguments which areRemoteObjects
are automatically mapped to the respective object on the worker. Moreover, keyword arguments which areimmutable
objects that have already been pushed to the workers will not be transmitted again. (Immutable
objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)Parameters
- function
The function to execute on each worker.
- args
The positional arguments for
function
.- kwargs
The keyword arguments for
function
.
Returns
List of return values of the function executions, ordered by worker number (from
0
tolen(pool) - 1
).
- apply_only(self, function, worker, *args, **kwargs)[source]¶
Apply function on a single worker.
This calls
function
on on the worker with numberworker
, passingargs
as positional andkwargs
as keyword arguments. Keyword arguments which areRemoteObjects
are automatically mapped to the respective object on the worker. Moreover, keyword arguments which areimmutable
objects that have already been pushed to the workers will not be transmitted again. (Immutable
objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)Parameters
- function
The function to execute.
- worker
The worker on which to execute the function. (Number between
0
andlen(pool) - 1
.)- args
The positional arguments for
function
.- kwargs
The keyword arguments for
function
.
Returns
Return value of the function execution.
- map(self, function, *args, **kwargs)[source]¶
Parallel version of the builtin
map
function.Each positional argument (after
function
) must be a sequence of same length n.map
callsfunction
in parallel on each of these n positional argument combinations, always passingkwargs
as keyword arguments. Keyword arguments which areRemoteObjects
are automatically mapped to the respective object on the worker. Moreover, keyword arguments which areimmutable
objects that have already been pushed to the workers will not be transmitted again. (Immutable
objects which have not been pushed before will be transmitted and the remote copy will be destroyed after function execution.)Parameters
- function
The function to execute on each worker.
- args
The sequences of positional arguments for
function
.- kwargs
The keyword arguments for
function
.
Returns
List of return values of the function executions, ordered by the sequence of positional arguments.
- class pymor.parallel.basic.GenericRemoteObject(pool, remote_id, uid=None)[source]¶
Bases:
pymor.parallel.interface.RemoteObject
Handle to remote data on the workers of a
WorkerPool
.See documentation of
WorkerPool
for usage of these handles in conjunction withapply
,scatter_array
,scatter_list
.Remote objects can be used as a context manager: when leaving the context, the remote object’s
remove
method is called to ensure proper cleanup of remote resources.