22. multi
— Framework for multiprocessing in pyFormex¶
This module contains some functions to perform multiprocessing in pyFormex in a unified way.
22.1. Functions defined in module multi¶

multi.
splitArgs
(args, mask=None, nproc=1, close=False)[source]¶ Split data blocks over multiple processors.
Parameters:  args (list or tuple of array_like) – A sequence of data blocks that may need to be split for parallel processing over multiple processors. Splitting is done along the first dimension, which should therefore be the same for all arrays in the sequence that need to be split.
 mask (list or tuple of bool) – If provided, this flags the items in
args
that should be split. The list should have the same length asargs
. If not provided, all array type items inargs
will be split.  nproc (int) – Intended number of processors. If negative (default), it is set equal to the number of processors detected on the host machine.
 close (bool) – If True, the elements where the arrays are split are included in both blocks delimited by the element. Thus splitting an array [1, 2, 3] in two results in [1, 2] and [2, 3], while the default split would be [1, 2] and [3].
Returns: list of tuples. – The list contains
nproc
tuples and each tuple contains the same number of items as the inputargs
and in the same order, whereby the (nonmasked) arrays are replaced by a slice of the array along its first axis, and the masked and nonarray items are replicated as is.See also
arraytools.splitar()
 the low level function used to do the splitting
Examples
>>> splitArgs([np.arange(5),'abcde'],nproc=3) [(array([0, 1]), 'abcde'), (array([2]), 'abcde'), (array([3, 4]), 'abcde')] >>> for i in splitArgs([np.eye(5),'=>',np.arange(5)],nproc=3): ... print("%s %s %s" % i) [[ 1. 0. 0. 0. 0.] [ 0. 1. 0. 0. 0.]] => [0 1] [[ 0. 0. 1. 0. 0.]] => [2] [[ 0. 0. 0. 1. 0.] [ 0. 0. 0. 0. 1.]] => [3 4] >>> for i in splitArgs([np.eye(5),'=>',np.arange(5)],mask=[1,0,0],nproc=3): ... print("%s %s %s" % i) [[ 1. 0. 0. 0. 0.] [ 0. 1. 0. 0. 0.]] => [0 1 2 3 4] [[ 0. 0. 1. 0. 0.]] => [0 1 2 3 4] [[ 0. 0. 0. 1. 0.] [ 0. 0. 0. 0. 1.]] => [0 1 2 3 4]

multi.
dofunc
(arg)[source]¶ Helper function for the multitask function.
Parameters: arg (tuple) – The first item of the tuple is a callable. The remaining items are its arguments. Returns: object – The value of the callable when passed the remaining items as arguments. Examples
>>> dofunc((max,(2, 5, 3))) 5

multi.
multitask
(tasks, nproc=1)[source]¶ Perform tasks in parallel.
Runs a number of tasks in parallel over a number of subprocesses.
Parameters:  tasks (list of tuples) – Each task in the list is a tuple where the first item is a callable and the other items are the arguments to be passed to the callable.
 nproc (int) – The number of subprocesses to be started. This may be
different from the number of tasks to run: processes finishing a
task will pick up a next one. There is no benefit in starting more
processes than the number of tasks or the number of processing units
available. The default will set
nproc
to the minimum of these two values.
Examples
>>> task1 = (int.__add__, (2, 3)) >>> task2 = (int.__mul__, (2, 3)) >>> multitask((task1, task2)) [5, 6]