Source Documentation

Module: calculon.Calculon

class calculon.Calculon(prod_func, prod_kwargs, prod_use_threads, cons_func, cons_kwargs, cons_use_threads)

Producer-consumer class. Responsible for initializing producers and consumers and controlling execution.

__init__(prod_func, prod_kwargs, prod_use_threads, cons_func, cons_kwargs, cons_use_threads)

Initializes Calculon.

Keyword arguments

• prod_func – producer function that accepts one argument (dictionary of values);
• prod_kwargs – a list of dictionaries, each representing a set of arguments for an instance of the producer function;
• prod_use_threads – a flag specifying if threads are used to run producer code (if False, processes are used);
• cons_func – consumer function that accepts one argument (dictionary of values)
• cons_kwargs – a list of dictionaries, each representing a set of arguments for an instance of the consumer function
• cons_use_threads – a flag specifying if threads are used to run consumer code (if False, processes are used).

start()

Starts producer and consumer threads / processes and controls the execution.

Keyword arguments

None

Returns

Returns a dictionary containing two elements:

• value for key “producers” contains a list of results returned by each of the producer instance.
• value for key “consumers” contains a list of results returned by each of the consumer instance.

Module: calculon.Producer

class calculon._Producer(func, kwargs, queue, pipe)

Producer class.

__init__(func, kwargs, queue, pipe)

Producer superclass, contains all of the functionality. Calculon does not deal with this class directly, rather it instantiates ProducerThread or ProducerProcess that inherit from _Producer and from either Thread or Process classes.

Keyword arguments

• queue – instance of multiprocessing.Queue;
• func – function that puts values into the queue, optional return value;
• kwargs – a dictionary of arguments passed to func;
• pipe – end of a multiprocessing.Pipe() to which producer can write the results.

run()

Runs the producer function once.

Keyword arguments

None

When the producer function is called, two additional arguments are passed to it:

• _name – unique name of the producer (uuid);
• _queue – the queue object where to put the results;.

Returns

• If the run completed successfully, the method returns a dictionary with two keys:

  • “name” – name of this producer (uuid);
  • “result” – the return value returned from the producer function.

• If the run completed unsuccessfully, the method returns a dictionary with two keys:

  • “name” – name of this producer (uuid);
  • “exception” – the exception object for the raised exception.

class calculon.ProducerThread(func, kwargs, queue)

Thread-based producer class.

__init__(func, kwargs, queue)

Instantiates _Producer and Thread superclasses.

class calculon.ProducerProcess(func, kwargs, queue, pipe)

Process-based producer class.

__init__(func, kwargs, queue, pipe)

Instantiates _Producer and Process superclasses.

Module calculon.Consumer

class calculon._Consumer(func, kwargs, queue, pipe)

Consumer class.

__init__(func, kwargs, queue, pipe)

Consumer superclass, contains all of the functionality. Calculon does not deal with this class directly, rather it instantiates ConsumerThread or ConsumerProcess that inherit from _Consumer and from either Thread or Process classes.

Keyword arguments

• queue – instance of multiprocessing.Queue;
• func – function that processes values received from the queue, once at a time;
• kwargs – a dictionary of arguments passed to func;
• pipe – end of a multiprocessing.Pipe() to which consumer can write the results.

run()

Runs the producer function once.

Keyword arguments

None

Consumer will repeteadly try to get a value from the queue until all producers have shutdown and the queue is not empty. Each time it succeeds, the value is passed to the consumer function to process the value. Once there are no more values in the queue and all of the producers have stopped, each of the consumers will be called once more, to allow to perform any sort of cleanup that might be required.

Along with each call to the consumer function, the following values are passed in the argument dictionary.

• _name – unique name of the consumer (uuid);
• _value – value from the queue to process during this call;
• _last_call – a flag that when set to True indicates that this is the last “cleanup” call to the consumer. Also note that if If _last_call is True, _value is None.
• _result – contains the return value of the previous call to the consumer function. Set to None on the first call.

shutdown()

Sets the exit flag. The exit flag is used by the run() code to check if the consumer can be shutdown safely. It is called from the Calculon instance and indicates that all of the producers have stopped running.

class calculon.ConsumerThread(func, kwargs, queue)

Instantiates _Consumer and Thread superclasses.

class calculon.ConsumerProcess(func, kwargs, queue, pipe)

Instantiates _Consumer and Process superclasses.