Class RMQPipeline¶
Defined in File stage.py
Inheritance Relationships¶
Base Type¶
public stage.Pipeline(Class Pipeline)
Class Documentation¶
- stage.RMQPipeline : public stage.Pipeline
A 'Pipeline' reading data from RabbitMQ and storing them back to the filesystem. Attributes: host: RabbitMQ host port: RabbitMQ port vhost: RabbitMQ virtual host user: RabbitMQ username password: RabbitMQ password for username cert: The TLS certificate rmq_queue: The RMQ queue to listen to.
Public Functions
- __init__(self, application_name, dest_dir, db_url, db_type, host, port, vhost, user, password, cert, data_queue, model_update_queue=None)¶
Initialize a RMQPipeline that will write data to the 'dest_dir' and optionally publish these files to the kosh-store 'store' by using the stage_dir as an intermediate directory.
- get_load_task(self, o_queue, policy)¶
Return a Task that loads data from the filesystem Args: o_queue: The queue the load task will push read data. Returns: An RMQDomainDataLoaderTask instance reading data from the filesystem and forwarding the values to the o_queue.
- get_model_update_task(self, o_queue, policy)¶
Return a Task receives messages from the training job regarding the status of new models Args: o_queue: The queue to push the model update message. Returns: An RMQControlMessageTask instance reading data from self._model_update_queue and forwarding the values to the o_queue.
- from_cli(cls, args)¶
Create RMQPipeline from the user provided CLI.
- requires_model_update(self)¶
Returns whether the pipeline provides a model-update message parsing mechanism
- shutdown(self, pid)¶
Public Static Functions
- add_cli_args(parser)¶
Add cli arguments to the parser required by this Pipelinereturn .