Class FSPipeline¶
Defined in File stage.py
Inheritance Relationships¶
Base Type¶
public stage.Pipeline(Class Pipeline)
Class Documentation¶
- stage.FSPipeline : public stage.Pipeline
A 'Pipeline' reading data from the Filesystem and storing them back to the filesystem. Attributes: src: The source directory to read data from. pattern: The pattern to glob files from. src_type: The file format of the source data
Public Functions
- __init__(self, application_name, dest_dir, db_url, db_type, src, src_type, pattern)¶
Initialize a FSPipeline that will write data to the 'dest_dir' and optionally publish these files to the kosh-store 'store' by using the stage_dir as an intermediate directory.
- get_load_task(self, o_queue, policy)¶
Return a Task that loads data from the filesystem Args: o_queue: The queue the load task will push read data. Returns: An FSLoaderTask instance reading data from the filesystem and forwarding the values to the o_queue.
- from_cli(cls, args)¶
Create FSPipeline from the user provided CLI.
- requires_model_update(self)¶
Returns whether the pipeline provides a model-update message parsing mechanism
- get_model_update_task(self, o_queue, policy)¶
- shutdown(self, pids)¶
Public Static Functions
- add_cli_args(parser)¶
Add cli arguments to the parser required by this Pipeline.
Public Static Attributes
- supported_readers = ("shdf5", "dhdf5")¶