Class FSLoaderTask¶

Defined in File stage.py

Inheritance Relationships¶

Base Type¶

public stage.Task (Class Task)

Class Documentation¶

stage.FSLoaderTask : public stage.Task

A FSLoaderTask reads files from the filesystem bundles the data of
the files into batches and forwards them to the next task waiting on the
output queuee.

Attributes:
    o_queue: The output queue to write the transformed messages
    loader: A child class inheriting from FileReader that loads data from the filesystem.
    pattern: The (glob-)pattern of the files to be read.

Public Functions

__init__(self, o_queue, loader, pattern)¶

__call__(self)¶

Busy loop of reading all files matching the pattern and creating
'100' batches which will be pushed on the queue. Upon reading all files
the Task pushes a 'Terminate' message to the queue and returns.

Public Members

o_queue¶

pattern¶

loader¶

datasize_byte¶

total_time_ns¶