Base interface

BaseFileLimit()	Base file limit interface.
BaseFileLimit.reset()	Resets the internal limit state.
BaseFileLimit.stops_at(path)	Update internal state and return current state.
BaseFileLimit.is_reached	Check if limit is reached.

class onetl.base.base_file_limit.BaseFileLimit

Base file limit interface.

Limits used by several onETL components, including File Downloader and File Mover, to determine if internal loop should be stopped.

Unlike file filters, limits have internal state which can be updated or reset.

abstract property is_reached: bool

Check if limit is reached.

Returns:

True if limit is reached, False otherwise.

Examples

from onetl.impl import LocalPath

assert not limit.is_reached

assert not limit.stops_at(LocalPath("/path/to/file.csv"))
assert not limit.is_reached

assert limit.stops_at(LocalPath("/path/to/file.csv"))
assert limit.is_reached

abstract reset() → Self

Resets the internal limit state.

Returns:

Returns a filter of the same type, but with non-reached state.

It could be the same filter or a new one, this is an implementation detail.

Examples

assert limit.is_reached

new_limit = limit.reset()
assert not new_limit.is_reached

abstract stops_at(path: PathProtocol) → bool

Update internal state and return current state.

Parameters:

pathonetl.base.path_protocol.PathProtocol

Path to check

Returns:

True if limit is reached, False otherwise.

Examples

from onetl.impl import LocalPath

assert not limit.stops_at(LocalPath("/path/to/file.csv"))
# do this multiple times
...

# stopped on some input
assert limit.stops_at(LocalPath("/path/to/another.csv"))

# at this point, .stops_at() and .is_reached will always return True,
# even on inputs that returned False before.
# it will be in the same state until .reset() is called
assert limit.stops_at(LocalPath("/path/to/file.csv"))