File Limit (legacy)

class onetl.core.file_limit.file_limit.FileLimit(*, count_limit: int = 100)

Limits the number of downloaded files.

Deprecated since version 0.8.0: Use MaxFilesCount instead.

Parameters:

count_limitint, default = 100

Number of downloaded files at a time.

Examples

Create a FileLimit object and set the amount in it:

from onetl.core import FileLimit

limit = FileLimit(count_limit=1500)

If you create a onetl.file.file_downloader.file_downloader.FileDownloader object without specifying the limit option, it will download with a limit of 100 files.

property is_reached: bool

Check if limit is reached.

Returns:

True if limit is reached, False otherwise.

Examples

from onetl.impl import LocalPath

assert not limit.is_reached

assert not limit.stops_at(LocalPath("/path/to/file.csv"))
assert not limit.is_reached

assert limit.stops_at(LocalPath("/path/to/file.csv"))
assert limit.is_reached

reset()

Resets the internal limit state.

Returns:

Returns a filter of the same type, but with non-reached state.

It could be the same filter or a new one, this is an implementation detail.

Examples

assert limit.is_reached

new_limit = limit.reset()
assert not new_limit.is_reached

stops_at(path: PathProtocol) → bool

Update internal state and return current state.

Parameters:

pathonetl.base.path_protocol.PathProtocol

Path to check

Returns:

True if limit is reached, False otherwise.

Examples

from onetl.impl import LocalPath

assert not limit.stops_at(LocalPath("/path/to/file.csv"))
# do this multiple times
...

# stopped on some input
assert limit.stops_at(LocalPath("/path/to/another.csv"))

# at this point, .stops_at() and .is_reached will always return True,
# even on inputs that returned False before.
# it will be in the same state until .reset() is called
assert limit.stops_at(LocalPath("/path/to/file.csv"))