File Downloader¶

Bases: FrozenModel

Allows you to download files from a remote source with specified file connection and parameters, and return an object with download result summary.

Note

FileDownloader can return different results depending on Read Strategies

Note

This class is used to download files only from remote directory to the local one.

It does NOT support direct file transfer between filesystems, like FTP -> SFTP. You should use FileDownloader + File Uploader to implement FTP -> local dir -> SFTP.

Added in 0.1.0

Changed in 0.8.0

Moved onetl.core.FileDownloader → onetl.file.FileDownloader

Parameters:

connection (FileConnection) –

Class which contains File system connection properties. See File Connections section.
local_path (PathLike | str) –

Local path where you download files
source_path (PathLike | str, default: None ) –

Remote path to download files from.

Could be None, but only if you pass absolute file paths directly to run method
temp_path (PathLike | str, default: None ) –

If set, this path will be used for downloading a file, and then renaming it to the target file path. If None is passed, files are downloaded directly to target_path.

Warning

In case of production ETL pipelines, please set a value for temp_path (NOT None). This allows to properly handle download interruption, without creating half-downloaded files in the target, because unlike file download, rename call is atomic.

Warning

In case of connections like SFTP or FTP, which can have multiple underlying filesystems, please pass to temp_path path on the SAME filesystem as target_path. Otherwise instead of rename, remote OS will move file between filesystems, which is NOT atomic operation.

Added in 0.5.0
filters (list[BaseFileFilter]) –

Return only files/directories matching these filters. See File Filters

Changed in 0.3.0

Replaces old source_path_pattern: str and exclude_dirs: str options.

Changed in 0.8.0

Renamed filter → filters
limits (list[BaseFileLimit]) –

Apply limits to the list of files/directories, and stop if one of the limits is reached. See File Limits

Added in 0.4.0

Changed in 0.8.0

Renamed limit → limits
options (Options | dict | None, default: None ) –

File downloading options. See FileDownloader.Options

Added in 0.3.0
hwm (type[HWM] | None, default: None ) –

HWM class to detect changes in incremental run. See File HWM

Warning

Used only in IncrementalStrategy.

Added in 0.5.0

Changed in 0.10.0

Replaces deprecated hwm_type attribute

Examples:

Minimal exampleFull exampleIncremental download (by tracking list of file paths)Incremental download (by tracking file modification time)

from onetl.connection import SFTP
from onetl.file import FileDownloader

sftp = SFTP(...)

# create downloader
downloader = FileDownloader(
    connection=sftp,
    source_path="/path/to/remote/source",
    local_path="/path/to/local",
)

# download files to "/path/to/local"
downloader.run()

from onetl.connection import SFTP
from onetl.file import FileDownloader
from onetl.file.filter import Glob, ExcludeDir
from onetl.file.limit import MaxFilesCount, TotalFileSize

sftp = SFTP(...)

# create downloader with a bunch of options
downloader = FileDownloader(
    connection=sftp,
    source_path="/path/to/remote/source",
    local_path="/path/to/local",
    temp_path="/tmp",
    filters=[
        Glob("*.txt"),
        ExcludeDir("/path/to/remote/source/exclude_dir"),
    ],
    limits=[MaxFilesCount(100), TotalFileSize("10GiB")],
    options=FileDownloader.Options(delete_source=True, if_exists="replace_file"),
)

# download files to "/path/to/local",
# but only *.txt,
# excluding files from "/path/to/remote/source/exclude_dir" directory
# and stop before downloading 101 file
downloader.run()

from onetl.connection import SFTP
from onetl.file import FileDownloader
from onetl.strategy import IncrementalStrategy
from etl_entities.hwm import FileListHWM

sftp = SFTP(...)

# create downloader
downloader = FileDownloader(
    connection=sftp,
    source_path="/path/to/remote/source",
    local_path="/path/to/local",
    hwm=FileListHWM(  # mandatory for IncrementalStrategy
        name="my_unique_hwm_name",
    ),
)

# download files to "/path/to/local", but only added since previous run
with IncrementalStrategy():
    downloader.run()

from onetl.connection import SFTP
from onetl.file import FileDownloader
from onetl.strategy import IncrementalStrategy
from etl_entities.hwm import FileModifiedTimeHWM

sftp = SFTP(...)

# create downloader
downloader = FileDownloader(
    connection=sftp,
    source_path="/path/to/remote/source",
    local_path="/path/to/local",
    hwm=FileModifiedTimeHWM(  # mandatory for IncrementalStrategy
        name="my_unique_hwm_name",
    ),
)

# download files to "/path/to/local", but only modified/created since previous run
with IncrementalStrategy():
    downloader.run()

`run(files=None)` ¶

Method for downloading files from source to local directory.

Note

This method can return different results depending on Read Strategies

Added in 0.1.0

Parameters:

files (Iterable[str | PathLike] | None, default: None ) –

File list to download.

If empty, download files from source_path to local_path, applying filter, limit and hwm to each one (if set).

If not, download to local_path all input files, ignoring filters, limits and HWM.

Added in 0.3.0

Returns:

DownloadResult –

Download result object

Raises:

DirectoryNotFoundError –

source_path does not found
NotADirectoryError –

source_path or local_path is not a directory

Examples:

Download files from source_path to local_path:

>>> from onetl.file import FileDownloader
>>> downloader = FileDownloader(source_path="/remote", local_path="/local", ...)
>>> download_result = downloader.run()
>>> download_result
DownloadResult(
    successful=FileSet([
        LocalPath("/local/file1.txt"),
        LocalPath("/local/file2.txt"),
        # directory structure is preserved
        LocalPath("/local/nested/path/file3.txt"),
    ]),
    failed=FileSet([
        FailedRemoteFile("/remote/failed.file"),
    ]),
    skipped=FileSet([
        RemoteFile("/remote/already.exists"),
    ]),
    missing=FileSet([
        RemotePath("/remote/missing.file"),
    ]),
)

Download only certain files from source_path:

>>> from onetl.file import FileDownloader
>>> downloader = FileDownloader(source_path="/remote", local_path="/local", ...)
>>> # paths could be relative or absolute, but all should be in "/remote"
>>> download_result = downloader.run(
...     [
...         "/remote/file1.txt",
...         "/remote/nested/path/file3.txt",
...         # excluding "/remote/file2.txt"
...     ]
... )
>>> download_result
DownloadResult(
    successful=FileSet([
        LocalPath("/local/file1.txt"),
        # directory structure is preserved
        LocalPath("/local/nested/path/file3.txt"),
    ]),
    failed=FileSet([]),
    skipped=FileSet([]),
    missing=FileSet([]),
)

Download certain files from any folder:

>>> from onetl.file import FileDownloader
>>> downloader = FileDownloader(local_path="/local", ...)  # no source_path set
>>> # only absolute paths
>>> download_result = downloader.run(
...     [
...         "/remote/file1.txt",
...         "/any/nested/path/file2.txt",
...     ]
... )
>>> download_result
DownloadResult(
    successful=FileSet([
        LocalPath("/local/file1.txt"),
        # directory structure is NOT preserved without source_path
        LocalPath("/local/file2.txt"),
    ]),
    failed=FileSet([]),
    skipped=FileSet([]),
    missing=FileSet([]),
)

`view_files()` ¶

Get file list in the source_path, after filter, limit and hwm applied (if any).

Note

This method can return different results depending on Read Strategies

Added in 0.3.0

Raises:

DirectoryNotFoundError –

source_path does not found
NotADirectoryError –

source_path is not a directory

Returns:

FileSet[RemoteFile] –

Set of files in source_path, which will be downloaded by run method

Examples:

View files:

>>> from onetl.file import FileDownloader
>>> downloader = FileDownloader(source_path="/remote", ...)
>>> downloader.view_files()
FileSet([
    RemoteFile("/remote/file1.txt"),
    RemoteFile("/remote/file3.txt"),
    RemoteFile("/remote/nested/file3.txt"),
])

File Downloader¶

run(files=None) ¶

view_files() ¶

`run(files=None)` ¶

`view_files()` ¶