File Downloader Options#

pydantic model onetl.file.file_downloader.options.FileDownloaderOptions#

File downloading options

field if_exists: FileExistBehavior = FileExistBehavior.ERROR (alias 'mode')#

How to handle existing files in the local directory.

Possible values:

  • error (default) - do nothing, mark file as failed

  • ignore - do nothing, mark file as ignored

  • replace_file - replace existing file with a new one

  • replace_entire_directory - delete local directory content before downloading files

field delete_source: bool = False#

If True, remove source file after successful download.

If download failed, file will left intact.

field workers: int = 1#

Number of workers to create for parallel file download.

1 (default) means files will me downloaded sequentially. 2 or more means files will be downloaded in parallel workers.

Recommended value is min(32, os.cpu_count() + 4), e.g. 5.