File Uploader Options

FileUploaderOptions

File uploading options.

Added in 0.3.0

Examples:

from onetl.file import FileUploader

options = FileUploader.Options(
    if_exists="replace_entire_directory",
    delete_local=True,
    workers=4,
)

Parameters:

• if_exists

  (FileExistBehavior, default: <FileExistBehavior.ERROR: 'error'> ) –

  How to handle existing files in the target directory.

  Possible values:

  • error (default) - mark file as failed
  • ignore - mark file as skipped
  • replace_file - replace existing file with a new one
  • replace_entire_directory - delete local directory content before downloading files

  Changed in 0.9.0

  Renamed mode → if_exists

• delete_local

  (bool, default: False ) –

  If True, remove local file after successful download.

  If download failed, file will left intact.

  Added in 0.2.0

  Changed in 0.3.0

  Move FileUploader.delete_local to FileUploaderOptions

• workers

  (int, default: 1 ) –

  Number of workers to create for parallel file upload.

  1 (default) means files will me uploaded sequentially. 2 or more means files will be uploaded in parallel workers.

  Recommended value is min(32, os.cpu_count() + 4), e.g. 5.

  Added in 0.8.1