File Uploader Result

Bases: FileResult

Representation of file upload result.

Container for file paths, divided into certain categories:

• successful
• failed
• skipped
• missing

Added in 0.3.0

Examples:

>>> from onetl.file import FileUploader
>>> uploader = FileUploader(target_path="/remote", ...)
>>> upload_result = uploader.run(
...     [
...         "/local/file1",
...         "/local/file2",
...         "/failed/file",
...         "/existing/file",
...         "/missing/file",
...     ]
... )
>>> upload_result
UploadResult(
    successful=FileSet([
        RemoteFile("/remote/file1"),
        RemoteFile("/remote/file2"),
    ]),
    failed=FileSet([
        FailedLocalFile("/failed/file")
    ]),
    skipped=FileSet([
        LocalPath("/existing/file")
    ]),
    missing=FileSet([
        LocalPath("/missing/file")
    ]),
)

successful = Field(default_factory=(lambda: FileSet({}))) class-attribute instance-attribute

File paths (remote) which were uploaded successfully

failed = Field(default_factory=(lambda: FileSet({}))) class-attribute instance-attribute

File paths (local) which were not uploaded because of some failure

skipped = Field(default_factory=(lambda: FileSet({}))) class-attribute instance-attribute

File paths (local) which were skipped because of some reason

missing = Field(default_factory=(lambda: FileSet({}))) class-attribute instance-attribute

File paths (local) which are not present in the local file system

successful_count property

Get number of successful files

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     successful={LocalPath("/some/file"), LocalPath("/some/another.file")},
... )
>>> file_result.successful_count
2

failed_count property

Get number of failed files

Examples:

>>> from onetl.impl import RemoteFile
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     failed={RemoteFile("/some/file"), RemoteFile("/some/another.file")},
... )
>>> file_result.failed_count
2

skipped_count property

Get number of skipped files

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     skipped={LocalPath("/some/file"), LocalPath("/some/another.file")},
... )
>>> file_result.skipped_count
2

missing_count property

Get number of missing files

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     missing={LocalPath("/some/file"), LocalPath("/some/another.file")},
... )
>>> file_result.missing_count
2

total_count property

Get total number of all files

Examples:

>>> from onetl.impl import RemoteFile
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     successful={LocalPath("/local/file"), LocalPath("/local/another.file")},
...     failed={RemoteFile("/remote/file"), RemoteFile("/remote/another.file")},
...     skipped={LocalPath("/skipped/file")},
...     missing={LocalPath("/missing/file")},
... )
>>> file_result.total_count
6

successful_size property

Get size (in bytes) of successful files

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     successful={LocalPath("/some/file"), LocalPath("/some/another.file")},
... )
>>> file_result.successful_size  # in bytes
1024

failed_size property

Get size (in bytes) of failed files

Examples:

>>> from onetl.impl import RemoteFile, RemotePathStat
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     failed={
...         RemoteFile("/some/file", stats=RemotePathStat(st_size=1024)),
...         RemoteFile("/some/another.file"), stats=RemotePathStat(st_size=1024)),
...     },
... )
>>> file_result.failed_size  # in bytes
2048

skipped_size property

Get size (in bytes) of skipped files

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     skipped={LocalPath("/some/file"), LocalPath("/some/another.file")},
... )
>>> file_result.skipped_size  # in bytes
1024

total_size property

Get total size (in bytes) of all files

Examples:

>>> from onetl.impl import RemoteFile, RemotePathStat, LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     successful={LocalPath("/local/file"), LocalPath("/local/another.file")},
...     failed={
...         RemoteFile("/remote/file", stats=RemotePathStat(st_size=1024)),
...         RemoteFile("/remote/another.file", stats=RemotePathStat(st_size=1024))
...     },
...     skipped={LocalPath("/skipped/file")},
...     missing={LocalPath("/missing/file")},
... )
>>> file_result.total_size  # in bytes
4096

is_empty property

Returns True if there are no files in successful, failed and skipped attributes

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result1 = FileResult()
>>> file_result1.is_empty
True
>>> file_result2 = FileResult(
...     successful={LocalPath("/local/file"), LocalPath("/local/another.file")},
... )
>>> file_result2.is_empty
False

details property

Return detailed information about files in the result object

Examples:

>>> from onetl.impl import FailedRemoteFile, LocalPath, RemoteFile, RemotePathStat
>>> from onetl.exception import NotAFileError
>>> from onetl.file.file_result import FileResult
>>> file_result1 = FileResult(
...     successful={
...         RemoteFile("/local/file", stats=RemotePathStat(st_size=1024)),
...         RemoteFile("/local/another.file", stats=RemotePathStat(st_size=1024)),
...     },
...     failed={
...         FailedRemoteFile(
...             path="/remote/file1",
...             stats=RemotePathStat(st_size=0),
...             exception=NotAFileError("'/remote/file1' is not a file"),
...         ),
...         FailedRemoteFile(
...             path="/remote/file2",
...             stats=RemotePathStat(st_size=0),
...             exception=PermissionError("'/remote/file2': [Errno 13] Permission denied"),
...         ),
...     },
...     skipped={LocalPath("/skipped/file1"), LocalPath("/skipped/file2")},
...     missing={LocalPath("/missing/file1"), LocalPath("/missing/file2")},
... )
>>> print(file_result1.details)
Total: 8 files (size='2.0 kB')
<BLANKLINE>
Successful 2 files (size='2.0 kB'):
    '/local/another.file' (size='1.0 kB')
    '/local/file' (size='1.0 kB')
<BLANKLINE>
Failed 2 files (size='0 Bytes'):
    '/remote/file2' (size='0 Bytes')
        PermissionError("'/remote/file2': [Errno 13] Permission denied")
    '/remote/file1' (size='0 Bytes')
        NotAFileError("'/remote/file1' is not a file")
<BLANKLINE>
Skipped 2 files (size='0 Bytes'):
    '/skipped/file1'
    '/skipped/file2'
<BLANKLINE>
Missing 2 files:
    '/missing/file2'
    '/missing/file1'

>>> file_result2 = FileResult()
>>> print(file_result2.details)
No successful files
<BLANKLINE>
No failed files
<BLANKLINE>
No skipped files
<BLANKLINE>
No missing files

summary property

Return short summary about files in the result object

Examples:

>>> from onetl.impl import FailedRemoteFile, LocalPath, RemoteFile, RemotePathStat
>>> from onetl.exception import NotAFileError
>>> from onetl.file.file_result import FileResult
>>> file_result1 = FileResult(
...     successful={
...         RemoteFile("/local/file", stats=RemotePathStat(st_size=1024)),
...         RemoteFile("/local/another.file", stats=RemotePathStat(st_size=1024)),
...     },
...     failed={
...         FailedRemoteFile(
...             path="/remote/file1",
...             stats=RemotePathStat(st_size=0),
...             exception=NotAFileError("'/remote/file1' is not a file"),
...         ),
...         FailedRemoteFile(
...             path="/remote/file2",
...             stats=RemotePathStat(st_size=0),
...             exception=PermissionError("'/remote/file2': [Errno 13] Permission denied"),
...         ),
...     },
...     skipped={LocalPath("/skipped/file1"), LocalPath("/skipped/file2")},
...     missing={LocalPath("/missing/file1"), LocalPath("/missing/file2")},
... )
>>> print(file_result1.summary)
Total: 8 files (size='2.0 kB')
<BLANKLINE>
Successful: 2 files (size='2.0 kB')
<BLANKLINE>
Failed: 2 files (size='0 Bytes')
<BLANKLINE>
Skipped: 2 files (size='0 Bytes')
<BLANKLINE>
Missing: 2 files

>>> file_result2 = FileResult()
>>> print(file_result2.summary)
No files

raise_if_failed()

Raise exception if there are some files in failed attribute

Raises:

• FailedFilesError –

  failed file set is not empty

Examples:

>>> from onetl.impl import FailedRemoteFile, RemotePathStat
>>> from onetl.exception import NotAFileError, FileMissingError
>>> from onetl.file.file_result import FileResult
>>> files_with_exception = [
...     FailedRemoteFile(
...         path="/remote/file1",
...         stats=RemotePathStat(st_size=0),
...         exception=NotAFileError("'/remote/file1' is not a file"),
...     ),
...     FailedRemoteFile(
...         path="/remote/file2",
...         stats=RemotePathStat(st_size=0),
...         exception=PermissionError("'/remote/file2': [Errno 13] Permission denied"),
...     ),
... ]
>>> file_result = FileResult(failed=files_with_exception)
>>> file_result.raise_if_failed()
Traceback (most recent call last)
...
onetl.exception.FailedFilesError: Failed 2 files (size='0 bytes'):
    '/remote/file1' (size='0 bytes')
        NotAFileError("'/remote/file1' is not a file")
<BLANKLINE>
    '/remote/file2' (size='0 Bytes')
        PermissionError("'/remote/file2': [Errno 13] Permission denied")

raise_if_missing()

Raise exception if there are some files in missing attribute

Raises:

• MissingFilesError –

  missing file set is not empty

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     missing={
...         LocalPath("/missing/file1"),
...         LocalPath("/missing/file2"),
...     },
... )
>>> file_result.raise_if_missing()
Traceback (most recent call last):
    ...
onetl.exception.MissingFilesError: Missing 2 files:
    '/missing/file1'
    '/missing/file2'

raise_if_skipped()

Raise exception if there are some files in skipped attribute

Raises:

• SkippedFilesError –

  skipped file set is not empty

Examples:

>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     skipped={
...         LocalPath("/skipped/file1"),
...         LocalPath("/skipped/file2"),
...     },
... )
>>> file_result.raise_if_skipped()
Traceback (most recent call last):
    ...
onetl.exception.SkippedFilesError: Skipped 2 files (15 kB):
    '/skipped/file1' (10kB)
    '/skipped/file2' (5 kB)

raise_if_empty()

Raise exception if there are no files in successful, failed and skipped attributes

Raises:

• EmptyFilesError –

  successful, failed and skipped file sets are empty

Examples:

>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult()
>>> file_result.raise_if_empty()
Traceback (most recent call last):
    ...
onetl.exception.EmptyFilesError: There are no files in the result

raise_if_contains_zero_size()

Raise exception if successful attribute contains a file with zero size

Raises:

• ZeroFileSizeError –

  successful file set contains a file with zero size

Examples:

>>> from onetl.exception import ZeroFileSizeError
>>> from onetl.impl import LocalPath
>>> from onetl.file.file_result import FileResult
>>> file_result = FileResult(
...     successful={
...         LocalPath("/local/empty1.file"),
...         LocalPath("/local/empty2.file"),
...         LocalPath("/local/normal.file"),
...     },
... )
>>> file_result.raise_if_contains_zero_size()
Traceback (most recent call last):
    ...
onetl.exception.ZeroFileSizeError: 2 files out of 3 have zero size:
    '/local/empty1.file'
    '/local/empty2.file'