S3 connection

S3

S3 file connection.

Based on minio-py client.

Warning

Since onETL v0.7.0 to use S3 connector you should install package as follows:

pip install "onetl[s3]"

# or
pip install "onetl[files]"

See DBR-onetl-install-files-file-connections installation instruction for more details.

Added in 0.5.1

Parameters:

• host

  (str) –

  Host of S3 source. For example: s3.domain.com

• port

  (int, default: 443 ) –

  Port of S3 source

• bucket

  (str) –

  Bucket name in the S3 file source

• access_key

  (str) –

  Access key (aka user ID) of an account in the S3 service

• secret_key

  (SecretStr) –

  Secret key (aka password) of an account in the S3 service

• protocol

  (Literal['http', 'https'], default: 'https' ) –

  Connection protocol.

  Changed in 0.6.0

  Renamed secure: bool to protocol: Literal["https", "http"]

• region

  (str | None, default: None ) –

  Region name of bucket in S3 service. Optional for some S3 implementations (MinIO, Ozone), but could be mandatory for others.

• session_token

  (SecretStr | None, default: None ) –

  Session token generated by S3 STS service, if used.

• extra

  (S3Extra, default: <dynamic> ) –

  Extra options for S3 connection.

  You can pass here any parameters supported by minio.Minio client.

  Added in 0.16.0

  Parameters

  timeout Timeout for requests, see urllib3 documentation. retry Retry for requests, see urllib3 documentation. ssl_verify One of:

  - a path to a file with SSL certificate.
  - a path to a directory with SSL certificates.
  - `True` to use default SSL certificates.
  - `False` to disable SSL certificate verification.
  

Examples:

Create and check S3 connectionCreate and check S3 connection with extra options

from onetl.connection import S3

s3 = S3(
    host="s3.domain.com",
    protocol="http",
    bucket="my-bucket",
    access_key="ACCESS_KEY",
    secret_key="SECRET_KEY",
    region="us-east-1",
).check()

from onetl.connection import S3
from urllib3.util.timeout import Timeout
from urllib3.util.retry import Retry

s3 = S3(
    host="s3.domain.com",
    protocol="http",
    bucket="my-bucket",
    access_key="ACCESS_KEY",
    secret_key="SECRET_KEY",
    region="us-east-1",
    extra=S3.Extra(
        timeout=Timeout(connect=10, read=60),
        retry=Retry(total=5, backoff_factor=0.2),
        ssl_verify=True,
    ),
).check()

path_exists(path)

resolve_dir(path)

resolve_file(path)

create_dir(path)

remove_dir(path, *, recursive=False)

S3Extra

Extra options for S3 connection.

You can pass here any parameters supported by minio.Minio client.

Added in 0.16.0

Parameters:

• timeout

  (Timeout, default: Timeout(connect=10, read=60, total=None) ) –

  Timeout for requests, see urllib3 documentation.

• retry

  (Retry, default: Retry(total=5, connect=None, read=None, redirect=None, status=None) ) –

  Retry for requests, see urllib3 documentation.

• ssl_verify

  (Path | Path | bool, default: True ) –

  One of:

  • a path to a file with SSL certificate.
  • a path to a directory with SSL certificates.
  • True to use default SSL certificates.
  • False to disable SSL certificate verification.