Skip to content

Filesystem Warehouse

IcebergFilesystemWarehouse

Iceberg Filesystem Warehouse.

Added in 0.15.0

Note

This warehouse uses FileDFConnection classes to access data at the warehouse location. It relies on Spark's filesystem configuration and behavior.

Parameters:

  • connection

    (SparkFileDFConnection) –

    File connection for data storage

  • path

    (PurePathProtocol) –

    Warehouse path

Examples:

from onetl.connection import Iceberg, SparkLocalFS

local_fs_connection = SparkLocalFS(spark=spark)

warehouse = Iceberg.FilesystemWarehouse(
    connection=local_fs_connection,
    path="/warehouse/path",
)

from onetl.connection import Iceberg, SparkHDFS

hdfs_connection = SparkHDFS(
    host="namenode",
    cluster="my-cluster",
    spark=spark,
)

warehouse = Iceberg.FilesystemWarehouse(
    connection=hdfs_connection,
    path="/warehouse/path",
)

from onetl.connection import Iceberg, SparkS3

s3_connection = SparkS3(
    host="s3.domain.com",
    protocol="http",
    bucket="my-bucket",
    access_key="access_key",
    secret_key="secret_key",
    path_style_access=True,
    region="us-east-1",
    spark=spark,
)

warehouse = Iceberg.FilesystemWarehouse(
    connection=s3_connection,
    path="/warehouse/path"
)