0.9.0 (2023-08-17)

Breaking Changes

• Rename methods:

  • DBConnection.read_df -> DBConnection.read_source_as_df

  • DBConnection.write_df -> DBConnection.write_df_to_target (#66)

• Rename classes:

  • HDFS.slots -> HDFS.Slots

  • Hive.slots -> Hive.Slots

  Old names are left intact, but will be removed in v1.0.0 (#103)

• Rename options to make them self-explanatory:

  • Hive.WriteOptions(mode="append") -> Hive.WriteOptions(if_exists="append")

  • Hive.WriteOptions(mode="overwrite_table") -> Hive.WriteOptions(if_exists="replace_entire_table")

  • Hive.WriteOptions(mode="overwrite_partitions") -> Hive.WriteOptions(if_exists="replace_overlapping_partitions")

  • JDBC.WriteOptions(mode="append") -> JDBC.WriteOptions(if_exists="append")

  • JDBC.WriteOptions(mode="overwrite") -> JDBC.WriteOptions(if_exists="replace_entire_table")

  • Greenplum.WriteOptions(mode="append") -> Greenplum.WriteOptions(if_exists="append")

  • Greenplum.WriteOptions(mode="overwrite") -> Greenplum.WriteOptions(if_exists="replace_entire_table")

  • MongoDB.WriteOptions(mode="append") -> Greenplum.WriteOptions(if_exists="append")

  • MongoDB.WriteOptions(mode="overwrite") -> Greenplum.WriteOptions(if_exists="replace_entire_collection")

  • FileDownloader.Options(mode="error") -> FileDownloader.Options(if_exists="error")

  • FileDownloader.Options(mode="ignore") -> FileDownloader.Options(if_exists="ignore")

  • FileDownloader.Options(mode="overwrite") -> FileDownloader.Options(if_exists="replace_file")

  • FileDownloader.Options(mode="delete_all") -> FileDownloader.Options(if_exists="replace_entire_directory")

  • FileUploader.Options(mode="error") -> FileUploader.Options(if_exists="error")

  • FileUploader.Options(mode="ignore") -> FileUploader.Options(if_exists="ignore")

  • FileUploader.Options(mode="overwrite") -> FileUploader.Options(if_exists="replace_file")

  • FileUploader.Options(mode="delete_all") -> FileUploader.Options(if_exists="replace_entire_directory")

  • FileMover.Options(mode="error") -> FileMover.Options(if_exists="error")

  • FileMover.Options(mode="ignore") -> FileMover.Options(if_exists="ignore")

  • FileMover.Options(mode="overwrite") -> FileMover.Options(if_exists="replace_file")

  • FileMover.Options(mode="delete_all") -> FileMover.Options(if_exists="replace_entire_directory")

  Old names are left intact, but will be removed in v1.0.0 (#108)

• Rename onetl.log.disable_clients_logging() to onetl.log.setup_clients_logging(). (#120)

Features

• Add new methods returning Maven packages for specific connection class:

  • Clickhouse.get_packages()

  • MySQL.get_packages()

  • Postgres.get_packages()

  • Teradata.get_packages()

  • MSSQL.get_packages(java_version="8")

  • Oracle.get_packages(java_version="8")

  • Greenplum.get_packages(scala_version="2.12")

  • MongoDB.get_packages(scala_version="2.12")

  • Kafka.get_packages(spark_version="3.4.1", scala_version="2.12")

  Deprecate old syntax:

  • Clickhouse.package

  • MySQL.package

  • Postgres.package

  • Teradata.package

  • MSSQL.package

  • Oracle.package

  • Greenplum.package_spark_2_3

  • Greenplum.package_spark_2_4

  • Greenplum.package_spark_3_2

  • MongoDB.package_spark_3_2

  • MongoDB.package_spark_3_3

  • MongoDB.package_spark_3_4 (#87)

• Allow to set client modules log level in onetl.log.setup_clients_logging().

  Allow to enable underlying client modules logging in onetl.log.setup_logging() by providing additional argument enable_clients=True. This is useful for debug. (#120)

• Added support for reading and writing data to Kafka topics.

  For these operations, new classes were added.

  • Kafka (#54, #60, #72, #84, #87, #89, #93, #96, #102, #104)

  • Kafka.PlaintextProtocol (#79)

  • Kafka.SSLProtocol (#118)

  • Kafka.BasicAuth (#63, #77)

  • Kafka.KerberosAuth (#63, #77, #110)

  • Kafka.ScramAuth (#115)

  • Kafka.Slots (#109)

  • Kafka.ReadOptions (#68)

  • Kafka.WriteOptions (#68)

  Currently, Kafka does not support incremental read strategies, this will be implemented in future releases.

• Added support for reading files as Spark DataFrame and saving DataFrame as Files.

  For these operations, new classes were added.

  FileDFConnections:

  • SparkHDFS (#98)

  • SparkS3 (#94, #100, #124)

  • SparkLocalFS (#67)

  High-level classes:

  • FileDFReader (#73)

  • FileDFWriter (#81)

  File formats:

  • Avro (#69)

  • CSV (#92)

  • JSON (#83)

  • JSONLine (#83)

  • ORC (#86)

  • Parquet (#88)

Improvements

• Remove redundant checks for driver availability in Greenplum and MongoDB connections. (#67)

• Check of Java class availability moved from .check() method to connection constructor. (#97)