Logging#

onetl.log.setup_logging(level: int | str = 20, enable_clients: bool = False) → None#

Set up onETL logging.

What this function does:

  • Adds stderr logging handler

  • Changes root logger format to 2023-05-31 11:22:33.456 [INFO] MainThread: message

  • Changes root logger level to level

  • Changes onETL logger level to level

  • Sets up logging level of underlying client modules

Note

Should be used only in IDEs (like Jupyter notebooks or PyCharm), or scripts (ETL pipelines).

Parameters:
levelint or str, default INFO

Log level for onETL module

enable_clientsbool, default False

If True, enable logging of underlying client modules. Otherwise, set client modules log level to DISABLED.

Note

For level="DEBUG" it is recommended to use enable_clients=True

onetl.log.setup_clients_logging(level: int | str = 9999) → None#

Set logging of underlying client modules used by onETL.

Affected modules:

  • ftputil

  • hdfs

  • minio

  • paramiko

  • py4j

  • pyspark

  • webdav3

Note

Can be used in applications, but it is recommended to set up these loggers according to your framework documentation.

Parameters:
levelint or str, default DISABLED

Log level for client modules

Note

For py4j, logging level with maximum verbosity is INFO because DEBUG logs are totally unreadable.

onetl.log.set_default_logging_format() → None#

Sets default logging format to preferred by onETL.

Example log message: 2023-05-31 11:22:33.456 [INFO] MainThread: message

Note

Should be used only in IDEs (like Jupyter notebooks or PyCharm), or scripts (ETL pipelines).

Warning

Should NOT be used in applications, you should set up logging settings manually, according to your framework documentation.