Postgres connection

Bases: JDBCConnection

PostgreSQL JDBC connection.

Based on Maven package org.postgresql:postgresql:42.7.11 (official Postgres JDBC driver).

See also

Before using this connector please take into account Postgres prerequisites

Added in 0.1.0

Parameters:

• host (str) –

  Host of Postgres database. For example: test.postgres.domain.com or 193.168.1.11

• port (int, default: 5432 ) –

  Port of Postgres database

• user (str) –

  User, which have proper access to the database. For example: some_user

• password (str) –

  Password for database connection

• database (str) –

  Database in RDBMS, NOT schema.

  See this page for more details

• spark (SparkSession) –

  Spark session.

• extra (dict, default: None ) –

  Specifies one or more extra parameters by which clients can connect to the instance.

  For example: {"ssl": "false"}

  See Postgres JDBC driver properties documentation for more details

Examples:

Create and check Postgres connection:

from onetl.connection import Postgres
from pyspark.sql import SparkSession

# Create Spark session with Postgres driver loaded
maven_packages = Postgres.get_packages()
spark = (
    SparkSession.builder.appName("spark-app-name")
    .config("spark.jars.packages", ",".join(maven_packages))
    .getOrCreate()
)

# Create connection
postgres = Postgres(
    host="database.host.or.ip",
    user="user",
    password="*****",
    database="target_database",
    spark=spark,
)

Create read-only connection:

...

# Create connection
postgres = Postgres(
    host="database.host.or.ip",
    user="user",
    password="*****",
    database="target_database",
    extra={"readOnly": True, "readOnlyMode": "always"},  # <--
    spark=spark,
).check()

get_packages(package_version=None) classmethod

Get package names to be downloaded by Spark. Allows specifying a custom JDBC driver version.

Added in 0.9.0

Parameters:

• package_version (str, default: None ) –

  Specifies the version of the PostgreSQL JDBC driver to use. Defaults to 42.7.11.

Examples:

from onetl.connection import Postgres

Postgres.get_packages()

# custom package version
Postgres.get_packages(package_version="42.6.0")

check()

Check source availability.

If not, an exception will be raised.

Returns:

• Self –

  Connection itself

Raises:

• RuntimeError –

  If the connection is not available

Examples:

connection.check()