Postgres connection

class onetl.connection.db_connection.postgres.connection.Postgres(*, spark: SparkSession, user: str, password: SecretStr, host: Host, port: int = 5432, database: str, extra: PostgresExtra = PostgresExtra(stringtype='unspecified'))

PostgreSQL JDBC connection.

Based on Maven package org.postgresql:postgresql:42.6.0 (official Postgres JDBC driver).

Warning

Before using this connector please take into account Prerequisites

Parameters:

hoststr

Host of Postgres database. For example: test.postgres.domain.com or 193.168.1.11

portint, default: 5432

Port of Postgres database

userstr

User, which have proper access to the database. For example: some_user

passwordstr

Password for database connection

databasestr

Database in RDBMS, NOT schema.

See this page for more details

sparkpyspark.sql.SparkSession

Spark session.

extradict, default: None

Specifies one or more extra parameters by which clients can connect to the instance.

For example: {"ssl": "false"}

See Postgres JDBC driver properties documentation for more details

Examples

Postgres connection initialization

from onetl.connection import Postgres
from pyspark.sql import SparkSession

# Create Spark session with Postgres driver loaded
maven_packages = Postgres.get_packages()
spark = (
    SparkSession.builder.appName("spark-app-name")
    .config("spark.jars.packages", ",".join(maven_packages))
    .getOrCreate()
)

# Create connection
postgres = Postgres(
    host="database.host.or.ip",
    user="user",
    password="*****",
    database="target_database",
    extra={"ssl": "false"},
    spark=spark,
)

check()

Check source availability.

If not, an exception will be raised.

Returns:

Connection itself

Raises:

RuntimeError

If the connection is not available

Examples

connection.check()

classmethod get_packages() → list[str]

Get package names to be downloaded by Spark.

Examples

from onetl.connection import Postgres

Postgres.get_packages()