Skip to content

Hive Connection

Bases: DBConnection

Spark connection with Hive MetaStore support. support hooks

See also

Before using this connector please take into account Hive prerequisites

Added in 0.1.0

Parameters:

  • cluster (str) –

    Cluster name. Used for HWM and lineage.

    Added in 0.7.0

  • spark (SparkSession) –

    Spark session with Hive metastore support enabled

Examples:

Execute kinit consome command before creating Spark Session

$ kinit -kt /path/to/keytab user

from onetl.connection import Hive
from pyspark.sql import SparkSession

# Create Spark session

spark = (
    SparkSession.builder.appName("spark-app-name")
    .option("spark.kerberos.access.hadoopFileSystems", "hdfs://cluster.name.node:8020")
    .option("spark.kerberos.principal", "user")
    .option("spark.kerberos.keytab", "/path/to/keytab")
    .enableHiveSupport()
    .getOrCreate()
)

# Create connection
hive = Hive(cluster="rnd-dwh", spark=spark).check()


from onetl.connection import Hive
from pyspark.sql import SparkSession

# Create Spark session
spark = SparkSession.builder.appName("spark-app-name").enableHiveSupport().getOrCreate()

# Create connection
hive = Hive(cluster="rnd-dwh", spark=spark).check()

get_current(spark) classmethod

Create connection for current cluster. support hooks

Note

Can be used only if there are some hooks bound to Slots.get_current_cluster slot.

Added in 0.7.0

Parameters:

  • spark (SparkSession) –

    Spark session

Examples:

from onetl.connection import Hive
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("spark-app-name").enableHiveSupport().getOrCreate()

# injecting current cluster name via hooks mechanism
hive = Hive.get_current(spark=spark)

check()

Check source availability. support hooks

If not, an exception will be raised.

Returns:

  • Self –

    Connection itself

Raises:

  • RuntimeError –

    If the connection is not available

Examples:

connection.check()