Writing to MongoDB using `DBWriter`#

For writing data to MongoDB, use DBWriter.

Warning

Please take into account MongoDB <-> Spark type mapping

Examples#

from onetl.connection import MongoDB
from onetl.db import DBWriter

mongodb = MongoDB(...)

df = ...  # data is here

writer = DBWriter(
    connection=mongodb,
    target="schema.table",
    options=MongoDB.WriteOptions(
        if_exists="append",
    ),
)

writer.run(df)

Write options#

Method above accepts JDBCWriteOptions

pydantic model onetl.connection.db_connection.mongodb.options.MongoDBWriteOptions#

Writing options for MongoDB connector.

Note

You can pass any value supported by connector, even if it is not mentioned in this documentation.

The set of supported options depends on connector version. See link above.

Warning

Options uri, database, collection are populated from connection attributes, and cannot be overridden by the user in WriteOptions to avoid issues.

Examples

Write options initialization

options = MongoDB.WriteOptions(
    if_exists="append",
    sampleSize=500,
    localThreshold=20,
)

field if_exists: MongoDBCollectionExistBehavior = MongoDBCollectionExistBehavior.APPEND (alias 'mode')#

Behavior of writing data into existing collection.

Possible values:

append (default)
Adds new objects into existing collection.
Behavior in details
Collection does not exist
Collection is created using options provided by user (shardkey and others).

Collection exists
Data is appended to a collection.

Warning

This mode does not check whether collection already contains objects from dataframe, so duplicated objects can be created.
replace_entire_collection
Collection is deleted and then created.
Behavior in details
Collection does not exist
Collection is created using options provided by user (shardkey and others).

Collection exists
Collection content is replaced with dataframe content.
ignore
Ignores the write operation if the collection already exists.
Behavior in details
Collection does not exist
Collection is created using options provided by user

Collection exists
The write operation is ignored, and no data is written to the collection.
error
Raises an error if the collection already exists.
Behavior in details
Collection does not exist
Collection is created using options provided by user

Collection exists
An error is raised, and no data is written to the collection.

Writing to MongoDB using DBWriter#

Examples#

Write options#

Writing to MongoDB using `DBWriter`#