Options

class onetl.file.file_df_reader.options.FileDFReaderOptions(*, recursiveFileLookup: bool | None = None, **kwargs)

Options for FileDFReader.

See Spark Generic File Data Source documentation for more details.

Note

You can pass any value supported by Spark, even if it is not mentioned in this documentation. Option names should be in camelCase!

The set of supported options depends on Spark version. See link above.

Examples

Created reader options

from onetl.file import FileDFReader

options = FileDFReader.Options(recursive=True)

field recursive: bool | None = None (alias 'recursiveFileLookup')

If True, perform recursive file lookup.

Warning

This disables partition inferring using file paths.

Warning

Can be used only in Spark 3+. See SPARK-27990.