pyspark.sql.DataFrameWriterV2.partitionedBy

DataFrameWriterV2.partitionedBy(col: pyspark.sql.column.Column, *cols: pyspark.sql.column.Column) → pyspark.sql.readwriter.DataFrameWriterV2

Partition the output table created by create, createOrReplace, or replace using the given columns or transforms.

When specified, the table data will be stored by these values for efficient reads.

For example, when a table is partitioned by day, it may be stored in a directory layout like:

• table/day=2019-06-01/

• table/day=2019-06-02/

Partitioning is one of the most widely used techniques to optimize physical data layout. It provides a coarse-grained index for skipping unnecessary data reads when queries have predicates on the partitioned columns. In order for partitioning to work well, the number of distinct values in each column should typically be less than tens of thousands.

col and cols support only the following functions:

• pyspark.sql.functions.years()

• pyspark.sql.functions.months()

• pyspark.sql.functions.days()

• pyspark.sql.functions.hours()

• pyspark.sql.functions.bucket()

New in version 3.1.

previous

pyspark.sql.DataFrameWriterV2.tableProperty

next

pyspark.sql.DataFrameWriterV2.create