pyspark.sql.datasource.DataSourceStreamReader.initialOffset

DataSourceStreamReader.initialOffset()

Return the initial offset of the streaming data source. A new streaming query starts reading data from the initial offset. If Spark is restarting an existing query, it will restart from the check-pointed offset rather than the initial one.

Returns

dict

A dict or recursive dict whose key and value are primitive types, which includes Integer, String and Boolean.

Examples

>>> def initialOffset(self):
...     return {"parititon-1": {"index": 3, "closed": True}, "partition-2": {"index": 5}}