pyspark.sql.streaming.DataStreamWriter.options#

DataStreamWriter.options(**options)[source]#

Adds output options for the underlying data source.

New in version 2.0.0.

Changed in version 3.5.0: Supports Spark Connect.

Notes

This API is evolving.

Examples

>>> df = spark.readStream.format("rate").load()
>>> df.writeStream.option("x", 1)
<...streaming.readwriter.DataStreamWriter object ...>

Specify options in a dictionary.

>>> df.writeStream.options(**{"k1": "v1", "k2": "v2"})
<...streaming.readwriter.DataStreamWriter object ...>

The example below specifies ‘numRows’ and ‘truncate’ options to Console source in order to print 3 rows for every batch without truncating the results.

>>> import time
>>> q = spark.readStream.format(
...     "rate").option("rowsPerSecond", 10).load().writeStream.format(
...         "console").options(numRows=3, truncate=False).start()
>>> time.sleep(3)
>>> q.stop()