Custom Spark DataFrameWriter for writing data to a given layer.

The layer type will be inferred from the layer configuration. Therefore the api to write to index layer or versioned layer is the same. To write to an index or versioned layer, your application must perform the following operations:

• Have some data as a DataFrame.
• Use LayerDataFrameWriter.DataFrameExt.writeLayer to create a LayerDataFrameWriter.
• Call withDataConverter to specify how the data groupings should be merged to a single data file. This is not necessary if the data is stored in avro, parquet, or protobuf as the appropriate DataConverter will be inferred by the layer's content type.
• Call save to save the data stored in the DataFrame to the given layer

Below is an example written in Scala that demonstrates how to write data to an index layer:

import com.here.platform.data.client.spark.LayerDataFrameWriter.DataFrameExt

val spark =
  SparkSession
    .builder()
    .appName(getClass.getSimpleName)
    .master("local[*]")
    .getOrCreate()

val dataFrame: DataFrame = ???
dataFrame
    .writeLayer(catalogHrn, indexLayer)
    .save()

Java developers should use com.here.platform.data.client.spark.javadsl.JavaLayerDataFrameWriter#writeLayer instead of dataFrame.writeLayer:

Dataset<Row> df = ???
JavaLayerDataFrameWriter.create(df)
    .writeLayer(catalogHrn, layerId)
    .save()

The batch size (number of Rows) for a grouping can be restricted to a certain amount by setting the option olp.groupedBatchSize (ie 2 for 2 Rows to be in each group):

val dataFrame: DataFrame = ???
dataFrame
    .writeLayer(catalogHrn, indexLayer)
    .option("olp.groupedBatchSize", 2)
    .save()