Package com.here.platform.data.client.spark.datasources.generic

Class GenericDataConverter

java.lang.Object
com.here.platform.data.client.spark.datasources.generic.GenericDataConverter
All Implemented Interfaces:
com.here.platform.data.client.spark.scaladsl.DataConverter, Serializable, org.apache.spark.internal.Logging
public class GenericDataConverter extends Object implements com.here.platform.data.client.spark.scaladsl.DataConverter, org.apache.spark.internal.Logging
GenericDataConverter is an implementation of DataConverter to store each message grouping as a single data file.
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    GenericDataConverter(org.apache.spark.util.SerializableConfiguration hadoopConf, org.apache.spark.sql.execution.datasources.OutputWriterFactory writerFactory)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    <M extends com.here.platform.data.client.spark.scaladsl.RowMetadata>
    com.here.platform.data.client.spark.scaladsl.GroupedData<M>
    serializeGroup(M metadata, scala.collection.Iterator<org.apache.spark.sql.Row> rows)
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Constructor Details

    • GenericDataConverter

      public GenericDataConverter(org.apache.spark.util.SerializableConfiguration hadoopConf, org.apache.spark.sql.execution.datasources.OutputWriterFactory writerFactory)

  • Method Details

    • serializeGroup

      public <M extends com.here.platform.data.client.spark.scaladsl.RowMetadata> com.here.platform.data.client.spark.scaladsl.GroupedData<M> serializeGroup(M metadata, scala.collection.Iterator<org.apache.spark.sql.Row> rows)
      Specified by:
      serializeGroup in interface com.here.platform.data.client.spark.scaladsl.DataConverter