trait StatefulCompilerExecutor extends Executor
Main interface to implement a compiler that saves a state for the next run, while the state of the previous run is provided as additional input. This compiler enables complex incremental compilation patterns.
- Alphabetic
- By Inheritance
- StatefulCompilerExecutor
- Executor
- OutputPartitioner
- OutputLayers
- InputPartitioner
- InputLayers
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def execute(inData: InData, inChanges: InChanges, existingState: ExistingState, parallelism: Int): (ToPublish, NewState)
The compile function that generates the changes of the output catalog and the new state.
The compile function that generates the changes of the output catalog and the new state. The input is the whole input catalog, information on what has changed from the previous compilation, as well as the state of that run.
- inData
The metadata of the input catalog for the layers requested. Keys are partitioned as specified in com.here.platform.data.processing.compiler.InputPartitioner.
- inChanges
The changes of the input catalog for the layers requested from the last run. Keys are partitioned as specified in com.here.platform.data.processing.compiler.InputPartitioner.
- existingState
The state generated the last run.
- parallelism
The parallelism level to be used when getting partitioners from com.here.platform.data.processing.compiler.InputPartitioner and/or from com.here.platform.data.processing.compiler.OutputPartitioner.
- returns
the payloads to be updated and committed incrementally and the state for the next run. The keys of the returned payloads shall be partitioned as specified in com.here.platform.data.processing.compiler.OutputPartitioner.
- Note
if an input catalog has not changed, inChanges does not contain it.
,please note and follow the RDD persistence policy described in com.here.platform.data.processing.driver.Executor.
- abstract def execute(inData: InData, parallelism: Int): (ToPublish, NewState)
The compile function that generates the whole output catalog taking as input the whole input catalog.
The compile function that generates the whole output catalog taking as input the whole input catalog.
- inData
The metadata of the input catalog for the layers requested.
- parallelism
The parallelism of both the input and the output RDDs. This parameter is normally needed to get partitioners from com.here.platform.data.processing.compiler.InputPartitioner and/or from com.here.platform.data.processing.compiler.OutputPartitioner traits.
- returns
the payloads to be uploaded as new snapshot of the output catalog. The keys of the returned payloads shall be partitioned as specified in com.here.platform.data.processing.compiler.OutputPartitioner.
- Note
please note and follow the RDD persistence policy described in com.here.platform.data.processing.driver.Executor.
- abstract def id: Id
Unique identifier of the com.here.platform.data.processing.driver.Executor.
Unique identifier of the com.here.platform.data.processing.driver.Executor.
- Definition Classes
- Executor
- abstract def inLayers: Map[Id, Set[Id]]
Represents layers of the input catalogs that you should query and provide to the compiler.
Represents layers of the input catalogs that you should query and provide to the compiler. These layers are grouped by input catalog and identified by catalog ID and layer ID.
- Definition Classes
- InputLayers
- abstract def inPartitioner(parallelism: Int): Partitioner[InKey]
Specifies the partitioner to use when querying the input catalogs.
Specifies the partitioner to use when querying the input catalogs.
- parallelism
The number of partitions the partitioner should partition the catalog into, this should match the parallelism of the Spark RDD containing the input partitions.
- returns
The input partitioner with the parallelism specified.
- Definition Classes
- InputPartitioner
- abstract def outLayers: Set[Id]
Layers to be produced by the compiler.
Layers to be produced by the compiler.
- Definition Classes
- OutputLayers
- abstract def outPartitioner(parallelism: Int): Partitioner[OutKey]
Specifies the partitioner to use when querying the output catalog and producing output data.
Specifies the partitioner to use when querying the output catalog and producing output data.
- parallelism
The number of partitions the partitioner should partition the catalog into, this should match the parallelism of the Spark RDD containing the output partitions.
- returns
The output partitioner with the parallelism specified.
- Definition Classes
- OutputPartitioner
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final val outCatalogId: Id
Identifier for the output catalog.
Identifier for the output catalog.
- Definition Classes
- OutputLayers
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)