class MultiExecutorsIncrementalMode extends MultiExecutorsBaseMode with ContextLogging
Runs a sequence of com.here.platform.data.processing.driver.Executors in incremental mode.
This driver task allows the running of a sequence of compilers where every subsequent executor is provided with data produced by the previous one. This allows the execution of multiple compilers without the need of creating a filter for each of them.
- Note
input layers list of subsequent compilers have to include the needed output layers from the previous ones
,no output layer overwriting is allowed in the current implementation
,users of this class are strongly encouraged to build the related driver task via the com.here.platform.data.processing.driver.MultiCompilerTaskBuilder. This guarantees a type safe approach in the construction. Moreover, the mentioned builders assigns to each executor a unique id based on the driver task id set by the user.
- Alphabetic
- By Inheritance
- MultiExecutorsIncrementalMode
- ContextLogging
- MultiExecutorsBaseMode
- ContextAwareMode
- TaskMode
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new MultiExecutorsIncrementalMode(context: DriverContext, id: Id, executors: Seq[Executor])
- context
the com.here.platform.data.processing.driver.DriverContext relative to this mode
- id
the identifier of the com.here.platform.data.processing.driver.DriverTask the mode belongs to, for logging reasons
- executors
the executor that should sequentially run
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def actualRun(): RDD[(Key, Commit)]
- Attributes
- protected
- def applyChanges(existingData: RDD[(OutKey, OutMeta)], outCommits: RDD[(OutKey, OutCommit)], version: Version): (RDD[(OutKey, InMeta)], RDD[(OutKey, Change)])
Propagates changes into the input metadata to simulate a commit.
Propagates changes into the input metadata to simulate a commit.
In multi-compiler full compilation intermediate results are only published to the blobstore, while metadata are kept and committed at the end of the executor sequence execution. This means that for a subsequent executor changes done by the previous one has to appear in the input metadata.
- existingData
the status of the out data before compilation
- outCommits
the just completed executor changes
- version
the version to be committed
- returns
the new state of the input metadata for the incoming execution
- Attributes
- protected
- Definition Classes
- MultiExecutorsBaseMode
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def classForLogger: Class[_]
Override this to change the class that is displayed in logs.
Override this to change the class that is displayed in logs.
- Attributes
- protected
- Definition Classes
- ContextLogging
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- val context: DriverContext
- Definition Classes
- ContextAwareMode
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getNextVersion(currentBase: BaseVersion): Version
Increments the base version of the output catalog.
Increments the base version of the output catalog.
In multi-compiler driver tasks we need to simulate a publishing and the related commit. For this reason we need to increase the version of some input data.
- currentBase
the current base version
- Attributes
- protected
- Definition Classes
- MultiExecutorsBaseMode
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def isApplicable: Boolean
This mode is only applicable if the following conditions are met: - disableIncremental config parameter is false - what's changed is available for every input catalog - output catalog is not empty
This mode is only applicable if the following conditions are met: - disableIncremental config parameter is false - what's changed is available for every input catalog - output catalog is not empty
- returns
if the mode is applicable, so if the mode should run and its result evaluated
- Definition Classes
- MultiExecutorsIncrementalMode → TaskMode
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final lazy val logger: ContextAwareLogger
The logger to use for normal and context-aware messages.
The logger to use for normal and context-aware messages.
- Attributes
- protected
- Definition Classes
- ContextLogging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- val parallelism: Int
- Attributes
- protected
- Definition Classes
- ContextAwareMode
- def run(): TaskResult
Runs the compiler only providing the changes since the last run for every input catalog.
Runs the compiler only providing the changes since the last run for every input catalog.
- returns
the result of the compilation
- Definition Classes
- MultiExecutorsIncrementalMode → TaskMode
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)