class MultiExecutorsIncrementalMode extends MultiExecutorsBaseMode with ContextLogging
Runs a sequence of com.here.platform.data.processing.driver.Executors in incremental mode.
This driver task allows the running of a sequence of compilers where every subsequent executor is provided with data produced by the previous one. This allows the execution of multiple compilers without the need of creating a filter for each of them.
- Note
input layers list of subsequent compilers have to include the needed output layers from the previous ones
,no output layer overwriting is allowed in the current implementation
,users of this class are strongly encouraged to build the related driver task via the com.here.platform.data.processing.driver.MultiCompilerTaskBuilder. This guarantees a type safe approach in the construction. Moreover, the mentioned builders assigns to each executor a unique id based on the driver task id set by the user.
- Alphabetic
- By Inheritance
- MultiExecutorsIncrementalMode
- ContextLogging
- MultiExecutorsBaseMode
- ContextAwareMode
- TaskMode
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
MultiExecutorsIncrementalMode(context: DriverContext, id: Id, executors: Seq[Executor])
- context
the com.here.platform.data.processing.driver.DriverContext relative to this mode
- id
the identifier of the com.here.platform.data.processing.driver.DriverTask the mode belongs to, for logging reasons
- executors
the executor that should sequentially run
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
actualRun(): RDD[(Key, Commit)]
- Attributes
- protected
-
def
applyChanges(existingData: RDD[(OutKey, OutMeta)], outCommits: RDD[(OutKey, OutCommit)], version: Version): (RDD[(OutKey, InMeta)], RDD[(OutKey, Change)])
Propagates changes into the input metadata to simulate a commit.
Propagates changes into the input metadata to simulate a commit.
In multi-compiler full compilation intermediate results are only published to the blobstore, while metadata are kept and committed at the end of the executor sequence execution. This means that for a subsequent executor changes done by the previous one has to appear in the input metadata.
- existingData
the status of the out data before compilation
- outCommits
the just completed executor changes
- version
the version to be committed
- returns
the new state of the input metadata for the incoming execution
- Attributes
- protected
- Definition Classes
- MultiExecutorsBaseMode
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
classForLogger: Class[_]
Override this to change the class that is displayed in logs.
Override this to change the class that is displayed in logs.
- Attributes
- protected
- Definition Classes
- ContextLogging
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
val
context: DriverContext
- Definition Classes
- ContextAwareMode
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getNextVersion(currentBase: BaseVersion): Version
Increments the base version of the output catalog.
Increments the base version of the output catalog.
In multi-compiler driver tasks we need to simulate a publishing and the related commit. For this reason we need to increase the version of some input data.
- currentBase
the current base version
- Attributes
- protected
- Definition Classes
- MultiExecutorsBaseMode
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
isApplicable: Boolean
This mode is only applicable if the following conditions are met: - disableIncremental config parameter is false - what's changed is available for every input catalog - output catalog is not empty
This mode is only applicable if the following conditions are met: - disableIncremental config parameter is false - what's changed is available for every input catalog - output catalog is not empty
- returns
if the mode is applicable, so if the mode should run and its result evaluated
- Definition Classes
- MultiExecutorsIncrementalMode → TaskMode
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
lazy val
logger: ContextAwareLogger
The logger to use for normal and context-aware messages.
The logger to use for normal and context-aware messages.
- Attributes
- protected
- Definition Classes
- ContextLogging
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
val
parallelism: Int
- Attributes
- protected
- Definition Classes
- ContextAwareMode
-
def
run(): TaskResult
Runs the compiler only providing the changes since the last run for every input catalog.
Runs the compiler only providing the changes since the last run for every input catalog.
- returns
the result of the compilation
- Definition Classes
- MultiExecutorsIncrementalMode → TaskMode
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()