abstract class PipelineRunner extends pipeline.PipelineRunner with DriverSetup
Implements the main entry points of applications based on the Data Processing Library.
The command line is parsed, after which a SparkContext is created. Then a com.here.platform.data.processing.driver.config.DriverConfig is constructed and passed to a new com.here.platform.data.processing.driver.Driver. Finally that com.here.platform.data.processing.driver.Driver is run with the version parameters.
You must implement a DriverSetup or use a more specialized setup pattern by mixing in and implementing com.here.platform.data.processing.driver.DriverSetupWithBuilder.
This trait implements two ways to run an application: - using the run() method that allows you to send a SparkContext - or using the main() method that forwards the call to run() without sending any SparkContext
- Alphabetic
- By Inheritance
- PipelineRunner
- DriverSetup
- PipelineRunner
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new PipelineRunner()
Abstract Value Members
- abstract def applicationVersion: String
The version for the application currently running.
The version for the application currently running.
It is good practice to always update the application version when you change the application code.
When the application version differs from a previous run, incremental compilation is disabled, as the library cannot rely on the dependencies between input and output set by a previous version of the application.
For this reason, failing to update the application version may result in an incorrect result in case dependencies between input and output have changed.
- abstract def setupDriver(completeConfig: CompleteConfig, context: DriverContext): Driver
Builds a new com.here.platform.data.processing.driver.Driver configured with driver tasks that implement the required compiler.
Builds a new com.here.platform.data.processing.driver.Driver configured with driver tasks that implement the required compiler.
- completeConfig
The complete configuration, including the processing library and compiler configuration.
- context
The DriverContext in which the Driver runs.
- returns
a fully-configured com.here.platform.data.processing.driver.Driver, that must include one or more DriverTasks.
- Definition Classes
- DriverSetup
- See also
com.here.platform.data.processing.driver.DriverBuilder to build a com.here.platform.data.processing.driver.Driver
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def appInfo(cmdLine: PipelineRunnerConfig): Option[AppInfo]
Specifies the optional legacy application info.
Specifies the optional legacy application info.
Noneby default.- Attributes
- protected
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def commitToken(cmdLine: PipelineRunnerConfig): Option[String]
Specifies an optional commit token.
Specifies an optional commit token.
Noneby default.- Attributes
- protected
- def completeConfig(cmdLine: PipelineRunnerConfig): CompleteConfig
Overrides the default complete configuration.
Overrides the default complete configuration.
- Attributes
- protected
- def createContext(cmdLine: CommandLineConfig): PipelineContext
- Attributes
- protected
- Definition Classes
- PipelineRunner
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def exitOnFailure: Boolean
- Attributes
- protected
- Definition Classes
- PipelineRunner
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def main(args: Array[String]): Unit
- Definition Classes
- PipelineRunner
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def parseCommandLine(args: Array[String]): CommandLineConfig
- Definition Classes
- PipelineRunner → PipelineRunner
- def run(pipelineContext: PipelineContext, cmdLine: CommandLineConfig): Unit
- Attributes
- protected
- Definition Classes
- PipelineRunner → PipelineRunner
- def runWith(args: Array[String], sparkContext: Option[SparkContext] = None, pipelineContext: Option[PipelineContext] = None): Unit
Runs the runner with optional overrides.
Runs the runner with optional overrides. This method is useful for testing, when you need to provide the Spark context or the pipeline context.
- args
The command line arguments.
- sparkContext
The optional Spark context,
Noneif not provided.- pipelineContext
The optional pipeline context,
Noneif not provided.
- def sparkMaster(cmdLine: PipelineRunnerConfig): Option[String]
Specifies an optional Spark master URL.
Specifies an optional Spark master URL. By default is either
Noneor the URL specified via the --master command line option.- Attributes
- protected
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)