abstract class PipelineRunner extends pipeline.PipelineRunner with DriverSetup
Implements the main entry points of applications based on the Data Processing Library.
The command line is parsed, after which a SparkContext is created. Then a com.here.platform.data.processing.driver.config.DriverConfig is constructed and passed to a new com.here.platform.data.processing.driver.Driver. Finally that com.here.platform.data.processing.driver.Driver is run with the version parameters.
You must implement a DriverSetup or use a more specialized setup pattern by mixing in and implementing com.here.platform.data.processing.driver.DriverSetupWithBuilder.
This trait implements two ways to run an application: - using the run() method that allows you to send a SparkContext - or using the main() method that forwards the call to run() without sending any SparkContext
- Alphabetic
- By Inheritance
- PipelineRunner
- DriverSetup
- PipelineRunner
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new PipelineRunner()
Abstract Value Members
- abstract def applicationVersion: String
The version for the application currently running.
The version for the application currently running.
It is good practice to always update the application version when you change the application code.
When the application version differs from a previous run, incremental compilation is disabled, as the library cannot rely on the dependencies between input and output set by a previous version of the application.
For this reason, failing to update the application version may result in an incorrect result in case dependencies between input and output have changed.
- abstract def setupDriver(completeConfig: CompleteConfig, context: DriverContext): Driver
Builds a new com.here.platform.data.processing.driver.Driver configured with driver tasks that implement the required compiler.
Builds a new com.here.platform.data.processing.driver.Driver configured with driver tasks that implement the required compiler.
- completeConfig
The complete configuration, including the processing library and compiler configuration.
- context
The DriverContext in which the Driver runs.
- returns
a fully-configured com.here.platform.data.processing.driver.Driver, that must include one or more DriverTasks.
- Definition Classes
- DriverSetup
- See also
com.here.platform.data.processing.driver.DriverBuilder to build a com.here.platform.data.processing.driver.Driver
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def appInfo(cmdLine: PipelineRunnerConfig): Option[AppInfo]
Specifies the optional legacy application info.
Specifies the optional legacy application info.
Noneby default.- Attributes
- protected
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def commitToken(cmdLine: PipelineRunnerConfig): Option[String]
Specifies an optional commit token.
Specifies an optional commit token.
Noneby default.- Attributes
- protected
- def completeConfig(cmdLine: PipelineRunnerConfig): CompleteConfig
Overrides the default complete configuration.
Overrides the default complete configuration.
- Attributes
- protected
- def createContext(cmdLine: CommandLineConfig): PipelineContext
Creates the context in which the pipeline runs.
Creates the context in which the pipeline runs. The default implementation creates a context using the configuration provided by the Pipeline Service.
- cmdLine
The command line configuration.
- returns
a new pipeline context.
- Attributes
- protected
- Definition Classes
- PipelineRunner
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def exitOnFailure: Boolean
Specifies whether the pipeline should exit (by calling
sys.exit(1)) when an exception is thrown by therunmethod; defaults tofalse.Specifies whether the pipeline should exit (by calling
sys.exit(1)) when an exception is thrown by therunmethod; defaults tofalse.- Attributes
- protected
- Definition Classes
- PipelineRunner
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def main(args: Array[String]): Unit
The entry point for the pipeline application, from where the
runmethod is called.The entry point for the pipeline application, from where the
runmethod is called.- args
The command line arguments.
- Definition Classes
- PipelineRunner
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def parseCommandLine(args: Array[String]): CommandLineConfig
Parses the command line arguments.
Parses the command line arguments. Override this method to parse custom CommandLineConfig subclasses.
- args
The command line arguments.
- returns
the command line configuration.
- Definition Classes
- PipelineRunner → PipelineRunner
- def run(pipelineContext: PipelineContext, cmdLine: CommandLineConfig): Unit
Runs a pipeline's processing logic, given the pipeline's context and the command line configuration.
Runs a pipeline's processing logic, given the pipeline's context and the command line configuration.
- cmdLine
The command line configuration.
- Attributes
- protected
- Definition Classes
- PipelineRunner → PipelineRunner
- def runWith(args: Array[String], sparkContext: Option[SparkContext] = None, pipelineContext: Option[PipelineContext] = None): Unit
Runs the runner with optional overrides.
Runs the runner with optional overrides. This method is useful for testing, when you need to provide the Spark context or the pipeline context.
- args
The command line arguments.
- sparkContext
The optional Spark context,
Noneif not provided.- pipelineContext
The optional pipeline context,
Noneif not provided.
- def sparkMaster(cmdLine: PipelineRunnerConfig): Option[String]
Specifies an optional Spark master URL.
Specifies an optional Spark master URL. By default is either
Noneor the URL specified via the --master command line option.- Attributes
- protected
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)