Packages

abstract class PipelineRunner extends pipeline.PipelineRunner with DriverSetup

Implements the main entry points of applications based on the Data Processing Library.

The command line is parsed, after which a SparkContext is created. Then a com.here.platform.data.processing.driver.config.DriverConfig is constructed and passed to a new com.here.platform.data.processing.driver.Driver. Finally that com.here.platform.data.processing.driver.Driver is run with the version parameters.

You must implement a DriverSetup or use a more specialized setup pattern by mixing in and implementing com.here.platform.data.processing.driver.DriverSetupWithBuilder.

This trait implements two ways to run an application: - using the run() method that allows you to send a SparkContext - or using the main() method that forwards the call to run() without sending any SparkContext

Linear Supertypes
DriverSetup, pipeline.PipelineRunner, AnyRef, Any
Known Subclasses
ImplPipelineRunner
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PipelineRunner
  2. DriverSetup
  3. PipelineRunner
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PipelineRunner()

Abstract Value Members

  1. abstract def applicationVersion: String

    The version for the application currently running.

    The version for the application currently running.

    It is good practice to always update the application version when you change the application code.

    When the application version differs from a previous run, incremental compilation is disabled, as the library cannot rely on the dependencies between input and output set by a previous version of the application.

    For this reason, failing to update the application version may result in an incorrect result in case dependencies between input and output have changed.

  2. abstract def setupDriver(completeConfig: CompleteConfig, context: DriverContext): Driver

    Builds a new com.here.platform.data.processing.driver.Driver configured with driver tasks that implement the required compiler.

    Builds a new com.here.platform.data.processing.driver.Driver configured with driver tasks that implement the required compiler.

    completeConfig

    The complete configuration, including the processing library and compiler configuration.

    context

    The DriverContext in which the Driver runs.

    returns

    a fully-configured com.here.platform.data.processing.driver.Driver, that must include one or more DriverTasks.

    Definition Classes
    DriverSetup
    See also

    com.here.platform.data.processing.driver.DriverBuilder to build a com.here.platform.data.processing.driver.Driver

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def appInfo(cmdLine: PipelineRunnerConfig): Option[AppInfo]

    Specifies the optional legacy application info.

    Specifies the optional legacy application info. None by default.

    Attributes
    protected
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. def commitToken(cmdLine: PipelineRunnerConfig): Option[String]

    Specifies an optional commit token.

    Specifies an optional commit token. None by default.

    Attributes
    protected
  8. def completeConfig(cmdLine: PipelineRunnerConfig): CompleteConfig

    Overrides the default complete configuration.

    Overrides the default complete configuration.

    Attributes
    protected
  9. def createContext(cmdLine: CommandLineConfig): PipelineContext

    Creates the context in which the pipeline runs.

    Creates the context in which the pipeline runs. The default implementation creates a context using the configuration provided by the Pipeline Service.

    cmdLine

    The command line configuration.

    returns

    a new pipeline context.

    Attributes
    protected
    Definition Classes
    PipelineRunner
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  12. def exitOnFailure: Boolean

    Specifies whether the pipeline should exit (by calling sys.exit(1)) when an exception is thrown by the run method; defaults to false.

    Specifies whether the pipeline should exit (by calling sys.exit(1)) when an exception is thrown by the run method; defaults to false.

    Attributes
    protected
    Definition Classes
    PipelineRunner
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. final def main(args: Array[String]): Unit

    The entry point for the pipeline application, from where the run method is called.

    The entry point for the pipeline application, from where the run method is called.

    args

    The command line arguments.

    Definition Classes
    PipelineRunner
  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. def parseCommandLine(args: Array[String]): CommandLineConfig

    Parses the command line arguments.

    Parses the command line arguments. Override this method to parse custom CommandLineConfig subclasses.

    args

    The command line arguments.

    returns

    the command line configuration.

    Definition Classes
    PipelineRunner → PipelineRunner
  22. def run(pipelineContext: PipelineContext, cmdLine: CommandLineConfig): Unit

    Runs a pipeline's processing logic, given the pipeline's context and the command line configuration.

    Runs a pipeline's processing logic, given the pipeline's context and the command line configuration.

    cmdLine

    The command line configuration.

    Attributes
    protected
    Definition Classes
    PipelineRunner → PipelineRunner
  23. def runWith(args: Array[String], sparkContext: Option[SparkContext] = None, pipelineContext: Option[PipelineContext] = None): Unit

    Runs the runner with optional overrides.

    Runs the runner with optional overrides. This method is useful for testing, when you need to provide the Spark context or the pipeline context.

    args

    The command line arguments.

    sparkContext

    The optional Spark context, None if not provided.

    pipelineContext

    The optional pipeline context, None if not provided.

  24. def sparkMaster(cmdLine: PipelineRunnerConfig): Option[String]

    Specifies an optional Spark master URL.

    Specifies an optional Spark master URL. By default is either None or the URL specified via the --master command line option.

    Attributes
    protected
  25. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  26. def toString(): String
    Definition Classes
    AnyRef → Any
  27. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from DriverSetup

Inherited from pipeline.PipelineRunner

Inherited from AnyRef

Inherited from Any

Ungrouped