sealed trait Executor extends InputLayers with InputPartitioner with OutputLayers with OutputPartitioner
Base class for all the executors.
Executors work at RDD level, meaning that RDDs are passed and returned to the functions that each executor implements. It is important to define a common policy regarding the persistence of RDDs that are passed and returned. Not respecting this policy introduces a risk of Spark throwing an exception due to the fact that some RDDs may be persisted twice with different storage levels. This policy may be applicable to classes other than executors, such as com.here.platform.data.processing.compiler.NonIncrementalCompiler, com.here.platform.data.processing.compiler.DepCompilerBase and derived.
- Note
The policy established is as follows: RDDs passed to every execute function are guaranteed to be reusable multiple times, efficiently, without the need for the implementations to persist them. Implementations shall not persist passed RDDs. These are either already persisted by the processing library or guaranteed to be reusable multiple time efficiently. Therefore, implementations shall not require() or assert() that passed RDDs are persisted, although it's guaranteed that they will be, or equivalent. RDDs returned by every execute function don't have to be persisted. They may be persisted if it's useful to the implementations. The processing library may persist the RDDs once returned, if not already persisted.
- Alphabetic
- By Inheritance
- Executor
- OutputPartitioner
- OutputLayers
- InputPartitioner
- InputLayers
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def id: Id
Unique identifier of the com.here.platform.data.processing.driver.Executor.
- abstract def inLayers: Map[Id, Set[Id]]
Represents layers of the input catalogs that you should query and provide to the compiler.
Represents layers of the input catalogs that you should query and provide to the compiler. These layers are grouped by input catalog and identified by catalog ID and layer ID.
- Definition Classes
- InputLayers
- abstract def inPartitioner(parallelism: Int): Partitioner[InKey]
Specifies the partitioner to use when querying the input catalogs.
Specifies the partitioner to use when querying the input catalogs.
- parallelism
The number of partitions the partitioner should partition the catalog into, this should match the parallelism of the Spark RDD containing the input partitions.
- returns
The input partitioner with the parallelism specified.
- Definition Classes
- InputPartitioner
- abstract def outLayers: Set[Id]
Layers to be produced by the compiler.
Layers to be produced by the compiler.
- Definition Classes
- OutputLayers
- abstract def outPartitioner(parallelism: Int): Partitioner[OutKey]
Specifies the partitioner to use when querying the output catalog and producing output data.
Specifies the partitioner to use when querying the output catalog and producing output data.
- parallelism
The number of partitions the partitioner should partition the catalog into, this should match the parallelism of the Spark RDD containing the output partitions.
- returns
The output partitioner with the parallelism specified.
- Definition Classes
- OutputPartitioner
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final val outCatalogId: Id
Identifier for the output catalog.
Identifier for the output catalog.
- Definition Classes
- OutputLayers
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)