trait DeltaContext extends AnyRef
Provides access to a set of common resources which a user of DeltaSets may require, that can be used by com.here.platform.data.processing.driver.DeltaSetup.
- Alphabetic
- By Inheritance
- DeltaContext
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract val autoIdAssigner: BaseSetIdAssigner
The ID assigner used for automatically assigning DeltaSet IDs if none is provided
- abstract val defaultParallelism: Int
Default parallelism (number of partitions), provided by Spark.
- abstract val defaultPartitioner: PartitionNamePartitioner
Default partitioner for input catalog data.
- abstract def disjointUnion[UK, UV](upstreams: Iterable[DeltaSet[UK, _ <: UV]], id: Id = autoIdAssigner("disjointUnion"), configOverride: Override = DeltaSetConfig.noOverride)(implicit arg0: ClassTag[UK], arg1: ClassTag[UV]): DeltaSet[UK, UV]
Computes the disjoint union of several DeltaSets.
Computes the disjoint union of several DeltaSets. If a key is present in two of the DeltaSets, the operation will throw an exception.
This function does not shuffle any data any data.
All DeltaSets must be partitioned by the same partitioner. If they are not, you must explicitly repartition the DeltaSets.
- upstreams
The operands.
- returns
A DeltaSet that represents the result of the union.
- abstract val driverContext: DriverContext
The context of the Driver executing the DriverTask corresponding to this DeltaContext
- abstract def id: Id
The ID of the DriverTask that owns this DeltaContext
- abstract def inRetriever(id: Id): Retriever
To retrieve payloads of partitions of an input catalog.
- abstract def incrementalProcessingAllowed: Boolean
- abstract def partitionKeyFiltering: PartitionKeyFiltering
Can be used to determine whether a specific input partition is filtered by the partition key filter defined in the configuration, if any.
- abstract def queryCatalog(catalogId: Id, layerIds: Set[Id], partitioner: Partitioner[Key] = defaultPartitioner, id: Id = autoIdAssigner("input"), configOverride: Override = DeltaSetConfig.noOverride): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.
- catalogId
The ID of the catalog.
- layerIds
The set of layers IDs.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
- abstract def queryCatalogLayer(catalogId: Id, layerId: Id, partitioner: Partitioner[Key] = defaultPartitioner, id: Id = autoIdAssigner("input"), configOverride: Override = DeltaSetConfig.noOverride): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
- catalogId
The ID of the catalog.
- layerId
The ID of the layer.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
- abstract def queryCatalogs(catalogsAndLayers: Map[Id, Set[Id]], partitioner: Partitioner[Key] = defaultPartitioner, id: Id = autoIdAssigner("input"), configOverride: Override = DeltaSetConfig.noOverride): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
- catalogsAndLayers
A map from catalog IDs to a set of layer IDs within the corresponding catalog.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
- abstract val stateManager: StateManager
The state manager used to retrieve the state of a DeltaSet from the state layer.
- abstract val transformations: Transformations
Import all functions in the implicits to enable transformations on DeltaSets.
- abstract def unionPublishedParts(publishedParts: Seq[PublishedPart], id: Id = autoIdAssigner("unionPublishedParts"), configOverride: Override = DeltaSetConfig.noOverride): PublishedSet
Creates a PublishedSet from the set of PublishedParts.
Creates a PublishedSet from the set of PublishedParts.
The set of PublishedParts must cover all parts the output layers are partitioned into, based on the PublishedPartMapper used during publishing.
This transformation shuffles the commit partitions in each PublishedPart if they don't use the same partitioner.
- publishedParts
Sequence of published parts.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A PublishedSet that reflects the metadata of the published partitions.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)