trait DeltaContext extends AnyRef
Provides access to a set of common resources which a user of DeltaSets may require, that can be used by com.here.platform.data.processing.driver.DeltaSetup.
- Alphabetic
- By Inheritance
- DeltaContext
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Abstract Value Members
-
abstract
val
autoIdAssigner: BaseSetIdAssigner
The ID assigner used for automatically assigning DeltaSet IDs if none is provided
-
abstract
val
defaultParallelism: Int
Default parallelism (number of partitions), provided by Spark.
-
abstract
val
defaultPartitioner: PartitionNamePartitioner
Default partitioner for input catalog data.
-
abstract
def
disjointUnion[UK, UV](upstreams: Iterable[DeltaSet[UK, _ <: UV]], id: Id = autoIdAssigner("disjointUnion"), configOverride: Override = DeltaSetConfig.noOverride)(implicit arg0: ClassTag[UK], arg1: ClassTag[UV]): DeltaSet[UK, UV]
Computes the disjoint union of several DeltaSets.
Computes the disjoint union of several DeltaSets. If a key is present in two of the DeltaSets, the operation will throw an exception.
This function does not shuffle any data any data.
All DeltaSets must be partitioned by the same partitioner. If they are not, you must explicitly repartition the DeltaSets.
- upstreams
The operands.
- returns
A DeltaSet that represents the result of the union.
-
abstract
val
driverContext: DriverContext
The context of the Driver executing the DriverTask corresponding to this DeltaContext
-
abstract
def
id: Id
The ID of the DriverTask that owns this DeltaContext
-
abstract
def
inRetriever(id: Id): Retriever
To retrieve payloads of partitions of an input catalog.
- abstract def incrementalProcessingAllowed: Boolean
-
abstract
def
partitionKeyFiltering: PartitionKeyFiltering
Can be used to determine whether a specific input partition is filtered by the partition key filter defined in the configuration, if any.
-
abstract
def
queryCatalog(catalogId: Id, layerIds: Set[Id], partitioner: Partitioner[Key] = defaultPartitioner, id: Id = autoIdAssigner("input"), configOverride: Override = DeltaSetConfig.noOverride): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.
- catalogId
The ID of the catalog.
- layerIds
The set of layers IDs.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
-
abstract
def
queryCatalogLayer(catalogId: Id, layerId: Id, partitioner: Partitioner[Key] = defaultPartitioner, id: Id = autoIdAssigner("input"), configOverride: Override = DeltaSetConfig.noOverride): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
- catalogId
The ID of the catalog.
- layerId
The ID of the layer.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
-
abstract
def
queryCatalogs(catalogsAndLayers: Map[Id, Set[Id]], partitioner: Partitioner[Key] = defaultPartitioner, id: Id = autoIdAssigner("input"), configOverride: Override = DeltaSetConfig.noOverride): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
- catalogsAndLayers
A map from catalog IDs to a set of layer IDs within the corresponding catalog.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
-
abstract
val
stateManager: StateManager
The state manager used to retrieve the state of a DeltaSet from the state layer.
-
abstract
val
transformations: Transformations
Import all functions in the implicits to enable transformations on DeltaSets.
-
abstract
def
unionPublishedParts(publishedParts: Seq[PublishedPart], id: Id = ..., configOverride: Override = DeltaSetConfig.noOverride): PublishedSet
Creates a PublishedSet from the set of PublishedParts.
Creates a PublishedSet from the set of PublishedParts.
The set of PublishedParts must cover all parts the output layers are partitioned into, based on the PublishedPartMapper used during publishing.
This transformation shuffles the commit partitions in each PublishedPart if they don't use the same partitioner.
- publishedParts
Sequence of published parts.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A PublishedSet that reflects the metadata of the published partitions.
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()