trait DeltaSet[K, V] extends BaseSet
The DeltaSet is the main processing abstraction to implement custom processing patterns.
Pipeline developers may implement their processing logic by applying transformations to DeltaSets.
Core of the incremental processing framework is the ability to expose not only the contents of a DeltaSet but also what has changed about the contents since a previous, reference run. This information may be used by transformed DeltaSets to also expose only what has changed, enabling differential processing across any DAG of transformations.
Each DeltaSet is characterized by the type of keys and values it contains. The format of the input data, when applicable, needed produce the output data is not specified here, and depends on the DeltaSet implementation.
Data in a DeltaSet is not only strongly typed but also strongly partitioned.
The provided DeltaContext exposes the input catalog as sources. When integrating with the com.here.platform.data.processing.driver.Driver via the com.here.platform.data.processing.driver.DeltaSetup interface and relative com.here.platform.data.processing.driver.DeltaDriverTaskBuilder or directly using the com.here.platform.data.processing.driver.DeltaDriverTask, the result of processing must be exposed as sink of fixed types.
DeltaSets are immutable, distributed, de-duplicated.
DeltaSet transformations are incremental, lazy.
- Alphabetic
- By Inheritance
- DeltaSet
- BaseSet
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
Abstract Value Members
- abstract def changedKeyValues: Determine[Changes[K, V]]
Returns the changed key-value pairs in the DeltaSet.
- abstract def config: DeltaSetConfig
The effective configuration of this BaseSet, taking into account all configuration files and programmatic overrides.
- abstract def fqn: FQN
The fully qualified name of the BaseSet, which consists of its BaseSet.Namespace and its BaseSet.Id.
The fully qualified name of the BaseSet, which consists of its BaseSet.Namespace and its BaseSet.Id.
- Definition Classes
- BaseSet
- abstract def id: Id
The ID of the BaseSet, that identifies it in configuration files and logs, and for storing metadata in the output catalog.
- abstract def inScope: Option[(K) => Boolean]
Defines that the DeltaSet is a scoped view on an underlying collection, for example a set of catalog layers.
Defines that the DeltaSet is a scoped view on an underlying collection, for example a set of catalog layers. Returns true if a given key is in the scope of the view.
If a key is in the scope of the view, but not contained in the DeltaSet, then it does not exist in the underlying collection. All keys in the DeltaSet must be in the scope of the view.
- abstract def keyValues(keys: Keys[K]): KeyValues[K, V]
Returns all key-value pairs in the DeltaSet, whose key is inside a given set of keys.
Returns all key-value pairs in the DeltaSet, whose key is inside a given set of keys.
The implementation of the DeltaSet can persist the result of this function, but this behavior not guaranteed. When you use the data multiple times, make sure to persist the result.
- keys
The set of keys. Must be partitioned with the
partitionerof this DeltaSet.
- abstract val keyValues: KeyValues[K, V]
Returns all key-value pairs in the DeltaSet.
- abstract def namespace: Namespace
The BaseSet.Namespace of the BaseSet.
The BaseSet.Namespace of the BaseSet.
- Definition Classes
- BaseSet
- abstract val partitioner: Partitioner[K]
The partitioner for the DeltaSet.
- abstract def upstreams: Iterable[BaseSet]
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)