Packages

trait DeltaSet[K, V] extends BaseSet

The DeltaSet is the main processing abstraction to implement custom processing patterns.

Pipeline developers may implement their processing logic by applying transformations to DeltaSets.

Core of the incremental processing framework is the ability to expose not only the contents of a DeltaSet but also what has changed about the contents since a previous, reference run. This information may be used by transformed DeltaSets to also expose only what has changed, enabling differential processing across any DAG of transformations.

Each DeltaSet is characterized by the type of keys and values it contains. The format of the input data, when applicable, needed produce the output data is not specified here, and depends on the DeltaSet implementation.

Data in a DeltaSet is not only strongly typed but also strongly partitioned.

The provided DeltaContext exposes the input catalog as sources. When integrating with the com.here.platform.data.processing.driver.Driver via the com.here.platform.data.processing.driver.DeltaSetup interface and relative com.here.platform.data.processing.driver.DeltaDriverTaskBuilder or directly using the com.here.platform.data.processing.driver.DeltaDriverTask, the result of processing must be exposed as sink of fixed types.

DeltaSets are immutable, distributed, de-duplicated.

DeltaSet transformations are incremental, lazy.

K

The type of the keys in the DeltaSet.

V

The type of the values in the DeltaSet.

Linear Supertypes
BaseSet, AnyRef, Any
Known Subclasses
DeltaSetImpl, InputCatalogDeltaSet, StatefulDeltaSet, AbstractFlatMap, AbstractFlatMapKeys, DetectChanges, DisjointUnion, FilterByKey, FlatMapGroup, FlatMapKeys, FlatMapKeysGroup, FlatMapKeysReduce, FlatMapReduce, FlatMapUnique, Join, LeftOuterJoin, MapValuesWithKey, MapValuesWithResolver, OuterJoin, Persist, ReadBack, Repartition, ResolveReferences, WrapRDD
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaSet
  2. BaseSet
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. type BaseSetType = DeltaSet[K, V]

    The type that will be returned by operations such as withId and overrideConfig.

    The type that will be returned by operations such as withId and overrideConfig.

    Attributes
    protected
    Definition Classes
    DeltaSet → BaseSet

Abstract Value Members

  1. abstract def changedKeyValues: Determine[Changes[K, V]]

    Returns the changed key-value pairs in the DeltaSet.

    Returns the changed key-value pairs in the DeltaSet.

    The implementation of the DeltaSet can persist the result of this function, but this behavior not guaranteed. When you use the data multiple times, make sure to persist the result.

  2. abstract def config: DeltaSetConfig

    The effective configuration of this BaseSet, taking into account all configuration files and programmatic overrides.

    The effective configuration of this BaseSet, taking into account all configuration files and programmatic overrides.

    Definition Classes
    BaseSet
  3. abstract def fqn: FQN

    The fully qualified name of the BaseSet, which consists of its BaseSet.Namespace and its BaseSet.Id.

    The fully qualified name of the BaseSet, which consists of its BaseSet.Namespace and its BaseSet.Id.

    Definition Classes
    BaseSet
  4. abstract def id: Id

    The ID of the BaseSet, that identifies it in configuration files and logs, and for storing metadata in the output catalog.

    The ID of the BaseSet, that identifies it in configuration files and logs, and for storing metadata in the output catalog. By default, the ID is automatically assigned.

    Definition Classes
    BaseSet
  5. abstract def inScope: Option[(K) ⇒ Boolean]

    Defines that the DeltaSet is a scoped view on an underlying collection, for example a set of catalog layers.

    Defines that the DeltaSet is a scoped view on an underlying collection, for example a set of catalog layers. Returns true if a given key is in the scope of the view.

    If a key is in the scope of the view, but not contained in the DeltaSet, then it does not exist in the underlying collection. All keys in the DeltaSet must be in the scope of the view.

  6. abstract def keyValues(keys: Keys[K]): KeyValues[K, V]

    Returns all key-value pairs in the DeltaSet, whose key is inside a given set of keys.

    Returns all key-value pairs in the DeltaSet, whose key is inside a given set of keys.

    The implementation of the DeltaSet can persist the result of this function, but this behavior not guaranteed. When you use the data multiple times, make sure to persist the result.

    keys

    The set of keys. Must be partitioned with the partitioner of this DeltaSet.

  7. abstract val keyValues: KeyValues[K, V]

    Returns all key-value pairs in the DeltaSet.

    Returns all key-value pairs in the DeltaSet.

    The implementation of the DeltaSet can persist this value, but this behavior not guaranteed. When you use the data multiple times, make sure to persist the result.

  8. abstract def namespace: Namespace

    The BaseSet.Namespace of the BaseSet.

    The BaseSet.Namespace of the BaseSet.

    Definition Classes
    BaseSet
  9. abstract val partitioner: Partitioner[K]

    The partitioner for the DeltaSet.

  10. abstract def upstreams: Iterable[BaseSet]

    BaseSets used to compute this BaseSet

    BaseSets used to compute this BaseSet

    Definition Classes
    BaseSet

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  16. def toString(): String
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from BaseSet

Inherited from AnyRef

Inherited from Any

Ungrouped