Packages

trait PreviousVersionsChecking extends DeltaContextImpl with ContextLogging

Verifies if, for every input catalog we're trying to compile incrementally, the output catalog has a corresponding dependency matching the "from" version.

This is used to strictly verify that we're about to compile incrementally from version V to any newer version on top of an output catalog that was actually derived from version V. If this doesn't hold for every input catalog, the result of incremental compilation is not correct, so we disable it if this trait is mixed in. As a consequence, the result is compute non-incrementally instead.

Linear Supertypes
ContextLogging, DeltaContextImpl, DeltaContext, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PreviousVersionsChecking
  2. ContextLogging
  3. DeltaContextImpl
  4. DeltaContext
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. val autoIdAssigner: BaseSetIdAssigner

    The ID assigner used for automatically assigning DeltaSet IDs if none is provided

    The ID assigner used for automatically assigning DeltaSet IDs if none is provided

    Definition Classes
    DeltaContextImpl → DeltaContext
  6. def classForLogger: Class[_]

    Override this to change the class that is displayed in logs.

    Override this to change the class that is displayed in logs.

    Attributes
    protected
    Definition Classes
    ContextLogging
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. val defaultParallelism: Int

    Default parallelism (number of partitions), provided by Spark.

    Default parallelism (number of partitions), provided by Spark.

    Definition Classes
    DeltaContextImpl → DeltaContext
  9. val defaultPartitioner: PartitionNamePartitioner

    Partitioner for input catalog data.

    Partitioner for input catalog data.

    Definition Classes
    DeltaContextImpl → DeltaContext
  10. def determineIncrementalProcessingAllowed: Boolean

    To be overwritten in traits

    To be overwritten in traits

    Definition Classes
    PreviousVersionsChecking → DeltaContextImpl
  11. def disjointUnion[UK, UV](upstreams: Iterable[DeltaSet[UK, _ <: UV]], id: Id, configOverride: Override)(implicit arg0: ClassTag[UK], arg1: ClassTag[UV]): DeltaSet[UK, UV]

    Computes the disjoint union of several DeltaSets.

    Computes the disjoint union of several DeltaSets. If a key is present in two of the DeltaSets, the operation will throw an exception.

    This function does not shuffle any data any data.

    All DeltaSets must be partitioned by the same partitioner. If they are not, you must explicitly repartition the DeltaSets.

    upstreams

    The operands.

    returns

    A DeltaSet that represents the result of the union.

    Definition Classes
    DeltaContextImpl → DeltaContext
  12. val driverContext: DriverContext

    The context of the Driver executing the DriverTask corresponding to this DeltaContext

    The context of the Driver executing the DriverTask corresponding to this DeltaContext

    Definition Classes
    DeltaContextImpl → DeltaContext
  13. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  15. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  16. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  18. val id: Id

    The ID of the DriverTask that owns this DeltaContext

    The ID of the DriverTask that owns this DeltaContext

    Definition Classes
    DeltaContextImpl → DeltaContext
  19. def inRetriever(id: Id): Retriever

    To retrieve payloads of partitions of an input catalog.

    To retrieve payloads of partitions of an input catalog.

    Definition Classes
    DeltaContextImpl → DeltaContext
  20. final val incrementalProcessingAllowed: Boolean
    Definition Classes
    DeltaContextImpl → DeltaContext
  21. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  22. final lazy val logger: ContextAwareLogger

    The logger to use for normal and context-aware messages.

    The logger to use for normal and context-aware messages.

    Attributes
    protected
    Definition Classes
    ContextLogging
  23. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  24. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  26. final val partitionKeyFiltering: PartitionKeyFiltering

    Can be used to determine whether a specific input partition is filtered by the partition key filter defined in the configuration, if any.

    Can be used to determine whether a specific input partition is filtered by the partition key filter defined in the configuration, if any.

    Definition Classes
    DeltaContextImpl → DeltaContext
  27. def queryCatalog(catalogId: Id, layerIds: Set[Id], partitioner: Partitioner[Key], id: Id, configOverride: Override): DeltaSet[Key, Meta]

    Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.

    Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.

    catalogId

    The ID of the catalog.

    layerIds

    The set of layers IDs.

    partitioner

    The partitioner to use for the input data. By default, defaultPartitioner is used.

    id

    Deprecated.

    configOverride

    Deprecated.

    returns

    A DeltaSet containing the combined key-meta pairs from all given catalogs.

    Definition Classes
    DeltaContextImpl → DeltaContext
  28. def queryCatalogLayer(catalogId: Id, layerId: Id, partitioner: Partitioner[Key], id: Id, configOverride: Override): DeltaSet[Key, Meta]

    Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.

    Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.

    catalogId

    The ID of the catalog.

    layerId

    The ID of the layer.

    partitioner

    The partitioner to use for the input data. By default, defaultPartitioner is used.

    id

    Deprecated.

    configOverride

    Deprecated.

    returns

    A DeltaSet containing the combined key-meta pairs from all given catalogs.

    Definition Classes
    DeltaContextImpl → DeltaContext
  29. def queryCatalogs(catalogsAndLayers: Map[Id, Set[Id]], partitioner: Partitioner[Key], id: Id, configOverride: Override): DeltaSet[Key, Meta]

    Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.

    Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.

    catalogsAndLayers

    A map from catalog IDs to a set of layer IDs within the corresponding catalog.

    partitioner

    The partitioner to use for the input data. By default, defaultPartitioner is used.

    id

    Deprecated.

    configOverride

    Deprecated.

    returns

    A DeltaSet containing the combined key-meta pairs from all given catalogs.

    Definition Classes
    DeltaContextImpl → DeltaContext
  30. val stateManager: StateManagerImpl

    The state manager used to retrieve the state of a DeltaSet from the state layer.

    The state manager used to retrieve the state of a DeltaSet from the state layer.

    Definition Classes
    DeltaContextImpl → DeltaContext
  31. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  32. def toString(): String
    Definition Classes
    AnyRef → Any
  33. val transformations: Transformations

    Import all functions in the implicits to enable transformations on DeltaSets.

    Import all functions in the implicits to enable transformations on DeltaSets.

    Definition Classes
    DeltaContextImpl → DeltaContext
  34. def unionPublishedParts(publishedParts: Seq[PublishedPart], id: Id, configOverride: Override): PublishedSet

    Creates a PublishedSet from the set of PublishedParts.

    Creates a PublishedSet from the set of PublishedParts.

    The set of PublishedParts must cover all parts the output layers are partitioned into, based on the PublishedPartMapper used during publishing.

    This transformation shuffles the commit partitions in each PublishedPart if they don't use the same partitioner.

    publishedParts

    Sequence of published parts.

    id

    Deprecated.

    configOverride

    Deprecated.

    returns

    A PublishedSet that reflects the metadata of the published partitions.

    Definition Classes
    DeltaContextImpl → DeltaContext
  35. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from ContextLogging

Inherited from DeltaContextImpl

Inherited from DeltaContext

Inherited from AnyRef

Inherited from Any

Ungrouped