trait GlobalIncrementalSwitch extends DeltaContextImpl with ContextLogging
Checks the global switch to disable incremental processing.
- Alphabetic
- By Inheritance
- GlobalIncrementalSwitch
- ContextLogging
- DeltaContextImpl
- DeltaContext
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- val autoIdAssigner: BaseSetIdAssigner
The ID assigner used for automatically assigning DeltaSet IDs if none is provided
The ID assigner used for automatically assigning DeltaSet IDs if none is provided
- Definition Classes
- DeltaContextImpl → DeltaContext
- def classForLogger: Class[_]
Override this to change the class that is displayed in logs.
Override this to change the class that is displayed in logs.
- Attributes
- protected
- Definition Classes
- ContextLogging
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- val defaultParallelism: Int
Default parallelism (number of partitions), provided by Spark.
Default parallelism (number of partitions), provided by Spark.
- Definition Classes
- DeltaContextImpl → DeltaContext
- val defaultPartitioner: PartitionNamePartitioner
Partitioner for input catalog data.
Partitioner for input catalog data.
- Definition Classes
- DeltaContextImpl → DeltaContext
- def determineIncrementalProcessingAllowed: Boolean
To be overwritten in traits
To be overwritten in traits
- Definition Classes
- GlobalIncrementalSwitch → DeltaContextImpl
- def disjointUnion[UK, UV](upstreams: Iterable[DeltaSet[UK, _ <: UV]], id: Id, configOverride: Override)(implicit arg0: ClassTag[UK], arg1: ClassTag[UV]): DeltaSet[UK, UV]
Computes the disjoint union of several DeltaSets.
Computes the disjoint union of several DeltaSets. If a key is present in two of the DeltaSets, the operation will throw an exception.
This function does not shuffle any data any data.
All DeltaSets must be partitioned by the same partitioner. If they are not, you must explicitly repartition the DeltaSets.
- upstreams
The operands.
- returns
A DeltaSet that represents the result of the union.
- Definition Classes
- DeltaContextImpl → DeltaContext
- val driverContext: DriverContext
The context of the Driver executing the DriverTask corresponding to this DeltaContext
The context of the Driver executing the DriverTask corresponding to this DeltaContext
- Definition Classes
- DeltaContextImpl → DeltaContext
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- val id: Id
The ID of the DriverTask that owns this DeltaContext
The ID of the DriverTask that owns this DeltaContext
- Definition Classes
- DeltaContextImpl → DeltaContext
- def inRetriever(id: Id): Retriever
To retrieve payloads of partitions of an input catalog.
To retrieve payloads of partitions of an input catalog.
- Definition Classes
- DeltaContextImpl → DeltaContext
- final val incrementalProcessingAllowed: Boolean
- Definition Classes
- DeltaContextImpl → DeltaContext
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final lazy val logger: ContextAwareLogger
The logger to use for normal and context-aware messages.
The logger to use for normal and context-aware messages.
- Attributes
- protected
- Definition Classes
- ContextLogging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final val partitionKeyFiltering: PartitionKeyFiltering
Can be used to determine whether a specific input partition is filtered by the partition key filter defined in the configuration, if any.
Can be used to determine whether a specific input partition is filtered by the partition key filter defined in the configuration, if any.
- Definition Classes
- DeltaContextImpl → DeltaContext
- def queryCatalog(catalogId: Id, layerIds: Set[Id], partitioner: Partitioner[Key], id: Id, configOverride: Override): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from a catalog.
- catalogId
The ID of the catalog.
- layerIds
The set of layers IDs.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
- Definition Classes
- DeltaContextImpl → DeltaContext
- def queryCatalogLayer(catalogId: Id, layerId: Id, partitioner: Partitioner[Key], id: Id, configOverride: Override): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
- catalogId
The ID of the catalog.
- layerId
The ID of the layer.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
- Definition Classes
- DeltaContextImpl → DeltaContext
- def queryCatalogs(catalogsAndLayers: Map[Id, Set[Id]], partitioner: Partitioner[Key], id: Id, configOverride: Override): DeltaSet[Key, Meta]
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
Constructs a DeltaSet containing the combined key-meta pairs from a set of layers from multiple catalogs.
- catalogsAndLayers
A map from catalog IDs to a set of layer IDs within the corresponding catalog.
- partitioner
The partitioner to use for the input data. By default, defaultPartitioner is used.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A DeltaSet containing the combined key-meta pairs from all given catalogs.
- Definition Classes
- DeltaContextImpl → DeltaContext
- val stateManager: StateManagerImpl
The state manager used to retrieve the state of a DeltaSet from the state layer.
The state manager used to retrieve the state of a DeltaSet from the state layer.
- Definition Classes
- DeltaContextImpl → DeltaContext
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- val transformations: Transformations
Import all functions in the implicits to enable transformations on DeltaSets.
Import all functions in the implicits to enable transformations on DeltaSets.
- Definition Classes
- DeltaContextImpl → DeltaContext
- def unionPublishedParts(publishedParts: Seq[PublishedPart], id: Id, configOverride: Override): PublishedSet
Creates a PublishedSet from the set of PublishedParts.
Creates a PublishedSet from the set of PublishedParts.
The set of PublishedParts must cover all parts the output layers are partitioned into, based on the PublishedPartMapper used during publishing.
This transformation shuffles the commit partitions in each PublishedPart if they don't use the same partitioner.
- publishedParts
Sequence of published parts.
- id
Deprecated.
- configOverride
Deprecated.
- returns
A PublishedSet that reflects the metadata of the published partitions.
- Definition Classes
- DeltaContextImpl → DeltaContext
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)