trait Pattern extends Function[Name, Name] with Serializable
Represents a leveling pattern.
A Pattern controls a leveling algorithm by deciding if a given partition should be mapped to another one, usually one of its parent and at a different level. Partitions are mapped to other partitions to balance the density of the content, such as map many small partitions to one single partition at a lower level; big partitions remain unmapped. Content is rebalanced and leveled, at the "leveling point".
Typically a leveling point may not be defined for every partition.
A Pattern can be used in multiple ways: - In com.here.platform.data.processing.compiler.direct.CompileInFn.mappingFn, com.here.platform.data.processing.compiler.mapgroup.CompileInFn.compileInFn, com.here.platform.data.processing.compiler.reftree.CompileInFnWithRefs.compileInFn or com.here.platform.data.processing.compiler.reftree.CompileInFnWithRefsReturnsReferences.compileInRefsFn to produce com.here.platform.data.processing.compiler.OutKeys that are functions of the density of the input, thus resulting in a density-based leveling of output partitions. - In the specialized com.here.platform.data.processing.spark.partitioner.AdaptiveLevelingPartitioner to define Spark partitions based on content density and distribute processing more uniformly across Spark partitions. This does not affect the output catalog but only the runtime characteristics of the process.
Leveling patterns are defined on com.here.platform.data.processing.catalog.Partition.Name. This concept is applicable to both the Generic and the HereTile partitioning scheme.
It is possible to use the pattern as a scala.Predef.Function to map each partition to its leveling point. In case the partition passed is not supposed to be aggregated, it is returned unchanged.
A Pattern must be scala.Serializable: it is usually calculated in the com.here.platform.data.processing.driver.Driver and transferred to worker nodes to implement the distributed adaptive leveling and/or the density-aware Spark partitioning.
Given the size and complexity of some Patterns, the org.apache.spark.broadcast.Broadcast mechanism should be used when capturing a Pattern. This happens for example when a Pattern is passed as parameter in the constructor of your implementation of com.here.platform.data.processing.compiler.direct.CompileInFn, com.here.platform.data.processing.compiler.mapgroup.CompileInFn or com.here.platform.data.processing.compiler.reftree.CompileInFn.
- Alphabetic
- By Inheritance
- Pattern
- Serializable
- Function1
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def andThen[A](g: (Name) => A): (Name) => A
- Definition Classes
- Function1
- Annotations
- @unspecialized()
- final def apply(partition: Name): Name
Enables using the pattern as a scala.Predef.Function.
Enables using the pattern as a scala.Predef.Function.
- partition
The name of a partition.
- returns
The partition to which the given partition should be mapped, if any; otherwise the partition name is returned unchanged.
- Definition Classes
- Pattern → Function1
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def compose[A](g: (A) => Name): (A) => Name
- Definition Classes
- Function1
- Annotations
- @unspecialized()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- Function1 → AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)