Packages

trait Pattern extends Function[Name, Name] with Serializable

Represents a leveling pattern.

A Pattern controls a leveling algorithm by deciding if a given partition should be mapped to another one, usually one of its parent and at a different level. Partitions are mapped to other partitions to balance the density of the content, such as map many small partitions to one single partition at a lower level; big partitions remain unmapped. Content is rebalanced and leveled, at the "leveling point".

Typically a leveling point may not be defined for every partition.

A Pattern can be used in multiple ways: - In com.here.platform.data.processing.compiler.direct.CompileInFn.mappingFn, com.here.platform.data.processing.compiler.mapgroup.CompileInFn.compileInFn, com.here.platform.data.processing.compiler.reftree.CompileInFnWithRefs.compileInFn or com.here.platform.data.processing.compiler.reftree.CompileInFnWithRefsReturnsReferences.compileInRefsFn to produce com.here.platform.data.processing.compiler.OutKeys that are functions of the density of the input, thus resulting in a density-based leveling of output partitions. - In the specialized com.here.platform.data.processing.spark.partitioner.AdaptiveLevelingPartitioner to define Spark partitions based on content density and distribute processing more uniformly across Spark partitions. This does not affect the output catalog but only the runtime characteristics of the process.

Leveling patterns are defined on com.here.platform.data.processing.catalog.Partition.Name. This concept is applicable to both the Generic and the HereTile partitioning scheme.

It is possible to use the pattern as a scala.Predef.Function to map each partition to its leveling point. In case the partition passed is not supposed to be aggregated, it is returned unchanged.

A Pattern must be scala.Serializable: it is usually calculated in the com.here.platform.data.processing.driver.Driver and transferred to worker nodes to implement the distributed adaptive leveling and/or the density-aware Spark partitioning.

Given the size and complexity of some Patterns, the org.apache.spark.broadcast.Broadcast mechanism should be used when capturing a Pattern. This happens for example when a Pattern is passed as parameter in the constructor of your implementation of com.here.platform.data.processing.compiler.direct.CompileInFn, com.here.platform.data.processing.compiler.mapgroup.CompileInFn or com.here.platform.data.processing.compiler.reftree.CompileInFn.

Linear Supertypes
Serializable, Serializable, (Name) ⇒ Name, AnyRef, Any
Known Subclasses
AdaptivePattern, FixedPattern
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Pattern
  2. Serializable
  3. Serializable
  4. Function1
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def levelingTargetFor(partition: Name): Option[Name]

    Finds the leveling point to which a partition should be mapped.

    Finds the leveling point to which a partition should be mapped.

    partition

    The name of a partition.

    returns

    The partition to which the given partition should be mapped, if any.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def andThen[A](g: (Name) ⇒ A): (Name) ⇒ A
    Definition Classes
    Function1
    Annotations
    @unspecialized()
  5. final def apply(partition: Name): Name

    Enables using the pattern as a scala.Predef.Function.

    Enables using the pattern as a scala.Predef.Function.

    partition

    The name of a partition.

    returns

    The partition to which the given partition should be mapped, if any; otherwise the partition name is returned unchanged.

    Definition Classes
    Pattern → Function1
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. def compose[A](g: (A) ⇒ Name): (A) ⇒ Name
    Definition Classes
    Function1
    Annotations
    @unspecialized()
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    Function1 → AnyRef → Any
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from (Name) ⇒ Name

Inherited from AnyRef

Inherited from Any

Ungrouped