final class AdaptivePattern extends Pattern
Levels partitions by parent com.here.platform.data.processing.catalog.Partition.HereTiles.
This Pattern is used to implement adaptive leveling of output tiles based on content density. Adaptive leveling can be used to output tiles at a lower level in geographic areas where the content is sparse or at a higher level in geographic areas where the content is dense.
This solution results in the following: - the sizes of the output tiles are more uniform and distributed closer to the average size - extremes such as few tiles that are too big or too many small tiles are avoided - download times are more uniform and predictable, especially for interactive applications
This Pattern can also be used to balance the sizes of Spark partitions to obtain a more even, uniformed distribution of content inside them, avoiding cases where the partitions are too heavy to process, or there are too many light partitions. This results in smoother processing and better cluster resource utilization, without affecting the output.
The Pattern is controlled by a set of parent tiles that represent leveling points in the tiles tree. If a partition is a HereTile with a parent included in that set, then the partition is mapped to that parent; the parent is the leveling point. If there are multiple parents present in the controlling set, the closest parent is the leveling point. This is determined by navigating from the HereTile upwards toward the root.
Partition names that are not HereTiles, or are orphan HereTiles in the controlling set are left unmapped.
In cases where every HereTile needs to be aggregated, make sure to include the root HereTile in the controlling set of parent tiles, so that every HereTile has at least one leveling point.
Consider passing this object to Spark worker nodes inside a org.apache.spark.broadcast.Broadcast, as the set of controlling parents may be very large.
This solution applies to HereTiles only, as it requires tiles to have a chain of parents.
However, developers may implement a similar pattern for the Generic partitioning scheme with a custom Pattern. Suppose you want to level generic data in single partitions, one per country for small countries; or in multiple partitions, one per region/state for large countries. You can establish a convention to use Generic partition names with ISO country codes for small countries, such as AND or SLO, and country codes followed by region/state codes for large countries, such as USA_CA or CAN_BC. Then, you can implement and use a custom Pattern that holds the set of ISO country codes of the large countries. Ultimately, given a country code and a region/state code, the pattern returns just the country code if it's not in the set. Otherwise, the pattern concatenates the country code and the region/state code.
- Note
Use a AdaptivePatternEstimator to compute the pattern and a com.here.platform.data.processing.spark.partitioner.AdaptiveLevelingPartitioner to balance the size of Spark partitions.
- Alphabetic
- By Inheritance
- AdaptivePattern
- Pattern
- Serializable
- Serializable
- Function1
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
andThen[A](g: (Name) ⇒ A): (Name) ⇒ A
- Definition Classes
- Function1
- Annotations
- @unspecialized()
-
final
def
apply(partition: Name): Name
Enables using the pattern as a scala.Predef.Function.
Enables using the pattern as a scala.Predef.Function.
- partition
The name of a partition.
- returns
The partition to which the given partition should be mapped, if any; otherwise the partition name is returned unchanged.
- Definition Classes
- Pattern → Function1
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
compose[A](g: (A) ⇒ Name): (A) ⇒ Name
- Definition Classes
- Function1
- Annotations
- @unspecialized()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(other: Any): Boolean
- Definition Classes
- AdaptivePattern → AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AdaptivePattern → AnyRef → Any
-
def
indexOfParentTile(parentTile: HereTile): Option[Int]
The stable index of a parent tile in the pattern.
The stable index of a parent tile in the pattern.
- parentTile
The parent tile.
- returns
The index of the parent tile if this is in the pattern, none otherwise.
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
levelingTargetFor(partition: Name): Option[Name]
Finds the leveling point to which a partition should be mapped.
Finds the leveling point to which a partition should be mapped.
- partition
The name of a partition
- returns
The partition to which the given partition should be mapped, if any
- Definition Classes
- AdaptivePattern → Pattern
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
val
numParentTiles: Int
The number of parent tiles that describe the pattern
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- Function1 → AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()