final class AdaptiveLevelingPartitioner extends PartitionNamePartitioner with ScalaPartitionNamePartitionerWrapper
A PartitionerOfKey that uses a precalculated leveling com.here.platform.data.processing.java.leveling.Pattern.
Keys are distributed to Spark partitions strictly following the leveling points that the pattern specifies. Keys left not aggregated by the pattern are distributed among a disjoint set of Spark partitions using a fallback partitioner, if specified. Otherwise they are uniformly distributed over the existing partitions.
The number of partitions used for aggregated keys is fixed and matches the number of leveling points of the pattern.
- Alphabetic
- By Inheritance
- AdaptiveLevelingPartitioner
- ScalaPartitionNamePartitionerWrapper
- Wrapper
- PartitionNamePartitioner
- PartitionerOfKey
- Serializable
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
AdaptiveLevelingPartitioner(pattern: AdaptivePattern, fallbackPartitioner: PartitionNamePartitioner)
- pattern
The adaptive leveling pattern that controls the partitioning.
- fallbackPartitioner
The partitioner used for non-aggregated keys.
-
new
AdaptiveLevelingPartitioner(pattern: AdaptivePattern)
- pattern
The adaptive leveling pattern that controls the partitioning.
- Note
With this constructor, non aggregated keys are distributed over the existing Spark partitions of the pattern.
- new AdaptiveLevelingPartitioner(impl: spark.partitioner.AdaptiveLevelingPartitioner)
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(o: Any): Boolean
- Definition Classes
- Wrapper → AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
getPartitionForKey(key: Key): Int
Gets the partition for a given key.
Gets the partition for a given key.
- key
the key for which the partition must be calculated
- returns
the partition, identified by one scala.Int, in which the key should be located
- Definition Classes
- PartitionNamePartitioner → PartitionerOfKey
-
def
getPartitionForName(name: Name): Int
- Definition Classes
- ScalaPartitionNamePartitionerWrapper → PartitionNamePartitioner
-
def
hashCode(): Int
- Definition Classes
- Wrapper → AnyRef → Any
-
val
impl: spark.partitioner.AdaptiveLevelingPartitioner
- Definition Classes
- AdaptiveLevelingPartitioner → Wrapper
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
numPartitions: Int
Gets the total number of partitions defined by the partitioner.
Gets the total number of partitions defined by the partitioner.
- returns
the total number of partitions defined
- Definition Classes
- ScalaPartitionNamePartitionerWrapper → PartitionerOfKey
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toSpark: Partitioner
Gets a standard Spark org.apache.spark.Partitioner wrapping this one.
Gets a standard Spark org.apache.spark.Partitioner wrapping this one.
- returns
this partitioner wrapped inside a standard Spark interface
- Definition Classes
- ScalaPartitionNamePartitionerWrapper → PartitionerOfKey
-
def
toString(): String
- Definition Classes
- Wrapper → AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()