implicit final class MapKeyValueWrapper[K, V] extends AnyVal
Maps RDDs without changing the partitioning.
- Alphabetic
- By Inheritance
- MapKeyValueWrapper
- AnyVal
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new MapKeyValueWrapper(rdd: RDD[(K, V)])
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- Any
-
final
def
##(): Int
- Definition Classes
- Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
collectWithKey[U](pf: PartialFunction[(K, V), U]): RDD[(K, U)]
Performs filtering and mapping on values of RDD preserving partitioning.
Performs filtering and mapping on values of RDD preserving partitioning.
- U
the type of the new values
- pf
the mapping partial function, providing the new value for the element
- returns
an RDD with mapping function applied to each element
-
def
flatMapValuesWithKey[U](f: ((K, V)) ⇒ GenIterable[U]): RDD[(K, U)]
Similar to flatMap on a pair RDD, but does not allow changing the key and preserves the partitioning.
Similar to flatMap on a pair RDD, but does not allow changing the key and preserves the partitioning.
- U
the type of the new values
- f
the mapping function providing the new value for the element
- returns
an RDD with mapping function applied to each element
-
def
flatMapValuesWithKeyParallel[U](numThreads: Int)(f: ((K, V)) ⇒ GenIterable[U]): RDD[(K, U)]
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
- U
the ype of the new values
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- f
the mapping function, providing the new value for the element
- returns
an RDD with mapping function applied to each element
-
def
flatMapValuesWithKeyParallel[U](f: ((K, V)) ⇒ GenIterable[U], numThreads: Int): RDD[(K, U)]
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
- U
the ype of the new values
- f
the mapping function, providing the new value for the element
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- returns
an RDD with mapping function applied to each element
-
def
flatMapValuesWithKeyParallelUnordered[U](numThreads: Int)(f: ((K, V)) ⇒ GenIterable[U]): RDD[(K, U)]
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1. Elements are emitted as soon as they are ready, therefore it is possible that the transformed elements in each Spark partition are not in the same order as in the upstream RDD.
- U
the ype of the new values
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- f
the mapping function, providing the new value for the element
- returns
an RDD with mapping function applied to each element
-
def
flatMapValuesWithKeyParallelUnordered[U](f: ((K, V)) ⇒ GenIterable[U], numThreads: Int): RDD[(K, U)]
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to flatMapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1. Elements are emitted as soon as they are ready, therefore it is possible that the transformed elements in each Spark partition are not in the same order as in the upstream RDD.
- U
the ype of the new values
- f
the mapping function, providing the new value for the element
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- returns
an RDD with mapping function applied to each element
-
def
getClass(): Class[_ <: AnyVal]
- Definition Classes
- AnyVal → Any
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
mapValuesParallel[U](f: (V) ⇒ U, numThreads: Int): RDD[(K, U)]
Similar to mapValues on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValues on a pair RDD, but uses a thread pool if numThreads is >1.
- U
the type of the new values
- f
the mapping function, providing the new value for the element
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- returns
an RDD with mapping function applied to each element
-
def
mapValuesParallelUnordered[U](f: (V) ⇒ U, numThreads: Int): RDD[(K, U)]
Similar to mapValues on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValues on a pair RDD, but uses a thread pool if numThreads is >1. Elements are emitted as soon as they are ready, therefore it is possible that the transformed elements in each Spark partition are not in the same order as in the upstream RDD.
- U
the type of the new values
- f
the mapping function, providing the new value for the element
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- returns
an RDD with mapping function applied to each element
-
def
mapValuesSortedWithKey[U, B](f: ((K, V)) ⇒ U, s: ((K, V)) ⇒ B)(implicit ord: Ordering[B]): RDD[(K, U)]
Similar to mapValues on a pair RDD, but provides the key as additional input to the mapping function and sorts the elements in each Spark partition before applying the function.
Similar to mapValues on a pair RDD, but provides the key as additional input to the mapping function and sorts the elements in each Spark partition before applying the function.
- U
the type of the new values
- B
the type of the ordering for the sort
- f
the mapping function, providing the new value for the element
- s
the mapping function, providing the ordering value of type
B- ord
the ordering assumed on type
B- returns
an RDD with mapping function applied to each element
-
def
mapValuesSortedWithKeyParallel[U, B](f: ((K, V)) ⇒ U, s: ((K, V)) ⇒ B, numThreads: Int)(implicit ord: Ordering[B]): RDD[(K, U)]
Similar to mapValuesSortedWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValuesSortedWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
- U
the type of the new values
- B
the type of the ordering for the sort
- f
the mapping function, providing the new value for the element
- s
the mapping function, providing the ordering value of type
B- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- ord
the ordering assumed on type
B- returns
an RDD with mapping function applied to each element
-
def
mapValuesSortedWithKeyParallelUnordered[U, B](f: ((K, V)) ⇒ U, s: ((K, V)) ⇒ B, numThreads: Int)(implicit ord: Ordering[B]): RDD[(K, U)]
Similar to mapValuesSortedWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValuesSortedWithKey on a pair RDD, but uses a thread pool if numThreads is >1. Elements are emitted as soon as they are ready, therefore it is possible that the transformed elements in each Spark partition are not in the same order as in the upstream RDD.
- U
the type of the new values
- B
the type of the ordering for the sort
- f
the mapping function, providing the new value for the element
- s
the mapping function, providing the ordering value of type
B- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- ord
the ordering assumed on type
B- returns
an RDD with mapping function applied to each element
-
def
mapValuesWithKey[U](f: ((K, V)) ⇒ U): RDD[(K, U)]
Similar to mapValues on a pair RDD, but provides the key as additional input to the mapping function.
Similar to mapValues on a pair RDD, but provides the key as additional input to the mapping function.
- U
the type of the new values
- f
the mapping function, providing the new value for the element
- returns
an RDD with mapping function applied to each element.
-
def
mapValuesWithKeyParallel[U](numThreads: Int)(f: ((K, V)) ⇒ U): RDD[(K, U)]
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
- U
the type of the new values
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- f
the mapping function, providing the new value for the element
- returns
an RDD with mapping function applied to each element
-
def
mapValuesWithKeyParallel[U](f: ((K, V)) ⇒ U, numThreads: Int): RDD[(K, U)]
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
- U
the type of the new values
- f
the mapping function, providing the new value for the element
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- returns
an RDD with mapping function applied to each element
-
def
mapValuesWithKeyParallelUnordered[U](numThreads: Int)(f: ((K, V)) ⇒ U): RDD[(K, U)]
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1. Elements are emitted as soon as they are ready, therefore it is possible that the transformed elements in each Spark partition are not in the same order as in the upstream RDD.
- U
the type of the new values
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- f
the mapping function, providing the new value for the element
- returns
an RDD with mapping function applied to each element
-
def
mapValuesWithKeyParallelUnordered[U](f: ((K, V)) ⇒ U, numThreads: Int): RDD[(K, U)]
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1.
Similar to mapValuesWithKey on a pair RDD, but uses a thread pool if numThreads is >1. Elements are emitted as soon as they are ready, therefore it is possible that the transformed elements in each Spark partition are not in the same order as in the upstream RDD.
- U
the type of the new values
- f
the mapping function, providing the new value for the element
- numThreads
the number of threads to use for parallel invocations of f(). Can only be set to a value larger than 1 if the operations include IO operations, or if multiple cores are available per executor
- returns
an RDD with mapping function applied to each element
-
def
toString(): String
- Definition Classes
- Any