package statistics
Common statistics utilities.
- Alphabetic
- By Inheritance
- statistics
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
case class
Buckets[T](bucketDelimiters: Array[T], itemsNumInBuckets: Array[Int])(implicit evidence$1: Ordering[T], evidence$2: Numeric[T]) extends Product with Serializable
Groups incoming values by user defined buckets.
Groups incoming values by user defined buckets. The bucket must be of type that supports ordering. User is not supposed to create the objects of this class directly, only by means of factory method.
- T
the type of element in the bucket. Defines the type of delimiters of your buckets and the type of value that this accumulable will accumulate
- bucketDelimiters
the delimiters of buckets as Array of Numeric. For example, given (1, 10, 100), four buckets will be created (-inf:1), [1:10), [10:100), [100:+inf)
- itemsNumInBuckets
Array that contains the number of elements in bucket for each bucket
-
class
BucketsAccumulator[T] extends AccumulatorV2[T, Buckets[T]]
Adapter for Spark 2.0 accumulatorV2.
Adapter for Spark 2.0 accumulatorV2.
- T
type of the value to accumulate, must be Numeric
-
class
CatalogStats extends AnyRef
Accumulates the catalog metrics using Spark.
Accumulates the catalog metrics using Spark.
- Note
A case class is not useful because there may be multiple accumulators with different versions.
-
case class
PayloadStats(calls: LongAccumulator, kBytes: LongAccumulator, kBytesBuckets: BucketsAccumulator[Int], time: LongAccumulator) extends Product with Serializable
Payload statistics used by both Retriever and Uploader.
-
case class
PublisherStats(compiled: LongAccumulator, compiledSize: LongAccumulator, added: LongAccumulator, changed: LongAccumulator, deleted: LongAccumulator) extends Product with Serializable
Spark accumulators for publishing statistics.
Spark accumulators for publishing statistics.
- compiled
The total number of output partitions compiled, but not necessarily published.
- added
The number of output partitions added.
- changed
The number of output partitions changed.
- deleted
The number of output partitions deleted.
-
class
Statistics extends ContextLogging
Container for all the statistics accumulators created.
Container for all the statistics accumulators created. All methods can be called in driver only.
Value Members
- val PayloadBucketLimits: Array[Int]
- def roundKBytes(bytes: Long): Int
- object Buckets extends Serializable
- object BucketsAccumulator extends Serializable
- object CatalogStats
- object PayloadStats extends Serializable
- object PublisherStats extends Serializable
-
object
RetrieveStats
Retriever statistics factory.
-
object
UploadStats
Uploader statistics factory.