Packages

package root

Definition Classes: root

package com

Definition Classes: root

package here

Definition Classes: com

package platform

Definition Classes: here

package data

Definition Classes: platform

package processing

This package provides the Data Processing Library for building distributed data processing applications.

A Runner both implements the interface with the environment for an application to run, and starts the application. The application, in turn, is driven by a Driver, that controls and performs the distributed processing.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to define com.here.platform.data.processing.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on some kind of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.DirectMToNCompiler: incremental compilation where every output tile depends on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally mixes in the a runner trait (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application is run. See the Main classes in the example compilers for more details.

com.here.platform.data.processing.catalog, com.here.platform.data.processing.blobstore, and com.here.platform.data.processing.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: data

package blobstore

Contains an abstract interface used to access BlobStore.

Obtain the retriever and uploader for a catalog directly from the com.here.platform.data.processing.catalog.Catalog instance.

Definition Classes: processing

package broadcast

This package should be considered as the preferred way of working with broadcast variables in a Data Processing Library compiler.

This package should be considered as the preferred way of working with broadcast variables in a Data Processing Library compiler. It provides the basic functionality to create an instance of org.apache.spark.broadcast.Broadcast.

This functionality is based on org.apache.spark.broadcast.Broadcast which offers a self-contained mechanism to create a single broadcast variable of a generic type.

The toBroadcast() method is provided to create a broadcast variable and add the hash to the fingerprint of the com.here.platform.data.processing.driver.DriverContext, which is required in order for incremental compilations to work correctly.

The package also enables developers to query the catalogs without the need to manage versions manually.

Definition Classes: processing

package catalog

Contains an abstract Scala interface for accessing catalogs from Spark.

Use the com.here.platform.data.processing.catalog.Catalog factory methods to obtain instances.

Definition Classes: processing

package clientfactory

Definition Classes: processing

package compiler

Definition Classes: processing

package driver

Definition Classes: processing

package exception

Definition Classes: processing

package java

This package provides Java bindings for the Data Processing Library, to build distributed data processing applications in Java.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.java.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to implement a com.here.platform.data.processing.java.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on different types of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.java.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.java.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.java.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.java.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.DirectMToNCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.java.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally extends a runner class (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application runs. For more details, see the Main classes in the example compilers.

com.here.platform.data.processing.java.catalog, com.here.platform.data.processing.java.blobstore, and com.here.platform.data.processing.java.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: processing

package leveling

Definition Classes: processing

package logging

Definition Classes: processing

package publisher

Definition Classes: processing

package spark

Definition Classes: processing

package statistics

Common statistics utilities.

Definition Classes: processing

package utils

Definition Classes: processing

package validation

Definition Classes: processing

package scalatest

Accumulator

AggregatedLongAccumulator

DefaultJsonSerializers

IncompatibleAccumulatorOpException

com.here.platform.data.processing

validation

package validation

Ordering

Alphabetic

Visibility

Public
Protected

Package Members

package scalatest

Type Members

trait Accumulator[T <: Accumulator[T]] extends AnyRef
Base trait for all accumulator classes.
Base trait for all accumulator classes.
T
Self-recursive type.
final case class AggregatedLongAccumulator(min: Long, max: Long, count: Long, accumulated: Long) extends Accumulator[AggregatedLongAccumulator] with Product with Serializable
Accumulates long values and computes aggregated statistics.
Accumulates long values and computes aggregated statistics.
min
The minimum value accumulated.
max
The maximum value accumulated.
count
The number of accumulated values.
accumulated
The total accumulated value.
final case class DoubleAccumulator(value: Double) extends Accumulator[DoubleAccumulator] with Product with Serializable
Accumulates double values.
Accumulates double values.
value
The accumulated value.
final case class Feature(geometry: Geometry, properties: Map[String, Any] = Map.empty) extends Product with Serializable
GeoJSON feature.
sealed trait Geometry extends AnyRef
Base trait for all GeoJSON geometry classes.
final case class IncompatibleAccumulatorOpException(type1: String, type2: String) extends Exception with Product with Serializable
An exception thrown when two incompatible accumulators are merged together.
An exception thrown when two incompatible accumulators are merged together.
type1
The class name of the first accumulator.
type2
The class name of the second accumulator.
class JsonDeserializer[T] extends Deserializer[T] with Serializable
Serialization.Deserializer that deserializes instances of T from their JSON representation.
Serialization.Deserializer that deserializes instances of T from their JSON representation.
T
The deserialized type.
class JsonSerializer[T <: AnyRef] extends Serializer[T] with Serializable
Serialization.Serializer that serializes instances of T into their JSON representation.
Serialization.Serializer that serializes instances of T into their JSON representation.
T
The serialized type.
final case class LineString(coordinates: Seq[Position]) extends Geometry with Product with Serializable
GeoJSON linestring geometry.
final case class LongAccumulator(value: Long) extends Accumulator[LongAccumulator] with Product with Serializable
Accumulates long values.
Accumulates long values.
value
The accumulated value.
final case class Metrics(stats: TestStats, accumulators: Map[String, Accumulator[_]], nested: Map[String, Metrics] = Map.empty) extends Product with Serializable
Test metrics.
Test metrics.
stats
Test statistics.
accumulators
Accumulator values created during the test execution.
nested
In aggregated metrics partitions this contains the lower level metrics partitions.
final case class MultiLineString(coordinates: Seq[LineStringCoordinates]) extends Geometry with Product with Serializable
GeoJSON multi-linestring geometry.
final case class MultiPoint(coordinates: Seq[Position]) extends Geometry with Product with Serializable
GeoJSON multi-point geometry.
final case class MultiPolygon(coordinates: Seq[PolygonCoordinates]) extends Geometry with Product with Serializable
GeoJSON multi-polygon geometry.
sealed trait Outcome extends AnyRef
Outcome of a test-case.
final case class Point(coordinates: Position) extends Geometry with Product with Serializable
GeoJSON point geometry.
final case class Polygon(coordinates: Seq[LineStringCoordinates]) extends Geometry with Product with Serializable
GeoJSON polygon geometry.
final case class Position(longitude: Double, latitude: Double) extends Product with Serializable
GeoJSON position.
final case class Report(results: Map[String, TestSuiteResult]) extends Product with Serializable
Test report of a set of test-suites.
Test report of a set of test-suites.
results
Result of each test-suite, indexed by suite Suite.id.
abstract class Suite[TestData] extends SuiteLike[TestData]
Base class for all test-suite implementations, parametrized on an instance of TestData.
class SuiteCompiler[TestData] extends Serializable
Runs a suite or a collection of suites object and produces the output payloads encoding the test Report and Metrics.
Runs a suite or a collection of suites object and produces the output payloads encoding the test Report and Metrics.
TestData
The parametric type of the data under test.
sealed trait SuiteLike[TestData] extends Serializable
Base trait for a suite or a collection of suites parametrized on the same test data.
Base trait for a suite or a collection of suites parametrized on the same test data.
TestData
The type of the data under test.
final case class Suites[TestData](suites: Suite[TestData]*) extends SuiteLike[TestData] with Product with Serializable
A collection of Suites parametrized on the same data.
A collection of Suites parametrized on the same data.
This class is useful to get single aggregated Report and Metrics when multiple Suites run on the same test data.
TestData
The type of the data under test.
suites
The sequence of suites.
final case class TestCaseDetails(title: String, payload: Option[Any], geometryIndex: Seq[Int], message: Option[String]) extends Product with Serializable
Details about a test-case.
Details about a test-case.
title
The title of the test-case.
payload
Optional payload.
geometryIndex
Index to the pool of geometry features.
message
Optional additional test outcome description.
final case class TestCaseStats(succeeded: Long = 0, failed: Long = 0, skipped: Long = 0) extends Product with Serializable
Statistics about a specific test-case.
Statistics about a specific test-case.
succeeded
Number of succeeded runs.
failed
Number of failed runs.
skipped
Number of skipped runs.
trait TestContext extends AnyRef
Context of the current test suite.
Context of the current test suite.
Every Suite implementation is provided with a TestContext to register test outcomes and accumulators.
final case class TestStats(suites: Map[String, TestSuiteStats]) extends Product with Serializable
Statistics about a set of test-suites.
Statistics about a set of test-suites.
suites
Statistics about each test-suite, indexed by suite Suite.id.
final case class TestSuiteResult(isSucceeded: Boolean, succeeded: Seq[TestCaseDetails], failed: Seq[TestCaseDetails], skipped: Seq[TestCaseDetails], geometry: Seq[Feature]) extends Product with Serializable
Result of a test-suite.
Result of a test-suite.
isSucceeded
false if at least a test-case has failed.
succeeded
Details about the succeeded test-cases.
failed
Details about the failed test-cases.
skipped
Details about the skipped test-cases.
geometry
Geometry pool referenced by the TestCaseDetails of each test-case.
final case class TestSuiteStats(testCases: Map[String, TestCaseStats]) extends Product with Serializable
Statistics about a specific test-suite.
Statistics about a specific test-suite.
testCases
Statistics about each test-case in the test-suite, indexed by test-case-title.

Value Members

object DefaultJsonSerializers
Default JSON Serialization.Serializers for the Report and Metrics types.
case object Failed extends Outcome with Product with Serializable
Outcome of a test-case that failed.
object GeoJson
object Serialization
case object Skipped extends Outcome with Product with Serializable
Outcome of a test-case that was skipped.
case object Succeeded extends Outcome with Product with Serializable
Outcome of a test-case that passed.
object Transformations
Set of implicit transformations added to com.here.platform.data.processing.driver.deltasets.DeltaSets.
Set of implicit transformations added to com.here.platform.data.processing.driver.deltasets.DeltaSets.
To enable them, you have to add an implicit com.here.platform.data.processing.driver.deltasets.DeltaContext in your scope.

Packages

validation

package validation

Package Members

Type Members

Value Members

Ungrouped

validation