Packages

package root

Definition Classes: root

package com

Definition Classes: root

package here

Definition Classes: com

package platform

Definition Classes: here

package data

Definition Classes: platform

package processing

This package provides the Data Processing Library for building distributed data processing applications.

A Runner both implements the interface with the environment for an application to run, and starts the application. The application, in turn, is driven by a Driver, that controls and performs the distributed processing.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to define com.here.platform.data.processing.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on some kind of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.DirectMToNCompiler: incremental compilation where every output tile depends on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally mixes in the a runner trait (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application is run. See the Main classes in the example compilers for more details.

com.here.platform.data.processing.catalog, com.here.platform.data.processing.blobstore, and com.here.platform.data.processing.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: data

package blobstore

Contains an abstract interface used to access BlobStore.

Obtain the retriever and uploader for a catalog directly from the com.here.platform.data.processing.catalog.Catalog instance.

Definition Classes: processing

package broadcast

This package should be considered as the preferred way of working with broadcast variables in a Data Processing Library compiler.

This package should be considered as the preferred way of working with broadcast variables in a Data Processing Library compiler. It provides the basic functionality to create an instance of org.apache.spark.broadcast.Broadcast.

This functionality is based on org.apache.spark.broadcast.Broadcast which offers a self-contained mechanism to create a single broadcast variable of a generic type.

The toBroadcast() method is provided to create a broadcast variable and add the hash to the fingerprint of the com.here.platform.data.processing.driver.DriverContext, which is required in order for incremental compilations to work correctly.

The package also enables developers to query the catalogs without the need to manage versions manually.

Definition Classes: processing

package catalog

Contains an abstract Scala interface for accessing catalogs from Spark.

Use the com.here.platform.data.processing.catalog.Catalog factory methods to obtain instances.

Definition Classes: processing

package datastore

package utils

Classes representing catalog versions.

AppCredentials

AppInfo

Catalog

CatalogFactory

CatalogLight

CatalogLightFactory

CatalogMetrics

CatalogNullMetrics

CatalogProxy

CatalogView

CatalogViewAtVersion

CatalogViewLight

CatalogViewSinceVersion

Configuration

Dependency

DependencyType

DirectDependency

EmptyCatalogView

EndpointLocatorSettings

Implicits

IndirectDependency

Layer

LayerConfiguration

NonEmptyCatalogView

Partition

Partitioning

QueryPersistence

QueryReuse

ServiceSupport

package clientfactory

Definition Classes: processing

package compiler

Definition Classes: processing

package driver

Definition Classes: processing

package exception

Definition Classes: processing

package java

This package provides Java bindings for the Data Processing Library, to build distributed data processing applications in Java.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.java.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to implement a com.here.platform.data.processing.java.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on different types of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.java.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.java.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.java.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.java.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.DirectMToNCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.java.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally extends a runner class (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application runs. For more details, see the Main classes in the example compilers.

com.here.platform.data.processing.java.catalog, com.here.platform.data.processing.java.blobstore, and com.here.platform.data.processing.java.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: processing

package leveling

Definition Classes: processing

package logging

Definition Classes: processing

package publisher

Definition Classes: processing

package spark

Definition Classes: processing

package statistics

Common statistics utilities.

Definition Classes: processing

package utils

Definition Classes: processing

package validation

Definition Classes: processing

com.here.platform.data.processing

catalog

package catalog

Contains an abstract Scala interface for accessing catalogs from Spark.

Use the com.here.platform.data.processing.catalog.Catalog factory methods to obtain instances.

Linear Supertypes

AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

catalog
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Type Members

case class AppCredentials(id: String, code: String) extends Product with Serializable
Models HERE app credentials, made up of an ID and a code.
Models HERE app credentials, made up of an ID and a code. Not used with the Data API.
id
the app_id
code
the app_code
case class AppInfo(appCredentials: AppCredentials, tag: Option[String] = None) extends Product with Serializable
Models a HERE app information, made up of credentials and a tag.
Models a HERE app information, made up of credentials and a tag.
appCredentials
The application credentials.
tag
The application tag.
trait Catalog extends CatalogLight with CatalogNullMetrics with ContextLogging
The interface to access the catalog metadata via Spark.
The interface to access the catalog metadata via Spark.
A catalog is partitioned logically into layers, to give clients the ability to only access the type of metadata that the client is interested in. A catalog can be configured to expose common properties and layer-specific properties. Metadata is versioned, so you can detect what has changed between two versions or retrieve metadata for a given version to provide a consistent view.
This extends the base interface CatalogLight by providing methods to process metadata using Spark.
You can retrieve the actual data by specifying the metadata and catalog configuration. This is implemented by com.here.platform.data.processing.blobstore.Retriever.
trait CatalogFactory extends ServiceSupport
A Catalog factory class.
A Catalog factory class. Different implementations of the catalog interface must provide an implementation of the create method and specify which service(s) they support.
trait CatalogLight extends AnyRef
The interface to access basic catalog properties and perform the catalog operations that do not involve metadata or Spark.
trait CatalogLightFactory extends ServiceSupport
A CatalogLight factory class.
A CatalogLight factory class. Different implementations of the catalog interface must provide an implementation of the create method and specify which service(s) they support.
trait CatalogMetrics extends CatalogNullMetrics
The mixin to add statistical calculation with Spark Accumulators.
trait CatalogNullMetrics extends AnyRef
The default mixin which does not collect any statistics.
class CatalogProxy extends Catalog
Proxy class to allow mixins for runtime loaded Catalog classes.
sealed trait CatalogView extends CatalogViewLight
Provides a view on a specific version of a catalog.
Provides a view on a specific version of a catalog. The view may reference a non-existing version EmptyCatalogView, an existing version of a catalog CatalogViewAtVersion or an existing version of catalog with a reference to a previous version CatalogViewSinceVersion.
final case class CatalogViewAtVersion(catalog: Catalog, version: Version) extends NonEmptyCatalogView with Product with Serializable
Provides a view on a specific version of a catalog, without a reference to a previous version of the catalog.
Provides a view on a specific version of a catalog, without a reference to a previous version of the catalog.
catalog
The catalog which is encapsulated by this view.
version
The version of the catalog. Must be non-negative.
trait CatalogViewLight extends AnyRef
Provides a view on a specific version of a CatalogLight.
final case class CatalogViewSinceVersion(catalog: Catalog, version: Version, sinceVersion: Version) extends NonEmptyCatalogView with Product with Serializable
Provides a view on a specific version of a catalog, with a reference to a previous version of the catalog, with respect to which changes are defined.
Provides a view on a specific version of a catalog, with a reference to a previous version of the catalog, with respect to which changes are defined.
catalog
The catalog which is encapsulated by this view.
version
The version of the catalog. Must be non-negative.
sinceVersion
The version of the catalog with respect to which changes are calculated. Must be non-negative.
case class Configuration(layers: Map[Id, LayerConfiguration]) extends Product with Serializable
The interface for catalog configuration.
The interface for catalog configuration.
The configuration is a mapping from the layer ID to the layer-specific configuration.
case class Dependency(hrn: HRN, version: Version, isDirect: Boolean) extends Product with Serializable
The dependencies that were used to generate the data in a catalog.
The dependencies that were used to generate the data in a catalog.
hrn
The HERE Resource Name for the catalog that was used to generate data for this version.
version
The version of the catalog that was used as a dependency.
isDirect
Whether the dependency is direct or indirect. Direct means that the data from this catalog was used by the filter processing the current version. Indirect means that the data from this catalog was not used by the filter processing the current version but it was only used by other upstream filters to generate the input data for this filter.
sealed trait DependencyType extends AnyRef
case class EmptyCatalogView(catalog: Catalog) extends CatalogView with Product with Serializable
Provides a view on an empty catalog.
Provides a view on an empty catalog. This can be used to model the access of a catalog in a version that does not exist.
catalog
The catalog which is encapsulated by this view.
case class EndpointLocatorSettings(datastoreBaseUrl: Option[URL], blobstoreBaseUrl: Option[URL]) extends Product with Serializable
Endpoint locator settings for a data store.
Endpoint locator settings for a data store. Not used with the Data API.
datastoreBaseUrl
The base URL to the data store.
blobstoreBaseUrl
The base URL to the blob store.
case class LayerConfiguration(partitioning: Scheme, dataUrl: Option[URL], properties: Map[String, String] = Map.empty, digest: Digest = DefaultDigest, compressed: Boolean = false) extends Product with Serializable
Layer-specific configuration.
Layer-specific configuration.
Contains layer-specific parameters and partition data common to every partition in the layer. A layer is a partitioned dataset, for example, geo-spatially, that contains homogeneous feature data to be compiled.
partitioning
The partitioning scheme for the layer, either generic or heretile.
dataUrl
The prefix to use in front of the dataHandle for every partition. Not used with the Data API.
properties
Generic layer properties, such as the content type and schema bundle name. Not used with the Data API.
digest
The digest object used to compute the checksums for this layer.
compressed
Whether the layer is compressed (its contentEncoding is gzip).
sealed trait NonEmptyCatalogView extends CatalogView
Provides a view on an existing version of a non-empty catalog.
trait QueryPersistence extends Catalog
Persists RDDs that are returned by the Catalog querying APIs.
trait QueryReuse extends QueryPersistence
Reuses catalog query RDDs by persisting the corresponding RDD references in private mutable maps, similar to caches.
trait ServiceSupport extends AnyRef
Specifies if a given service is supported.

Value Members

object AppInfo extends Serializable
object Catalog
Identifier to refer to a catalog in the compiler configuration.
object CatalogLight
object Dependency extends Serializable
object DirectDependency extends DependencyType with Product with Serializable
Catalog A is said to be directly dependent on Catalog B, if data from Catalog A was used to create Catalog B.
object EndpointLocatorSettings extends Serializable
object Implicits
Implicit conversions for working with catalogs and partitions.
object IndirectDependency extends DependencyType with Product with Serializable
Catalog A is indirectly dependent on catalog B, if there is a catalog C from which data was used to create catalog A and data of Catalog B was used to create Catalog C.
object Layer
The model properties for layers and references to these layers.
object Partition
Groups common types to represent partitions, both generic and tiles, as well as various types of metadata that are used in the various stages of processing.
Groups common types to represent partitions, both generic and tiles, as well as various types of metadata that are used in the various stages of processing. The information includes metadata returned by querying the catalog for a particular version, metadata returned by querying the differences between two versions, and metadata provided when committing a new version of a catalog.
object Partitioning
Layer partitioning traits.

Packages

catalog

package catalog

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

catalog 

package catalog

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

catalog