Packages

package root

Definition Classes: root

package com

Definition Classes: root

package here

Definition Classes: com

package platform

Definition Classes: here

package data

Definition Classes: platform

package processing

This package provides the Data Processing Library for building distributed data processing applications.

A Runner both implements the interface with the environment for an application to run, and starts the application. The application, in turn, is driven by a Driver, that controls and performs the distributed processing.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to define com.here.platform.data.processing.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on some kind of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.DirectMToNCompiler: incremental compilation where every output tile depends on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally mixes in the a runner trait (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application is run. See the Main classes in the example compilers for more details.

com.here.platform.data.processing.catalog, com.here.platform.data.processing.blobstore, and com.here.platform.data.processing.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: data

package java

This package provides Java bindings for the Data Processing Library, to build distributed data processing applications in Java.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.java.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to implement a com.here.platform.data.processing.java.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on different types of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.java.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.java.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.java.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.java.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.DirectMToNCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.java.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally extends a runner class (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application runs. For more details, see the Main classes in the example compilers.

com.here.platform.data.processing.java.catalog, com.here.platform.data.processing.java.blobstore, and com.here.platform.data.processing.java.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: processing

package blobstore

Java bindings for the blobstore package.

Definition Classes: java

package broadcast

Provides the basic functionality to perform org.apache.spark.broadcast.Broadcast creation.

To use a broadcast variable in the processing library, developers are required to add the variable's hash to the fingerprints. This ensures that incremental compilations are not compromised. This package object provides the method toBroadcast() to use for that purpose. The package also offers a functionality to query catalogs by properly managing versions. The functionality is based on the org.apache.spark.broadcast.Broadcast which offers a self-contained mechanism to create a single broadcast variable of a generic type.

This compiler class is the preferred way of working with broadcast variables in the Data Processing Library.

Definition Classes: java

package catalog

Java bindings for the catalog package.

Definition Classes: java

package partition

package partitioning

package utils

CatalogViewSinceVersion

package compiler

Definition Classes: java

package driver

Definition Classes: java

package impl

Definition Classes: java

package leveling

Definition Classes: java

package publisher

Definition Classes: java

package spark

Definition Classes: java

package utils

Definition Classes: java

com.here.platform.data.processing.java

catalog

package catalog

Java bindings for the catalog package.

Linear Supertypes

AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

catalog
AnyRef
Any

Hide All
Show All

Visibility

Public
Protected

Package Members

package partition
package partitioning
package utils

Type Members

trait Catalog extends CatalogLight
The interface to access the catalog metadata via Spark.
The interface to access the catalog metadata via Spark.
A catalog is partitioned logically into layers, to give clients the ability to only access the type of metadata that the client is interested in. A catalog can be configured to expose common properties and layer-specific properties. Metadata is versioned, so you can detect what has changed between two versions or retrieve metadata for a given version to provide a consistent view.
This extends the base interface CatalogLight by providing methods to process metadata using Spark.
You can retrieve the actual data by specifying the metadata and catalog configuration. This is implemented by com.here.platform.data.processing.java.blobstore.Retriever.
Note
This is a Java friendly wrapper of com.here.platform.data.processing.catalog.Catalog.
trait CatalogLight extends AnyRef
The interface to access basic catalog properties and perform the catalog operations that do not involve metadata or Spark.
The interface to access basic catalog properties and perform the catalog operations that do not involve metadata or Spark.
Note
This a Java friendly wrapper of com.here.platform.data.processing.catalog.CatalogLight.
sealed abstract class CatalogView extends AnyRef
Provides a view on a specific version of a catalog.
Provides a view on a specific version of a catalog. The view may reference a non-existing version EmptyCatalogView, an existing version of a catalog CatalogViewAtVersion or an existing version of catalog with a reference to a previous version CatalogViewSinceVersion.
final class CatalogViewAtVersion extends CatalogView with NonEmptyCatalogView
Provides a view on a specific version of a catalog, without a reference to a previous version of the catalog.
final class CatalogViewSinceVersion extends CatalogView with NonEmptyCatalogView
Provides a view on a specific version of a catalog, with a reference to a previous version of the catalog, with respect to which changes are defined.
trait CommitPart extends WrappedAbstractType
The type of each part identifier of a multipart commit.
trait CommitToken extends WrappedAbstractType
The type of a commit token.
case class Configuration(layers: Map[String, LayerConfiguration]) extends Product with Serializable
The interface for catalog configuration.
The interface for catalog configuration.
The configuration is a mapping from the layer ID to the layer-specific configuration.
Note
This is a Java friendly version of com.here.platform.data.processing.catalog.Configuration.
final class EmptyCatalogView extends CatalogView
Provides a view on an empty catalog.
Provides a view on an empty catalog. This can be used to model the access of a catalog in a version that does not exist.
case class LayerConfiguration(partitioning: Scheme, dataUrl: Option[URL], properties: Map[String, String], digest: Digest) extends Product with Serializable
Layer-specific configuration.
Layer-specific configuration.
Contains layer-specific parameters and partition data common to every partition in the layer. A layer is a partitioned dataset, for example, geo-spatially, that contains homogeneous feature data to be compiled.
partitioning
The partitioning scheme for the layer, either generic or heretile.
dataUrl
The prefix to use in front of the dataHandle for every partition. Not used with the Data API.
properties
Generic layer properties, such as the content type and schema bundle name. Not used with the Data API.
digest
The digest object used to compute the checksums for this layer.
Note
This is a Java friendly version of com.here.platform.data.processing.catalog.LayerConfiguration.
sealed trait NonEmptyCatalogView extends CatalogView
Provides a view on an existing version of a non-empty catalog.
trait WrappedAbstractType extends AnyRef
Wraps a Scala abstract type, to make it easily accessible from Java.

Packages

catalog

package catalog

Package Members

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

catalog

package catalog

Package Members

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

catalog