Packages

package root

Definition Classes: root

package com

Definition Classes: root

package here

Definition Classes: com

package platform

Definition Classes: here

package data

Definition Classes: platform

package processing

This package provides the Data Processing Library for building distributed data processing applications.

A Runner both implements the interface with the environment for an application to run, and starts the application. The application, in turn, is driven by a Driver, that controls and performs the distributed processing.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to define com.here.platform.data.processing.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on some kind of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.DirectMToNCompiler: incremental compilation where every output tile depends on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally mixes in the a runner trait (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application is run. See the Main classes in the example compilers for more details.

com.here.platform.data.processing.catalog, com.here.platform.data.processing.blobstore, and com.here.platform.data.processing.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: data

package java

This package provides Java bindings for the Data Processing Library, to build distributed data processing applications in Java.

Choose a Runner best suited for the environment where the application runs.

The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.

The main entry point in the processing library is the com.here.platform.data.processing.java.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.

Tasks are implemented using one or more compilers.

The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to implement a com.here.platform.data.processing.java.compiler.Direct1ToNCompiler.

Other more complex compilation patterns are based on different types of dependency tracking between input partitions and output partitions.

The processing Library supports the following patterns:

- com.here.platform.data.processing.java.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.java.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.java.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.java.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.DirectMToNCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.java.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic

The application's main object normally extends a runner class (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application runs. For more details, see the Main classes in the example compilers.

com.here.platform.data.processing.java.catalog, com.here.platform.data.processing.java.blobstore, and com.here.platform.data.processing.java.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes: processing

package blobstore

Java bindings for the blobstore package.

Definition Classes: java

package broadcast

Provides the basic functionality to perform org.apache.spark.broadcast.Broadcast creation.

To use a broadcast variable in the processing library, developers are required to add the variable's hash to the fingerprints. This ensures that incremental compilations are not compromised. This package object provides the method toBroadcast() to use for that purpose. The package also offers a functionality to query catalogs by properly managing versions. The functionality is based on the org.apache.spark.broadcast.Broadcast which offers a self-contained mechanism to create a single broadcast variable of a generic type.

This compiler class is the preferred way of working with broadcast variables in the Data Processing Library.

Definition Classes: java

package catalog

Java bindings for the catalog package.

Definition Classes: java

package compiler

Definition Classes: java

package driver

Definition Classes: java

package config

package filter

package job

Default

Driver

DriverBuilder

DriverContext

DriverTask

MultiCompilerTaskBuilder

TaskBuilder

TaskResult

package impl

Definition Classes: java

package leveling

Definition Classes: java

package publisher

Definition Classes: java

package spark

Definition Classes: java

package utils

Definition Classes: java

com.here.platform.data.processing.java

driver

package driver

Ordering

Alphabetic

Visibility

Public
All

Type Members

trait Driver extends AnyRef
Instantiates a Driver using a com.here.platform.data.processing.java.driver.DriverBuilder to run a Spark pipeline from one or more input catalogs to one output catalog, using the versions specified for all of the catalogs and the library configuration.
Instantiates a Driver using a com.here.platform.data.processing.java.driver.DriverBuilder to run a Spark pipeline from one or more input catalogs to one output catalog, using the versions specified for all of the catalogs and the library configuration.
This trait conceptually wraps together Catalogs, Retrievers, Uploaders, com.here.platform.data.processing.java.publisher.Publisher and processing code in the form of one or more DriverTasks.
class DriverBuilder extends AnyRef
Constructs a Driver from one or more DriverTasks.
Constructs a Driver from one or more DriverTasks.
You can build com.here.platform.data.processing.java.driver.DriverTasks using the TaskBuilders provided by this class and then you can add them to the DriverBuilder.

Note
This is a Java friendly version of com.here.platform.data.processing.driver.DriverBuilder.
abstract class DriverContext extends AnyRef
Provides access to a set of common resources a com.here.platform.data.processing.driver.Driver has that can be used by DriverTasks and Compilers.
Provides access to a set of common resources a com.here.platform.data.processing.driver.Driver has that can be used by DriverTasks and Compilers.

Note
This is a Java friendly version of com.here.platform.data.processing.driver.DriverContext.
trait DriverTask extends AnyRef
A DriverTask processes input catalogs, given pairs of versions, and publishes the result to an output catalog, incrementing its version.
A DriverTask processes input catalogs, given pairs of versions, and publishes the result to an output catalog, incrementing its version. One or more DriverTasks are attached to a Driver, which executes these DriverTasks sequentially.

Note
This is a Java friendly wrapper of com.here.platform.data.processing.driver.DriverTask.
trait MultiCompilerTaskBuilder extends AnyRef
Constructs a multi-compiler com.here.platform.data.processing.java.driver.DriverTask by creating com.here.platform.data.processing.driver.Executors for each compiler and their applicable com.here.platform.data.processing.driver.modes.TaskModes.
Constructs a multi-compiler com.here.platform.data.processing.java.driver.DriverTask by creating com.here.platform.data.processing.driver.Executors for each compiler and their applicable com.here.platform.data.processing.driver.modes.TaskModes.
Executors' IDs can be supplied or can be left out so that they are auto-generated. When supplied they must be unique or an exception will be thrown at runtime. When they are automatically generated they have the form of taskId-n where n is the index of the position of the executor in the constructed task sequence
The resulting applicable mode is incremental, if all compilers in the driver task are incremental. Otherwise, it is non-incremental, even if only one non-incremental compiler is added to the task.
When the task is executed, each compiler is executed sequentially. An output layer produced by a compiler can be used as input data for the subsequent steps.

Note
the class DriverBuilder offers the method DriverBuilder.newMultiCompilerTaskBuilder to create this builder, it is suggested to use it.
,
This is a Java friendly version of com.here.platform.data.processing.driver.MultiCompilerTaskBuilder.
trait TaskBuilder extends AnyRef
Constructs a com.here.platform.data.processing.java.driver.DriverTask by creating an com.here.platform.data.processing.driver.Executor and its applicable com.here.platform.data.processing.driver.modes.TaskModes.
Constructs a com.here.platform.data.processing.java.driver.DriverTask by creating an com.here.platform.data.processing.driver.Executor and its applicable com.here.platform.data.processing.driver.modes.TaskModes.

Note
This is a Java friendly version of com.here.platform.data.processing.driver.TaskBuilder.
case class TaskResult(toCommit: Option[JavaPairRDD[Key, Commit]]) extends Product with Serializable
The result obtained from processing a single DriverTask.
The result obtained from processing a single DriverTask.
toCommit
The metadata to commit, if the DriverTask produced any. An empty java.util.Optional, otherwise.

Value Members

object Default
object DriverContext

Packages

driver 

package driver

Type Members

Value Members

Ungrouped

driver