HERE Data SDK - Scala API references < Back

Packages

package root

Definition Classes
root
package com

Definition Classes
root
package here

Definition Classes
com
package platform

Definition Classes
here
package data

Definition Classes
platform
package processing
This package provides the Data Processing Library for building distributed data processing applications.
This package provides the Data Processing Library for building distributed data processing applications.
A Runner both implements the interface with the environment for an application to run, and starts the application. The application, in turn, is driven by a Driver, that controls and performs the distributed processing.
Choose a Runner best suited for the environment where the application runs.
The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.
The main entry point in the processing library is the com.here.platform.data.processing.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.
Tasks are implemented using one or more compilers.
The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to define com.here.platform.data.processing.compiler.Direct1ToNCompiler.
Other more complex compilation patterns are based on some kind of dependency tracking between input partitions and output partitions.
The processing Library supports the following patterns:
- com.here.platform.data.processing.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.DirectMToNCompiler: incremental compilation where every output tile depends on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic
The application's main object normally mixes in the a runner trait (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application is run. See the Main classes in the example compilers for more details.
com.here.platform.data.processing.catalog, com.here.platform.data.processing.blobstore, and com.here.platform.data.processing.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes
data
package blobstore
Contains an abstract interface used to access BlobStore.
Contains an abstract interface used to access BlobStore.
Obtain the retriever and uploader for a catalog directly from the com.here.platform.data.processing.catalog.Catalog instance.

Definition Classes
processing
package broadcast
This package should be considered as the preferred way of working with broadcast variables in a Data Processing Library compiler.
This package should be considered as the preferred way of working with broadcast variables in a Data Processing Library compiler. It provides the basic functionality to create an instance of org.apache.spark.broadcast.Broadcast.
This functionality is based on org.apache.spark.broadcast.Broadcast which offers a self-contained mechanism to create a single broadcast variable of a generic type.
The toBroadcast() method is provided to create a broadcast variable and add the hash to the fingerprint of the com.here.platform.data.processing.driver.DriverContext, which is required in order for incremental compilations to work correctly.
The package also enables developers to query the catalogs without the need to manage versions manually.

Definition Classes
processing
package catalog
Contains an abstract Scala interface for accessing catalogs from Spark.
Contains an abstract Scala interface for accessing catalogs from Spark.
Use the com.here.platform.data.processing.catalog.Catalog factory methods to obtain instances.

Definition Classes
processing
package clientfactory

Definition Classes
processing
package compiler

Definition Classes
processing
package driver

Definition Classes
processing
package config
This package contains the configuration classes for all components of a Driver.
This package contains the configuration classes for all components of a Driver. Configuration is read from application.conf and provided to the developer as a com.here.platform.data.processing.driver.config.CompleteConfig instance, when the driver is setup.
package deltasets
package filter
package impl
package job
package modes
package runner
Default
DeltaDriverTask
DeltaDriverTaskBuilder
DeltaSetup
DeltaSimpleSetup
Driver
DriverBuilder
DriverContext
DriverSetup
DriverSetupManual
DriverSetupWithBuilder
DriverTask
Executor
Fingerprints
IncrementalCompilerExecutor
MultiCompilerTaskBuilder
MultiModeDriverTask
NonIncrementalCompilerExecutor
PartitionKeyFiltering
StatefulCompilerExecutor
TaskBuilder
TaskResult
package exception

Definition Classes
processing
package java
This package provides Java bindings for the Data Processing Library, to build distributed data processing applications in Java.
This package provides Java bindings for the Data Processing Library, to build distributed data processing applications in Java.
A Runner both implements the interface with the environment for an application to run, and starts the application. The application, in turn, is driven by a Driver, that controls and performs the distributed processing.
Choose a Runner best suited for the environment where the application runs.
The Driver performs one of more tasks which read layers from input catalogs and write to one or more layers of an output catalog.
The main entry point in the processing library is the com.here.platform.data.processing.java.driver.DriverBuilder class where you can add different kinds of tasks to the driver. The driver runs the tasks, and commits the final results to the output catalog.
Tasks are implemented using one or more compilers.
The simplest compiler is the direct compiler which maps each input tile to N output tiles. The application needs to implement a com.here.platform.data.processing.java.compiler.Direct1ToNCompiler.
Other more complex compilation patterns are based on different types of dependency tracking between input partitions and output partitions.
The processing Library supports the following patterns:
- com.here.platform.data.processing.java.compiler.NonIncrementalCompiler: non-incremental compilation only - com.here.platform.data.processing.java.compiler.DepCompiler: non-incremental dependency calculation and incremental compilation - com.here.platform.data.processing.java.compiler.IncrementalDepCompiler: incremental dependency calculation and compilation - com.here.platform.data.processing.java.compiler.Direct1ToNCompiler: incremental compilation where every output tile depends only on one input tile, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.DirectMToNCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping is independent from tile content - com.here.platform.data.processing.java.compiler.MapGroupCompiler: incremental compilation where every output tile can depend on multiple input tiles, and this mapping depend on the tile content - com.here.platform.data.processing.java.compiler.RefTreeCompiler: fully-managed two phases incremental compilation that can resolve references between input partitions. Input/Output dependency management is implemented and the developer doesn't need to provide this logic
The application's main object normally extends a runner class (like PipelineRunner) to setup the Driver, and interfaces with the environment where the application runs. For more details, see the Main classes in the example compilers.
com.here.platform.data.processing.java.catalog, com.here.platform.data.processing.java.blobstore, and com.here.platform.data.processing.java.publisher contain utilities for accessing catalogs and payloads in a Spark-friendly way, providing an RDD-based abstraction over data and metadata. These classes are used by the processing library, but can also be used independently.

Definition Classes
processing
package leveling

Definition Classes
processing
package logging

Definition Classes
processing
package publisher

Definition Classes
processing
package spark

Definition Classes
processing
package statistics
Common statistics utilities.
Common statistics utilities.

Definition Classes
processing
package utils

Definition Classes
processing
package validation

Definition Classes
processing

com.here.platform.data.processing

driver

package driver

Ordering

Alphabetic

Visibility

Public
All

Type Members

class DeltaDriverTask extends DriverTask with ContextLogging
A DriverTask that commits a collection of com.here.platform.data.processing.driver.deltasets.PublishedSets.
class DeltaDriverTaskBuilder extends AnyRef
A builder to construct a DeltaDriverTask.
A builder to construct a DeltaDriverTask. Extend the trait DeltaSetup to setup the com.here.platform.data.processing.driver.deltasets.DeltaSets to be evaluated by the DeltaDriverTask, and then use this builder to construct the DeltaDriverTask.

Note
If the driver is set up manually, this builder does not need to be used.
trait DeltaSetup extends AnyRef
Implement this trait to setup a DeltaDriverTask via the DeltaDriverTaskBuilder.
Implement this trait to setup a DeltaDriverTask via the DeltaDriverTaskBuilder.

Note
If the driver is set up manually, this trait does not need to be used.
trait DeltaSimpleSetup extends DriverSetup with DeltaSetup
Simplifies the setup of a Driver for the use with DeltaSets.
Simplifies the setup of a Driver for the use with DeltaSets.
It will create a Driver with a single DeltaDriverTask, evaluating a set of com.here.platform.data.processing.driver.deltasets.DeltaSets, which in turn are constructed by the DeltaSetup.setupSets, which must be implemented by the inheritor of this class.
trait Driver extends AnyRef
Instantiates a Driver using a com.here.platform.data.processing.driver.DriverBuilder to run a Spark pipeline from one or more input catalogs to one output catalog, using the versions specified for all of the catalogs and the library configuration.
Instantiates a Driver using a com.here.platform.data.processing.driver.DriverBuilder to run a Spark pipeline from one or more input catalogs to one output catalog, using the versions specified for all of the catalogs and the library configuration.
This trait conceptually wraps together Catalogs, Retrievers, Uploaders, com.here.platform.data.processing.publisher.Publisher and processing code in the form of one or more DriverTasks.
class DriverBuilder extends AnyRef
Constructs a Driver from one or more DriverTasks.
Constructs a Driver from one or more DriverTasks.
You can build DriverTasks using the TaskBuilders provided by this class and then you can add them to the DriverBuilder.
Finally, you must call DriverBuilder!.build to construct the Driver.
trait DriverContext extends AnyRef
Provides access to a set of common resources which a Driver has, that can be used by DriverTasks and Compilers.
trait DriverSetup extends AnyRef
The root interface to provide the library with a fully-configured Driver.
The root interface to provide the library with a fully-configured Driver. The Driver is then executed by the library.
This setup procedure has two aspects: - Configuring the Driver - which DriverTasks and which compilers. - Interfacing with the caller - parsing command line as in a PipelineRunner.
Developers should extend one of the runners and mix in one of the configuration patterns to obtain a valid executable class.
Possible setup patterns: - DriverSetupWithBuilder - DriverSetupManual
trait DriverSetupManual extends DriverSetup
Implements the Driver setup pattern DriverSetup to enable developers to manually configure each DriverTask.
Implements the Driver setup pattern DriverSetup to enable developers to manually configure each DriverTask. It is the developer's task to return the list of manually constructed DriverTasks, as required. In turn, the processing library instantiates a corresponding Driver to run these tasks sequentially.
trait DriverSetupWithBuilder extends DriverSetup
Implements the Driver setup pattern DriverSetup using a com.here.platform.data.processing.driver.DriverBuilder.
Implements the Driver setup pattern DriverSetup using a com.here.platform.data.processing.driver.DriverBuilder. It is the developer's task to configure the passed builder as required and return it.
trait DriverTask extends AnyRef
A DriverTask processes input catalogs, given pairs of versions, and publishes the result to an output catalog, incrementing its version.
A DriverTask processes input catalogs, given pairs of versions, and publishes the result to an output catalog, incrementing its version. One or more DriverTasks are attached to a Driver, which executes these DriverTasks sequentially.

See also
Driver
sealed trait Executor extends InputLayers with InputPartitioner with OutputLayers with OutputPartitioner
Base class for all the executors.
Base class for all the executors.
Executors work at RDD level, meaning that RDDs are passed and returned to the functions that each executor implements. It is important to define a common policy regarding the persistence of RDDs that are passed and returned. Not respecting this policy introduces a risk of Spark throwing an exception due to the fact that some RDDs may be persisted twice with different storage levels. This policy may be applicable to classes other than executors, such as com.here.platform.data.processing.compiler.NonIncrementalCompiler, com.here.platform.data.processing.compiler.DepCompilerBase and derived.

Note
The policy established is as follows: RDDs passed to every execute function are guaranteed to be reusable multiple times, efficiently, without the need for the implementations to persist them. Implementations shall not persist passed RDDs. These are either already persisted by the processing library or guaranteed to be reusable multiple time efficiently. Therefore, implementations shall not require() or assert() that passed RDDs are persisted, although it's guaranteed that they will be, or equivalent. RDDs returned by every execute function don't have to be persisted. They may be persisted if it's useful to the implementations. The processing library may persist the RDDs once returned, if not already persisted.
final class Fingerprints extends AnyRef
Fingerprints of compilation.
Fingerprints of compilation. They are constructed from both source code and configuration. The library uses the fingerprints, for example, to verify that the incremental compilation is done using the same setup as a previous full compilation, and to disable incremental compilation otherwise.
trait IncrementalCompilerExecutor extends Executor
The main interface to implement a compiler that uses the changes from the previous compilation to optimize the amount of input data to be processed, and incrementally publishes the resulting output.
class MultiCompilerTaskBuilder extends AnyRef
Constructs a multi-compiler DriverTask by creating Executors for each compiler and their applicable com.here.platform.data.processing.driver.modes.TaskModes.
Constructs a multi-compiler DriverTask by creating Executors for each compiler and their applicable com.here.platform.data.processing.driver.modes.TaskModes.
Executors' IDs can be supplied or can be left out so that they are auto-generated. When supplied they must be unique or an exception will be thrown at runtime. When they are automatically generated they have the form of taskId-n where n is the index of the position of the executor in the constructed task sequence
The resulting applicable mode is incremental, if all compilers in the driver task are incremental. Otherwise, it is non-incremental, even if only one non-incremental compiler is added to the task.
When the task is executed, each compiler is executed sequentially. An output layer produced by a compiler can be used as input data for the subsequent steps.

Note
the class DriverBuilder offers the method DriverBuilder.newMultiCompilerTaskBuilder to create this builder, it is suggested to use it.
class MultiModeDriverTask extends DriverTask
The DriverTask implementation that runs the first com.here.platform.data.processing.driver.modes.TaskMode that is applicable.
The DriverTask implementation that runs the first com.here.platform.data.processing.driver.modes.TaskMode that is applicable. Throws an exception if no mode is applicable.
trait NonIncrementalCompilerExecutor extends Executor
The main interface to implement a compiler that always compiles the complete input catalog into the complete output catalog.
The main interface to implement a compiler that always compiles the complete input catalog into the complete output catalog. This compiler provides developers with the maximum level of freedom to develop a compiler, but does not perform well when compiling small changes.
case class PartitionKeyFiltering(partitionKeyFilters: Seq[PartitionKeyFilterConfig]) extends Product with Serializable
Takes care of applying configured com.here.platform.data.processing.driver.filter.PartitionKeyFilter on the input data.
trait StatefulCompilerExecutor extends Executor
Main interface to implement a compiler that saves a state for the next run, while the state of the previous run is provided as additional input.
Main interface to implement a compiler that saves a state for the next run, while the state of the previous run is provided as additional input. This compiler enables complex incremental compilation patterns.
class TaskBuilder extends AnyRef
Constructs a DriverTask by creating an Executor and its applicable com.here.platform.data.processing.driver.modes.TaskModes.
Constructs a DriverTask by creating an Executor and its applicable com.here.platform.data.processing.driver.modes.TaskModes.

Note
Use the factory method in com.here.platform.data.processing.driver.DriverBuilder to instantiate a TaskBuilder.
case class TaskResult(toCommit: Option[RDD[(Key, Commit)]]) extends Product with Serializable
The result obtained from the processing of a single DriverTask.
The result obtained from the processing of a single DriverTask.
toCommit
The metadata to commit, if the DriverTask produced a result. None, if the DriverTask did not produce any metadata to commit.

Value Members

object Default
object DriverContext
object DriverTask
object Executor
Executor factory, to create an Executor corresponding to each compiler pattern
object Fingerprints

Packages

driver 

package driver

Type Members

Value Members

Ungrouped

driver