Type Members

1.  case class BaseVersion(baseVersion: Long) extends Product with Serializable

   The base version of the output catalog.

   The base version of the output catalog. It corresponds to the version of the catalog when the job is submitted. This information can be used to validate the job configuration, preventing an old configuration, specified for an older version of the output catalog, from being used.

   baseVersion

   Base version of the output catalog.

2.  case class Changes(sinceVersion: Long, version: Long) extends ProcessingType with Product with Serializable

   Indicates that there are new changes to process for this upstream catalog.

   Indicates that there are new changes to process for this upstream catalog.

   This processing type consists of two versions: the version processed in the previous run and the current version. A pipeline can use this additional information to implement incremental processing: calculate and process only the changes that have occurred since the last time the job was submitted.

   If incremental processing is not implemented, a pipeline can ignore sinceVersion and use version only.

   sinceVersion

   The version processed in the previous run.

   version

   The current version to process.

3.  trait CommandLineConfig extends AnyRef

   Base interface for all custom command line configurations.

4.  class ConfigLoaderException extends RuntimeException

   Exception class representing a configuration loader error.

5.  case class InputCatalogDescription(hrn: HRN, processing: Option[ProcessingType]) extends Product with Serializable

   Describes one of the pipeline input catalogs.

   Describes one of the pipeline input catalogs.

   hrn

   Catalog HRN

   processing

   The ProcessingType object that holds the version and a change description. This description details whether it's possible to process the catalog incrementally.

6.  case class NoChanges(version: Long) extends ProcessingType with Product with Serializable

   Indicates that there are no new changes for this upstream catalog.

   Indicates that there are no new changes for this upstream catalog.

   version

   The current version to use as input

7.  case class OutputCatalogDescription(hrn: HRN, baseVersion: Option[Long]) extends Product with Serializable

   Describes a pipeline output catalog.

   Describes a pipeline output catalog.

   hrn

   Catalog HRN

   baseVersion

   An optional version used to base the incremental processing on

8.  case class PipelineConfig(outputCatalog: HRN, inputCatalogs: Map[String, HRN] = Map.empty, billingTag: Option[String] = None) extends Product with Serializable

   A pipeline configuration.

   A pipeline configuration.

   outputCatalog

   HRN of the output catalog.

   inputCatalogs

   Map of input catalogs' IDs and their HRNs.

   billingTag

   Optional billing tag.

9.  case class PipelineContext(config: PipelineConfig, job: Option[PipelineJob] = None) extends Product with Serializable

   The complete configuration context for a pipeline.

   The complete configuration context for a pipeline.

   config

   The pipeline configuration.

   job

   An optional job configuration; None if not specified.

10.  case class PipelineIdentity(pipelineId: String, versionId: String, deploymentId: String, realm: String) extends Product with Serializable

    Uniquely identifies a Pipeline and a run.

    Uniquely identifies a Pipeline and a run.

    pipelineId

    The UUID of the pipeline.

    versionId

    The UUID of the pipeline version.

    deploymentId

    The UUID of the instance of the pipeline job run.

    realm

    The customer's realm.

11.  case class PipelineJob(outputCatalog: Option[BaseVersion] = None, inputCatalogs: Map[String, ProcessingType] = Map.empty) extends Product with Serializable

    Configuration of a pipeline management system job.

    Configuration of a pipeline management system job.

    outputCatalog

    The optional base version of the output catalog. If it's not specified, the latest version should be used.

    inputCatalogs

    Specifies which versions and how to process each input catalog.

12.  abstract class PipelineRunner extends AnyRef

    A simple base trait for main classes that implement the pipeline processing logic.

    A simple base trait for main classes that implement the pipeline processing logic. PipelineRunner that automatically generates a pipeline context. Subclasses need only to implement a run method. Additionally, you can use command line arguments to customize the context construction.

13.  sealed trait ProcessingType extends AnyRef

    Describes the versions that should be processed for a given upstream catalog.

14.  class RawCommandLineConfig extends CommandLineConfig

    Basic command line config that only exposes the raw command line arguments.

15.  case class Reprocess(version: Long) extends ProcessingType with Product with Serializable

    Indicates that it is needed to reprocess the whole catalog.

    Indicates that it is needed to reprocess the whole catalog.

    This happens when upstream catalog versions are retired. Reprocessing may also be requested on purpose to prevent incremental processing.

    version

    The version that should be fully reprocessed