Package com.here.platform.data.client.engine.javadsl

Interface WriteEngine

All Known Implementing Classes:
WriteEngineImpl
public interface WriteEngine
Engine supporting write operations to the underlying storage

  • Method Details

    • blobChecksum

      CompletionStage<Optional<String>> blobChecksum(String layer, NewPartition.BufferedBlob blob)
      Computes a blob's checksum based on its layer configuration.

      The algorithm used is defined accordingly to the layer's configuration. If the layer doesn't have the respective field set, then an empty optional is returned.

      Parameters:
      layer - the layer that the specified blob is associated to.
      blob - the blob that will have its checksum computed.
      Returns:
      a checksum or empty if the layer's configuration field is not set.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • blobCrc

      CompletionStage<Optional<String>> blobCrc(String layer, NewPartition.BufferedBlob blob)
      Computes a blob's crc based on its layer configuration.

      The algorithm used is defined accordingly to the layer's configuration. If the layer doesn't have the respective field set, then an empty optional is returned.

      Parameters:
      layer - the layer that the specified blob is associated to.
      blob - the blob that will have its crc computed.
      Returns:
      a crc or empty if the layer's configuration field is not set.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • completeObjectUpload

      CompletionStage<org.apache.pekko.Done> completeObjectUpload(String layerId, String key, String multipartToken, ObjectStoreUploadResult parts)
      Complete multi-part upload for object in objectstore

      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      multipartToken - the the token of the multi-part process
      parts - the information of all the parts uploaded
      Returns:
      future which completes when blob copy request is sent to server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • copyObject

      CompletionStage<org.apache.pekko.Done> copyObject(String layerId, String key, String sourceKey)
      Copies object from one location to other

      Parameters:
      layerId - the id of the layer
      key - the key of the destination object
      sourceKey - the key of the source object
      Returns:
      future which completes when blob copy request is sent to server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • deleteObject

      CompletionStage<org.apache.pekko.Done> deleteObject(String layerId, String key)
      Deletes the specified object from object store.

      Parameters:
      layerId - the id of the layer
      key - the key of the object to be deleted
      Returns:
      future which completes when blob deletion request is sent to server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • publish

      CompletionStage<org.apache.pekko.Done> publish(Iterator<? extends PendingPartition> partitions)
      Publish new partitions to a catalog.

      This method can be used to publish new partitions to stream layer.

      Parameters:
      partitions - partitions to publish.
      Returns:
      future which completes when all partitions are published to server.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • publish

      CompletionStage<org.apache.pekko.Done> publish(org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions)
      Publish new partitions to a catalog.

      This method can be used to publish new partitions to stream layer.

      Parameters:
      partitions - partitions to publish.
      Returns:
      future which completes when all partitions are published to server.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • publish

      CompletionStage<org.apache.pekko.Done> publish(org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions, PartitionEncoder encoder)
      Publish new partitions to a catalog.

      This method can be used to publish new partitions to stream layer.

      Parameters:
      partitions - partitions to publish.
      encoder - encodes partition data before adding.
      Returns:
      future which completes when all partitions are published to server.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • publishBatch2

      CompletionStage<org.apache.pekko.Done> publishBatch2(int parallelism, Optional<List<String>> layerIds, List<VersionDependency> dependencies, Iterator<? extends PendingPartition> partitions)
      Performs new batch publication to a catalog.

      This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
      Please note that there can only be one active publication at a time per versioned layer.

      Parameters:
      parallelism - define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
      layerIds - List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
      dependencies - List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
      partitions - partitions to publish.
      Returns:
      future which completes when publication is fully processed by server and new version is published to catalog.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • publishBatch2

      CompletionStage<org.apache.pekko.Done> publishBatch2(int parallelism, Optional<List<String>> layerIds, List<VersionDependency> dependencies, org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions)
      Performs new batch publication to a catalog.

      This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
      Please note that there can only be one active publication at a time per versioned layer.

      Parameters:
      parallelism - define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
      layerIds - List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
      dependencies - List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
      partitions - partitions to publish.
      Returns:
      future which completes when publication is fully processed by server and new version is published to catalog.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • publishBatch2

      CompletionStage<org.apache.pekko.Done> publishBatch2(int parallelism, Optional<List<String>> layerIds, List<VersionDependency> dependencies, org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions, PartitionEncoder encoder)
      Performs new batch publication to a catalog.

      This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
      Please note that there can only be one active publication at a time per versioned layer.

      Parameters:
      parallelism - define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
      layerIds - List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
      dependencies - List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
      partitions - partitions to publish.
      encoder - encodes partition data before adding.
      Returns:
      future which completes when publication is fully processed by server and new version is published to catalog.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • put

      CompletionStage<CommitPartition> put(PendingPartition partition)
      Adds the selected partition to the underlying blob service.

      Parameters:
      partition - the partition to add
      Returns:
      future which completes when all data was sent to publish service.
      Throws:
      ConflictException - if a different blob already exists for the generated BlobId
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • put

      CompletionStage<CommitPartition> put(PendingPartition partition, PartitionEncoder encoder)
      Adds the selected partition to the underlying blob service.

      Parameters:
      partition - the partition to add
      encoder - encodes partition data before adding.
      Returns:
      future which completes when all data was sent to publish service.
      Throws:
      ConflictException - if a different blob already exists for the generated BlobId
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • simpleUploadObject2

      CompletionStage<org.apache.pekko.Done> simpleUploadObject2(String layerId, String key, org.apache.pekko.util.ByteString data, Optional<String> contentType, Optional<ContentEncoding> contentEncoding)
      Upload object to objectstore in single request

      NOTE: This function simply uploads 1:1 to object store without any fancy logic. For most use cases we suggest to rather use function uploadObject2. This function does *NOT* - split large files in smaller upload pieces (more error prone) - automatically gzip (means if you define contentEncoding = ContentEncoding.gzip then you need to make sure your data is already gzipped; if not it will lead to an error when you try to read it later or want to see it in HERE platform portal)

      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      data - the data of the object to be uploaded
      Returns:
      future which completes when blob copy request is sent to server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • startUploadObject2

      CompletionStage<ObjectMultipartToken> startUploadObject2(String layerId, String key, Optional<String> contentType, Optional<ContentEncoding> contentEncoding)
      Start multi-part upload for object in objectstore

      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      Returns:
      future which completes when blob copy request is sent to server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • updateObjectTtl

      CompletionStage<org.apache.pekko.Done> updateObjectTtl(String layerId, String key, long ttlSeconds)
      Updates the TTL for the object stored in Volatile Layer

      Parameters:
      layerId - the id of the layer
      key - the key of the target object
      ttlSeconds - new value of ttl for the object
      Returns:
      future which completes when server update request is sent to server.

      Throws:
      UnsupportedOperationException - in case method called for non-volatile layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadAndIndex

      CompletionStage<org.apache.pekko.Done> uploadAndIndex(Iterator<NewPartition> partitions)
      Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.

      Parameters:
      partitions - the partitions to upload and to index
      Returns:
      future which completes when partition are published to server.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadAndIndex

      CompletionStage<org.apache.pekko.Done> uploadAndIndex(org.apache.pekko.stream.javadsl.Source<NewPartition,org.apache.pekko.NotUsed> partitions)
      Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.

      Parameters:
      partitions - the partitions to upload and to index
      Returns:
      future which completes when partition are published to server.
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadIMLFeatureCollection

      CompletionStage<org.apache.pekko.Done> uploadIMLFeatureCollection(String layer, FeatureCollection featureCollection, OptionalInt batchSize)
      Upload a large FeatureCollection to the Interactive Map Layer

      Parameters:
      layer - the id of the layer
      featureCollection - the FeatureCollection to upload
      batchSize - the number of features in an upload batch
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadIMLFeatureModificationList

      CompletionStage<FeatureCollection> uploadIMLFeatureModificationList(String layer, FeatureModificationList featureModificationList)
      Uploads a FeatureModificationList to the Interactive Map Layer

      Parameters:
      layer - the id of the layer
      featureModificationList - the FeatureModificationList to upload
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadIMLFeaturesAsSource

      CompletionStage<org.apache.pekko.Done> uploadIMLFeaturesAsSource(String layer, org.apache.pekko.stream.javadsl.Source<Feature,org.apache.pekko.NotUsed> features, OptionalInt batchSize)
      Upload features to the Interactive Map Layer

      Parameters:
      layer - the id of the layer
      features - source of features to be uploaded
      batchSize - the number of features in an upload batch
      Throws:
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadObject

      CompletionStage<org.apache.pekko.Done> uploadObject(String layerId, String key, com.here.platform.data.client.scaladsl.NewPartition.Blob blob)
      Uploads the specified object to object store

      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      blob - he blob of data that is to be uploaded
      Returns:
      future which completes when the data blob is uploaded to the server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadObject2

      CompletionStage<org.apache.pekko.Done> uploadObject2(String layerId, String key, com.here.platform.data.client.scaladsl.NewPartition.Blob blob, Optional<String> contentType, Optional<ContentEncoding> contentEncoding)
      Uploads the specified object to object store

      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      blob - the blob of data that is to be uploaded
      contentType - the contentType of the data that is to be uploaded. This field is optional. Its default value is application/octet-stream.
      contentEncoding - the contentEncoding of the data that is to be uploaded.This field is optional.
      Returns:
      future which completes when the data blob is uploaded to the server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error

    • uploadObjectPart

      CompletionStage<ObjectUploadPart> uploadObjectPart(String layerId, String multipartToken, int partIndex, org.apache.pekko.util.ByteString data)
      Upload part of object in objectstore

      Parameters:
      layerId - the id of the layer
      multipartToken - token of the multipart process
      partIndex - the number of part
      data - the data of part to be uploaded
      Returns:
      future which completes when blob copy request is sent to server.
      Throws:
      UnsupportedOperationException - in case method called for non-objectstore layer
      DataClientNonRetriableException - in case of non-retriable error
      DataClientRetriableException - in case of retriable error