Package com.here.platform.data.client.engine.javadsl

Class WriteEngineImpl

java.lang.Object
com.here.platform.data.client.engine.javadsl.WriteEngineImpl
All Implemented Interfaces:
WriteEngine, com.here.platform.data.client.javadsl.internal.Implicits
public final class WriteEngineImpl extends Object implements WriteEngine, com.here.platform.data.client.javadsl.internal.Implicits

  • Constructor Details

    • WriteEngineImpl

      public WriteEngineImpl(com.here.platform.data.client.engine.scaladsl.WriteEngine delegate)

  • Method Details

    • blobChecksum

      public CompletionStage<Optional<String>> blobChecksum(String layer, NewPartition.BufferedBlob blob)
      Description copied from interface: WriteEngine
      Computes a blob's checksum based on its layer configuration.

      The algorithm used is defined accordingly to the layer's configuration. If the layer doesn't have the respective field set, then an empty optional is returned.

      Specified by:
      blobChecksum in interface WriteEngine
      Parameters:
      layer - the layer that the specified blob is associated to.
      blob - the blob that will have its checksum computed.
      Returns:
      a checksum or empty if the layer's configuration field is not set.

    • blobCrc

      public CompletionStage<Optional<String>> blobCrc(String layer, NewPartition.BufferedBlob blob)
      Description copied from interface: WriteEngine
      Computes a blob's crc based on its layer configuration.

      The algorithm used is defined accordingly to the layer's configuration. If the layer doesn't have the respective field set, then an empty optional is returned.

      Specified by:
      blobCrc in interface WriteEngine
      Parameters:
      layer - the layer that the specified blob is associated to.
      blob - the blob that will have its crc computed.
      Returns:
      a crc or empty if the layer's configuration field is not set.

    • completeObjectUpload

      public CompletionStage<org.apache.pekko.Done> completeObjectUpload(String layerId, String key, String multipartToken, ObjectStoreUploadResult parts)
      Description copied from interface: WriteEngine
      Complete multi-part upload for object in objectstore

      Specified by:
      completeObjectUpload in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      multipartToken - the the token of the multi-part process
      parts - the information of all the parts uploaded
      Returns:
      future which completes when blob copy request is sent to server.

    • copyObject

      public CompletionStage<org.apache.pekko.Done> copyObject(String layerId, String key, String sourceKey)
      Description copied from interface: WriteEngine
      Copies object from one location to other

      Specified by:
      copyObject in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the destination object
      sourceKey - the key of the source object
      Returns:
      future which completes when blob copy request is sent to server.

    • deleteObject

      public CompletionStage<org.apache.pekko.Done> deleteObject(String layerId, String key)
      Description copied from interface: WriteEngine
      Deletes the specified object from object store.

      Specified by:
      deleteObject in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the object to be deleted
      Returns:
      future which completes when blob deletion request is sent to server.

    • publish

      public CompletionStage<org.apache.pekko.Done> publish(Iterator<? extends PendingPartition> partitions)
      Description copied from interface: WriteEngine
      Publish new partitions to a catalog.

      This method can be used to publish new partitions to stream layer.

      Specified by:
      publish in interface WriteEngine
      Parameters:
      partitions - partitions to publish.
      Returns:
      future which completes when all partitions are published to server.

    • publish

      public CompletionStage<org.apache.pekko.Done> publish(org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions)
      Description copied from interface: WriteEngine
      Publish new partitions to a catalog.

      This method can be used to publish new partitions to stream layer.

      Specified by:
      publish in interface WriteEngine
      Parameters:
      partitions - partitions to publish.
      Returns:
      future which completes when all partitions are published to server.

    • publish

      public CompletionStage<org.apache.pekko.Done> publish(org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions, PartitionEncoder encoder)
      Description copied from interface: WriteEngine
      Publish new partitions to a catalog.

      This method can be used to publish new partitions to stream layer.

      Specified by:
      publish in interface WriteEngine
      Parameters:
      partitions - partitions to publish.
      encoder - encodes partition data before adding.
      Returns:
      future which completes when all partitions are published to server.

    • publishBatch2

      public CompletionStage<org.apache.pekko.Done> publishBatch2(int parallelism, Optional<List<String>> layerIds, List<VersionDependency> dependencies, Iterator<? extends PendingPartition> partitions)
      Description copied from interface: WriteEngine
      Performs new batch publication to a catalog.

      This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
      Please note that there can only be one active publication at a time per versioned layer.

      Specified by:
      publishBatch2 in interface WriteEngine
      Parameters:
      parallelism - define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
      layerIds - List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
      dependencies - List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
      partitions - partitions to publish.
      Returns:
      future which completes when publication is fully processed by server and new version is published to catalog.

    • publishBatch2

      public CompletionStage<org.apache.pekko.Done> publishBatch2(int parallelism, Optional<List<String>> layerIds, List<VersionDependency> dependencies, org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions)
      Description copied from interface: WriteEngine
      Performs new batch publication to a catalog.

      This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
      Please note that there can only be one active publication at a time per versioned layer.

      Specified by:
      publishBatch2 in interface WriteEngine
      Parameters:
      parallelism - define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
      layerIds - List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
      dependencies - List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
      partitions - partitions to publish.
      Returns:
      future which completes when publication is fully processed by server and new version is published to catalog.

    • publishBatch2

      public CompletionStage<org.apache.pekko.Done> publishBatch2(int parallelism, Optional<List<String>> layerIds, List<VersionDependency> dependencies, org.apache.pekko.stream.javadsl.Source<PendingPartition,org.apache.pekko.NotUsed> partitions, PartitionEncoder encoder)
      Description copied from interface: WriteEngine
      Performs new batch publication to a catalog.

      This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
      Please note that there can only be one active publication at a time per versioned layer.

      Specified by:
      publishBatch2 in interface WriteEngine
      Parameters:
      parallelism - define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
      layerIds - List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
      dependencies - List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
      partitions - partitions to publish.
      encoder - encodes partition data before adding.
      Returns:
      future which completes when publication is fully processed by server and new version is published to catalog.

    • put

      public CompletionStage<CommitPartition> put(PendingPartition partition)
      Description copied from interface: WriteEngine
      Adds the selected partition to the underlying blob service.

      Specified by:
      put in interface WriteEngine
      Parameters:
      partition - the partition to add
      Returns:
      future which completes when all data was sent to publish service.

    • put

      public CompletionStage<CommitPartition> put(PendingPartition partition, PartitionEncoder encoder)
      Description copied from interface: WriteEngine
      Adds the selected partition to the underlying blob service.

      Specified by:
      put in interface WriteEngine
      Parameters:
      partition - the partition to add
      encoder - encodes partition data before adding.
      Returns:
      future which completes when all data was sent to publish service.

    • simpleUploadObject2

      public CompletionStage<org.apache.pekko.Done> simpleUploadObject2(String layerId, String key, org.apache.pekko.util.ByteString data, Optional<String> contentType, Optional<ContentEncoding> contentEncoding)
      Description copied from interface: WriteEngine
      Upload object to objectstore in single request

      NOTE: This function simply uploads 1:1 to object store without any fancy logic. For most use cases we suggest to rather use function uploadObject2. This function does *NOT* - split large files in smaller upload pieces (more error prone) - automatically gzip (means if you define contentEncoding = ContentEncoding.gzip then you need to make sure your data is already gzipped; if not it will lead to an error when you try to read it later or want to see it in HERE platform portal)

      Specified by:
      simpleUploadObject2 in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      data - the data of the object to be uploaded
      Returns:
      future which completes when blob copy request is sent to server.

    • startUploadObject2

      public CompletionStage<ObjectMultipartToken> startUploadObject2(String layerId, String key, Optional<String> contentType, Optional<ContentEncoding> contentEncoding)
      Description copied from interface: WriteEngine
      Start multi-part upload for object in objectstore

      Specified by:
      startUploadObject2 in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      Returns:
      future which completes when blob copy request is sent to server.

    • updateObjectTtl

      public CompletionStage<org.apache.pekko.Done> updateObjectTtl(String layerId, String key, long ttlSeconds)
      Description copied from interface: WriteEngine
      Updates the TTL for the object stored in Volatile Layer

      Specified by:
      updateObjectTtl in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the target object
      ttlSeconds - new value of ttl for the object
      Returns:
      future which completes when server update request is sent to server.

    • uploadAndIndex

      public CompletionStage<org.apache.pekko.Done> uploadAndIndex(Iterator<NewPartition> partitions)
      Description copied from interface: WriteEngine
      Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.

      Specified by:
      uploadAndIndex in interface WriteEngine
      Parameters:
      partitions - the partitions to upload and to index
      Returns:
      future which completes when partition are published to server.

    • uploadAndIndex

      public CompletionStage<org.apache.pekko.Done> uploadAndIndex(org.apache.pekko.stream.javadsl.Source<NewPartition,org.apache.pekko.NotUsed> partitions)
      Description copied from interface: WriteEngine
      Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.

      Specified by:
      uploadAndIndex in interface WriteEngine
      Parameters:
      partitions - the partitions to upload and to index
      Returns:
      future which completes when partition are published to server.

    • uploadIMLFeatureCollection

      public CompletionStage<org.apache.pekko.Done> uploadIMLFeatureCollection(String layer, FeatureCollection featureCollection, OptionalInt batchSize)
      Description copied from interface: WriteEngine
      Upload a large FeatureCollection to the Interactive Map Layer

      Specified by:
      uploadIMLFeatureCollection in interface WriteEngine
      Parameters:
      layer - the id of the layer
      featureCollection - the FeatureCollection to upload
      batchSize - the number of features in an upload batch

    • uploadIMLFeatureModificationList

      public CompletionStage<FeatureCollection> uploadIMLFeatureModificationList(String layer, FeatureModificationList featureModificationList)
      Description copied from interface: WriteEngine
      Uploads a FeatureModificationList to the Interactive Map Layer

      Specified by:
      uploadIMLFeatureModificationList in interface WriteEngine
      Parameters:
      layer - the id of the layer
      featureModificationList - the FeatureModificationList to upload

    • uploadIMLFeaturesAsSource

      public CompletionStage<org.apache.pekko.Done> uploadIMLFeaturesAsSource(String layer, org.apache.pekko.stream.javadsl.Source<Feature,org.apache.pekko.NotUsed> features, OptionalInt batchSize)
      Description copied from interface: WriteEngine
      Upload features to the Interactive Map Layer

      Specified by:
      uploadIMLFeaturesAsSource in interface WriteEngine
      Parameters:
      layer - the id of the layer
      features - source of features to be uploaded
      batchSize - the number of features in an upload batch

    • uploadObject

      public CompletionStage<org.apache.pekko.Done> uploadObject(String layerId, String key, com.here.platform.data.client.scaladsl.NewPartition.Blob blob)
      Description copied from interface: WriteEngine
      Uploads the specified object to object store

      Specified by:
      uploadObject in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      blob - he blob of data that is to be uploaded
      Returns:
      future which completes when the data blob is uploaded to the server.

    • uploadObject2

      public CompletionStage<org.apache.pekko.Done> uploadObject2(String layerId, String key, com.here.platform.data.client.scaladsl.NewPartition.Blob blob, Optional<String> contentType, Optional<ContentEncoding> contentEncoding)
      Description copied from interface: WriteEngine
      Uploads the specified object to object store

      Specified by:
      uploadObject2 in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      key - the key of the object to be uploaded
      blob - the blob of data that is to be uploaded
      contentType - the contentType of the data that is to be uploaded. This field is optional. Its default value is application/octet-stream.
      contentEncoding - the contentEncoding of the data that is to be uploaded.This field is optional.
      Returns:
      future which completes when the data blob is uploaded to the server.

    • uploadObjectPart

      public CompletionStage<ObjectUploadPart> uploadObjectPart(String layerId, String multipartToken, int partIndex, org.apache.pekko.util.ByteString data)
      Description copied from interface: WriteEngine
      Upload part of object in objectstore

      Specified by:
      uploadObjectPart in interface WriteEngine
      Parameters:
      layerId - the id of the layer
      multipartToken - token of the multipart process
      partIndex - the number of part
      data - the data of part to be uploaded
      Returns:
      future which completes when blob copy request is sent to server.