trait WriteEngine extends AnyRef
Engine supporting write operations to the underlying storage
- Alphabetic
- By Inheritance
- WriteEngine
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Abstract Value Members
-
abstract
def
blobChecksum(layer: String, blob: BufferedBlob): Future[Option[String]]
Computes a blob's checksum based on its layer configuration.
Computes a blob's checksum based on its layer configuration.
The algorithm used is defined accordingly to the layer's configuration. If the layer doesn't have the respective field set, then scala.None is returned.
- layer
the layer that the specified blob is associated to.
- blob
the blob that will have its checksum computed.
- returns
a checksum or None if the layer's configuration field is not set.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
blobCrc(layer: String, blob: BufferedBlob): Future[Option[String]]
Computes a blob's CRC based on its layer configuration.
Computes a blob's CRC based on its layer configuration.
The algorithm used is defined accordingly to the layer's configuration. If the layer doesn't have the respective field set, then scala.None is returned.
- layer
the layer that the specified blob is associated to.
- blob
the blob that will have its CRC computed.
- returns
a CRC or None if the layer's configuration field is not set.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
completeObjectUpload(layerId: String, key: String, multipartToken: String, parts: ObjectStoreUploadResult): Future[Done]
Complete multi-part upload for object in objectstore
Complete multi-part upload for object in objectstore
- layerId
the id of the layer
- key
the key of the object to be uploaded
- multipartToken
the the token of the multi-part process
- parts
the information of all the parts uploaded
- returns
future which completes when blob copy request is sent to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-objectstore layer
-
abstract
def
copyObject(layerId: String, key: String, sourceKey: String): Future[Done]
Copies object from one location to other
Copies object from one location to other
- layerId
the id of the layer
- key
the key of the destination object
- sourceKey
the key of the source object
- returns
future which completes when blob copy request is sent to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-objectstore layer
-
abstract
def
deleteObject(layerId: String, key: String): Future[Done]
Deletes the specified object from object store
Deletes the specified object from object store
- layerId
the id of the layer
- key
the key of the object to be deleted
- returns
future which completes when blob deletion request is sent to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-objectstore layer
-
abstract
def
publish(partitions: Source[PendingPartition, NotUsed], metricsReporter: Option[MetricReporter], encoder: Option[PartitionEncoder] = None): Future[Done]
Publish new partitions to a catalog and allows to report metrics.
Publish new partitions to a catalog and allows to report metrics.
This method can be used to publish new partitions to stream layer.
- partitions
partitions to publish.
- metricsReporter
callback used to report metrics
- encoder
encodes partition data before adding.
- returns
future which completes when all partitions are published to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
publish(partitions: Source[PendingPartition, NotUsed]): Future[Done]
Publish new partitions to a catalog.
Publish new partitions to a catalog.
This method can be used to publish new partitions to stream layer.
- partitions
partitions to publish.
- returns
future which completes when all partitions are published to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
publish(partitions: Iterator[PendingPartition]): Future[Done]
Publish new partitions to a catalog.
Publish new partitions to a catalog.
This method can be used to publish new partitions to stream layer.
- partitions
partitions to publish.
- returns
future which completes when all partitions are published to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
publishBatch2(parallelism: Int, layerIds: Option[Seq[String]], dependencies: Seq[VersionDependency], partitions: Source[PendingPartition, NotUsed], encoder: Option[PartitionEncoder] = None): Future[Done]
Performs new batch publication to a catalog.
Performs new batch publication to a catalog.
This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
Please note that there can only be one active publication at a time per versioned layer.- parallelism
define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
- layerIds
List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
- dependencies
List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
- partitions
partitions to commit
- encoder
encodes partition data before adding.
- returns
future which completes when all data was sent to blobstore and new version is committed to catalog.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
publishBatch2(parallelism: Int, layerIds: Option[Seq[String]], dependencies: Seq[VersionDependency], partitions: Iterator[PendingPartition]): Future[Done]
Performs new batch publication to a catalog.
Performs new batch publication to a catalog.
This call is identical invoking put for each partitions and later uploading metadata with PublishApi.startBatch2 + PublishApi.publishToBatch + PublishApi.completeBatch
Please note that there can only be one active publication at a time per versioned layer.- parallelism
define number of parallel operations to upload data to blob service. Optimal value depends on node configuration, RAM, CPU, network, recommended value <= 100 parallel operations. Too high parallelism affecting performance in negative way.
- layerIds
List of layer ids which should be updated. Note: Publishing versioned and volatile layer partitions in the same publication is not advisable as in this case it increases version number even for volatile layers.
- dependencies
List of the dependencies for this publication. This is only relevant for publishing to versioned layer.
- partitions
partitions to commit
- returns
future which completes when all data was sent to blobstore and new version is committed to catalog.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
put(partition: PendingPartition, encoder: Option[PartitionEncoder] = None): Future[CommitPartition]
Adds the selected partition to the underlying to blob service.
Adds the selected partition to the underlying to blob service.
- partition
the partition to add
- encoder
encodes partition data before adding.
- returns
future which completes when data was sent to publish service
- Exceptions thrown
com.here.platform.data.client.ConflictExceptionif a different blob already exists for the generated BlobIdcom.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
simpleUploadObject2(layerId: String, key: String, data: ByteString, contentType: Option[String] = None, contentEncoding: Option[ContentEncoding] = None): Future[Done]
Upload object to objectstore in single request
Upload object to objectstore in single request
NOTE: This function simply uploads 1:1 to object store without any fancy logic. For most use cases we suggest to rather use function
uploadObject2. This function does *NOT*- split large files in smaller upload pieces (more error prone)
- automatically gzip (means if you define contentEncoding = ContentEncoding.gzip then you need to make sure your data is already gzipped; if not it will lead to an error when you try to read it later or want to see it in HERE platform portal)
- layerId
the id of the layer
- key
the key of the object to be uploaded
- data
the data of the object to be uploaded
- returns
future which completes when blob copy request is sent to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-objectstore layer
-
abstract
def
startUploadObject2(layerId: String, key: String, contentType: Option[String] = None, contentEncoding: Option[ContentEncoding] = None): Future[ObjectMultipartToken]
Start multi-part upload for object in objectstore
Start multi-part upload for object in objectstore
- layerId
the id of the layer
- key
the key of the object to be uploaded
- returns
future which completes when blob copy request is sent to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-objectstore layer
-
abstract
def
updateObjectTtl(layerId: String, key: String, ttlSeconds: Long): Future[Done]
Updates the TTL for the object stored in Volatile Layer
Updates the TTL for the object stored in Volatile Layer
- layerId
the id of the layer
- key
the key of the target object
- ttlSeconds
new value of ttl for the object
- returns
future which completes when server update request is sent to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-volatile layer
-
abstract
def
uploadAndIndex(partitions: Source[NewPartition, NotUsed]): Future[Done]
Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.
Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.
- partitions
the partitions to upload and to index
- returns
future which completes when partition are published to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
uploadAndIndex(partitions: Iterator[NewPartition]): Future[Done]
Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.
Uploads the specified index partition to the underlying to blob service, and indexes it with specified fields in index layer.
- partitions
the partitions to upload and to index
- returns
future which completes when partition are published to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
uploadIMLFeatureCollection(layer: String, featureCollection: FeatureCollection, batchSize: Option[Int] = None): Future[Done]
Upload a large FeatureCollection to the Interactive Map Layer
Upload a large FeatureCollection to the Interactive Map Layer
- layer
the id of the layer
- featureCollection
the FeatureCollection to upload
- batchSize
the number of features in an upload batch
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
uploadIMLFeatureModificationList(layer: String, featureModificationList: FeatureModificationList): Future[FeatureCollection]
Upload a FeatureModificationList to the Interactive Map Layer
Upload a FeatureModificationList to the Interactive Map Layer
- layer
the id of the layer
- featureModificationList
the FeatureModificationList to upload
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
uploadIMLFeaturesAsSource(layer: String, features: Source[Feature, NotUsed], batchSize: Option[Int] = None): Future[Done]
Upload features to the Interactive Map Layer
Upload features to the Interactive Map Layer
- layer
the id of the layer
- features
source of features to be uploaded
- batchSize
the number of features in an upload batch
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable error
-
abstract
def
uploadObject2(layerId: String, key: String, blob: Blob, contentType: Option[String] = None, contentEncoding: Option[ContentEncoding] = None): Future[Done]
Uploads the specified object to object store
Uploads the specified object to object store
- layerId
the id of the layer
- key
the key of the object to be uploaded
- blob
the blob of data that is to be uploaded
- contentType
the contentType of the data that is to be uploaded. This field is optional. Its default value is "application/octet-stream".
- contentEncoding
the contentEncoding of the data that is to be uploaded.This field is optional.
- returns
future which completes when blob is published to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-objectstore layer
-
abstract
def
uploadObjectPart(layerId: String, multipartToken: String, partIndex: Int, data: ByteString): Future[ObjectUploadPart]
Upload part of object in objectstore
Upload part of object in objectstore
- layerId
the id of the layer
- multipartToken
token of the multipart process
- partIndex
the number of part
- data
the data of part to be uploaded
- returns
future which completes when blob copy request is sent to server.
- Exceptions thrown
com.here.platform.data.client.DataClientNonRetriableExceptionin case of non-retriable errorcom.here.platform.data.client.DataClientRetriableExceptionin case of retriable errorjava.lang.UnsupportedOperationExceptionin case method called for non-objectstore layer
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()