Error: InvalidPartOrder

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
dalzielmiller
Posts: 1
Joined: Tue Dec 27, 2022 3:07 am

Error: InvalidPartOrder

Post by dalzielmiller »

Zimbra 9.0.0_GA_4258.NETWORK
Hello. We are testing HSM with secondary volume on custom S3.
After manually starting doMoveBlobs operation we see move blob errors in mailbox.log:

Code: Select all

2022-12-26 11:12:35,245 WARN  [HSM Blob Mover Copy] [tid=137;operationId=ea7856b6-63d3-4ffe-99de-44f5888ccdcc;module=ZxPowerstore;operation=MoveBlobs;] mailbox - Failed to copy 3461 of mailbox 232 from volume 1: Error occurred in underlying filesystem com.amazonaws.services.s3.model.AmazonS3Exception: The list of parts was not in ascending order. The parts list must be specified in order by part number. (Service: Amazon S3; Status Code: 400; Error Code: InvalidPartOrder; Request ID: a5b9bedac20d3bb8; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null Skipping.
2022-12-26 14:12:35,725 ERROR [HSM Blob Mover Copy] [tid=194;operationId=ea7856b6-63d3-4ffe-99de-44f5888ccdcc;module=ZxPowerstore;operation=MoveBlobs;] mailbox - com.zextras.lib.vfs.s3.S3Error: Error occurred in underlying filesystem com.amazonaws.services.s3.model.AmazonS3Exception: The list of parts was not in ascending order. The parts list must be specified in order by part number. (Service: Amazon S3; Status Code: 400; Error Code: InvalidPartOrder; Request ID: 06d91f2cb40bf913; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
        at com.zextras.lib.vfs.s3.S3FS.wrapException (S3FS.java:373)
        at com.zextras.lib.vfs.s3.S3StreamWriter$2.run (S3StreamWriter.java:248)
        at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:144)
Caused by: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The list of parts was not in ascending order. The parts list must be specified in order by part number. (Service: Amazon S3; Status Code: 400; Error Code: InvalidPartOrder; Request ID: 06d91f2cb40bf913; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
        at com.zextras.lib.vfs.s3.S3Error.<init> (S3Error.java:16)
        at com.zextras.lib.vfs.s3.S3FS.wrapException (S3FS.java:373)
        at com.zextras.lib.vfs.s3.S3StreamWriter$2.run (S3StreamWriter.java:248)
        at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:144)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The list of parts was not in ascending order. The parts list must be specified in order by part number. (Service: Amazon S3; Status Code: 400; Error Code: InvalidPartOrder; Request ID: 06d91f2cb40bf913; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse (AmazonHttpClient.java:1811)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse (AmazonHttpClient.java:1395)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest (AmazonHttpClient.java:1371)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper (AmazonHttpClient.java:1145)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute (AmazonHttpClient.java:802)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer (AmazonHttpClient.java:770)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute (AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500 (AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute (AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:550)
        at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:530)
        at com.amazonaws.services.s3.AmazonS3Client.invoke (AmazonS3Client.java:5062)
        at com.amazonaws.services.s3.AmazonS3Client.invoke (AmazonS3Client.java:5008)
        at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload (AmazonS3Client.java:3490)
        at com.zextras.lib.s3.S3Client.completeMultipartUpload (S3Client.java:346)
        at com.zextras.lib.vfs.s3.S3StreamWriter$2.run (S3StreamWriter.java:237)
        at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:144)
In hsm monitor we see skipped blobs (about 2% of moved blobs). Also in S3 console we see failed multipart uploads.

1. Is there any way to force right part order?
2. Do this failed blobs stay in primary volume?
3. If we run doMoveBlobs again will zimbra try to move failed blobs again, creating another failed multipart uploads in S3?
Post Reply