Duplicate email uploads exceeding users quota - ZBUG-4613

[Matt]

Hi all,

I have a support ticket open for this (case 01747918) which resulted in a bug being opened (ZBUG-4613). I'm wondering if anyone else is experiencing the same problem....

I'm seeing several IMAP users uploading large amounts of data, exceeding their quota limits. Repeated uploads of the same emails (duplicate message id's) which does not increase their reported mailbox size but does increase actual storage consumed on disk. In the worst case, I have a user with a 30GB quota, 24GB reported quota usage, consuming 600GB of storage.

All of the occurrences I've seen, the email client has been New Outlook. However, it's only a few users so that might be unrelated.

If I search for the top repeat occurrences of the same message ID being added by an IMAP client, I see emails with many thousands of uploads:

Code: Select all

$ grep "mailop - Adding Message:" /opt/zimbra/log/mailbox.log | grep "ImapSSLServer" | grep ', Message-ID=' | awk -F ', Message-ID=' '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10
  14783 <01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>,
   4385 <CAB+tdd=+TfupKG=3c+z-=gYJpzouW8FN826B2FhvOKgksT_J5Q@mail.gmail.com>,
   2763 <CANOjgt5OXy=wsTv4dOqerC9AxekYgO_O6BAqcRpHig@mail.gmail.com>,
   2551 <DB7PR05MB5738EFF32FC94CA949F78C6PR05MB5738.eurprd05.prod.outlook.com>,
   2341 <CWXP265MB37516E2283DB34E58BD31BCXP265MB3751.GBRP265.PROD.OUTLOOK.COM>,
   2193 <f4488561080440a9aee6d4f7ca4babc9123MB2438.GBRP123.PROD.OUTLOOK.COM>,
   2193 <CAB+BJom-DjHVVo2f4iAmXEksCgPvtg50hC20XARZvpQ@mail.gmail.com>,
   2192 <LO3P123MB3162DAF07B678C33939F102@LO3PMB3162.GBRP123.PROD.OUTLOOK.COM>,
   1887 <DB7PR05MB5738FC539745C56053D260A4F@DB7PB5738.eurprd05.prod.outlook.com>,
   1754 <CWXP265MB3671C178899F1BDEA102@CWXMB3671.GBR265.PROD.OUTLOOK.COM>,

If I then search for the top entry in this list, I see it's the same user uploading it over and over (I had to redact the username and IP below):

Code: Select all

$ grep '01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com' /opt/zimbra/log/mailbox.log
2025-01-07 12:33:21,112 INFO  [ImapSSLServer-31] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492816, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:22,099 INFO  [ImapSSLServer-2] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492817, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:23,740 INFO  [ImapSSLServer-49] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492818, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:25,378 INFO  [ImapSSLServer-18] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492819, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:27,095 INFO  [ImapSSLServer-19] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492820, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.

The users reported quota usage is ok but the actual disk space consumed by this user far exceeds their quota.

The Zimbra version is 10.0.12. It was also occurring on 10.0.10.

Is anyone else experiencing this problem?

Thanks,

[Lostaff]

Hello,

An identical problem happened to us too in November 2022 with zimbra 8.8.15.
The occupied disk space in the main store had increased overnight by about 175 GB .

Checking the logs we found that an outlook client via the imap protocol kept writing the exact same message of about 7.5 MB over and over again, it did it about 25000 times. The strange thing is that the user quota of 5GB was not increased at all, and the zimbraPrefMessageIdDedupingEnabled option (automatically deletes duplicate copies of the same message upon receipt) that account has enabled apparently did not work.
These are some lines from the mailbox.log

2022-11-22 23:59:46,415 INFO [ImapServer-1307] [name=xxxxxx@istat.it;mid=1098;ip=xxxxxx;oip=xxxxxx;via=xxxxxx(nginx/1.20.0);ua=Microsoft Outlook/16.0.5257.1000;cid=4013125;] mailop - Adding Message: id=868455, Message-ID=<bmgDH.0000@8a8025be-6a7b-11ed-8000-9c934ef39eca>, parentId=858294, folderId=2, folderName=Inbox acct=c94d016f-db9d-44cc-b5aa-5f834c1b5f2e.
2022-11-22 23:59:48,765 INFO [ImapServer-1312] [name=xxxxxx@istat.it;mid=1098;ip=xxxxxx;oip=xxxxxx;via=xxxxxx(nginx/1.20.0);ua=Microsoft Outlook/16.0.5257.1000;cid=4013125;] mailop - Adding Message: id=868456, Message-ID=<bmgDH.0000@8a8025be-6a7b-11ed-8000-9c934ef39eca>, parentId=858294, folderId=2, folderName=Inbox acct=c94d016f-db9d-44cc-b5aa-5f834c1b5f2e.
2022-11-22 23:59:51,111 INFO [ImapServer-1307] [name=xxxxxx@istat.it;mid=1098;ip=xxxxxx;oip=xxxxxx;via=xxxxxx(nginx/1.20.0);ua=Microsoft Outlook/16.0.5257.1000;cid=4013125;] mailop - Adding Message: id=868457, Message-ID=<bmgDH.0000@8a8025be-6a7b-11ed-8000-9c934ef39eca>, parentId=858294, folderId=2, folderName=Inbox acct=c94d016f-db9d-44cc-b5aa-5f834c1b5f2e.
2022-11-22 23:59:53,455 INFO [ImapServer-1312] [name=xxxxxx@istat.it;mid=1098;ip=xxxxxx;oip=xxxxxx;via=xxxxxx(nginx/1.20.0);ua=Microsoft Outlook/16.0.5257.1000;cid=4013125;] mailop - Adding Message: id=868458, Message-ID=<bmgDH.0000@8a8025be-6a7b-11ed-8000-9c934ef39eca>, parentId=858294, folderId=2, folderName=Inbox acct=c94d016f-db9d-44cc-b5aa-5f834c1b5f2e.


Fortunately the problem has not recurred.

Since that day we have put in a check that alerts us to any excessive message writing.


Greetings

[Matt]

Thanks for the reply. That's very useful to know. It's a shame that this bug has been known about and unfixed for so long (unless it's a regression).

Can you remember if you did anything to 'fix' the users email client? Presumably asking them to use ZCO instead of IMAP. Once they've stopped uploading data, I'll need to look into cleaning up the duplicates.

Thanks.

[Lostaff]

Hi Matt,

we certainly contacted the user and told him to delete the message that had created the problem, the user was not aware of the problem as the file was not being indexed by zimbra, it was being written directly to the file system, then perhaps we restarted the mailboxd.

To solve the duplicate file problem i don't remember exactly what we did, most likely we did a restore of the account, then renamed the old one and assigned the email address to the restored one, then deleted the account with the duplicate file problem.

Now the user is still using outlook 16 with imap and we haven't had the problem anymore, maybe we had him upgrade version of outlook, now he has version Outlook/16.0.5461.1001.

In addition to the ZCO connector you can also use activesync instead of imap.

Greetings

[adrian.gibanel.btactic]

If I then search for the top entry in this list, I see it's the same user uploading it over and over (I had to redact the username and IP below):

Code: Select all

$ grep '01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com' /opt/zimbra/log/mailbox.log
2025-01-07 12:33:21,112 INFO  [ImapSSLServer-31] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492816, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:22,099 INFO  [ImapSSLServer-2] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492817, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:23,740 INFO  [ImapSSLServer-49] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492818, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:25,378 INFO  [ImapSSLServer-18] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492819, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.
2025-01-07 12:33:27,095 INFO  [ImapSSLServer-19] [name=<snip>(nginx/1.24.0);ua=Microsoft Outlook/16.0.16026.20146;cid=1074452;] mailop - Adding Message: id=4492820, Message-ID=<01020193b595c050-365e541e-8bde-000000@eu-west-1.amazonses.com>, parentId=4472817, folderId=2, folderName=Inbox acct=11951a74-db8f-4384-9970-6b14981eb952.

The users reported quota usage is ok but the actual disk space consumed by this user far exceeds their quota.

I wonder if this is a Enormous IMAP traffic from Outlook clients to ZSC server duplicate.

Do you have EXPUNGE commands net to the APPEND ones in the mailbox.log file?
If that's the case what you are actually experiencing is the same email being created and deleted again and again.

Regarding the quota what actually happens is that if you have dumpster enabled for the account or its associated CoS this translates onto the deleted emails consuming Dumpster quota.
'Dumpster quota' doesn't exist per se in size but in maximum of days saved if I'm not mistaken.

[Matt]

That is extremely helpful, thank you. Yes, it looks like this is a duplicate of that problem:

Code: Select all

$ grep "$mailbox" /opt/zimbra/log/mailbox.log | grep "imap - APPEND" | wc -l
1233
$ grep "$mailbox" /opt/zimbra/log/mailbox.log | grep "imap - UID EXPUNGE" | wc -l
1232
$ grep "$mailbox" /opt/zimbra/log/mailbox.log | grep "mailop - Deleting Message" | wc -l
1233
$ zgrep "$mailbox" /opt/zimbra/log/mailbox.log.2025-01-09.gz | grep "imap - APPEND" | wc -l
36372
$ zgrep "$mailbox" /opt/zimbra/log/mailbox.log.2025-01-09.gz | grep "imap - UID EXPUNGE" | wc -l
36353
$ zgrep "$mailbox" /opt/zimbra/log/mailbox.log.2025-01-09.gz | grep "mailop - Deleting Message" | wc -l
36348

Checking the dumpster count:

Code: Select all

$ mailboxid=`/opt/zimbra/bin/zmprov gmi $mailbox | grep "mailboxId" | awk '{print $2}'`
$ echo "$mailboxid"
804
$ expr 804 % 100
4
$ mysql mboxgroup4
MariaDB [mboxgroup4]> select count(*) from mail_item_dumpster where mailbox_id=804;
+----------+
| count(*) |
+----------+
|  2842712 |
+----------+
1 row in set (2.90 sec)

It doesn't look like the dumpster purge is working for this user. I'll dig into that.

This user is still uploading data which I still need to resolve. I've been attempting to fix another affected user as their mailbox is smaller... I performed a zmmboxmove to a different server and, so far, the problem for them has stopped. It's only been ~16 hours though so I need to wait longer and see. I'll then try moving this larger mailbox. I'm hoping this is a workaround whilst we wait for a fix from Zimbra.

EDIT: Moving the mailbox did not help. The problem occurred again for this user today.

[ghen]

Hi

We are having the same issue but caused by Apple mailclients (ua=com.apple.email.maild/...).
They keep copying the same message to Trash, which causes their mailbox to go overquota by orders of magnitude.

It turns out that as of Zimbra 10.0, the Trash folder is excluded from quota on purpose:
https://github.com/Zimbra/zm-mailbox/pull/1276
https://github.com/Zimbra/zm-mailbox/co ... 32bcd15c7a
Zimbra is reconsidering that logic due to multiple customers reporting this ZBUG-4613.

[ghen]

I'm evaluating the same approach as discussed in viewtopic.php?p=317312, patching zimbrastore.jar with rebuilt versions of Mailbox.class and MailItem.class with ZBUG-1375 reverted. It works in our lab tests, I'm going to test it in production next week.

[adrian.gibanel.btactic]

It turns out that as of Zimbra 10.0, the Trash folder is excluded from quota on purpose:
https://github.com/Zimbra/zm-mailbox/pull/1276
https://github.com/Zimbra/zm-mailbox/co ... 32bcd15c7a
Zimbra is reconsidering that logic due to multiple customers reporting this ZBUG-4613.

If you are to trust public Zimbra Github repos this might get fixed in ZCS 10.1.17 thanks to a newer zimbraMailQuotaSoftLimitPercent which would be a Soft quota limit percentage applied over zimbraMailQuota set to 110 by default.

What I'm concerned is if this is going to have as a unintended consequence that given a 1 GB quota you would be able to have 1 GB and then an additional 1100 MB occupied quota for the Trash.

Maybe that's not a problem because of Trash messages being actually deleted after some days of being there.

[ghen]

That was exactly my feedback as well in that PR. But the percentage is configurable.

The rationale for the "need" for this behaviour is in ZBUG-1375. We never had that problem, but we now see customers with huge Trash folders because there's currently no limit there.