lmtp - Not delivering message with duplicate Message-ID

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
mps
Posts: 7
Joined: Fri Sep 12, 2014 10:27 pm

lmtp - Not delivering message with duplicate Message-ID

Post by mps »

Hello,
I'm taking a look to the mailbox.log and I'm seeing this message:

"lmtp - Not delivering message with duplicate Message-ID"
The incoming message is for multiple recipients.
The users are not receiving this message? Or is there some internal control for this in Zimbra?
Version of Zimbra: 5.0.4_GA_2101.DEBIAN4.0
Thanks in advance.
mmorse
Ambassador
Ambassador
Posts: 6036
Joined: Fri Sep 12, 2014 10:24 pm

lmtp - Not delivering message with duplicate Message-ID

Post by mmorse »

When LMTP receives the same message for the same user twice in a short period of time, it drops the second copy - this all takes place in memory. The server will keep that many recipient/Message-ID pairs in a FIFO map (first in, first out) dropping any duplicates it sees. Though I don't think you need to, it's possible to disable this by setting zimbraMessageIdDedupeCacheSize to 0 in the global LDAP config. Useful when you're using zmlmtpinject over and over & want it to actually make more copies. For normal operation default is 3000.
mps
Posts: 7
Joined: Fri Sep 12, 2014 10:27 pm

lmtp - Not delivering message with duplicate Message-ID

Post by mps »

mmorse, thanks for the explanation!
sfilipov
Posts: 2
Joined: Fri Sep 12, 2014 11:36 pm

lmtp - Not delivering message with duplicate Message-ID

Post by sfilipov »

Hi,
I have an issue with messages which are not coming and after some investigation I've found what is the problem.

Fisrt at 13:55 user jatanasova have sent message with subject "info profile diageo" to user mbobeva. Message was received normally.

Then at 14:07 and 14:25 user jatanasova have sent message with subject "Fwd: info profile diageo" to user mbobeva, but messages didn't come at all. Mailbox.log says: "lmtp - Not delivering message with duplicate Message-ID".

Then at 14:40 user jatanasova have sent message with subject "Fwd: info profile diageo" to distribution list in which mbobeva is participating, but again message was stopped with the same error.
Is this normal, and why is this happening - messages are different.
And from where/how can I change zimbraMessageIdDedupeCacheSize to 0. And did this hide some risks?
Thanks in advance.

Bellow is the log:


2008-06-30 13:55:04,849 INFO [LmtpServer-4952] [ip=212.72.217.102;] lmtp - Delivering message: size=25967 bytes, nrcpts=1, sender=jatanasova@manager.bg, msgid=

2008-06-30 13:55:04,883 INFO [LmtpServer-4952] [name=mbobeva@manager.bg;mid=68;] mailop - Adding Message: id=3174, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.
2008-06-30 14:07:05,587 INFO [LmtpServer-4965] [ip=212.72.217.102;] lmtp - Delivering message: size=34200 bytes, nrcpts=1, sender=jatanasova@manager.bg, msgid=

2008-06-30 14:07:05,588 INFO [LmtpServer-4965] [name=mbobeva@manager.bg;mid=68;] lmtp - Not delivering message with duplicate Message-ID
2008-06-30 14:25:21,555 INFO [LmtpServer-4993] [ip=212.72.217.102;] lmtp - Delivering message: size=35543 bytes, nrcpts=1, sender=jatanasova@manager.bg, msgid=

2008-06-30 14:25:21,556 INFO [LmtpServer-4993] [name=mbobeva@manager.bg;mid=68;] lmtp - Not delivering message with duplicate Message-ID
2008-06-30 14:40:45,608 INFO [LmtpServer-5018] [ip=212.72.217.102;] lmtp - Delivering message: size=36629 bytes, nrcpts=9, sender=jatanasova@manager.bg, msgid=

2008-06-30 14:40:45,635 INFO [LmtpServer-5018] [name=dlukanova@manager.bg;mid=69;] mailop - Adding Message: id=2307, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.

2008-06-30 14:40:45,679 INFO [LmtpServer-5018] [name=ggoranov@manager.bg;mid=70;] mailop - Adding Message: id=2077, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.

2008-06-30 14:40:45,692 INFO [LmtpServer-5018] [name=ghristova@manager.bg;mid=71;] mailop - Adding Message: id=2108, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.

2008-06-30 14:40:45,716 INFO [LmtpServer-5018] [name=iivanova@manager.bg;mid=72;] mailop - Adding Message: id=2133, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.

2008-06-30 14:40:45,735 INFO [LmtpServer-5018] [name=jatanasova@manager.bg;mid=66;] mailop - Adding Message: id=3343, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.

2008-06-30 14:40:45,744 INFO [LmtpServer-5018] [name=mbobeva@manager.bg;mid=68;] lmtp - Not delivering message with duplicate Message-ID

2008-06-30 14:40:45,746 INFO [LmtpServer-5018] [name=mmarinova@manager.bg;mid=73;] mailop - Adding Message: id=2153, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.

2008-06-30 14:40:45,876 INFO [LmtpServer-5018] [name=sfilipov@manager.bg;mid=48;] mailop - Adding Message: id=1798, Message-ID=, parentId=-1, folderId=1494, folderName=ADV.

2008-06-30 14:40:45,895 INFO [LmtpServer-5018] [name=sgeorgieva@manager.bg;mid=74;] mailop - Adding Message: id=2206, Message-ID=, parentId=-1, folderId=2, folderName=Inbox.
fowler
Posts: 31
Joined: Sat Sep 13, 2014 12:01 am

lmtp - Not delivering message with duplicate Message-ID

Post by fowler »

[quote user="mmorse"]When LMTP receives the same message for the same user twice in a short period of time, it drops the second copy

...

it's possible to disable this by setting zimbraMessageIdDedupeCacheSize to 0 in the global LDAP config. [/QUOTE]


zmprov gcf zimbraMessageIdDedupeCacheSize

3000

zmprov mcf zimbraMessageIdDedupeCacheSize 0

zmprov gcf zimbraMessageIdDedupeCacheSize

0


NOTE: Even after flushing the config cache (zmprov fc config), the change was not reflected. It appears a restart is necessary for the change to take affect.
See also: Flushing LDAP Cache
liuk
Posts: 18
Joined: Sat Sep 13, 2014 12:22 am

lmtp - Not delivering message with duplicate Message-ID

Post by liuk »

Hello Mike,

we are facing similare issues here with Clients using Outlook 2010. For example it the user forward the same email again changing the attachment the message is discarded for the recipient.

Since from what I can read on RFC 2822 :

the existence of a message id is an optional (SHOULD have) requirement, but its uniqueness if it is supplied is necessary (MUST be)

I suppose that the fault is from the Outlook client.
Now my other questions are: what is the purpose of this caching ? And what can be the side effects of setting it to ZERO ?
Thanks!

Luca
mattg1
Posts: 39
Joined: Sat Sep 13, 2014 12:30 am
ZCS/ZD Version: RHEL7_64 NETWORK, Patch 8.8.15_P41

lmtp - Not delivering message with duplicate Message-ID

Post by mattg1 »

Hi... We just encountered this issue here too. In fact, it has been going on for a long time, but was just noticed now. Users use Outlook templates to send messages via ZCO. The messages from that template always have the same Message-ID. Apparently this is a known Outlook problem for a long time. Awesome. Anyway, I guess changing zimbraMessageIdDedupeCacheSize is a workaround, but may have unintended consequences in the case of mail loops and such. (Perhaps merely reducing the value rather than zeroing it would help the problem, while still providing some loop protection.)
Another option might be for ZCO to remove the Message-ID header (which is optional from the MUA's end) so that the Zimbra server can generate a new unique one. Or, even configure the Zimbra server to ignore client-generated Message-IDs on submitted emails. I don't know if those would violate some RFC, though.
Any other ideas besides tweaking zimbraMessageIdDedupeCacheSize ? (Or getting rid of Outlook?)
jerryipe
Posts: 1
Joined: Sat Sep 13, 2014 3:12 am

lmtp - Not delivering message with duplicate Message-ID

Post by jerryipe »

I have read the above post, i am also facing the same issue mentioned above,

Any idea about zimbraMessageIdDedupeCacheSize, what does the value 3000 means.

Any one please give an idea.
Regards,

Jerry I. George


[quote user="mattg1"]Hi... We just encountered this issue here too. In fact, it has been going on for a long time, but was just noticed now. Users use Outlook templates to send messages via ZCO. The messages from that template always have the same Message-ID. Apparently this is a known Outlook problem for a long time. Awesome. Anyway, I guess changing zimbraMessageIdDedupeCacheSize is a workaround, but may have unintended consequences in the case of mail loops and such. (Perhaps merely reducing the value rather than zeroing it would help the problem, while still providing some loop protection.)
Another option might be for ZCO to remove the Message-ID header (which is optional from the MUA's end) so that the Zimbra server can generate a new unique one. Or, even configure the Zimbra server to ignore client-generated Message-IDs on submitted emails. I don't know if those would violate some RFC, though.
Any other ideas besides tweaking zimbraMessageIdDedupeCacheSize ? (Or getting rid of Outlook?)[/QUOTE]
liuk
Posts: 18
Joined: Sat Sep 13, 2014 12:22 am

lmtp - Not delivering message with duplicate Message-ID

Post by liuk »

In the release notes of 7.1.4:

Global LDAP attrib zimbraMessageIdDedupeCacheTimeout for LMTP de-dupe cache to workaround clients incorrectly using the same ID for different messages. If non-zero, zimbraMessageIdDedupeCacheSize limit is ignored. (44564)
Did you try to first reduce the zimbraMessageIdDedupeCacheSize to a small value ?
If you set it to zero users will receive duplicate e-mails under certain conditions, but there are no other side effects AFAIK.
HTH

Luca
mattg1
Posts: 39
Joined: Sat Sep 13, 2014 12:30 am
ZCS/ZD Version: RHEL7_64 NETWORK, Patch 8.8.15_P41

lmtp - Not delivering message with duplicate Message-ID

Post by mattg1 »

[quote user="jerryipe"]Any idea about zimbraMessageIdDedupeCacheSize, what does the value 3000 means.[/QUOTE]
The way I understand it, it means that Zimbra will remember the last 3000 Message-IDs it sees, and prevent delivery of any "duplicates" of those 3000 messages. It's up to you to figure out how much email your server sees, and adjust that value to estimate how long in seconds this might be. On my server, I had to set it to 0 to allow Outlook users to do what they wanted reliably.
[quote user="liuk"]Global LDAP attrib zimbraMessageIdDedupeCacheTimeout for LMTP de-dupe cache to workaround clients incorrectly using the same ID for different messages. If non-zero, zimbraMessageIdDedupeCacheSize limit is ignored. (44564)[/QUOTE]
Thanks.... I will have to look into zimbraMessageIdDedupeCacheTimeout to see if it might be an alternative for me.
Post Reply