Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Ask questions about your setup or get help installing ZCS server (ZD section below).
yogesh.dasi
Zimbra Employee
Zimbra Employee
Posts: 54
Joined: Tue Oct 18, 2022 11:22 am

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by yogesh.dasi »

gmelis wrote:The /tmp directory was and is writeable. It was not a permissions problem, it was more like not finding values for the variables defined in the XML files.
Right its the variables defined in the XML are not getting updated but that could happen if there are no write permission for zimbra user to /tmp folder.
While updating the variables in the XML there are some rights required to create some temp configs in /tmp folder for which write rights are required by zimbra user.

Check getfacl /tmp

Also please check if there are any errors seen while zmcontrol restart in /op/zimbra/zmmailboxd.out
gmelis
Posts: 6
Joined: Sun Mar 03, 2019 6:46 pm

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by gmelis »

Ok, so is zimbra-ldap-patch a prerequisite or not? Could the problem have been caused by its absence?
stasouv
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 2:25 am
ZCS/ZD Version: 8.8.15_GA_3869.RHEL7_64_20190917004

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by stasouv »

We still have not managed to get the zimbraAdmin to function.

We have tried changing zimbraAdminPort and zimbraAdminURL, to no avail, as yet.

Any suggestion would be nice, using the zmprov commandline is quite tedious, when having to deal with circa 3000 users.

Also, a script to reinstall could be handy, should there actually have been an incomplete installation.
gmelis
Posts: 6
Joined: Sun Mar 03, 2019 6:46 pm

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by gmelis »

yogesh.dasi wrote: Right its the variables defined in the XML are not getting updated but that could happen if there are no write permission for zimbra user to /tmp folder.
While updating the variables in the XML there are some rights required to create some temp configs in /tmp folder for which write rights are required by zimbra user.

Check getfacl /tmp

Also please check if there are any errors seen while zmcontrol restart in /op/zimbra/zmmailboxd.out
There's no acl on /tmp, security policies or anything. It's just a standard drwxrwxrwt. In fact zimbra uses /tmp a lot during normal operatoin, creating a lot of tmpXXXXXX files. There's absolutely nothing that would prohibit the usage of /tmp to zimbra. Just to be on the safe side, I checked the past 4 audit.log files for any references of "tmp", and there were none relevant. I strongly believe we can safely put the permissions matter to rest.

Attached are yesterday's and today's zmmailboxd.out files.
Attachments
zmmailboxd.out.202210200000.gz
(92.61 KiB) Downloaded 107 times
zmmailboxd.out.202210190000b.gz
(956.24 KiB) Downloaded 265 times
zmmailboxd.out.202210190000a.gz
(935 KiB) Downloaded 100 times
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by L. Mark Stone »

gmelis wrote:Ok, so is zimbra-ldap-patch a prerequisite or not? Could the problem have been caused by its absence?
I am finding that I need to manually install zimbra-ldap-patch, on both newly-built Zimbra systems as well as existing systems being patched. On single-server systems, if zimbra-ldap-patch is not present, my experience has been that Zimbra will in most cases run and restart OK. But on multi-server systems with LDAP-only servers, without zimbra-ldap-patch, zmconfigd won't start.

In conducting certified Zimbra Administrator Training this week, we built several single- and multi-server 8.8.15 and 9.0.0 environments, and this zimbra-ldap-patch package bug (which I previously reported to Zimbra) was still present.

Bottom Lines:

1. After you do a fresh Zimbra instal, and after you run "apt-get update && apt-get dist-upgrade", before you restart Zimbra, also run "apt-get install zimbra-ldap-patch".
2. After you patch a Zimbra server running LDAP, before you restart Zimbra, also run "apt-get install zimbra-ldap-patch".


Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
stasouv
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 2:25 am
ZCS/ZD Version: 8.8.15_GA_3869.RHEL7_64_20190917004

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by stasouv »

Hello, Mark.

This is good to know, but it doesn't seem to relate to our dis-functionality.

Our problem is, and remains, though we managed to drive around it, that the Jetty configuration files do not pick up important variables, ones that are otherwise accessible with zmprov and zmlocalconfig.

Regards,
Stavros
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by L. Mark Stone »

I just reread your earlier post. The value of 100 for DoSFilter Max Requests per second is too low. I would increase it to at least 350 as the Admin Console after login makes a lot of requests.

Alternatively, I would turn off DoSFilter entirely and use Fail2Ban instead.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
stasouv
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 2:25 am
ZCS/ZD Version: 8.8.15_GA_3869.RHEL7_64_20190917004

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by stasouv »

L. Mark Stone wrote:Alternatively, I would turn off DoSFilter entirely and use Fail2Ban instead.
Wait. What? Can you actually do that???

I've never found anything documented, I thought this was imposed functionality.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by L. Mark Stone »

The DoSFilter has two components: the connection throttling component and the block-an-ip-after-too-many-failed-logins component.

The second component can be turned off entirely:

Code: Select all

zmprov ms `zmhostname` zimbraInvalidLoginFilterMaxFailedLogin 0
For the first component, you can "turn it off" by adding your Zimbra servers (required) and any other desired IP addresses in CIDR format to the zimbraHttpThrottleSafeIPs global attribute. I think you know what to do here if you wish... ;-)

But, you could also just increase zimbraHttpDosFilterMaxRequestsPerSec to a really big number. There is apparently no maximum for this attribute:

Code: Select all

zmprov desc -a zimbraHttpDosFilterMaxRequestsPerSec
Regardless, for most single- and multi-server Zimbra servers, here's what I do:

Code: Select all

zmprov mcf zimbraHttpDosFilterDelayMillis 20
zmprov mcf zimbraHttpDosFilterMaxRequestsPerSec 500
zmprov ms `zmhostname` zimbraInvalidLoginFilterMaxFailedLogin 0
zmprov mcf +zimbraHttpThrottleSafeIPs xx.xx.xx.xx/32
zmprov mcf +zimbraHttpThrottleSafeIPs yy.yy.yy.yy/16
Multiple Zimbra servers _should_ be added automatically to zimbraHttpThrottleSafeIPs, but I find that that doesn't always happen for some reason, so I just manually add all the Zimbra servers's IP addresses (and any other networks I don't want to be analyzed by DosFilter connection throttling) to zimbraHttpThrottleSafeIPs.

For some really busy systems, where for example multiple delegated Admins are logged in to the Admin Console at the same time from the same IP, I find I need to increase zimbraHttpDosFilterMaxRequestsPerSec to as much as 1500. I think it's a good idea to have _some_ connection throttling always in place, but you have some flexibility as to how, and how high, you want to set the threshold. And you can always exempt any IP/network you wish from connection throttling entirely too.

Hope that helps,
Mark

P.S. As re the documentation, I agree it could be better. IMHO, the DoSFilter Wiki page tells you the rules of the game, but not how to play the game.
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
stasouv
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 2:25 am
ZCS/ZD Version: 8.8.15_GA_3869.RHEL7_64_20190917004

Re: Zimbra 9: zmmailboxdctl is not starting after applying latest patches(p26)

Post by stasouv »

So, basically, you can't turn it off, as in OFF.

We are already using the throttle safe ips configuration option, plus the values set on the xml posted here are arbitrary, not real.

We do also have the values for large deployments set, to small millis and big PerSec.

I have cursed and cursed for this functionality all in all, as sometimes throttle safe ips DO get blocked (well, not in the most recent versions). Plus, the only thing blocked is the web interface (pop/imap are still available).

We can't use Fail2Ban, we are behind a proxy (for web), so we do the best next thing, use the data collected to feed an IPS.

Thanks for the tips, though.
Post Reply