FOSS 8.8.15P20 problem with mail queue monitoring

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Andy
Posts: 17
Joined: Mon Jun 08, 2020 2:09 pm

FOSS 8.8.15P20 problem with mail queue monitoring

Post by Andy »

Hi,

I have FOSS version on CentOS6. After install last patch i can't view mail queue - get error like this:

Code: Select all

system failure: exception executing command: zmqstat all with {RemoteManager: mail.domain.com->zimbra@mail.domain.com:22} Error code: service.FAILURE Method: [unknown] Details:soap:Receiver 
and in /opt/zimbra/log/mailbox like this:

Code: Select all

2021-04-03 14:16:28,123 WARN  [qtp103536485-152:https://mail.domain.com:7071/service/admin/soap/GetMailQueueInfoRequest] [name=user@domain.com;mid=7;ip=a.b.c.d;port=59016;ua=Zimbra
WebClient - FF87 (Win);soapId=7ba85bdb;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: exception executing command: zmqstat all with {RemoteManager: mail.domain.com->zimbra@mail.domain.com:22}
ExceptionId:qtp103536485-152:https://mail.domain.com:7071/service/admin/soap/GetMailQueueInfoRequest:1617452188123:83b127f201cd4dae
Code:service.FAILURE
and

Code: Select all

Caused by: org.apache.sshd.common.SshException: DefaultAuthFuture[ssh-connection]: Failed (IllegalArgumentException) to execute: No signer could be located for key type=ssh-dss
How to fix it?

Thank you in advance for your help
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by JDunphy »

I am also running RHEL6/Centos6 with the April 2 patch 20. This is really strange... It is also failing for my NETWORK version as you are reporting. I thought they were doing something like this:

Code: Select all

# su - zimbra
%  echo "HOST:mail.example.com showqueue" | ssh -i ~/.ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@mail.example.com "/opt/zimbra/libexec/zmrcd"
where zmrcd has a list of commands they support that you can put in place of showqueue such as status, slapcat, etc that we see in the graphical console. When I run it from the command line it works even with the existing ssh-dss keys.
Regenerating the keys only changes it to ssh-rsa for the keys:

Code: Select all

# su - zimbra
% zmsshkeygen
% zmupdateauthkeys
And the command still works as with the previous keys.

Code: Select all

# su - zimbra
%  echo "HOST:mail.example.com showqueue" | ssh -i ~/.ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@mail.example.com "/opt/zimbra/libexec/zmrcd"
But fails from the console.

For NETWORK version customers we also now have our backups failing in addition to the server status and mail queues, etc with patch 20 as you are reporting. I guess I need to turn on some debugging with both sshd and zmrcd to see if they have changed the way they query for this data. The message we have is:

Code: Select all

system: system failure: LDAP backup failed: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/full-20210403.225113.538/ldap --zip with {RemoteManager: mail.example.com->zimbra@mail.example.com:22}
com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/full-20210403.225113.538/ldap --zip with {RemoteManager: mail.example.com->zimbra@mail.example.com:22}
ExceptionId:FullBackupThread:1617490298496:f66adc45c22e7db5
Code:service.FAILURE
...
Caused by: org.apache.sshd.common.SshException: DefaultAuthFuture[ssh-connection]: Failed (IllegalArgumentException) to execute: No signer could be located for key type=ssh-dss
...
Caused by: java.lang.IllegalArgumentException: No signer could be located for key type=ssh-dss
I have also opened a bug report on this with them.

Ref: https://wiki.zimbra.com/wiki/Mail_Queue_Monitoring
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by Klug »

Is only the LDAP part of backup failing (once a day) or the whole backup (continuous)?
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by JDunphy »

Klug wrote:Is only the LDAP part of backup failing (once a day) or the whole backup (continuous)?
Yes just the ldap part once per day and only with the communication to kick that ldap part off it would appear. I have something like this followed by a mv to push it back into place after if fails until I get around to digging in and finding a better resolution.

Code: Select all

# su - zimbra
% /opt/zimbra/libexec/zmbackupldap --outdir /tmp/ldaptest --zip
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by Klug »

I've upgraded some servers yesterday and I have the same issue.

I use zmslapcat (twice, once alone and once with "-c" ) rather than zmbackupldap, because it creates a dated file.
It also create a non-dated file per backup and as it is able to erase/replace it with the new/current one, I find thie easier to deal with.

Code: Select all

/opt/zimbra/libexec/zmslapcat /home/zimbra/ldap	; /opt/zimbra/libexec/zmslapcat -c /home/zimbra/ldap
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: FOSS 8.8.15P20 problem with mail queue status and legacy backups

Post by JDunphy »

I have a working solution that fixes this problem but I need to make this easier or find a better way. The root cause is that zimbra has embedded a ssh client inside their software which doesn't appear to be compatible with the provided sshd server after patch20 we run on RHEL6/CENTOS6 or at least in the standard configuration. That is why running the standard ssh client from the shell worked but from within their java code would fail. What I did was put sshd into debug mode and saw this:

Code: Select all

broken-relay2:# /usr/sbin/sshd -d
debug1: sshd version OpenSSH_5.3p1
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from X.X.X.X port 55874
debug1: Client protocol version 2.0; client software version APACHE-SSHD-2.6.0
debug1: no match: APACHE-SSHD-2.6.0
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: permanently_set_uid: 74/74
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-ctr hmac-sha2-256 none
debug1: kex: server->client aes128-ctr hmac-sha2-256 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user zimbra service ssh-connection method none
debug1: attempt 0 failures 0
debug1: PAM: initializing for "zimbra"
debug1: PAM: setting PAM_RHOST to "mail.example.com"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user zimbra service ssh-connection method publickey
debug1: attempt 1 failures 0
userauth_pubkey: unsupported public key algorithm: rsa-sha2-512
Connection closed by X.X.X.X
debug1: do_cleanup
debug1: do_cleanup
debug1: PAM: cleanup
Next, I grabbed a newer openssh (not newest but something like openss-7.4p1) and did:

Code: Select all

% wget https://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.4p1.tar.gz
% tar zxvf openssh-7.4p1.tar.gz
% cd openssh/openssh-7.4p1
% configure
% make
% make install
This will install in /usr/local where /usr/local/sbin/sshd and /usr/local/etc/sshd_config are the important files, etc.
But it doesn't end here... It complained the zimbra user is locked which it is. They look for ! so I replaced ! with * via:

Code: Select all

# vipw -s
# /usr/local/sbin/sshd 
This appears to fix both the mail queue issue and the backups. I misspoke about the server status as that is my weird issue with rsyslog because I have some oddities in there to change the format for a central logging server and I didn't realize they had updated this during the patch so I needed to put the zimbra entries above my changes.

I need to rethink this but the good news from my perspective at least is that it is a simple problem to fix and not fundamentally bad patch. My support ticket can be worked on now that it's business hours so I'll see what they recommend.

Jim
Last edited by JDunphy on Fri Apr 09, 2021 2:57 pm, edited 2 times in total.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by JDunphy »

An update. I didn't find a simpler solution and Zimbra is researching and investigating. If your backups are not working and you can't see your mail queues via the gui console, please reference ZBUG-2191 if you open a support ticket.

It's an incompatibility with the built-in ssh client in mailboxd and the RHEL6/Centos6 sshd daemon.

Here is the output from the failed converstation with the default sshd if anyone has any ideas:

Code: Select all

debug1: Client protocol version 2.0; client software version APACHE-SSHD-2.6.0
debug1: kex: client->server aes128-ctr hmac-sha2-256 none
debug1: kex: server->client aes128-ctr hmac-sha2-256 none
root cause: userauth_pubkey: unsupported public key algorithm: rsa-sha2-512
userauth_pubkey: unsupported public key algorithm: rsa-sha2-512
followed by:
Connection closed by X.X.X.X
This is the default RHEL6 sshd -T output:

Code: Select all

ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
macs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

kexalgorithms diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
and the working compiled sshd (7.4p1)

Code: Select all


ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
macs umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

kexalgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1

hostbasedacceptedkeytypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
I was hoping to find a way to downgrade this protocol exchange but mailboxd is pretty much a mystery to me how it works and what capabilities it has for unit testing if any. I am fine with updating my sshd but was hoping to find a simpler solution for those in the community that might not work for.

Jim
Andy
Posts: 17
Joined: Mon Jun 08, 2020 2:09 pm

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by Andy »

Hi again,

Can you count on support in this topic? Or is the only solution given by JDunphy?
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by JDunphy »

Providing a tar ball for those that can't compile it themselves and need a solution now. I realize this is difficult when EOL and repositories are all down for CentOS release 6.10 (Final). I think it should work as I used the standard libs for linking with RHEL/centos 6.10.

There are 3 steps to this. You will need to convert from ssh-dss to ssh-rsa keys which is done like this:

Code: Select all

# su - zimbra
% zmsshkeygen
% zmupdateauthkeys
Next you need to replace the '!!' for the zimbra user that is locked with something else.. '**' will do and it will still be locked. The newer sshd doesn't want to work with locked accounts so this is a quick workaround.

Code: Select all

% su -
# vipw -s
Lastly you will pull and extract a self-contained and hopefully working sshd in /usr/local. It is done like this:

Code: Select all

% wget --no-check-certificate "https://onedrive.live.com/download?cid=42DE3C953BCD8071&resid=42DE3C953BCD8071%2127694&authkey=AE4sIqmp3v7ChRI" -O /tmp/sshd-7.4p1.install-usr-local.tar
% su -
# cd /usr/local
# tar xvf /tmp/sshd-7.4p1.install-usr-local.tar
# /etc/init.d/sshd stop
# /usr/local/sbin/sshd
If it works out, add /usr/local/sbin/sshd to /etc/rc.local and disable the default sshd via chkconfig to make it survive after reboots.

Note: ... stopping sshd will not kill your sshd session you are running when you execute /etc/init.d/sshd stop... and provided you have changed directory to /usr/local before extracting the tar image will not modify your existing sshd installation with RHEL6/CENTOS6

Jim
Andy
Posts: 17
Joined: Mon Jun 08, 2020 2:09 pm

Re: FOSS 8.8.15P20 problem with mail queue monitoring

Post by Andy »

JDunphy wrote:Providing a tar ball for those that can't compile it themselves and need a solution now. I realize this is difficult when EOL and repositories are all down for CentOS release 6.10 (Final). I think it should work as I used the standard libs for linking with RHEL/centos 6.10.
.....
Jim
Yeah I known but zimbra version on rhel6 is still supported and why don't they do anything about it with this...
I will be migrating to another system (Ubuntu) anyway, but I thought they would release some mini patch for this bug.
Andy
Last edited by Andy on Tue Apr 13, 2021 6:49 am, edited 2 times in total.
Post Reply