Hi,
A little update on issues I faced in several upgrades to 8.8.12 in some different installations. It's been a looong week, somedays working from 7:30am until 4:00am. So here it goes, I hope it helps someone...next week I must face new migrations so who knows, maybe we can make this list a little bit longer I'm afraid:
- Issue 1
Documentation on this link, is still providing a wrong path for ldif files.
https://zimbra.github.io/installguides/ ... grade.html
--> As it turned out, this step as most of that guide is completly outdated and absolutly not necessary. Sadly, even some of the zimbra support staff did not know this. Lucky me, Teodor was there to save the day and clarify things. So, here are other things for which I have confirmation that are not necessary in that guide:
- Upgrading LDAP Replica Servers or Multi-Master Server.
- Disable SSLv3 Support.
- After all MTA nodes are upgraded to Zimbra Collaboration 8.8.12, the following commands may be run to fix the default globalconfig values, if necessary.
- You should run zmldapupgrade -b 66387 after upgrading.
- Issue 2
Even when when zimbra talk was previously installed in server and migration method was an in-place update, instead of asking to "update" zimbra talk, it asks for "install zimbra talk".
--> This is "normal" behaviour. As per Zimbra support "zimlets are reinstalled, together with the talk zimlet. The installer reinstalls them." Yet...zimbra is the only one zimlet asking to "be installed". No prob...say "Y" and keep moving with the install wizard
- Issue 3
Done on a RHEL7.x, I got the warning about not having "libreoffice-headless".
Checking required packages for zimbra-store
FOUND: libreoffice-5.3.6.1-19
MISSING: libreoffice-headless
###WARNING###
yum install libreoffice-headless
Loaded plugins: product-id, search-disabled-repos, subscription-manager
zimbra | 2.9 kB 00:00:00
zimbra-8812-oss | 2.9 kB 00:00:00
Package 1:libreoffice-headless-4.3.7.2-5.el7_2.1.x86_64 is obsoleted by 1:libreoffice-core-5.3.6.1-19.el7.x86_64 which is already installed
Nothing to do
--> According to the technician from Zimbra support, he thinks it may be only a problelm with the installer not realizing the package is already installed in a newer version. He is trying to confirm this information internally.
- Issue 4
After update was done on mailstores, even when services were up and running, in /opt/zimbra/log/mailbox.log I was able to see these errors. Really big stack, this is just a portion:
2019-04-02 00:09:24,104 INFO [main] [] misc - System property sun.os.patch.level=unknown
2019-04-02 00:09:24,138 INFO [main] [] system - Setting mysql connector property: maxActive=100
2019-04-02 00:09:24,138 INFO [main] [] system - Setting mysql connector property: maxActive=100
2019-04-02 00:09:24,224 FATAL [main] [] soap - Unable to start servlet
java.lang.NoClassDefFoundError: java/sql/Connection
at com.compuware.apm.agent.introspection.jdbc.ConnectionPoolIntrospection.updateConnectionClass(ConnectionPoolIntrospection.java:214)
at org.mariadb.jdbc.MySQLConnection.createStatement(MySQLConnection.java)
at org.mariadb.jdbc.MySQLConnection.newConnection(MySQLConnection.java:128)
at org.mariadb.jdbc.Driver.connect(Driver.java:113)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
at com.zimbra.cs.db.ZimbraConnectionFactory.createConnection(ZimbraConnectionFactory.java:67)
Visible problem was not the log itself, but having a "blank page" in admiUI and having a "network encountered error" when trying to access webUI via proxies...in other woards, mail system down as per user perspective.
A support ticket is opened (actually 2 support tickets are opened) about this issue. We are nos suspecting that the database takes too long to start and that's may be the reason why the servlet fails to start when connecting to the database. I'll keep you posted
--> This was caused by something very specific to one of my customers. The problem in this case was with "dynatrace" agent, which was slowing down the db start, thus 'causing the servlet to fail the connection to it during strat time. So even when all services were showing as running, functionality was broken. Once disabled this agent from dynatrace, problem was solved.
- Issue 5
I was not able to find any documentation about how to upgrade zimbra docs. I understand that this should be easy and I feel is almost intuitive, but I would like to see some docs about it instead of again, having to write a piece of doc by try and error in my lab.
--> You can' find documentation if there is no documentation (insert roll safe think about it meme here). So there, there is no documentation on how the upgrade process should be. Mark guessing is absolutly right and as per Zimbra support, we should: 1. Install the new zDocs server in a new machine. 2. Remove the previous version (zdocs remove-local-server). The instructions for point 1. are as follow:
Prerequisites for zimbra Docs
In case of multi-server, all server should be resolvable with each other.
Mailbox servers should be reachable to the Docs server & vice versa.
Proxy servers should be reachable to the Docs server & vice versa.
Check host files of each server.
The zimbraPublicServiceHostname, zimbraPublicServiceProtocol, zimbraPublicServicePort should be configured on a domain so that it can proxy all request to the corresponding mailbox server.
All mailbox servers will need to be able to directly communicate with the Docs Server over port 8443 (HTTPS Backend), which must be open on both ends.
The Supported OS (Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, Red Hat Enterprise Linux 7, CentOS 7) for the Docs server as of now.
Installation steps for the Docs server
At this stage, we assume that all above prerequisites are done.
Login into the Docs server.
Download the zimbra-docs-OSv.tgz standalone installer, extract it and as the root user execute the install.sh script contained in the package.
https://www.zimbra.com/downloads/zimbra-collaboration/
We need to provide the ldap url, zimbra DN and zimbra ldap password.
e.g:
LDAP url (ldap://proxy.abc.com:389): ldap://proxy.abc.com:389
LDAP dn (uid=zimbra,cn=admins,cn=zimbra): uid=zimbra,cn=admins,cn=zimbra
LDAP password (fadpXcun5): fadpXcun5
Trying to connect to LDAP...OK
Configuration written
................++\+
..............................................................................................++\+
Generated key in /opt/zimbra/conf/docs/key.pem
Updating server 590fae5c-2cf3-4e1f-8537-df7989216945 with hostname 'docs.abc.com' in LDAP
Changes applied
No changed in the configuration
systemctl restart zimbra-docs-server
Setup successfully completed.
If we get the above message, then docs is installed successfully.
To check if the installation was successful,
We'll verify the ldap connectivity from the docs server
[]# zdocs ldap-test
Trying to connect to LDAP...OK
To restart a docs server if needed:
zdocs zdocs restart
Regenerate the config and restart the server if config changed.
zdocs --auto-restart generate-config
Then we continue the installation
Run the following on all proxy servers
We need to re-generate to proxy server configuration - just restart the services.
su - zimbra
zmproxyctl restart
To verify: Run the following command:
grep -i server /opt/zimbra/conf/nginx/includes/nginx.conf.docs.upstream
The expected output is:
[]$ grep -i server /opt/zimbra/conf/nginx/includes/nginx.conf.docs.upstream
server docs.abc.com:9980;
Run the following on the all mailbox servers
Now the com_zextras_docs Zimlet needs to be deployed on the server and enable on all users and COS that need to have access to the Zimbra Docs features.
RHEL/CentOS
yum install zimbra-docs
Ubuntu
apt-get install zimbra-docs
su - zimbra
zmmailboxdctl restart
Troubleshooting
Check the zimlet (com_zextras_docs) is installed and enabled on all mailbox nodes.
Check the docs server status from Docs server and from mailbox server as well.
zdocs status # From Docs Server
zxsuite docs status # From mailbox nodes
Check the network connection between the servers.
You can check the error by browser debugging (?dev=1)
We see below logs which indicate that docs server is working fine.
- Issue 6
Latetly, a new "localhost" is showing under supervision in adminUI. I'll fix that today tho...easy thing to fix, although I don't understand why is there.
I did not know it could dissapear by itself when the conditions you mention are met! In any case, following this wiki fixed the problem right away
https://wiki.zimbra.com/wiki/Server_sta ... rossed_out
- Issue 7
IMAP clients can't show/sync non-english characters like "ñ" or "á"
We are able to see them in the webUI, but nor Outlooks, thunderbirds or others (like the mac client) can properly see or even create such folders anymore. All of these custemers sync via IMAP...which I suspect has something to do with this "imap deserialze" bug being fix". I'll open a support ticket (once the others are solved) to ask about this
--> Bug. This is Zimbra support response...no comments from my side on their answer (of course there is no way for us to follow the bug state as bug tracking tool is now private):
Thank you for contacting Zimbra Support.
After a close investigation, we have identified that your case has resulted into a bug and we have reported the bug to Development using our new bug process.
For more information about this process, please see the [
https://blog.zimbra.com/2018/05/zimbras ... g-process/ | Zimbra New Bug Fix / Reporting Process ] wiki. We will be keeping this case open and the support team will be sending you updates on the status of the bug.
If you have any questions, please let us know by replying to the case and support will followup with you.
We appreciate your commitment to Zimbra.
- Issue 8
- zextras was not enabled even when it was enabled during install
In this case, if we "zmprov gacf zimbraNetworkModulesNGEnabled", the value was set to TRUE, yet, at a server level (zmprov gs), the value was set to "FALSE". The sympthom was the lack of the zimlet in the adminUI, and an error saying no server available when doing a "zxsuite core getservices".
--> The fix was to simply modify the value to TRUE at a server level and restart.
- Issue 9
- Legacy backups schedule is restored to default, regardless it was dissabled before it.
--> This is a long time error, that I must assume that provoked more than one problem to unaware admins.
viewtopic.php?t=65096
Fix is "easy", basically you need to fulsh the leagacy backups (zmschedulebackup -F) and also set the deleteonrollover for redologs to "TRUE" as there is not going to be more legacy backups to remove them for you. ("zmprov mcf zimbraRedoLogDeleteOnRollover TRUE" for the global config and then double check it at a server level or just do it one server by one with "zmprov ms `zmhostname` zimbraRedoLogDeleteOnRollover TRUE")
- Issue 10
- Zextras (backup_ng) backups fail when trying to backup ldap files.
--> I remember phoenix recenlty explaining someone in this forum how to fix it but sorry, I can't find the post and I'll go nuts if I keep waiting for this forum pages to load. So basicall, permissions on those files need to be fixed. This is done by executing this as root user:
chmod 0644 /opt/zimbra/common/etc/openldap/slpad.*
- Issue 11
- The stats service was not able to start.
--> Fixed by removing the .pid file which had wrong permissions/owner. In this particular case was what I did was to move away this file "/opt/zimbra/zmstat/pid/zmstat-fd-real.pid" and restart the service via "zmstatsctl restart"
- Issue 12
- Some specific android devices, see the mails with date of 1/1/1970. I did not see this one by myself, but I know my customer opened a support ticket about it. I'll try to find out the details about which exact android model is having the problem also.