Continous problems with fresh multi-server install

[pup_seba]

Hi,

In a new environment I'm unable to install zimbra 8.8.12 and having working properly.

The environment has 2 sites, with 1 store + 1 mta/proxy each (4 serves in total). Site 1, has the ldap master along the store. Site 2 has a ldap replica along its store server. Logger is deployed on store 1.

Problems:
- I can login to the adminUI of the store in site one. But I can't login to the adminUI of the store in site 2 as it gets stuck on "loading screen". On logs, I see this:

2019-08-07 12:39:24,974 INFO [main] [] misc - Thread pool was configured to max=250
2019-08-07 12:39:25,996 INFO [ Activity Thread] [] extensions - Zimbra docs read an empty configuration
2019-08-07 12:39:29,681 INFO [qtp1258084361-19:https:https://10.0.10.8:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider
2019-08-07 12:39:29,682 INFO [qtp1258084361-19:https:https://10.0.10.8:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: sampleoauth com.zimbra.cs.service.ZimbraAuthProviderForOAuth
2019-08-07 12:39:29,724 INFO [qtp1258084361-19:https:https://10.0.10.8:7071/service/admin/soap/AuthRequest] [name=admin@algo.es;ip=10.0.0.213;port=50025;ua=ZimbraWebClient - GC75 (Win);soapId=3086576a;] extensions - Using two-factor auth factory ZimbraTwoFactorAuth
2019-08-07 12:39:29,947 INFO [qtp1258084361-19:https:https://10.0.10.8:7071/service/admin/soap/AuthRequest] [name=admin@algo.es;ip=10.0.0.213;port=50025;ua=ZimbraWebClient - GC75 (Win);soapId=3086576a;] soap - AuthRequest elapsed=245
2019-08-07 12:39:30,007 INFO [qtp1258084361-21:https:https://10.0.10.8:7071/service/admin/soap/GetInfoRequest] [name=admin@algo.es;ip=10.0.0.213;port=50025;ua=ZimbraWebClient - GC75 (Win);soapId=3086576b;] soap - Proxying request: requestedAccountId=9a3f0944-4c47-4dda-9138-8c27100f1b2b authAcct name=admin@algo.es id=9a3f0944-4c47-4dda-9138-8c27100f1b2b reason: onLocalSvr=false isLocal=false target=correo_mbox01.algo.es localhost=email_mbox02.algo.es account=admin@algo.es
2019-08-07 12:39:30,069 INFO [qtp1258084361-21:https:https://10.0.10.8:7071/service/admin/soap/GetInfoRequest] [name=admin@algo.es;ip=10.0.0.213;port=50025;ua=ZimbraWebClient - GC75 (Win);soapId=3086576b;] HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused)
2019-08-07 12:39:30,069 INFO [qtp1258084361-21:https:https://10.0.10.8:7071/service/admin/soap/GetInfoRequest] [name=admin@algo.es;ip=10.0.0.213;port=50025;ua=ZimbraWebClient - GC75 (Win);soapId=3086576b;] HttpMethodDirector - Retrying request
2019-08-07 12:39:30,094 WARN [qtp1258084361-21:https:https://10.0.10.8:7071/service/admin/soap/GetInfoRequest] [name=admin@algo.es;ip=10.0.0.213;port=50025;ua=ZimbraWebClient - GC75 (Win);soapId=3086576b;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: operation sent to wrong host (you want 'correo_mbox01.algo.es')
ExceptionId:qtp1258084361-21:https:https://10.0.10.8:7071/service/admin/soap/GetInfoRequest:1565174370093:2cdb3d47894236c3

- If I try to access to store in site 2, from the admin console on the store in site one (configuration, servers, mbox02), I get another error on the screen:
pop up with this message and no further details: "JavaScript error encountered in method ZaOverviewPanelController.prototype._overviewTreeListener"

- I can't run "zmmailbox" to any email from any of the 2 stores. I always get this error (no matter to which account, no matter from which store):
[zimbra@correo_mbox01 log]$ zmmailbox -z -m admin@algo.es
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused (Connection refused)
[] INFO: Retrying request
ERROR: remote.CONNECT_FAILURE (https://correo_mbox01.algo.es:7071/service/admin/soap/) (cause: java.net.ConnectException Connection refused (Connection refused))

- Even more, even with these errors, I've imported information from a backup_ng from an older server (the ones we are migrating from) and my logs are full with messages like these:
2019-08-07 12:49:18,760 WARN [Index-6] [name=c.benavides@algo.es;mid=85;] ParsedMessage - Unable to parse part=2 filename=Esquema CIP La Polesa.pdf content-type=application/pdf message-id=<9636C319D2F5A54CA6DB3570C5CBB2035CBD10@mailserver.intranet.inoxpa.com>
com.zimbra.cs.mime.MimeHandlerException: extraction failed
at com.zimbra.cs.mime.handler.ConverterHandler.getContentImpl(ConverterHandler.java:119)
at com.zimbra.cs.mime.MimeHandler.getContent(MimeHandler.java:180)
at com.zimbra.cs.mime.ParsedMessage.analyzePart(ParsedMessage.java:1152)
at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:442)
at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:473)
at com.zimbra.cs.mailbox.Message.generateIndexData(Message.java:1405)
at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:774)
at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:411)
at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:85)
at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1434)
at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1413)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.zimbra.cs.convert.ConversionException: connect failed
... 16 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at com.zimbra.common.net.ProtocolSocketFactoryWrapper.createSocket(ProtocolSocketFactoryWrapper.java:48)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at com.zimbra.cs.convert.AbstractConverterClient.call(AbstractConverterClient.java:117)
at com.zimbra.cs.convert.AbstractConverterClient.putOrPost(AbstractConverterClient.java:165)
at com.zimbra.cs.convert.AbstractConverterClient.post(AbstractConverterClient.java:149)
at com.zimbra.cs.convert.LegacyConverterClient.extract(LegacyConverterClient.java:38)
at com.zimbra.cs.convert.PooledConverterClient.extract(PooledConverterClient.java:42)
at com.zimbra.cs.mime.handler.ConverterHandler.getContentImpl(ConverterHandler.java:109)
... 15 more

- That which seems "just an convertd" indexing problem, wouldn't scare me but the fact that i got 3M items backed up on origin, 3M items restored in these new servers, yet when I access one of the accounts (via webclient as I can't use zmmailbox), I can see folders missing, and 6 thousand mails in inbox after restore in new servers when they see 24 thousand mail in inbox on origin server (one particular case).

I've configured and reconfigured the DoSfilters making sure that config is ok (delays, throttleips, maxrequests, trustedips). I even disabled all tls and interprocess security just in case connections were being refuded because of that.

Firewalld is disabled, selinux is disabled and no other firewall or "weird thing" (not even nagios agents,etc) on these machines, which btw, they are deployed from the same CentOS vSphere template I use for all my deployments, and I configured using the same ansible playbooks I use for all my deployments.

I can telnet in any direction and from any of the store servers to ports 7071, 8443, 7047, 8080.

Any hint/sugestion/experience is more than welcome, I'm quite stucked here :/

[pup_seba]

Bumping this one. Please help.

[pup_seba]

Hi,

Everything seems to be fixed, thanks to Zimbra support (kudos to Gopal in this case). I overlooked the name my customer gave to the servers (lame way to try to excuse me and say it was not my fault ...but it is), which included a "_". So slap me and call me noob, as apache has a bug with such symbol in the names (https://bugs.debian.org/cgi-bin/bugrepo ... bug=851357), everything was working just awfully wrong given the prevously described sympthoms.

Anyways, hope this helps someone someday.
PS: No ego, no shame...always learning

[Klug]

IIRC, it's not just about apache, the "_" is not DNS compliant.
https://www.quora.com/Why-are-underscor ... host-names