I hope someone can help me on this....
I have recently upgraded my Zimbra system from zcs-8.7.11_GA_1854.RHEL6_64 to zcs-8.8.10_GA_3039.RHEL6_64 on CentOS 6. This was done using the official packages, though I had major issues with ldap but finally got that resolved.
The issue I am experiencing at present is that I am simply unable to connect to the Zimbra instance or even issue the zmprov command.
If I try to run zmprov as the Zimbra user I get this output:
Code: Select all
zmprov
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)
Code: Select all
amavis Running
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
snmp Running
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Running
IMAP was working a few hours ago but now I am getting for all clients in the nginx.log:
Code: Select all
[error] 3225#0: *238 An error occurred in mail zmauth: while SSL handshaking to lookup handler
Code: Select all
2018-12-11 09:09:15,815 ERROR [qtp1286783232-13:https:https://IP/] [] webclient - Error during GetDomainInfoRequest invocation
com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
ExceptionId:qtp1286783232-13:https:https://IP/:1544519355815:361bf23be2d440db
Code:service.PROXY_ERROR Arg:(url, STR, "https://localhost:7071/service/admin/soap/GetDomainInfoRequest")
at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:379)
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:296)
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:193)
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:186)
at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:447)
at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:433)
at com.zimbra.client.ZSoapProvisioning.invokeRequest(ZSoapProvisioning.java:80)
at com.zimbra.client.ZSoapProvisioning.invoke(ZSoapProvisioning.java:58)
at com.zimbra.client.ZSoapProvisioning.invokeJaxb(ZSoapProvisioning.java:51)
at com.zimbra.client.ZSoapProvisioning.getDomainInfo(ZSoapProvisioning.java:32)
at com.zimbra.cs.taglib.tag.GetDomainInfoTag.getInfo(GetDomainInfoTag.java:102)
at com.zimbra.cs.taglib.tag.GetDomainInfoTag.checkCache(GetDomainInfoTag.java:84)
at com.zimbra.cs.taglib.tag.GetDomainInfoTag.doTag(GetDomainInfoTag.java:70)
at org.apache.jsp.tag.web.infra.skin_tag.doTag(skin_tag.java:254)
at org.apache.jsp.tag.web.infra.skinAndRedirect_tag._jspx_meth_app_005fskin_005f0(skinAndRedirect_tag.java:233)
at org.apache.jsp.tag.web.infra.skinAndRedirect_tag.doTag(skinAndRedirect_tag.java:156)
at org.apache.jsp.public_.login_jsp._jspx_meth_app_005fskinAndRedirect_005f0(login_jsp.java:964)
at org.apache.jsp.public_.login_jsp._jspService(login_jsp.java:220)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
at org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:107)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:199)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:74)
at org.eclipse.jetty.servlet.DefaultServlet.sendWelcome(DefaultServlet.java:593)
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:479)
at javax.servlet.http.HttpServlet.doHead(HttpServlet.java:288)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:704)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.webClient.filters.ForwardFilter.doFilter(ForwardFilter.java:88)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
at com.zimbra.webClient.filters.SetHeaderFilter.doFilter(SetHeaderFilter.java:248)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.webClient.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:37)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:748)
- restart zimbra service
- run extended permission check: /opt/zimbra/libexec/zmfixperms -extended
- rerun the install.sh script
- reset the proxy to defaults: https://wiki.zimbra.com/wiki/Enabling_Z ... _memcached
Checking that the proxy and ports are running and listening:
Code: Select all
# lsof -i :443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 3222 zimbra 11u IPv4 975451 0t0 TCP *:https (LISTEN)
nginx 3224 zimbra 11u IPv4 975451 0t0 TCP *:https (LISTEN)
nginx 3225 zimbra 11u IPv4 975451 0t0 TCP *:https (LISTEN)
nginx 3226 zimbra 11u IPv4 975451 0t0 TCP *:https (LISTEN)
nginx 3227 zimbra 11u IPv4 975451 0t0 TCP *:https (LISTEN)
Code: Select all
lsof -i :11211
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
memcached 3201 zimbra 26u IPv4 975387 0t0 TCP *:memcache (LISTEN)
memcached 3201 zimbra 27u IPv6 975388 0t0 TCP *:memcache (LISTEN)
memcached 3201 zimbra 28u IPv4 975465 0t0 TCP server.domain.com:memcache->server.domain.com:40642 (ESTABLISHED)
memcached 3201 zimbra 29u IPv4 975467 0t0 TCP server.domain.com:memcache->server.domain.com:40644 (ESTABLISHED)
memcached 3201 zimbra 30u IPv4 975469 0t0 TCP server.domain.com:memcache->server.domain.com:40646 (ESTABLISHED)
memcached 3201 zimbra 31u IPv4 975471 0t0 TCP server.domain.com:memcache->server.domain.com:40648 (ESTABLISHED)
nginx 3224 zimbra 14u IPv4 975464 0t0 TCP server.domain.com:40642->server.domain.com:memcache (ESTABLISHED)
nginx 3225 zimbra 16u IPv4 975466 0t0 TCP server.domain.com:40644->server.domain.com:memcache (ESTABLISHED)
nginx 3226 zimbra 18u IPv4 975468 0t0 TCP server.domain.com:40646->server.domain.com:memcache (ESTABLISHED)
nginx 3227 zimbra 20u IPv4 975470 0t0 TCP server.domain.com:40648->server.domain.com:memcache (ESTABLISHED)
Code: Select all
22 4964/sshd
25 4173/master
80 3222/nginx
110 3222/nginx
111 1265/rpcbind
143 3222/nginx
199 1567/snmpd
389 871/slapd
443 3222/nginx
465 4173/master
587 4173/master
993 3222/nginx
995 3222/nginx
2049 -
3310 3925/clamd
4949 1651/perl
5666 1212/nrpe
7025 2600/java
7071 2600/java
7072 2600/java
7073 2600/java
7110 2600/java
7143 2600/java
7171 893/java
7306 2415/mysqld
7780 3971/httpd
7993 2600/java
7995 2600/java
8080 2600/java
8443 2600/java
8465 3946/opendkim
10024 3289/amavisd
10025 4173/master
10026 3289/amavisd
10027 4173/master
10028 4173/master
10029 4173/master
10030 4173/master
10032 3289/amavisd
10050 1621/zabbix_agentd
11211 3201/memcached
23232 3248/perl
23233 3250/perl
33890 -
36602 1595/xinetd
36915 1287/rpc.statd
39079 1411/rpc.mountd
40286 1411/rpc.mountd
40852 1344/rpc.statd
42675 1411/rpc.mountd
43635 -
45228 1339/rpc.statd
45334 1411/rpc.mountd
46344 1344/rpc.statd
46544 1411/rpc.mountd
55998 1411/rpc.mountd
56829 1287/rpc.statd
58861 1339/rpc.statd
I think the issue started when trying to access a very large folder inside a mailbox. For this particular folder it was very difficult to get any client to open it as it kept timing out so after reading around I issued the:
Code: Select all
zmprov ms `zmhostname` zimbraImapInactiveSessionEhcacheSize 104857600
What more information could I provide to be of help or does anyone have a solution for this??
Many thanks for any ideas!
