Zimbra stoped working

antonio.meireles · Post by **antonio.meireles** » Wed Jan 18, 2006 10:48 am

I had M3 (build 436) working fine (upgraded from M2) until this morning.
Suddently zimbra started misbehaving. First it doesn 't allow user to login via web mail. It Reports ...

[QUOTE]"An unknown application error has occurred. Please correct any errors and retry. If the problem persists, please contact your System Aministrator. (INVALID_PDU)".

[/QUOTE]

Second it *isn't* receiving mail, with the following in the logs...
[QUOTE](delivery temporarily suspended: connect to zimbra.xxx.xxx.xxx [zzz.zzz.zz.zzz]: Connection refused[/QUOTE]
although a zmcontrol status says that everithing is ok

The only abnormal situation is in the tomcat logs where i find ...

[QUOTE]2006-01-18 16:31:00,354 INFO [main] [] RedoLogManager - Starting pre-startup crash recovery

2006-01-18 16:31:00,355 INFO [IndexWritersSweeper] [] MailboxIndex - IndexWritersSweeper thread starting

2006-01-18 16:31:02,174 FATAL [main] [] soap - Unable to start servlet

java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2006-01-18 16:31:02,174 INFO [main] [] [/service] - Marking servlet AdminServlet as unavailable

2006-01-18 16:31:02,175 ERROR [main] [] [/service] - Servlet /service threw load() exception[/QUOTE]
The box is a Dell 1850 running CentOS with 4GB of RAM and *nothing* changed from yesterday to today :-S

Any HELP will be greatly appreciated
Thanxs in Advance.
AntÃƒÂ³nio
P.S. I searched in the forums and nome of the problems mentioned seems fits this one

marcmac · Post by **marcmac** » Wed Jan 18, 2006 10:53 am

Sound like tomcat is in a bad way, predicated by some kind of java memory problem.
Stop and restart tomcat:

su - zimbra

tomcat stop

ps auxww | grep tomcat (is it really stopped?)

tomcat start

antonio.meireles · Post by **antonio.meireles** » Wed Jan 18, 2006 11:06 am

Problem persists... after start/stop tomcat, server restart, whatever

Any idea ?

TIA
AntÃƒÂ³nio
[QUOTE][zimbra@zimbra ~]$ tomcat stop

Using CATALINA_BASE: /opt/zimbra/tomcat

Using CATALINA_HOME: /opt/zimbra/tomcat

Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp

Using JRE_HOME: /opt/zimbra/java

Waiting ..... trying again

Using CATALINA_BASE: /opt/zimbra/tomcat

Using CATALINA_HOME: /opt/zimbra/tomcat

Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp

Using JRE_HOME: /opt/zimbra/java

Jan 18, 2006 5:03:18 PM org.apache.catalina.startup.Catalina stopServer

SEVERE: Catalina.stop:

java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)

at java.net.Socket.connect(Socket.java:507)

at java.net.Socket.connect(Socket.java:457)

at java.net.Socket.(Socket.java:365)

at java.net.Socket.(Socket.java:178)

at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:394)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:320)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:411)

Waiting ..... trying again

Using CATALINA_BASE: /opt/zimbra/tomcat

Using CATALINA_HOME: /opt/zimbra/tomcat

Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp

Using JRE_HOME: /opt/zimbra/java

Jan 18, 2006 5:03:23 PM org.apache.catalina.startup.Catalina stopServer

SEVERE: Catalina.stop:

java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)

at java.net.Socket.connect(Socket.java:507)

at java.net.Socket.connect(Socket.java:457)

at java.net.Socket.(Socket.java:365)

at java.net.Socket.(Socket.java:178)

at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:394)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:320)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:411)

Waiting ..... trying again

Using CATALINA_BASE: /opt/zimbra/tomcat

Using CATALINA_HOME: /opt/zimbra/tomcat

Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp

Using JRE_HOME: /opt/zimbra/java

Jan 18, 2006 5:03:29 PM org.apache.catalina.startup.Catalina stopServer

SEVERE: Catalina.stop:

java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)

at java.net.Socket.connect(Socket.java:507)

at java.net.Socket.connect(Socket.java:457)

at java.net.Socket.(Socket.java:365)

at java.net.Socket.(Socket.java:178)

at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:394)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:320)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:411)

Waiting ..... trying again

Giving up - killing 5151

[zimbra@zimbra ~]$ ps auxww | grep tomcat

zimbra 8387 0.0 0.0 1660 476 pts/1 R+ 17:03 0:00 grep tomcat

[zimbra@zimbra ~]$ tomcat start[/QUOTE]

marcmac · Post by **marcmac** » Wed Jan 18, 2006 11:39 am

restarting tomcat did not help.
Have you rebooted the server?
What is this set to:

zmlocalconfig tomcat_java_options
You've got 4 gigs of ram - is this box running in a virtual server?
Is this a 64 bit machine (more importantly, are you running a 64 bit OS?)
What else is running on the box?
Which version are you running (FOSS or network?)

antonio.meireles · Post by **antonio.meireles** » Wed Jan 18, 2006 11:47 am

a) Yes, I have rebooted the thing

b) [zimbra@zimbra ~]$ zmlocalconfig tomcat_java_options

tomcat_java_options = -client -XX:NewRatio=2

c) this is running as a Xen VM instance. 32bits. canonic CentOS/RHEL4 install

d) the box *only* runs zimbra

e) Running FOSS version
All was working *fine* until today
See plz the private msg that i have sent you.

Thanxs in advance
AntÃƒÂ³nio

[quote user="marcmac"]restarting tomcat did not help.
Have you rebooted the server?
What is this set to:

zmlocalconfig tomcat_java_options
You've got 4 gigs of ram - is this box running in a virtual server?
Is this a 64 bit machine (more importantly, are you running a 64 bit OS?)
What else is running on the box?
Which version are you running (FOSS or network?)[/QUOTE]

antonio.meireles · Post by **antonio.meireles** » Wed Jan 18, 2006 1:03 pm

I' m writing here to thank publicly to Marc Mac, and zimbra, for the awesome job he did. We get in touch on MSN, he prontified to log via ssh on my zimbra box, and promply, with another zimbra engineer, identified the problem which turned to be a corrupted (tomcat) redolog.
keep the awesome job.
Cheers, from Portugal
AntÃƒÂ³nio

vogon1 · Post by **vogon1** » Wed Jan 18, 2006 1:19 pm

[quote user="antonio.meireles"]identified the problem which turned to be a corrupted (tomcat) redolog.

[/QUOTE]
Is it known now how the corruption occurred, how it can be prevented, and if it happens how it can be resolved?

Might be useful information here.
Sietse

marcmac · Post by **marcmac** » Wed Jan 18, 2006 1:29 pm

Not known yet - but we've got the log file, and we're looking at it.

Isi · Post by **Isi** » Thu Jan 19, 2006 3:44 pm

Zimbra Team,
Was there ever a fix for this? I have recently upgraded from a working install of M3 to the M3 Update and am experiencing this problem.
So far i have done the following:
1. My install slightly deviates from standard by changing the port from 80 to 8080 in /opt/zimbra//conf/server.xml and the webapps/service/WEB-INF/web.xml files.
2. I have restarted my maching (suggested earlier in the post)
3. I have looked through the logs that I am aware of and don't see any points of concern or errors.
Any thoughts or advise i could use to get to the bottom of this?
exact error i get when trying to login:

"An unknown application error has occurred. Please correct any errors and retry. If the problem persists, please contact your System Aministrator. (INVALID_PDU)"
-- Isi

Isi · Post by **Isi** » Thu Jan 19, 2006 4:35 pm

Zimbra Team,
I am still getting this error:

"An unknown application error has occurred. Please correct any errors and retry. If the problem persists, please contact your System Aministrator. (INVALID_PDU)"
but i think i know why.
I run my server on port 8080 and not the default of 80. To do this I modify a couple of files as suggested else where in the forum and as noted in my post above.
It seems though that after I modify these and start the zimbra service the files are modified back including the ports being changed.
Is this normal procedure? If so why? If not how can i fix it?
Here is a transcript of the process I am taking so you can see exactly what I am doing and exactly what happens (so that you see i am not crazy).
[root@abbadon ~]# service zimbra stop

Host abbadon.ibancsolutions.com

Stopping antispam...Done

Stopping antivirus...Done

Stopping ldap...Done

Stopping logger...Done

Stopping mailbox...Done

Stopping mta...Done

Stopping snmp...Done

Stopping spell...Done
[root@abbadon ~]# vi /opt/zimbra/jakarta-tomcat-5.5.7/conf/server.xml
[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/conf/server.xml

enableLookups="false" redirectPort="443"

[root@abbadon ~]# vi /opt/zimbra/jakarta-tomcat-5.5.7/webapps/service/WEB-INF/web.xml
[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/webapps/service/WEB-INF/web.xml

8080, 8443, 7070, 7443

8080, 8443, 7070, 7443, 7071

8080, 8443, 7070, 7443, 7071

8080, 8443, 7070, 7443, 7071

8080, 8443, 7070, 7443, 7071

8080, 8443, 7070, 7443, 7071

8080, 8443, 7070

8080, 8443, 7070, 7443, 7071
[root@abbadon ~]# service zimbra start

Host abbadon.ibancsolutions.com

Starting ldap...Done.

Starting antispam...Done.

Starting antivirus...Done.

Starting logger...Done.

Starting mailbox...Done.

Starting mta...Done.

Starting snmp...Done.

Starting spell...Done.
[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/conf/server.xml

enableLookups="false" redirectPort="443"

[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/webapps/service/WEB-INF/web.xml

80, 443, 7070, 7443

80, 443, 7070, 7443, 7071

80, 443, 7070, 7443, 7071

80, 443, 7070, 7443, 7071

80, 443, 7070, 7443, 7071

80, 443, 7070, 7443, 7071

80, 443, 7070

80, 443, 7070, 7443, 7071
Any feedback or advice is greatly appreciated.
-- Isi