Proxy service fails to start

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
desi
Posts: 7
Joined: Thu Oct 25, 2018 1:17 pm

Proxy service fails to start

Post by desi »

Hi Guys:

We just upgraded from Zimbra 8.7.1 on Ubuntu 12 to 8.8.11 on Ubuntu 16 a few weeks ago.

All went well except a few days after we started having an issue with the proxy service.

We have a script that stops Zimbra (/etc/init.d/zimbra stop) and does an rsync backup in the wee hours then issues a /etc/init.d/zimbra start. We have been doing it this way for years.

What is happening since the upgrade is that one or twice a week the proxy service will not start. Message we get is 'Proxy service failed to start, unknown reason'.

When this happens, logging on as the Zimbra user and issuing 'zmproxyctl restart' will sometimes get the proxy started. Other times it requires a reboot of the server.

Does anyone have any ideas as to what may be wrong or can point to specific logs to check? mailbox.log is not showing anything. Does the proxy service have a specific log?
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Proxy service fails to start

Post by JDunphy »

I will take a wild guess here...I dislike this kill -0 pid pattern zimbra uses to see if a daemon is already running. All that tells us if we have permission to send a signal to some pid - not necessarily the pid they assume it is. I don't find this clever myself and just asking for some pid wrap collision with the wrong process.

I wonder if we have another bug associated with this paradigm because it thinks nginx is already running so it never issues the start for you. That failed kill pattern got me with the MTA restart recently. It would be different here because the ngnix.pid file is removed normally by nginx unlike postfix that does not remove its created pid file... My wild guess would be - what if nginx was not able to shutdown because a socket in a SO_FIN_WAIT2 state? You might want to check the OS parameters to see what the time-outs for those TCP states given you have recently updated the OS and it sounds like you think it related to the upgrade of zimbra+OS... I think FIN_WAIT2 can be up to 8 mins but that is from memory from my unix days. That you have had to reboot the server is really strange in some instances... next time run a netstat and ps to see if nginx is running and what ports if any it has opened. You should be able to kill it before attempting to zmproxyctl restart it without having to restart the physical server. That you are forced to do a reboot, makes me think that nginx.pid file contains a pid from some other running process if nginx is not running.

PS... look in /opt/zimbra/log/nginx.log ... the reason for it not starting should be there. Report back and perhaps we can come up with a proper fix. While I don't restart every night, I have certificates that are renewed which requires a zmcontrol restart so this is of interest to me even though I have never seen this failure. I ran for 12+ years before I saw the MTA failure so it is just a matter of time perhaps. :-)
PS 2... I don't run version 8.8.X+ so if you do a zmcontrol stop and /opt/zimbra/log/nginx.pid still exists... The bug is the same as the MTA bug I reported. That is the second thing I would try after looking in those nginx logs.

Code: Select all

 
pidfile=${zimbra_log_directory}/nginx.pid
pid=""

getpid()
{
  if [ -f ${pidfile} ]; then
    pid=$(cat ${pidfile})
  fi
}


checkrunning()
{
  getpid
  if [ "x$pid" = "x" ]; then
    running=0
  else
    kill -0 $pid 2> /dev/null
    if [ $? != 0 ]; then
      pid=""
      running=0
    else
      running=1
    fi
  fi
}
BTW, I have already reported this about the MTA and is Bug798 and very simple to reproduce. See my thread for the MTA viewtopic.php?f=15&t=65332

HTH,

Jim
desi
Posts: 7
Joined: Thu Oct 25, 2018 1:17 pm

Re: Proxy service fails to start

Post by desi »

Hi Jim:

Thanks for the response. The system has behaved itself the last couple days, I will post any further details when it happens again.
desi
Posts: 7
Joined: Thu Oct 25, 2018 1:17 pm

Re: Proxy service fails to start

Post by desi »

It happened again last night.

nginx.log was no help. it showed the shut down at 3:00 a.m. and then nothing else until I did a successful manual restart at 6:01. The failed startup from my script generated no entries.
Post Reply