Help! my zimbra server is crashing with a rlimit error

kechols · Post by **kechols** » Thu Mar 01, 2007 5:37 pm

I'm running Suse Enterprise 9 SP3 and Zimbra 3.0.1GA (yes I know it's old, they won't let me upgrade until it's stable again).

My message logs show this error

Mar 1 15:18:35 mailserv6 kernel: open files rlimit 1024 reached for uid 1001 pid 14156

Mar 1 15:18:35 mailserv6 kernel: open files rlimit 1024 reached for uid 1001 pid 14408

Mar 1 15:18:35 mailserv6 kernel: open files rlimit 1024 reached for uid 1001 pid 14408

Then my loads jump from no higher than 15 - 25 (peak) to 85+ immediately

and I can't run more than one command at a console without losing access to that shell. Thus forcing me to press the reset button to get control of the server back.
I've determined that uid 1001 is the zimbra user, once this happens I get repeated auth failures and a few minutes later
auth failed: curl_easy_perform: error(7): Connect failed

starts showing up in my logs.
Has anyone seen something like this ? I've checked and it looks like the limit should be 10240 files for zimbra instead of 1024. So I'm not sure why this keeps happening, it's happened twice now.
If it helps I'm running zimbra over an IBM ds300 SAN with 4 146 GB u320 drives (firmware on the drives and SAN are current per IBM). I'll be happy to provide any information I can, just ask and I'll get it posted asap.

this seems to be happening every couple of days now.
Thanks

marcmac · Post by **marcmac** » Thu Mar 01, 2007 5:46 pm

on rh/fedora, we set the hard limit for open files to 10000 in /etc/security/limits.conf, then do ulimit -n 10000 in the zimbra environment. Do the analagous on suse, and you should be fine.

kechols · Post by **kechols** » Thu Mar 01, 2007 5:48 pm

Would this be a one time setting or do I need to add this to a boot/init script?

kechols · Post by **kechols** » Fri Mar 02, 2007 3:35 pm

So i ran ulimit -n 10000 this morning, and verified the following entries in my /etc/security/limits.conf

zimbra soft nofile 10000

zimbra hard nofile 10000

And the server just crashed again with another rlimit error. This time loads jumped up to 120. After rebooting, ulimit shows:

zimbra@mailserv6:~> ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

file size (blocks, -f) unlimited

max locked memory (kbytes, -l) unlimited

max memory size (kbytes, -m) unlimited

open files (-n) 1024

pipe size (512 bytes, -p) 8

stack size (kbytes, -s) unlimited

cpu time (seconds, -t) unlimited

max user processes (-u) 32251

virtual memory (kbytes, -v) unlimited

I then do the following:

mailserv6:~ # ulimit -n 10000

mailserv6:~ # ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

file size (blocks, -f) unlimited

max locked memory (kbytes, -l) unlimited

max memory size (kbytes, -m) unlimited

open files (-n) 10000

pipe size (512 bytes, -p) 8

stack size (kbytes, -s) unlimited

cpu time (seconds, -t) unlimited

max user processes (-u) 32251

virtual memory (kbytes, -v) unlimited

is there something I need to do to make these settings stick between reboots ? How about to make these take effect immediately? should it be run before zimbra is started everytime?

Thanks.

marcmac · Post by **marcmac** » Fri Mar 02, 2007 4:01 pm

the ulimit -n is in zimbra's login shell, too, so the startup scripts should get the change.
Have you changed the .bashrc? Try adding something to the zmcontrol script to output the ulimit -a data to a file, then reboot the machine and check the file. IF the file has 1024, not 10000, then SuSE is doing something you don't want it to.

kechols · Post by **kechols** » Fri Mar 02, 2007 4:13 pm

at the bottom of zimbra's .bashrc is

ulimit -n 10000 > /dev/null 2>&1

think i'd do better to place it at the top of the zimbra startup script?