[Solved] Clamd failes and subject of messages shows up as Unchecked every few days

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

[Solved] Clamd failes and subject of messages shows up as Unchecked every few days

Post by davidkillingsworth »

Server:

Code: Select all

Release 8.8.12.GA.3794.UBUNTU14.64 UBUNTU14_64 FOSS edition, Patch 8.8.12_P4.
I just updated today from Patch 2 to Patch 4.

Clamd version (from freshclam.log):

Code: Select all

WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.99.4 Recommended version: 0.101.2
Every few days, I keep getting the situation where messages are delivered with "Unchecked" in their subject line.

I can manually restart zimbra with a "zontrol restart" but it comes back every few days.

Today, I decided to take a look at what might be causing this and found this in the log when the zimbra services start

Code: Select all

Jul 30 11:51:30 mail clamd[30211]: TCP: Bound to [127.0.0.1]:3310
Jul 30 11:51:30 mail clamd[30211]: TCP: Setting connection queue length to 200
Jul 30 11:51:30 mail clamd[30211]: LOCAL: Unix socket file /opt/zimbra/data/clamav/clamav.sock
Jul 30 11:51:30 mail clamd[30211]: LOCAL: Setting connection queue length to 200
Jul 30 11:51:30 mail clamd[30211]: daemonize() failed: Cannot allocate memory
Jul 30 11:51:30 mail clamd[30211]: Socket file removed.
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!)connect to /opt/zimbra/data/clamav/clamav.sock failed, attempt #1: Can't connect to a UNIX socket /opt/zimbra/data/clamav/clamav.sock: No such file or directory
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!)ClamAV-clamd av-scanner FAILED: run_av error: Too many retries to talk to /opt/zimbra/data/clamav/clamav.sock (All attempts (1) failed connecting to /opt/zimbra/data/clamav/clamav.sock) at (eval 148) line 613.\n
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!)WARN: all primary virus scanners failed, considering backups
Jul 30 11:51:30 mail amavis[30041]: (30041-01) (!!)AV: ALL VIRUS SCANNERS FAILED
When I try to manually check the services, I see the folowing, which doesn't make any sense.

Code: Select all

zimbra@mail:~/data/clamav$ zmcontrol status
Host mail.myhost.com
	amavis                  Running
	antispam                Running
	antivirus               Running
	dnscache                Running
	ldap                    Running
	logger                  Running
	mailbox                 Running
	memcached               Running
	mta                     Running
	opendkim                Running
	proxy                   Running
	service webapp          Running
	snmp                    Running
	spell                   Running
	stats                   Running
	zimbra webapp           Running
	zimbraAdmin webapp      Running
	zimlet webapp           Running
	zmconfigd               Running
zimbra@mail:~/data/clamav$ zmantivirusctl stop
Stopping clamd...done.
Stopping freshclam...done.
zimbra@mail:~/data/clamav$ zmantivirusctl start
Starting amavisd-mc...amavisd-mc is already running.
Starting amavisd...amavisd is already running.
Starting clamd...failed.
Starting freshclam...done.
zimbra@mail:~/data/clamav$ zmantivirusctl restart
Stopping amavisd... done.
Stopping amavisd-mc... done.
Starting amavisd-mc...done.
Starting amavisd...done.
Stopping clamd...done.
Starting clamd...failed.
Stopping freshclam...done.
Starting freshclam...done.
zimbra@mail:~/data/clamav$ zmclamdctl status
clamd is running.
zimbra@mail:~/data/clamav$ zmclamdctl stop  
Stopping clamd...done.
zimbra@mail:~/data/clamav$ zmclamdctl start
Starting clamd...failed.
zimbra@mail:~/data/clamav$ zmantivirusctl status
antivirus is running
zimbra@mail:~/data/clamav$ zmclamdctl status
clamd is running.
Clamd fails to start, but then when I check it's status, it is running.

Any idea what's going on?

I did find that I had some duplicates in /opt/zimbra/data/clamav/db/

Code: Select all

zimbra@mail:~/data/clamav/db$ ls -al
total 479508
drwxr-xr-x 10 zimbra zimbra      4096 Jul 30 12:16 .
drwxrwxr-x  4 zimbra zimbra      4096 Jul 30 12:16 ..
-rw-r-----  1 zimbra zimbra   1013248 Jan  2  2019 bytecode.cld
-rw-r-----  1 zimbra zimbra    207879 Jul 30 11:52 bytecode.cvd
drwxr-x---  2 zimbra zimbra      4096 May 21  2016 clamav-067d7d74e7db25496b87dbf761186fe1.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov  9  2016 clamav-278a82d9b2c3fd7be2d4619e1652882b.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov  9  2016 clamav-4d234370c55f3805833208b2e6a4870c.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov  9  2016 clamav-4e0ab5775afb14c96bf117438cd53466.tmp
drwxr-x---  2 zimbra zimbra      4096 Nov 13  2016 clamav-60e9d3622749eee696cda4a47066900e.tmp
drwxr-x---  2 zimbra zimbra      4096 May 13  2016 clamav-79abe18ba0ab500e60858078dd42eb79.tmp
drwxr-x---  2 zimbra zimbra      4096 Jun  1  2016 clamav-97fe75be43aacdf44e1b5a983578b096.tmp
drwxr-x---  2 zimbra zimbra      4096 May 21  2016 clamav-c4caf97b60239b9a82cc51161cb5398b.tmp
-rw-r-----  1 zimbra zimbra 136675328 Jul 29 20:02 daily.cld
-rw-r-----  1 zimbra zimbra  45067320 Jul 30 11:50 daily.cvd
-rw-r-----  1 zimbra zimbra 307499008 Jul 30 11:50 main.cld
-rw-------  1 zimbra zimbra      2548 Jul 30 12:16 mirrors.dat
I ran the following:

Code: Select all

cd /opt/zimbra/data/clamav/db/
mv *.* /tmp/clamavdbback/
/opt/zimbra/common/bin/freshclam --config-file=/opt/zimbra/conf/freshclam.conf
This cleaned out duplicates and old .tmp files in the db folder that I had and this is the result of what's there now.

Code: Select all

zimbra@mail:~/data/clamav/db$ ls -al
total 159532
drwxr-xr-x 2 zimbra zimbra      4096 Jul 30 14:16 .
drwxrwxr-x 4 zimbra zimbra      4096 Jul 30 13:18 ..
-rw-r----- 1 zimbra zimbra    207879 Jul 30 12:48 bytecode.cvd
-rw-r----- 1 zimbra zimbra  45067320 Jul 30 12:46 daily.cvd
-rw-r----- 1 zimbra zimbra 117892267 Jul 30 12:46 main.cvd
-rw------- 1 zimbra zimbra        52 Jul 30 14:16 mirrors.dat
I still get the clamd start failures though.

I will wait a few days to see if the unchecked issue comes back.

Any suggestions on what the problems are with the lock file?
Last edited by davidkillingsworth on Tue Jul 30, 2019 11:05 am, edited 1 time in total.
User avatar
DualBoot
Elite member
Elite member
Posts: 1326
Joined: Mon Apr 18, 2016 8:18 pm
Location: France - Earth
ZCS/ZD Version: ZCS FLOSS - 8.8.15 Mutli servers
Contact:

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Post by DualBoot »

Hello,

related to the log you provided, it is a problem of memory :
Jul 30 11:51:30 mail clamd[30211]: daemonize() failed: Cannot allocate memory
Regards,
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Post by davidkillingsworth »

DualBoot wrote:Hello,

related to the log you provided, it is a problem of memory :
Jul 30 11:51:30 mail clamd[30211]: daemonize() failed: Cannot allocate memory
Regards,
Thanks for pointing that out. This is a VPS with less than 10 users on it. The VPS has 4GB of memory.

I see the following:

Code: Select all

zimbra@mail:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          3952       3716        236          0         11        113
-/+ buffers/cache:       3591        361
Swap:            0          0          0

Code: Select all

top - 18:04:04 up 1 day,  7:27,  1 user,  load average: 0.56, 0.56, 0.58
Tasks: 185 total,   2 running, 183 sleeping,   0 stopped,   0 zombie
%Cpu(s): 57.1 us, 11.7 sy,  0.0 ni, 30.9 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem:   4047820 total,  3768496 used,   279324 free,    12432 buffers
KiB Swap:        0 total,        0 used,        0 free.   116900 cached Mem
Is it a case of memory running out after a certain number of days? Any way to mitigate other than paying for more memory with the VPS provider?

Thanks,
David
phoenix
Ambassador
Ambassador
Posts: 27262
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Post by phoenix »

Your 4GB is below the recommended minimum of 8GB.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Post by davidkillingsworth »

phoenix wrote:Your 4GB is below the recommended minimum of 8GB
Got it. Will bump it up then.

Thanks for the responses.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 883
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Post by JDunphy »

davidkillingsworth wrote: Tasks: 185 total, 2 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 57.1 us, 11.7 sy, 0.0 ni, 30.9 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 4047820 total, 3768496 used, 279324 free, 12432 buffers
KiB Swap: 0 total, 0 used, 0 free. 116900 cached Mem[/code]

Is it a case of memory running out after a certain number of days? Any way to mitigate other than paying for more memory with the VPS provider?
This is why you still want some swap. Add a simple swap file is more than enough so that a few pages can reside on there from time to time during peak shortfalls. Kernel will move least recently used pages out which is much faster than accesses back through the fs for something aborting for all its pages again. Just enough to ride through some memory peaks. Long term, more memory is probably warranted but you need to watch the server with vmstat, etc to know for sure. A swap file should do the trick given you probably don't want to re-partition for a few memory short falls. You could also tune down some of the big memory hogs but that is a hard recommendation to make without watching in more detail.You can tell my age because I always configure a small amount of swap (reliability) vs (pure performance).

Note: you can add a swap file without a reboot or restarting zimbra so it fairly quick for a production machine. If you need it to remain, put it in your /etc/fstab for future reboots.

Ref: https://www.cyberciti.biz/faq/linux-add ... ile-howto/
Ref: https://www.linux.com/news/all-about-linux-swap-space
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Post by davidkillingsworth »

JDunphy wrote:
davidkillingsworth wrote: Tasks: 185 total, 2 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 57.1 us, 11.7 sy, 0.0 ni, 30.9 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 4047820 total, 3768496 used, 279324 free, 12432 buffers
KiB Swap: 0 total, 0 used, 0 free. 116900 cached Mem[/code]

Is it a case of memory running out after a certain number of days? Any way to mitigate other than paying for more memory with the VPS provider?
This is why you still want some swap. Add a simple swap file is more than enough so that a few pages can reside on there from time to time during peak shortfalls. Kernel will move least recently used pages out which is much faster than accesses back through the fs for something aborting for all its pages again. Just enough to ride through some memory peaks. Long term, more memory is probably warranted but you need to watch the server with vmstat, etc to know for sure. A swap file should do the trick given you probably don't want to re-partition for a few memory short falls. You could also tune down some of the big memory hogs but that is a hard recommendation to make without watching in more detail.You can tell my age because I always configure a small amount of swap (reliability) vs (pure performance).

Note: you can add a swap file without a reboot or restarting zimbra so it fairly quick for a production machine. If you need it to remain, put it in your /etc/fstab for future reboots.

Ref: https://www.cyberciti.biz/faq/linux-add ... ile-howto/
Ref: https://www.linux.com/news/all-about-linux-swap-space
Interesting. Thanks for this.

I have checked about adding physical memory and my VPS provider only allows upgrades of the whole VPS size. I don't need more CPU or disk space, so it's a case of spending a bunch of extra money just for 4GB more RAM. That sort of makes running my own zimbra server a little less practical. However, having the knowledge of being able to add a swap file effectively, make that more feasible.

I read both articles, but what would you recommend in terms of size for a standard default Ubuntu \ Zimbra installation? Would you mind writing up a little guide? I think this could help for those of us that are running servers in environments that are limited by how much physical memory we can assign, such as VPS environments.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 883
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Clamd failes and subject of messages shows up as Unchecked every few days

Post by JDunphy »

davidkillingsworth wrote: I read both articles, but what would you recommend in terms of size for a standard default Ubuntu \ Zimbra installation? Would you mind writing up a little guide? I think this could help for those of us that are running servers in environments that are limited by how much physical memory we can assign, such as VPS environments.
Hi David,

The old school answer was 2x memory but I have not done that in many years given how big memory sizes have grown. I run memory size for my swap size if I have plenty of disk. So 4GB would be plenty for you and then watch the system and your logs for memory short falls and any disk bottlenecks. If your disk controller has enough bandwidth and your disks are keeping up than I wouldn't worry too much about the odd swap usage. If you are still getting memory allocation fails than it is time to tune memory usage down or add more memory. If you are not happy with your search responsiveness with the web clients than you might need to get a little more creative to find that happy mix for your usage. Given the cost of ram, many admin's will double the ram sizes vs invest in the time and effort to tune. For some VPS providers, that generally doubles the cost of your hosting so I understand where you are coming from.

This is a better link explaining how to size them which is about 20% of memory these days and provides a little more background including how to size swap usage.

ref: https://www.redhat.com/en/blog/do-we-re ... rn-systems

Jim
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: [Solved] Clamd failes and subject of messages shows up as Unchecked every few days

Post by davidkillingsworth »

FYI, for anyone on Ubuntu who is having the memory problem that I was having, I found a package called swapspace

It dynamically manages the swap file for you, which seems like a way easier way to manage this.

Installation.

Code: Select all

sudo apt-get install swapspace
Man pages are here: https://manpages.ubuntu.com/manpages/co ... ace.8.html

After installation, I rebooted and let zimbra start up. Once again, the zmclamdctl service failed. I manually started it and I watch the swap file get created and used and the server started properly.

Thanks for pushing me in the right direction JDunphy.
omer.nasim
Posts: 4
Joined: Fri Feb 17, 2023 7:22 am

Re: [Solved] Clamd failes and subject of messages shows up as Unchecked every few days

Post by omer.nasim »

True, i faced the same issue and increasing the swap to 2GB made clam somewhat stable. single zimbra server deployed on 4cpu 4GB ram. As already mentioned recommended specs for single server deployment are min 8GB ram.
zimbra-swap.png
zimbra-swap.png (328.82 KiB) Viewed 1634 times
Post Reply