8.8.15 Patch32 message log filling on Centos

Ask questions about your setup or get help installing ZCS server (ZD section below).
Post Reply
gmills
Posts: 13
Joined: Sat Mar 11, 2017 10:13 pm

8.8.15 Patch32 message log filling on Centos

Post by gmills »

Hi

Latest 8.8.15 OSE Patch 32 (patched again to fix LDAP issue) is filling the /var/log/messages on Centos 7
The message log is filled with entries like this:
Jun 18 11:11:35 store.xxxxxx.co.nz mailboxd: INFO [mailboxd.csv] [] slogger - ....
Jun 18 11:28:35 store.xxxxxx.co.nz mailboxd: INFO [soap.csv] [] slogger - soap.csv: ...
Jun 18 11:28:35 store.xxxxxx.co.nz mailboxd: INFO [ldap.csv] [] slogger - ldap.csv: ...

These entries were not present before P32
Bug number: ZBUG-2835 though relevant does not seem to work with Centos. I cannot work out how to apply the second change "2. Update line no 210 and remove SLOGGER from statement. "

/opt/zimbra/conf/log4j.properties.in line 210: %%comment VAR:!zimbraLogHostname%%logger.slogger.level=INFO

Thanks
Geoffrey Mills
gmills
Posts: 13
Joined: Sat Mar 11, 2017 10:13 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by gmills »

Apologies for the multiple posting, I was getting an SQL error on posting but turns out the messages were being posted.
Maybe an administrator can purge the duplicates?

Geoffrey Mills
User avatar
helper003
Posts: 23
Joined: Fri Feb 12, 2021 3:59 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by helper003 »

Try to restart the Zimbra services after updating to the latest patch.

su - zimbra
zmcontrol restart

Thanks
gmills
Posts: 13
Joined: Sat Mar 11, 2017 10:13 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by gmills »

Solved, workaround the excess logging in /var/log/messages on Centos 7
Reference: https://wiki.zimbra.com/wiki/Log4j2
Reference: https://wiki.zimbra.com/wiki/Current_Known_Issues

Bug number: ZBUG-2835
1. Open /opt/zimbra/conf/log4j.properties.in and update line no 195 and change value to info from debug.
2. Update line no 210 and remove SLOGGER from statement.

I managed to "decode" this line: '2. Update line no 210 and remove SLOGGER from statement'

In the file "/opt/zimbra/conf/log4j.properties.in" locate the block shown below and comment this block out using # prefix:

%%comment VAR:!zimbraLogHostname%%logger.slogger.name = zimbra.slogger
%%comment VAR:!zimbraLogHostname%%logger.slogger.level=INFO
%%uncomment VAR:!zimbraLogHostname%%logger.slogger.level=ERROR
%%comment VAR:!zimbraLogHostname%%logger.slogger.additivity = false
%%comment VAR:!zimbraLogHostname%%logger.slogger.appenderRef.SLOGGER.ref = sloggerAppender

MUST restart mailbox - zmmailboxdctl restart

"/opt/zimbra/conf/log4j.properties.in" has appenders defined for both syslog (LOCAL0) and slogger (LOCAL1) for mailboxd traffic

If there is a better way please do let me know

Regards
Geoffrey Mills
jerome3859
Posts: 2
Joined: Wed Jul 06, 2022 2:52 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by jerome3859 »

Hello

I have a log4j issue after applying 8.8.15 Patch 32.

No logs are written to zimbra.log file but only in messages.
The log4j configuration fle is correctly set.

IIt seems that the logs are sent to Rsyslog with a wrong format.

The logs are correctly redirected to zimbra.log if comment the following lines in log4j.properties
appender.SYSLOG.layout.type = PatternLaout
appender.SYSLOG.layout.pattern = imailboxd: %-5p [%t] [%z] %c{1} - %m

My system is CentOS 7 with Rsyslog 8.24.0.57


Thanks
Jerome
Last edited by jerome3859 on Thu Jul 07, 2022 9:48 am, edited 2 times in total.
gmills
Posts: 13
Joined: Sat Mar 11, 2017 10:13 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by gmills »

Hi,

Sounds more like an OS issue, except you only updated zimbra.
Check rsyslog? /etc/rsyslog.conf and /etc/rsyslogconf.d/, try to follow the logging process

My /etc/rsyslog.conf has these entries at the end of the file:
local0.* -/var/log/zimbra.log
local1.* -/var/log/zimbra-stats.log
auth.* -/var/log/zimbra.log
mail.* -/var/log/zimbra.log

has /var/log/zimbra-stat.log got data coming in (local1.*)

And in /opt/zimbra/conf/log4j.provider:
# Syslog appender
appender.SYSLOG.type = Syslog
appender.SYSLOG.name = syslogAppender
appender.SYSLOG.host = localhost <-- in /etc/hosts?
appender.SYSLOG.port = 514 <-- not blocked?
appender.SYSLOG.protocol = UDP
appender.SYSLOG.facility = LOCAL0
appender.SYSLOG.layout.type = PatternLayout
appender.SYSLOG.layout.pattern = mailboxd: %-5p [%t] [%z] %c{1} - %m

# Logger service appender
appender.SLOGGER.type = Syslog
appender.SLOGGER.name = sloggerAppender
appender.SLOGGER.host = store.xxxxx.co.nz
appender.SLOGGER.facility = LOCAL1
appender.SLOGGER.port = 514
appender.SLOGGER.protocol = UDP
appender.SLOGGER.layout.type = PatternLayout
appender.SLOGGER.layout.pattern = mailboxd: %-5p [%t] [%z] %c{1} - %m

Are the log files in /opt/zimbra/log/ working?
Restart rsyslog service and look for errors in /var/log/messages?
I also ran zmfixperms at some stage https://wiki.zimbra.com/wiki/Fix_the_Zi ... ermissions
Suspect you have looked at most of these though...
I vaguely remember some Zimbra Wiki pages on getting logging working properly in multi-server environment, had to use these when I set Zimbra up

Regards
Geoffrey
jerome3859
Posts: 2
Joined: Wed Jul 06, 2022 2:52 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by jerome3859 »

Hello Geoffroy

Thans for you advice

But in fact, it's a PatternLayout issue
The faliciy is not trasmitted by default

The pattern must be changed to
appender.SYSLOG.layout.pattern = <134> %d{MMM dd HH:mm:ss} ${hostName} mailboxd: %-5p [%t] [%z] %c{1} - %m



But some issue persist
In release 8.8.15 P31 the zimbra.log contains all mailboxd messages
And qith the new Patch these messages are only store in the /opt/zimbra/log/mailboxd.log file


Perhaps something missing at the rootlogger level

I only have these 2 lines
rootLogger.level=INFO
rootLogger.appenderRef.LOGFILE.ref = mailboxFile

Thanks
Jerome
gmills
Posts: 13
Joined: Sat Mar 11, 2017 10:13 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by gmills »

Hi Jerome,

OK, my 2cents worth and I could be totally wrong so ignore me if required.
The pattern must be changed to
appender.SYSLOG.layout.pattern = <134> %d{MMM dd HH:mm:ss} ${hostName} mailboxd: %-5p [%t] [%z] %c{1} - %m
OK, this redirects (appends) data from existing logs into SYSLOG but this is the *wrong* data.
No logs are written to zimbra.log file but only in messages.
I think postfix, amavis, etc log *direct* to SYSLOG (via local0), not sure if they use the java log4j logging. These do not log in /opt/zimbra/log/mailbox.log anyway.

https://wiki.zimbra.com/wiki/Log_Files
The MTA (postfix) logs via syslog to the mail facility. By default, this goes to /var/log/maillog, but Zimbra sets it to /var/log/zimbra.log

On my Zimbra store server, use zimbra "postconf" utility to display logging destination for postfix:
[zimbra@store conf]$ /opt/zimbra/common/sbin/postconf maillog_file
maillog_file =

The "maillog_file=" is empty therefore postfix is logging to syslog

Is our Centos logging working? https://www.poftut.com/linux-logger-com ... -examples/
Looking at the end of /etc/rsyslog.conf, "local0" logs to /var/log/zimbra.log

Test it:
logger -p "local0.info" Test log to local0.info
This message is displayed in /var/log/zimbra.log, so logging to local0 is working

Reset logging for multi-server setup:
https://wiki.zimbra.com/wiki/Configuring-Logger-Host
is Zimbra zmlogger running?

PS: Are you using Centos syslog or rsyslog - I remember I changed my one over to rsyslog
PPS: selinux running - caused me many issues originally - /opt/zimbra/libexec/zmfixperms -e -v

/opt/zimbra/log/mailbox.log: <-- mailboxd logs, traffic volumes, etc but not mail send/receive data
2022-07-09 09:47:38,045 INFO [MailboxPurge] [name=primesure@gamix.co.nz;mid=17;] purge - Purging messages.
2022-07-09 09:47:56,916 INFO [mailboxd.csv] [] slogger - mailboxd.csv:

/var/log/zimbra.log:
Jul 9 09:50:46 ldap zmconfigd[1698]: Watchdog: skipping service antivirus. Service not yet available for restarts.
Jul 9 09:50:46 ldap zmconfigd[1698]: All rewrite threads completed in 0.00 sec
Jul 9 09:50:46 ldap zmconfigd[1698]: All restarts completed in 0.00 sec
Jul 9 09:50:48 store /postfix-script[18264]: the Postfix mail system is running: PID: 11811
Jul 9 10:20:15 store postfix/qmgr[11813]: 6B44823B16D5: from=<admin@gamix.co.nz>, size=2495, nrcpt=1 (queue active)
Jul 9 10:20:15 store amavis[12695]: (12695-14) Passed CLEAN {RelayedOutbound},

Regards
Geoffrey
Blueberry
Posts: 19
Joined: Thu Jan 25, 2018 12:14 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by Blueberry »

Hi Jerome,

Did you find a solution to this bug? It's a bug since zimbraLogToSyslog is a ZCS feature and it's not working anymore.

The only way I could get some logs coming to the Syslog server is with the zimbra.security logger and by commenting the 2 layout Appender lines. But poor logs then (with thread names, IP adresses, user agent, etc.).
Since log4j2 sometimes manages to send logs to Syslog, there has to be a way to set a pattern so that we can continue receiving rich / extended logs through Syslog.

I've opened a support case this morning, I'll let you know how it goes.

Regards,
Blue.
gcakici2
Posts: 4
Joined: Mon Jan 18, 2021 9:45 pm

Re: 8.8.15 Patch32 message log filling on Centos

Post by gcakici2 »

Did any solution comes up with your support case? Can not send our mailbox logs to rsyslog after applying patch 32.

Thanks
Gokalp
Post Reply