8.8.15 Patch32 message log filling on Centos
8.8.15 Patch32 message log filling on Centos
Hi
Latest 8.8.15 OSE Patch 32 (patched again to fix LDAP issue) is filling the /var/log/messages on Centos 7
The message log is filled with entries like this:
Jun 18 11:11:35 store.xxxxxx.co.nz mailboxd: INFO [mailboxd.csv] [] slogger - ....
Jun 18 11:28:35 store.xxxxxx.co.nz mailboxd: INFO [soap.csv] [] slogger - soap.csv: ...
Jun 18 11:28:35 store.xxxxxx.co.nz mailboxd: INFO [ldap.csv] [] slogger - ldap.csv: ...
These entries were not present before P32
Bug number: ZBUG-2835 though relevant does not seem to work with Centos. I cannot work out how to apply the second change "2. Update line no 210 and remove SLOGGER from statement. "
/opt/zimbra/conf/log4j.properties.in line 210: %%comment VAR:!zimbraLogHostname%%logger.slogger.level=INFO
Thanks
Geoffrey Mills
Latest 8.8.15 OSE Patch 32 (patched again to fix LDAP issue) is filling the /var/log/messages on Centos 7
The message log is filled with entries like this:
Jun 18 11:11:35 store.xxxxxx.co.nz mailboxd: INFO [mailboxd.csv] [] slogger - ....
Jun 18 11:28:35 store.xxxxxx.co.nz mailboxd: INFO [soap.csv] [] slogger - soap.csv: ...
Jun 18 11:28:35 store.xxxxxx.co.nz mailboxd: INFO [ldap.csv] [] slogger - ldap.csv: ...
These entries were not present before P32
Bug number: ZBUG-2835 though relevant does not seem to work with Centos. I cannot work out how to apply the second change "2. Update line no 210 and remove SLOGGER from statement. "
/opt/zimbra/conf/log4j.properties.in line 210: %%comment VAR:!zimbraLogHostname%%logger.slogger.level=INFO
Thanks
Geoffrey Mills
Re: 8.8.15 Patch32 message log filling on Centos
Apologies for the multiple posting, I was getting an SQL error on posting but turns out the messages were being posted.
Maybe an administrator can purge the duplicates?
Geoffrey Mills
Maybe an administrator can purge the duplicates?
Geoffrey Mills
Re: 8.8.15 Patch32 message log filling on Centos
Try to restart the Zimbra services after updating to the latest patch.
su - zimbra
zmcontrol restart
Thanks
su - zimbra
zmcontrol restart
Thanks
Re: 8.8.15 Patch32 message log filling on Centos
Solved, workaround the excess logging in /var/log/messages on Centos 7
Reference: https://wiki.zimbra.com/wiki/Log4j2
Reference: https://wiki.zimbra.com/wiki/Current_Known_Issues
Bug number: ZBUG-2835
1. Open /opt/zimbra/conf/log4j.properties.in and update line no 195 and change value to info from debug.
2. Update line no 210 and remove SLOGGER from statement.
I managed to "decode" this line: '2. Update line no 210 and remove SLOGGER from statement'
In the file "/opt/zimbra/conf/log4j.properties.in" locate the block shown below and comment this block out using # prefix:
%%comment VAR:!zimbraLogHostname%%logger.slogger.name = zimbra.slogger
%%comment VAR:!zimbraLogHostname%%logger.slogger.level=INFO
%%uncomment VAR:!zimbraLogHostname%%logger.slogger.level=ERROR
%%comment VAR:!zimbraLogHostname%%logger.slogger.additivity = false
%%comment VAR:!zimbraLogHostname%%logger.slogger.appenderRef.SLOGGER.ref = sloggerAppender
MUST restart mailbox - zmmailboxdctl restart
"/opt/zimbra/conf/log4j.properties.in" has appenders defined for both syslog (LOCAL0) and slogger (LOCAL1) for mailboxd traffic
If there is a better way please do let me know
Regards
Geoffrey Mills
Reference: https://wiki.zimbra.com/wiki/Log4j2
Reference: https://wiki.zimbra.com/wiki/Current_Known_Issues
Bug number: ZBUG-2835
1. Open /opt/zimbra/conf/log4j.properties.in and update line no 195 and change value to info from debug.
2. Update line no 210 and remove SLOGGER from statement.
I managed to "decode" this line: '2. Update line no 210 and remove SLOGGER from statement'
In the file "/opt/zimbra/conf/log4j.properties.in" locate the block shown below and comment this block out using # prefix:
%%comment VAR:!zimbraLogHostname%%logger.slogger.name = zimbra.slogger
%%comment VAR:!zimbraLogHostname%%logger.slogger.level=INFO
%%uncomment VAR:!zimbraLogHostname%%logger.slogger.level=ERROR
%%comment VAR:!zimbraLogHostname%%logger.slogger.additivity = false
%%comment VAR:!zimbraLogHostname%%logger.slogger.appenderRef.SLOGGER.ref = sloggerAppender
MUST restart mailbox - zmmailboxdctl restart
"/opt/zimbra/conf/log4j.properties.in" has appenders defined for both syslog (LOCAL0) and slogger (LOCAL1) for mailboxd traffic
If there is a better way please do let me know
Regards
Geoffrey Mills
-
- Posts: 2
- Joined: Wed Jul 06, 2022 2:52 pm
Re: 8.8.15 Patch32 message log filling on Centos
Hello
I have a log4j issue after applying 8.8.15 Patch 32.
No logs are written to zimbra.log file but only in messages.
The log4j configuration fle is correctly set.
IIt seems that the logs are sent to Rsyslog with a wrong format.
The logs are correctly redirected to zimbra.log if comment the following lines in log4j.properties
appender.SYSLOG.layout.type = PatternLaout
appender.SYSLOG.layout.pattern = imailboxd: %-5p [%t] [%z] %c{1} - %m
My system is CentOS 7 with Rsyslog 8.24.0.57
Thanks
Jerome
I have a log4j issue after applying 8.8.15 Patch 32.
No logs are written to zimbra.log file but only in messages.
The log4j configuration fle is correctly set.
IIt seems that the logs are sent to Rsyslog with a wrong format.
The logs are correctly redirected to zimbra.log if comment the following lines in log4j.properties
appender.SYSLOG.layout.type = PatternLaout
appender.SYSLOG.layout.pattern = imailboxd: %-5p [%t] [%z] %c{1} - %m
My system is CentOS 7 with Rsyslog 8.24.0.57
Thanks
Jerome
Last edited by jerome3859 on Thu Jul 07, 2022 9:48 am, edited 2 times in total.
Re: 8.8.15 Patch32 message log filling on Centos
Hi,
Sounds more like an OS issue, except you only updated zimbra.
Check rsyslog? /etc/rsyslog.conf and /etc/rsyslogconf.d/, try to follow the logging process
My /etc/rsyslog.conf has these entries at the end of the file:
local0.* -/var/log/zimbra.log
local1.* -/var/log/zimbra-stats.log
auth.* -/var/log/zimbra.log
mail.* -/var/log/zimbra.log
has /var/log/zimbra-stat.log got data coming in (local1.*)
And in /opt/zimbra/conf/log4j.provider:
# Syslog appender
appender.SYSLOG.type = Syslog
appender.SYSLOG.name = syslogAppender
appender.SYSLOG.host = localhost <-- in /etc/hosts?
appender.SYSLOG.port = 514 <-- not blocked?
appender.SYSLOG.protocol = UDP
appender.SYSLOG.facility = LOCAL0
appender.SYSLOG.layout.type = PatternLayout
appender.SYSLOG.layout.pattern = mailboxd: %-5p [%t] [%z] %c{1} - %m
# Logger service appender
appender.SLOGGER.type = Syslog
appender.SLOGGER.name = sloggerAppender
appender.SLOGGER.host = store.xxxxx.co.nz
appender.SLOGGER.facility = LOCAL1
appender.SLOGGER.port = 514
appender.SLOGGER.protocol = UDP
appender.SLOGGER.layout.type = PatternLayout
appender.SLOGGER.layout.pattern = mailboxd: %-5p [%t] [%z] %c{1} - %m
Are the log files in /opt/zimbra/log/ working?
Restart rsyslog service and look for errors in /var/log/messages?
I also ran zmfixperms at some stage https://wiki.zimbra.com/wiki/Fix_the_Zi ... ermissions
Suspect you have looked at most of these though...
I vaguely remember some Zimbra Wiki pages on getting logging working properly in multi-server environment, had to use these when I set Zimbra up
Regards
Geoffrey
Sounds more like an OS issue, except you only updated zimbra.
Check rsyslog? /etc/rsyslog.conf and /etc/rsyslogconf.d/, try to follow the logging process
My /etc/rsyslog.conf has these entries at the end of the file:
local0.* -/var/log/zimbra.log
local1.* -/var/log/zimbra-stats.log
auth.* -/var/log/zimbra.log
mail.* -/var/log/zimbra.log
has /var/log/zimbra-stat.log got data coming in (local1.*)
And in /opt/zimbra/conf/log4j.provider:
# Syslog appender
appender.SYSLOG.type = Syslog
appender.SYSLOG.name = syslogAppender
appender.SYSLOG.host = localhost <-- in /etc/hosts?
appender.SYSLOG.port = 514 <-- not blocked?
appender.SYSLOG.protocol = UDP
appender.SYSLOG.facility = LOCAL0
appender.SYSLOG.layout.type = PatternLayout
appender.SYSLOG.layout.pattern = mailboxd: %-5p [%t] [%z] %c{1} - %m
# Logger service appender
appender.SLOGGER.type = Syslog
appender.SLOGGER.name = sloggerAppender
appender.SLOGGER.host = store.xxxxx.co.nz
appender.SLOGGER.facility = LOCAL1
appender.SLOGGER.port = 514
appender.SLOGGER.protocol = UDP
appender.SLOGGER.layout.type = PatternLayout
appender.SLOGGER.layout.pattern = mailboxd: %-5p [%t] [%z] %c{1} - %m
Are the log files in /opt/zimbra/log/ working?
Restart rsyslog service and look for errors in /var/log/messages?
I also ran zmfixperms at some stage https://wiki.zimbra.com/wiki/Fix_the_Zi ... ermissions
Suspect you have looked at most of these though...
I vaguely remember some Zimbra Wiki pages on getting logging working properly in multi-server environment, had to use these when I set Zimbra up
Regards
Geoffrey
-
- Posts: 2
- Joined: Wed Jul 06, 2022 2:52 pm
Re: 8.8.15 Patch32 message log filling on Centos
Hello Geoffroy
Thans for you advice
But in fact, it's a PatternLayout issue
The faliciy is not trasmitted by default
The pattern must be changed to
appender.SYSLOG.layout.pattern = <134> %d{MMM dd HH:mm:ss} ${hostName} mailboxd: %-5p [%t] [%z] %c{1} - %m
But some issue persist
In release 8.8.15 P31 the zimbra.log contains all mailboxd messages
And qith the new Patch these messages are only store in the /opt/zimbra/log/mailboxd.log file
Perhaps something missing at the rootlogger level
I only have these 2 lines
rootLogger.level=INFO
rootLogger.appenderRef.LOGFILE.ref = mailboxFile
Thanks
Jerome
Thans for you advice
But in fact, it's a PatternLayout issue
The faliciy is not trasmitted by default
The pattern must be changed to
appender.SYSLOG.layout.pattern = <134> %d{MMM dd HH:mm:ss} ${hostName} mailboxd: %-5p [%t] [%z] %c{1} - %m
But some issue persist
In release 8.8.15 P31 the zimbra.log contains all mailboxd messages
And qith the new Patch these messages are only store in the /opt/zimbra/log/mailboxd.log file
Perhaps something missing at the rootlogger level
I only have these 2 lines
rootLogger.level=INFO
rootLogger.appenderRef.LOGFILE.ref = mailboxFile
Thanks
Jerome
Re: 8.8.15 Patch32 message log filling on Centos
Hi Jerome,
OK, my 2cents worth and I could be totally wrong so ignore me if required.
https://wiki.zimbra.com/wiki/Log_Files
The MTA (postfix) logs via syslog to the mail facility. By default, this goes to /var/log/maillog, but Zimbra sets it to /var/log/zimbra.log
On my Zimbra store server, use zimbra "postconf" utility to display logging destination for postfix:
[zimbra@store conf]$ /opt/zimbra/common/sbin/postconf maillog_file
maillog_file =
The "maillog_file=" is empty therefore postfix is logging to syslog
Is our Centos logging working? https://www.poftut.com/linux-logger-com ... -examples/
Looking at the end of /etc/rsyslog.conf, "local0" logs to /var/log/zimbra.log
Test it:
logger -p "local0.info" Test log to local0.info
This message is displayed in /var/log/zimbra.log, so logging to local0 is working
Reset logging for multi-server setup:
https://wiki.zimbra.com/wiki/Configuring-Logger-Host
is Zimbra zmlogger running?
PS: Are you using Centos syslog or rsyslog - I remember I changed my one over to rsyslog
PPS: selinux running - caused me many issues originally - /opt/zimbra/libexec/zmfixperms -e -v
/opt/zimbra/log/mailbox.log: <-- mailboxd logs, traffic volumes, etc but not mail send/receive data
2022-07-09 09:47:38,045 INFO [MailboxPurge] [name=primesure@gamix.co.nz;mid=17;] purge - Purging messages.
2022-07-09 09:47:56,916 INFO [mailboxd.csv] [] slogger - mailboxd.csv:
/var/log/zimbra.log:
Jul 9 09:50:46 ldap zmconfigd[1698]: Watchdog: skipping service antivirus. Service not yet available for restarts.
Jul 9 09:50:46 ldap zmconfigd[1698]: All rewrite threads completed in 0.00 sec
Jul 9 09:50:46 ldap zmconfigd[1698]: All restarts completed in 0.00 sec
Jul 9 09:50:48 store /postfix-script[18264]: the Postfix mail system is running: PID: 11811
Jul 9 10:20:15 store postfix/qmgr[11813]: 6B44823B16D5: from=<admin@gamix.co.nz>, size=2495, nrcpt=1 (queue active)
Jul 9 10:20:15 store amavis[12695]: (12695-14) Passed CLEAN {RelayedOutbound},
Regards
Geoffrey
OK, my 2cents worth and I could be totally wrong so ignore me if required.
OK, this redirects (appends) data from existing logs into SYSLOG but this is the *wrong* data.The pattern must be changed to
appender.SYSLOG.layout.pattern = <134> %d{MMM dd HH:mm:ss} ${hostName} mailboxd: %-5p [%t] [%z] %c{1} - %m
I think postfix, amavis, etc log *direct* to SYSLOG (via local0), not sure if they use the java log4j logging. These do not log in /opt/zimbra/log/mailbox.log anyway.No logs are written to zimbra.log file but only in messages.
https://wiki.zimbra.com/wiki/Log_Files
The MTA (postfix) logs via syslog to the mail facility. By default, this goes to /var/log/maillog, but Zimbra sets it to /var/log/zimbra.log
On my Zimbra store server, use zimbra "postconf" utility to display logging destination for postfix:
[zimbra@store conf]$ /opt/zimbra/common/sbin/postconf maillog_file
maillog_file =
The "maillog_file=" is empty therefore postfix is logging to syslog
Is our Centos logging working? https://www.poftut.com/linux-logger-com ... -examples/
Looking at the end of /etc/rsyslog.conf, "local0" logs to /var/log/zimbra.log
Test it:
logger -p "local0.info" Test log to local0.info
This message is displayed in /var/log/zimbra.log, so logging to local0 is working
Reset logging for multi-server setup:
https://wiki.zimbra.com/wiki/Configuring-Logger-Host
is Zimbra zmlogger running?
PS: Are you using Centos syslog or rsyslog - I remember I changed my one over to rsyslog
PPS: selinux running - caused me many issues originally - /opt/zimbra/libexec/zmfixperms -e -v
/opt/zimbra/log/mailbox.log: <-- mailboxd logs, traffic volumes, etc but not mail send/receive data
2022-07-09 09:47:38,045 INFO [MailboxPurge] [name=primesure@gamix.co.nz;mid=17;] purge - Purging messages.
2022-07-09 09:47:56,916 INFO [mailboxd.csv] [] slogger - mailboxd.csv:
/var/log/zimbra.log:
Jul 9 09:50:46 ldap zmconfigd[1698]: Watchdog: skipping service antivirus. Service not yet available for restarts.
Jul 9 09:50:46 ldap zmconfigd[1698]: All rewrite threads completed in 0.00 sec
Jul 9 09:50:46 ldap zmconfigd[1698]: All restarts completed in 0.00 sec
Jul 9 09:50:48 store /postfix-script[18264]: the Postfix mail system is running: PID: 11811
Jul 9 10:20:15 store postfix/qmgr[11813]: 6B44823B16D5: from=<admin@gamix.co.nz>, size=2495, nrcpt=1 (queue active)
Jul 9 10:20:15 store amavis[12695]: (12695-14) Passed CLEAN {RelayedOutbound},
Regards
Geoffrey
Re: 8.8.15 Patch32 message log filling on Centos
Hi Jerome,
Did you find a solution to this bug? It's a bug since zimbraLogToSyslog is a ZCS feature and it's not working anymore.
The only way I could get some logs coming to the Syslog server is with the zimbra.security logger and by commenting the 2 layout Appender lines. But poor logs then (with thread names, IP adresses, user agent, etc.).
Since log4j2 sometimes manages to send logs to Syslog, there has to be a way to set a pattern so that we can continue receiving rich / extended logs through Syslog.
I've opened a support case this morning, I'll let you know how it goes.
Regards,
Blue.
Did you find a solution to this bug? It's a bug since zimbraLogToSyslog is a ZCS feature and it's not working anymore.
The only way I could get some logs coming to the Syslog server is with the zimbra.security logger and by commenting the 2 layout Appender lines. But poor logs then (with thread names, IP adresses, user agent, etc.).
Since log4j2 sometimes manages to send logs to Syslog, there has to be a way to set a pattern so that we can continue receiving rich / extended logs through Syslog.
I've opened a support case this morning, I'll let you know how it goes.
Regards,
Blue.
Re: 8.8.15 Patch32 message log filling on Centos
Did any solution comes up with your support case? Can not send our mailbox logs to rsyslog after applying patch 32.
Thanks
Gokalp
Thanks
Gokalp