Missing activiy, logs and status not being updated anymore

MaxAuray · Post by **MaxAuray** » Mon Jun 03, 2019 7:58 am

Hi all,

My server was attacked recently through CVE-2019-9670. I finally patch my server to 8.7.11_GA_3800.NETWORK (patch 11). Since, I cannot have my services status updated and server statistics remain empty.

I verified zmlogswatch.out and nothing seems abnormal to me.

Code: Select all

zmrrdfetch server pid: 8304
utime > 15ms (0.02): line: [Jun  3 05:31:30 mail zimbramon[10327]: :::14190D54-85B0-11E9-842F-9242188343F4:::,0,0,0,0,3,155829800,0,23,0,379,391,611,0,379,398,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1031328,0,0,0,0,0,1,90014,90014,54143836,13764513,854991,AUTH_MASTER,0,0,2470,0,2133,0,0.000,0,0,0,OFF,0,0,0,0,457,53,1122,1838,6555952,9,0,0,0,,,0,0,0,0,0,0,0,,,0,0,NONE,0,0,0,0,0,0,0,,0,1335,2,21422,0,0,4096,0,0,0,0,3,3,1,50818,50818,18446744073709551615,0,,Disconnected,OFF,0,18446744073709551615,,,,OFF,0]
utime > 15ms (0.0199999999999999): line: [Jun  3 05:51:01 mail zimbramon[10327]: :::CDB6E810-85B2-11E9-842F-9242188343F4:::,0,0,0,0,3,155829800,0,23,0,379,391,615,0,379,398,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1031328,0,0,0,0,0,1,91060,91060,54802885,13799123,874686,AUTH_MASTER,0,0,2483,0,2184,0,0.000,0,0,0,OFF,0,0,0,0,469,53,1154,1876,6555953,9,0,0,0,,,0,0,0,0,0,0,0,,,0,0,NONE,0,0,0,0,0,0,0,,0,1335,2,21639,0,0,4096,0,0,0,0,3,3,1,51989,51989,18446744073709551615,0,,Disconnected,OFF,0,18446744073709551615,,,,OFF,0]
utime > 15ms (0.02): line: [Jun  3 06:03:30 mail zimbramon[10327]: :::8C5AC0A6-85B4-11E9-842F-9242188343F4:::,0,0,0,0,3,155829800,0,23,0,379,391,615,0,379,398,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1031328,0,0,0,0,0,1,91662,91662,55218388,13927164,887311,AUTH_MASTER,0,0,2492,0,2216,0,0.000,0,0,0,OFF,0,0,0,0,476,53,1176,1902,6555953,9,0,0,0,,,0,0,0,0,0,0,0,,,0,0,NONE,0,0,0,0,0,0,0,,0,1335,2,21773,0,0,4096,0,0,0,0,3,3,1,52738,52738,18446744073709551615,0,,Disconnected,OFF,0,18446744073709551615,,,,OFF,0]
utime > 15ms (0.0199999999999998): line: [Jun  3 06:18:00 mail zimbramon[10327]: :::92E3457C-85B6-11E9-842F-9242188343F4:::,0,0,0,0,3,155829800,0,23,0,379,391,617,0,379,398,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1031328,0,0,0,0,0,1,92271,92271,55598860,14056259,901956,AUTH_MASTER,0,0,2505,0,2247,0,0.000,0,0,0,OFF,0,0,0,0,478,54,1196,1931,6655427,9,0,0,0,,,0,0,0,0,0,0,0,,,0,0,NONE,0,0,0,0,0,0,0,,0,1335,2,21886,0,0,4096,0,0,0,0,3,3,1,53608,53608,18446744073709551615,0,,Disconnected,OFF,0,18446744073709551615,,,,OFF,0]
Mon Jun  3 08:06:18 2019: Spawned: 18573 -- Connected from 127.0.0.1:55669
Mon Jun  3 08:06:18 2019: Spawned: 18574 -- Connected from 127.0.0.1:55670
Mon Jun  3 08:06:18 2019: Spawned: 18575 -- Connected from 127.0.0.1:55671
Mon Jun  3 08:06:18 2019: Spawned: 18578 -- Connected from 127.0.0.1:55672
Mon Jun  3 08:06:18 2019: Spawned: 18579 -- Connected from 127.0.0.1:55673
Mon Jun  3 08:06:18 2019: Spawned: 18829 -- Connected from 127.0.0.1:55674
Mon Jun  3 08:06:18 2019: Spawned: 18830 -- Connected from 127.0.0.1:55675
Mon Jun  3 08:06:18 2019: Spawned: 18831 -- Connected from 127.0.0.1:55676
Mon Jun  3 08:06:18 2019: Spawned: 18832 -- Connected from 127.0.0.1:55677
Mon Jun  3 08:06:18 2019: Spawned: 18833 -- Connected from 127.0.0.1:55678
Mon Jun  3 08:06:19 2019: Spawned: 19079 -- Connected from 127.0.0.1:55679
Mon Jun  3 08:06:21 2019: Spawned: 19179 -- Connected from 127.0.0.1:55680
Mon Jun  3 08:06:21 2019: Spawned: 19180 -- Connected from 127.0.0.1:55681
Mon Jun  3 08:06:21 2019: Spawned: 19181 -- Connected from 127.0.0.1:55682
Mon Jun  3 08:06:21 2019: Spawned: 19182 -- Connected from 127.0.0.1:55683
Mon Jun  3 08:06:28 2019: Spawned: 19538 -- Connected from 127.0.0.1:55686
Mon Jun  3 08:06:50 2019: Spawned: 19968 -- Connected from 127.0.0.1:55693
Mon Jun  3 08:06:52 2019: Spawned: 20020 -- Connected from 127.0.0.1:55694
Mon Jun  3 08:06:52 2019: Spawned: 20022 -- Connected from 127.0.0.1:55695
Mon Jun  3 08:06:52 2019: Spawned: 20023 -- Connected from 127.0.0.1:55696
Mon Jun  3 08:06:52 2019: Spawned: 20024 -- Connected from 127.0.0.1:55697
Mon Jun  3 08:29:46 2019: Spawned: 5117 -- Connected from 127.0.0.1:56064
Mon Jun  3 08:29:46 2019: Spawned: 5132 -- Connected from 127.0.0.1:56065
Mon Jun  3 08:29:46 2019: Spawned: 5134 -- Connected from 127.0.0.1:56066
Mon Jun  3 08:29:46 2019: Spawned: 5143 -- Connected from 127.0.0.1:56067
Mon Jun  3 08:29:52 2019: Spawned: 5327 -- Connected from 127.0.0.1:56078
Mon Jun  3 08:29:55 2019: Spawned: 5380 -- Connected from 127.0.0.1:56081
Mon Jun  3 08:29:55 2019: Spawned: 5382 -- Connected from 127.0.0.1:56082
Mon Jun  3 08:29:55 2019: Spawned: 5383 -- Connected from 127.0.0.1:56083
Mon Jun  3 08:29:55 2019: Spawned: 5384 -- Connected from 127.0.0.1:56084
Mon Jun  3 08:48:27 2019: Spawned: 17314 -- Connected from 127.0.0.1:56330
Mon Jun  3 08:48:27 2019: Spawned: 17319 -- Connected from 127.0.0.1:56331
Mon Jun  3 08:48:27 2019: Spawned: 17321 -- Connected from 127.0.0.1:56332
Mon Jun  3 08:48:27 2019: Spawned: 17331 -- Connected from 127.0.0.1:56333
Mon Jun  3 08:48:34 2019: Spawned: 17599 -- Connected from 127.0.0.1:56339
Mon Jun  3 08:48:37 2019: Spawned: 17650 -- Connected from 127.0.0.1:56341
Mon Jun  3 08:48:37 2019: Spawned: 17651 -- Connected from 127.0.0.1:56342
Mon Jun  3 08:48:37 2019: Spawned: 17652 -- Connected from 127.0.0.1:56343
Mon Jun  3 08:48:37 2019: Spawned: 17653 -- Connected from 127.0.0.1:56344
Mon Jun  3 08:48:44 2019: Spawned: 17850 -- Connected from 127.0.0.1:56349
Mon Jun  3 08:48:44 2019: Spawned: 17852 -- Connected from 127.0.0.1:56350
Mon Jun  3 08:48:44 2019: Spawned: 17853 -- Connected from 127.0.0.1:56351
Mon Jun  3 08:48:44 2019: Spawned: 17854 -- Connected from 127.0.0.1:56352
Mon Jun  3 08:48:48 2019: Spawned: 18051 -- Connected from 127.0.0.1:56353
Mon Jun  3 08:48:48 2019: Spawned: 18052 -- Connected from 127.0.0.1:56354
Mon Jun  3 08:48:48 2019: Spawned: 18053 -- Connected from 127.0.0.1:56355

Also checked /var/log/zimbra-stats.log, same thing, seems normal. Noticed zimbra-stats.log is owned by syslog:adm, not sure if this is normal or not (not zimbra:zimbra). Nevertheless, zimbra user can read zimbra-stats.log seamlessly.

Code: Select all

Jun  3 09:23:00 mail zimbramon[10327]: 10327:info: :::6AF561C0-85D0-11E9-842F-9242188343F4:::Performance_schema_table_handles_lost, Performance_schema_table_instances_lost, Performance_schema_thread_classes_lost, Performance_schema_thread_instances_lost, Performance_schema_users_lost, Prepared_stmt_count, Qcache_free_blocks, Qcache_free_memory, Qcache_hits, Qcache_inserts, Qcache_lowmem_prunes, Qcache_not_cached, Qcache_queries_in_cache, Qcache_total_blocks, Queries, Questions, Rows_read, Rows_sent, Rows_tmp_read, Rpl_status, Select_full_join, Select_full_range_join, Select_range, Select_range_check, Select_scan, Slave_connections, Slave_heartbeat_period, Slave_open_temp_tables, Slave_received_heartbeats, Slave_retried_transactions, Slave_running, Slave_skipped_errors, Slaves_connected, Slaves_running, Slow_launch_threads, Slow_queries, Sort_merge_passes, Sort_priority_queue_sorts:::6AF562E2-85D0-11E9-842F-9242188343F4:::
Jun  3 09:23:00 mail zimbramon[10327]: 10327:info: :::6AF562E2-85D0-11E9-842F-9242188343F4:::, Sort_range, Sort_rows, Sort_scan, Ssl_accept_renegotiates, Ssl_accepts, Ssl_callback_cache_hits, Ssl_cipher, Ssl_cipher_list, Ssl_client_connects, Ssl_connect_renegotiates, Ssl_ctx_verify_depth, Ssl_ctx_verify_mode, Ssl_default_timeout, Ssl_finished_accepts, Ssl_finished_connects, Ssl_server_not_after, Ssl_server_not_before, Ssl_session_cache_hits, Ssl_session_cache_misses, Ssl_session_cache_mode, Ssl_session_cache_overflows, Ssl_session_cache_size, Ssl_session_cache_timeouts, Ssl_sessions_reused, Ssl_used_session_cache_entries, Ssl_verify_depth, Ssl_verify_mode, Ssl_version, Subquery_cache_hit, Subquery_cache_miss, Syncs, Table_locks_immediate, Table_locks_waited, Tc_log_max_pages_used, Tc_log_page_size, Tc_log_page_waits, Threadpool_idle_threads, Threadpool_threads, Threads_cached, Thr:::6AF563FA-85D0-11E9-842F-9242188343F4:::
Jun  3 09:23:00 mail zimbramon[10327]: 10327:info: :::6AF563FA-85D0-11E9-842F-9242188343F4:::eads_connected, Threads_created, Threads_running, Uptime, Uptime_since_flush_status, wsrep_cluster_conf_id, wsrep_cluster_size, wsrep_cluster_state_uuid, wsrep_cluster_status, wsrep_connected, wsrep_local_bf_aborts, wsrep_local_index, wsrep_provider_name, wsrep_provider_vendor, wsrep_provider_version, wsrep_ready, wsrep_thread_count:: 06/03/2019 09:23:00,0,0,0,0,6,0,0,2,0,0,0,8,0,0,0,0,0,,0,0,0,0,0,0,0.000000,8037764,790786890,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,13866,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,474,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,465,462,0,0,0,0,0,0,0,0,0,0,0,0,0,1246,0,0,0,0,0,0,16244,0,0,18767,48732,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1078,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3157,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4672,20,0,0,0,0,0,0,OFF,0,0,0,0,0,0,3162,0.000000,:::6AF5651C-85D0-11E9-842F-9242188343F4:::
Jun  3 09:23:00 mail zimbramon[10327]: 10327:info: :::6AF5651C-85D0-11E9-842F-9242188343F4:::20,124,2518,0,0,0,7168,0,0,0,0,0,0,0,3591,0,0,0,1,37816,558,0,0,109397,108505,0,0,0,0,147,30859,0,56924614,1033,7510906,0,1460679,1299,0,0,1,1090135,5758,2259,128,64136,671612928,0,Dumping buffer pool(s) not yet started,Loading buffer pool(s) not yet started,40992,0,20951,575115,0,0,12,653,15286,616760,7491,0,0,56312915,33328,0,56886,0,848635454,11688,0,0,0,670142464,42743,26695,691470848,20951,1771,0,ON,2392,0,0,0,0,140,47,92,106,2,1,0,6554,2192,8827643834,8827643834,8827643834,2336,61805,30728648,170689648,51312953,10421010432,296,10185,1254,0,3944,0,0,4051456,16384,95,40897,20951,30728642,0,432,0,0,0,0,0,1248,1674,64820334,5758,0,0,0,0,2742,98928,3646,0,70,2176,12,0,0,0,0,0,0,0,0,10779,15916,0,0,0,0,0,0,0,OFF,OFF,OFF,ON,OFF,0,0,0,0,0,0,600233,0,0,0,0,0,0,0,0,0,0,0,0,0,107163,1,0,3,0,6,0:::6AF5662A-85D0-11E9-842F-9242188343F4:::
Jun  3 09:23:00 mail zimbramon[10327]: :::6AF5662A-85D0-11E9-842F-9242188343F4:::,0.000000,0,0,0,0,3,155963480,0,23,0,383,398,635,0,383,405,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1031328,0,0,0,0,0,1,111342,111342,64814244,17557884,1088806,AUTH_MASTER,0,0,2875,0,2695,0,0.000,0,0,0,OFF,0,0,0,0,558,59,1412,2343,7504567,9,0,0,0,,,0,0,0,0,0,0,0,,,0,0,NONE,0,0,0,0,0,0,0,,0,1335,2,26013,0,0,4096,0,0,0,0,3,3,1,64708,64708,18446744073709551615,0,,Disconnected,OFF,0,18446744073709551615,,,,OFF,0
Jun  3 09:23:02 mail zimbramon[10314]: 10314:info: zmstat vm.csv: timestamp, r, b, swpd, free, buff, cache, si, so, bi, bo, in, cs, us, sy, id, wa, st, MemTotal, MemFree, MemAvailable, Buffers, Cached, SwapCached, Active, Inactive, Active(anon), Inactive(anon), Active(file), Inactive(file), Unevictable, Mlocked, SwapTotal, SwapFree, Dirty, Writeback, AnonPages, Mapped, Shmem, Slab, SReclaimable, SUnreclaim, KernelStack, PageTables, NFS_Unstable, Bounce, WritebackTmp, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, HardwareCorrupted, DirectMap4k, DirectMap2M, DirectMap1G, loadavg:: 06/03/2019 09:23:01, 2, 0, 47756, 2626024, 1012044, 22533072, 0, 0, 1, 200, 4309, 5804, 4, 7, 86, 2, 0, 32894444, 2626236, 24408748, 1012044, 22022660, 1444, 8532984, 20837288, 6794888, 1131844, 1738096, 19705444, 0, 0, 2616300, 2568544, 752, :::6B737D4E-85D0-11E9-8D7B-F5B8A60F866D:::
Jun  3 09:23:02 mail zimbramon[10314]: :::6B737D4E-85D0-11E9-8D7B-F5B8A60F866D:::0, 6334528, 103704, 1591144, 510412, 424964, 85448, 9456, 30348, 0, 0, 0, 19063520, 24260892, 34359738367, 344960, 34359178892, 0, 9512, 1982464, 33554432, 2.20
Jun  3 09:23:11 mail zimbramon[11006]: 11006:info: zmstat fd.csv: timestamp, fd_count, mailboxd_fd_count:: 06/03/2019 09:23:11, 4736, 1611
Jun  3 09:23:14 mail zimbramon[10331]: 10331:info: zmstat nginx.csv: timestamp, utime, stime, cputime, rchar, wchar, read_bytes, write_bytes, rss, processes, threads:: 06/03/2019 09:23:14, 4, 0, 4, 80137, 75906, 0, 12288, 48500, 5, 5

Also verified /opt/zimbra/logger/db/data/logger.sqlitedb with sqlite3, data seem consistent.

I finally found this as being abnormal:

$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog

Code: Select all

1559546130,,,,,,,,,,,,,,,,,,
1559546160,,,,,,,,,,,,,,,,,,
1559546190,,,,,,,,,,,,,,,,,,
1559546220,,,,,,,,,,,,,,,,,,
1559546250,,,,,,,,,,,,,,,,,,
1559546280,,,,,,,,,,,,,,,,,,
1559546310,,,,,,,,,,,,,,,,,,
1559546340,,,,,,,,,,,,,,,,,,
1559546370,,,,,,,,,,,,,,,,,,
1559546400,,,,,,,,,,,,,,,,,,
1559546430,,,,,,,,,,,,,,,,,,
1559546460,,,,,,,,,,,,,,,,,,
1559546490,,,,,,,,,,,,,,,,,,
1559546520,,,,,,,,,,,,,,,,,,
1559546550,,,,,,,,,,,,,,,,,,
1559546580,,,,,,,,,,,,,,,,,,
1559546610,,,,,,,,,,,,,,,,,,
1559546640,,,,,,,,,,,,,,,,,,
1559546670,,,,,,,,,,,,,,,,,,
1559546700,,,,,,,,,,,,,,,,,,
1559546730,,,,,,,,,,,,,,,,,,
1559546760,,,,,,,,,,,,,,,,,,
1559546790,,,,,,,,,,,,,,,,,,
1559546820,,,,,,,,,,,,,,,,,,
1559546850,,,,,,,,,,,,,,,,,,
1559546880,,,,,,,,,,,,,,,,,,
1559546910,,,,,,,,,,,,,,,,,,
1559546940,,,,,,,,,,,,,,,,,,
1559546970,,,,,,,,,,,,,,,,,,
1559547000,,,,,,,,,,,,,,,,,,
1559547030,,,,,,,,,,,,,,,,,,
1559547060,,,,,,,,,,,,,,,,,,

I would there expect lines like below:

Code: Select all

1559547000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1559547030,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1559547060,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1

Any idea, please?

Thanks.