Updated live sync script

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Updated live sync script

Post by zimico »

Dear,

I set up live sync for OSE NE (trial) 8.7.11 on Centos 7.
Failover server seems to be synced ok when I check it. However in the log file I see the following:
[zimbra@mail live_sync]$ tail -f log/live_sync.log
***************************************
Sun Jun 18 07:36:16 +07 2017 :Starting live_syncd
Sun Jun 18 07:36:16 +07 2017 :Incremental backups enabled : true
Sun Jun 18 07:36:16 +07 2017 :Convertd enabled : true
Sun Jun 18 07:36:16 +07 2017 :Starting redo log live sync process
Sun Jun 18 07:36:16 +07 2017 :Starting ldap live sync process
Sun Jun 18 07:36:16 +07 2017 :Detecting if HSM used
Sun Jun 18 07:37:41 +07 2017 :No HSM Detected
Sun Jun 18 07:37:41 +07 2017 :Syncing redologs...
Sun Jun 18 07:37:41 +07 2017 :Syncing incremental backup file: /opt/zimbra/backup/sessions/incr-20170617.180020.104/redologs/redo-20170616.172226.447-seq0.log
Sun Jun 18 07:37:46 +07 2017 :Syncing incremental backup file: /opt/zimbra/backup/sessions/incr-20170617.180020.104/redologs/redo-20170616.181004.169-seq1.log
Sun Jun 18 07:37:46 +07 2017 :Syncing incremental backup file: /opt/zimbra/backup/sessions/incr-20170617.180020.104/redologs/redo-20170616.184905.562-seq2.log
Sun Jun 18 07:37:47 +07 2017 :Syncing incremental backup file: /opt/zimbra/backup/sessions/incr-20170617.180020.104/redologs/redo-20170617.180021.658-seq3.log
Sun Jun 18 07:37:47 +07 2017 :Syncing incremental backup file: /opt/zimbra/redolog/archive/redo-20170618.115357.012-seq10.log
Connection refused
usage: zmhsm <options>
-a,--abort Abort the current HSM session.
-h,--help Displays this help message.
-s,--server <arg> Mail server hostname. Default is localhost.
-t,--start Start the HSM process.
-u,--status Get status on the last HSM session.
Sun Jun 18 07:37:54 +07 2017 :Syncing /opt/zimbra/redolog/redo.log
Sun Jun 18 07:37:54 +07 2017 :Syncing /opt/zimbra/redolog/redo.log done
Sun Jun 18 07:37:54 +07 2017 :Syncing redologs done
Sun Jun 18 07:37:54 +07 2017 :Purging redolog directory and archives
Sun Jun 18 07:37:54 +07 2017 :Purge redolog directory and archives done
Sun Jun 18 07:37:59 +07 2017 :Replaying redologs...
Sun Jun 18 07:38:26 +07 2017 :Replaying redologs done
Sun Jun 18 07:38:26 +07 2017 :Live streaming redolog
Sun Jun 18 07:38:31 +07 2017 :Live streaming redolog established
Setting up watches. Beware: since -r was given, this may take a while!
I see Connection refused and don't know what does it mean. Could you please let me know about this warning/error?

Thank for your time and sharing.
Minh.
myzamri
Posts: 20
Joined: Tue Mar 28, 2017 2:46 am

Re: Updated live sync script

Post by myzamri »

I think you should debug and check the script to find out what command was running at that particular error.
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Updated live sync script

Post by zimico »

Hi all,

I managed to run live sync on zimbra 8.8 and centos 7. Because centos 7 has new ifconfig format such as name of interfce, so I updated the script part which relating to finding IP address
This is the old one

Code: Select all

#Find all local addresses
server_addresses=$( /sbin/ifconfig | grep inet | \
  grep -Eio "addr:[[:space:]]*(([0-9]+\.){3}[0-9]+|[0-9a-f]+(:[0-9a-f]*){5})" | \
  sed "s/addr://" | tr -d " \t" )
This is the new one:

Code: Select all

#Find all local addresses
server_addresses=$( /sbin/ifconfig | \
  grep -Eio "inet[[:space:]]*(([0-9]+\.){3}[0-9]+|[0-9a-f]+(:[0-9a-f]*){5})" | \
  sed "s/inet//" | tr -d " \t" )
Hope this help someone.
Regards,
T00thless
Posts: 1
Joined: Fri Apr 06, 2018 12:45 pm

Re: Updated live sync script

Post by T00thless »

HI Zimico,

Could you please share your script?
Mine got error when running ./live_syncd start

Code: Select all

bash: /opt/zimbra/live_sync/sync_commands: /bin/bash^M: bad interpreter: No such file or directory
Unable to run commands on remote server
Thanks,
Julius
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: Updated live sync script

Post by maxxer »

myzamri wrote:I increased the memory and the problem solved.
system memory or some zimbra param? thanks
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Updated live sync script

Post by zimico »

HI Zimico,

Could you please share your script?
Mine got error when running ./live_syncd start
You should be careful with typo and should follow carefully the wiki for all steps.
Here is my script:

Code: Select all

#!/bin/bash
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.

#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.

#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>

##########################################################################
# Title      :  live_syncd
# Author     :  Simon Blandford <simon -at- onepointltd -dt- com>
# Date       :  2013-03-12
# Requires   :  zimbra sync_commands inotify-tools
# Category   :  Administration
# Version    :  2.1.5
# Copyright  :  Simon Blandford, Onepoint Consulting Limited
# License    :  GPLv3 (see above)
##########################################################################
# Description
# Keep two Zimbra servers synchronised in near-realtime
##########################################################################


#******************************************************************************
#********************** Constants *********************************************
#******************************************************************************

LOG_LEVEL=5
REDO_LOG_HISTORY_DAYS=10
ERROR_CLEAR_MINUTES=10
LDAP_CHECK_MINUTES_INTERVAL=10

ZIMBRA_DIR="/opt/zimbra"
BASE_DIR="$ZIMBRA_DIR""/live_sync"
LOCKING_DIR="$BASE_DIR""/lock"
PID_DIR="$BASE_DIR""/pid"
LOG_DIR="$BASE_DIR""/log"
LOG_FILE="$LOG_DIR""/live_sync.log"
LDAP_TEMP_DIR="$BASE_DIR""/ldap"
LDAP_TEMP_LDIF="$BASE_DIR""/ldif.bak"
STATUS_DIR="$BASE_DIR""/status"
SSH_IDENTITY_FILE="$ZIMBRA_DIR""/.ssh/live_sync"
REDOLOG_DIR="$ZIMBRA_DIR""/redolog"
REDO_LOG_FILE="$REDOLOG_DIR""/redo.log"
ARCHIVE_DIR="$REDOLOG_DIR""/archive"
LIVE_SYNC_ARCHIVE_DIR="$REDOLOG_DIR""/live_sync_archives"
LDAP_DATA_DIR="$ZIMBRA_DIR""/data/ldap/"
BACKUP_DIR="$ZIMBRA_DIR""/backup"
SYNC_COMMANDS_SCRIPT="$BASE_DIR""/sync_commands"
SSH="ssh -i ""$SSH_IDENTITY_FILE"" -o StrictHostKeyChecking=no -o CheckHostIP=no"\
" -o PreferredAuthentications=hostbased,publickey"
LOCK_STATE_DIR="$LOCKING_DIR""/live_sync.lock"
STOP_FILE="$STATUS_DIR""/live_sync.stop"
LAST_GOOD_REDO_REPLAY="$STATUS_DIR""/last_good_redo_replay"
LAST_GOOD_REDO_SYNC="$STATUS_DIR""/last_good_redo_sync"
LAST_GOOD_REDO_STREAM="$STATUS_DIR""/last_good_redo_stream"
LAST_GOOD_LDAP_SYNC="$STATUS_DIR""/last_good_ldap_sync"
LAST_GOOD_LDAP_START="$STATUS_DIR""/last_good_ldap_start"
WATCHES_FILE="$STATUS_DIR""/watches"
PID_FILE_LDAP="$PID_DIR""/ldap_live_sync.pid"
PID_FILE_REDO="$PID_DIR""/redo_log_live_sync.pid"
CONF_FILE="$BASE_DIR""/live_sync.conf"

#******************************************************************************
#********************** Functions *********************************************
#******************************************************************************

#Format for log output with errors and warnings going to >&2
logit () {
  logit_1 () {
    echo -n "$( date ) :"
    case $ in
      1)
        echo -n "Error :"
        ;;
      2)
        echo -n "Warning :"
        ;;
      3)
        echo -n "Info :"
        ;;
      4)
        echo -n "Debug :"
        ;;
    esac
    echo $@
  }
  local msg_level output_chan
  if [ $1 -le $LOG_LEVEL ]; then
    msg_level=$1
    shift
    if [ $msg_level -le 2 ]; then
      logit_1 $@ >&2
    else
      logit_1 $@
    fi
  fi
}

#Detect HSM
detect_hsm () {
  local retval
  #LDAP must be running
  ldap status &>/dev/null || ldap start &>/dev/null
  #MySQL must be running
  mysql.server status &>/dev/null || mysql.server start &>/dev/null
  #Preserve mailbox running state
  zmmailboxdctl status &>/dev/null
  prev_zmmailbox_status=$?
  zmmailboxdctl start &>/dev/null
  zxsuite hsm getAllVolumes | grep "secondaries" >/dev/null
  retval=$?
   if [ $prev_zmmailbox_status -ne 0 ]; then
    zmmailboxdctl stop &>/dev/null
  fi
  return $retval
}

#Ensure ldap, convertd and mysql servers are running and then replay redo logs
replay_redo_logs () {
  local server_failed

  ldap status &>/dev/null || ldap start &>/dev/null
  mysql.server status &>/dev/null || mysql.server start &>/dev/null
  server_failed=0
  if ! ldap status &>/dev/null; then
    logit 1 "Start of local ldap server failed"
    ldap status >&2
    #Return error to trigger a "break" in while loop
    server_failed=1
  fi
  if ! mysql.server status &>/dev/null; then
    logit 1 "Start of local mysql server failed"
    mysql.server status >&2
    #Return error to trigger a "break" in while loop
    server_failed=1
  fi
  if [ "x""$convertd_enabled" == "xtrue" ]; then
    #Make sure indexing works while replaying redo log
    zmconvertctl status &>/dev/null || zmconvertctl start &>/dev/null
    if ! zmconvertctl status &>/dev/null; then
      logit 2 "Start of local convertd servers failed"
      zmconvertctl status >&2
    fi
  fi
  [ $server_failed -eq 1 ] && return 1
  logit 3 "Replaying redologs..."
  if ! zmplayredo >/dev/null; then
    logit 2 "Replay of redolog failed"
    #No error returned here since "break" is not necessary
  else
    #If no errors then archive redo log files
    if ! mkdir -p "$LIVE_SYNC_ARCHIVE_DIR"; then
      logit 1 "Unable to create directory $LIVE_SYNC_ARCHIVE_DIR"
      exit 1
    fi
    mv -f "$ARCHIVE_DIR""/"* "$LIVE_SYNC_ARCHIVE_DIR""/" 2>/dev/null
    touch "$LAST_GOOD_REDO_REPLAY"
  fi
  logit 3 "Replaying redologs done"
  return 0
}

#The redo log sync daemon
redo_log_live_sync () {
  local stream_pid archived_file i archived_redo_log_file prev_zmmailbox_status secondary_storage

  logit 3 "Starting redo log live sync process"

  #Wait for lock directory to be successfully created
  while ! mkdir "$LOCK_STATE_DIR" &>/dev/null; do
    sleep 2
  done
  logit 3 "Detecting if HSM used"
  if detect_hsm; then
    logit 3 "HSM Detected"
    secondary_storage="yes"
  else
    logit 3 "No HSM Detected"
  fi
  rmdir "$LOCK_STATE_DIR"

  while [ ! -f "$STOP_FILE" ]; do
    while [ ! -f "$STOP_FILE" ]; do
      #Wait for lock directory to be successfully created
      while ! mkdir "$LOCK_STATE_DIR" &>/dev/null; do
        sleep 2
      done
      [ -f "$STOP_FILE" ] && break
      logit 3 "Syncing redologs..."
      #If incremental backups are enabled then gather redo logs from backups and copy
      #to local archive directory
      redo_sync_fail="false"
      for archived_redo_log_file in $( echo "gather""$REDO_LOG_HISTORY_DAYS" | \
          $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" ); do
        if [ -f "$LIVE_SYNC_ARCHIVE_DIR""/""$( basename "$archived_redo_log_file" )" ]; then
          logit 4 "Already processed so skipping: $archived_redo_log_file"
        else
          logit 4 "Syncing incremental backup file: $archived_redo_log_file"
          if ! rsync -z -e "$SSH" --size-only "$remote_address"":""$archived_redo_log_file" \
              "$ARCHIVE_DIR""/".; then
            logit 2 "Rsync of a redolog, $archived_redo_log_file, failed"
            redo_sync_fail="true"
          fi
        fi
      done

      #Suspend if HSM is running
      if which zxsuite >/dev/null && zxsuite hsm getAllOperations | grep "MoveBlobs" >/dev/null; then
        logit 3 "Replaying redologs is suspended while HSM process is active"
      else

        #Mailbox process must not be running now. Preserve state and stop.
        zmmailboxdctl status &>/dev/null
        prev_zmmailbox_status=$?
        if [ $prev_zmmailbox_status -eq 0 ]; then
          zmmailboxdctl stop &>/dev/null
        fi
        sleep 2
        if zmmailboxdctl status &>/dev/null; then
          logit 1 "Unable to stop local Zimbra mailbox service"
          return 1
        fi

        logit 4 "Syncing $REDO_LOG_FILE"
        if ! rsync -e "$SSH" -z \
          "$remote_address"":$REDO_LOG_FILE" "$REDO_LOG_FILE"; then
          logit 2 "Rsync of $REDO_LOG_FILE failed"
          redo_sync_fail="true"
        fi
        logit 4 "Syncing $REDO_LOG_FILE done"
        if [ "x""$redo_sync_fail" == "xfalse" ]; then
          touch "$LAST_GOOD_REDO_SYNC"
        else
          break
        fi
        logit 4 "Syncing redologs done"
        logit 4 "Purging redolog directory and archives"
        #Purge local redolog directory
        find $REDOLOG_DIR -mtime +$REDO_LOG_HISTORY_DAYS -type f -exec rm {} \;
        #Purge any interrupted rsync files
        find $REDOLOG_DIR -name '.redo*' -type f -exec rm {} \;
        logit 4 "Purge redolog directory and archives done"
        replay_redo_logs || break

        #Restore mailboxd to previous running state or start if HSM is being used
        if [ $prev_zmmailbox_status -eq 0 ] || \
            [ "x""$secondary_storage" == "xyes" ] >/dev/null; then
          logit 4 "Re-starting Zimbra mailbox service"
          zmmailboxdctl start &>/dev/null
          if ! zmmailboxdctl status &>/dev/null; then
            logit 2 "Unable to re-start local Zimbra mailbox service"
          fi
        fi
      fi

      #If there are no incremental backups then remote archive directory will need purging
      if [ "x""$incremental_backups" != "xtrue" ]; then
        logit 4 "Purging remote redolog directory"
        echo "purge""$REDO_LOG_HISTORY_DAYS" | \
          $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT"
        logit 4 "Purging remote redolog directory done"
      fi
      #Establish copy-and-live-stream of current redo.log file
      logit 4 "Live streaming redolog"
      echo stream | \
        $SSH "$remote_address" \
        "$SYNC_COMMANDS_SCRIPT" >"$REDO_LOG_FILE" &
      stream_pid=$!
      disown $stream_pid
      #Delay as PID was sometimes not being found if checked immediately
      sleep 5
      #If successfully established stream then sit and wait for move to archive
      if ps $stream_pid | grep "$SYNC_COMMANDS_SCRIPT" &>/dev/null; then
        logit 4 "Live streaming redolog established"
        touch "$LAST_GOOD_REDO_STREAM"
        #Remove lock file, this is resting point
        rmdir "$LOCK_STATE_DIR" &>/dev/null
        #Wait for name to be passed of new archive file after redo.log is moved on remote server
        #This is normal resting point of this process
        archived_file=$( echo wait_redo | \
          $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" | \
          tail -n 1 | grep -Eo "redo-.*log" )
        #Kill stream
        kill -KILL $( ps aux | grep "$SYNC_COMMANDS_SCRIPT" | \
          grep -v grep | awk '{print $2}' ) &>/dev/null
        #Mirror move operation on local server
        if echo "$archived_file" | grep -E "redo-.*log" &>/dev/null; then
          logit 4 "Moving redo.log to $archived_file"
          mv -f "$REDO_LOG_FILE" "$ARCHIVE_DIR""/""$archived_file" 2>/dev/null
        else
          logit 2 "Archive file name not found"
        fi
        [ -f "$STOP_FILE" ] && break
      else
        logit 2 "Failed to start redolog streaming, PID=$stream_pid"
        break
      fi
    done
    rmdir "$LOCK_STATE_DIR" &>/dev/null
    #Wait $ERROR_CLEAR_MINUTES minutes for error to error to clear
    i=0
    while [ $(( i++ )) -lt 60 ] && [ ! -f "$STOP_FILE" ]; do
      sleep $ERROR_CLEAR_MINUTES
    done
  done
  logit 3 "Ending redo log live sync process"
}

#The ldap sync daemon
ldap_live_sync () {
  local ldap_wait_pid i last_ldap_success_state

  last_ldap_success_state="false"

  logit 3  "Starting ldap live sync process"
  while [ ! -f "$STOP_FILE" ]; do
    while [ ! -f "$STOP_FILE" ]; do
      #Wait for lock directory to be successfully created
      while ! mkdir "$LOCK_STATE_DIR" &>/dev/null; do
        sleep 3
      done
      if [ $zimbra_version -lt 8 ]; then
        logit 3 "Syncing ldap using rsync"
        #Use rsync for Zimbra older than verion 8
        while [ 1 ]; do
          #Check for changes during ldap sync operation
          echo wait_ldap | \
            $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" &>"$WATCHES_FILE" &
          ldap_wait_pid=$!
          disown $ldap_wait_pid
          if ! ps "$ldap_wait_pid" &>/dev/null; then
            logit 2 "Unable to establish watch on remote LDAP directory, no ldap sync performed"
            break
          fi
          #Wait for watches to be established
          while ! grep "established" "$WATCHES_FILE" &>/dev/null && \
              ps "$ldap_wait_pid" &>/dev/null; do
            sleep 1
          done
          #Echo out status
          cat "$WATCHES_FILE"
          rm -f "$WATCHES_FILE"


          #Rsync remote server to temporary local ldap directory
          if ! rsync -e "$SSH" -aHz --sparse --force --delete \
            "$remote_address"":$LDAP_DATA_DIR""/" "$LDAP_TEMP_DIR""/"; then
            logit 2 "Rsync of ldap failed"
            break
          else
            touch "$LAST_GOOD_LDAP_SYNC"
          fi
          ps $ldap_wait_pid &>/dev/null && break
          logit 3 "Ldap changed during rsync. Re-syncing."
          sleep 10
        done
        kill -KILL $ldap_wait_pid &>/dev/null
      else
        #Use ldif export for Zimbra 8 and over
        logit 3 "Syncing ldap using ldif"
        if ! echo dump_ldap | \
          $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" >"$LDAP_TEMP_LDIF"; then
          logit 2 "Unable to fetch remote LDIF, no LDAP sync performed"
          break
        else
          touch "$LAST_GOOD_LDAP_SYNC"
        fi
      fi
      if which zxsuite >/dev/null && zxsuite hsm getAllOperations | grep "MoveBlobs" >/dev/null; then
        logit 3 "LDAP update is suspended while HSM process is active"
      else
        #Stop ldap
        ldap status &>/dev/null && ldap stop &>/dev/null
        if ldap status &>/dev/null; then
          logit 1 "Unable to stop local ldap server"
          break
        fi
        if [ $zimbra_version -lt 8 ]; then
          #Use rsync for Zimbra older than verion 8
          #rsync temporary local ldap directory to real local ldap directory
          rsync -aH --sparse "$LDAP_TEMP_DIR""/" "$LDAP_DATA_DIR""/"
        else
          #Use LDIF import for Zimbra 8 and over
          rm -rf "$LDAP_DATA_DIR""/mdb" && \
          mkdir -p "$LDAP_DATA_DIR""/mdb/db" && \
          mkdir -p "$LDAP_DATA_DIR""/mdb/log" && \
          /opt/zimbra/libexec/zmslapadd "$LDAP_TEMP_LDIF"
          if [ $? != 0 ]; then
            logit 2 "Unable to import LDIF into local LDAP"
            break
          fi
        fi
        #Restart ldap
        ldap status &>/dev/null || ldap start &>/dev/null
        if ! ldap status &>/dev/null; then
          logit 1 "Unable to restart local ldap server"
          last_ldap_success_state="false"
        else
          last_ldap_success_state="true"
        fi
        logit 4 "Syncing LDAP done"
      fi
      rmdir "$LOCK_STATE_DIR" &>/dev/null
      [ -f "$STOP_FILE" ] && break
      #Wait for change in remote ldap over $LDAP_CHECK_MINUTES_INTERVAL intervals
      echo wait_ldap | \
        $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" &
      ldap_wait_pid=$!
      disown $ldap_wait_pid
      while [ ! -f "$STOP_FILE" ]; do
        logit 4 "Start new LDAP monitor period"
        #Repeat last ldap success so that no ldap change is not
        #interpreted by Nagios as no ldap success.
        if [ "x""$last_ldap_success_state" == "xtrue" ]; then
          touch "$LAST_GOOD_LDAP_START"
        fi
        #Restart wait for ldap change if required
        if ! ps $ldap_wait_pid &>/dev/null; then
          echo wait_ldap | \
            $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" &
          ldap_wait_pid=$!
          disown $ldap_wait_pid
        fi
        #Wait $LDAP_CHECK_MINUTES_INTERVAL minutes
        i=0
        while [ $(( i++ )) -lt 60 ] && [ ! -f "$STOP_FILE" ]; do
          sleep $LDAP_CHECK_MINUTES_INTERVAL
        done
        #If wait process is not still running then there was a change
        ps $ldap_wait_pid &>/dev/null || break
      done
    done
    rmdir "$LOCK_STATE_DIR" &>/dev/null
    #Wait $ERROR_CLEAR_MINUTES minutes for error to error to clear
    i=0
    while [ $(( i++ )) -lt 60 ] && [ ! -f "$STOP_FILE" ]; do
      sleep $ERROR_CLEAR_MINUTES
    done
  done
  logit 3 "Ending ldap live sync process"
}

get_zimbra_config_globals () {
  #Query whether incremental backups are enabled
  incremental_backups=$( echo "query_incremental" | \
    $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" )

  #Query whether convertd is installed and enabled
  ldap status &>/dev/null || ldap start &>/dev/null
  if ! ldap status &>/dev/null; then
    logit 1 "Unable to start local ldap server"
    exit 1
  fi
  if [ $( zmprov -l  gs `zmhostname` | \
          grep -E "(zimbraServiceInstalled|zimbraServiceEnabled):[[:space:]]*convertd" | \
          wc -l  ) -eq 2 ]; then
    convertd_enabled="true"
  else
    convertd_enabled="false"
  fi
}

kill_everything () {
  touch "$STOP_FILE"
  kill -KILL $( head -n 1 "$PID_FILE_LDAP" 2>/dev/null ) &>/dev/null
  kill -KILL $( head -n 1 "$PID_FILE_REDO" 2>/dev/null ) &>/dev/null
  kill -KILL $( ps aux | grep "live_syncd start" | grep -v grep | awk '{print $2}' ) &>/dev/null
  kill -KILL $( ps aux | grep "redo_log_live_sync" | grep -v grep | awk '{print $2}' ) &>/dev/null
  kill -KILL $( ps aux | grep "ldap_live_sync" | grep -v grep | awk '{print $2}' ) &>/dev/null
  kill -KILL $( ps aux | \
    grep "$SYNC_COMMANDS_SCRIPT" | grep -v grep | awk '{print $2}' ) &>/dev/null
  kill -KILL $( ps aux | grep "rsync" | grep -E "$REDOLOG_DIR""|""$LDAP_DATA_DIR""|""$BACKUP_DIR" | \
    awk '{print $2}' ) &>/dev/null
  #Kill redolog playback if running
  kill -KILL $( ps aux | grep -E "zimbra.*java.*PlaybackUtil" | grep -v grep | \
    awk '{print $2}' ) &>/dev/null
  rm -f "$STOP_FILE"
  rm -f "$PID_FILE_LDAP"
  rm -f "$PID_FILE_REDO"
  rmdir "$LOCK_STATE_DIR" &>/dev/null
}

quitting () {
  echo "Quitting"
  #Kill any hanging processes
  kill_everything
  trap - INT TERM SIGINT SIGTERM
  echo 'kill -KILL $( ps aux | grep live_syncd | grep -v grep | awk '"'"'{print $2}'"'"' ) &>/dev/null' | \
    at now && sleep 1 && rmdir "$LOCK_STATE_DIR" &>/dev/null
  exit
}


#******************************************************************************
#********************** Main Program ******************************************
#******************************************************************************

if [ "$( whoami )" != "zimbra" ]; then
  echo "Must run as zimbra user" >&2
  exit 1
fi

mkdir -p "$LOCKING_DIR"
mkdir -p "$PID_DIR"
mkdir -p "$LOG_DIR"
mkdir -p "$LDAP_TEMP_DIR"
mkdir -p "$STATUS_DIR"
chmod 755 "$STATUS_DIR"

if [ ! -f "$CONF_FILE" ]; then
  echo "Configuration file, $CONF_FILE, not found" >&2
  exit 1
fi

source "$CONF_FILE"

#Find all local addresses
server_addresses=$( /sbin/ifconfig | \
  grep -Eio "inet[[:space:]]*(([0-9]+\.){3}[0-9]+|[0-9a-f]+(:[0-9a-f]*){5})" | \
  sed "s/inet//" | tr -d " \t" )

#Check configured server addresses are valid
if ! echo "$server1" | \
    grep -Ei "([0-9]+\.){3}[0-9]+|[0-9a-f]+(:[0-9a-f]*){5}" &>/dev/null; then
  echo "No valid IP address found for server1 in configuration file" >&2
  exit 1
fi
if ! echo "$server2" | \
    grep -Ei "([0-9]+\.){3}[0-9]+|[0-9a-f]+(:[0-9a-f]*){5}" &>/dev/null; then
  echo "No valid IP address found for server2 in configuration file" >&2
  exit 1
fi

#Deduce local address and assume other address is remote machine
if echo "$server_addresses" | grep "$server1" &>/dev/null; then
  local_address="$server1"
  remote_address="$server2"
else
  if echo "$server_addresses" | grep "$server2" &>/dev/null; then
    local_address="$server2"
    remote_address="$server1"
  else
    echo "Unable to identify local server address and assume remote address" >&2
    exit 1
  fi
fi

#Check remote server is OK
remote_server_status=$( echo "test" | \
  $SSH "$remote_address" "$SYNC_COMMANDS_SCRIPT" )

if [ "x""$remote_server_status" == "xbusy" ]; then
  echo "Remote server appears to have live_syncd process running" >&2
  echo "This can not run on both servers" >&2
  exit 1
fi

if [ "x""$remote_server_status" != "xOK" ]; then
  echo "Unable to run commands on remote server" >&2
  exit 1
fi

#Get major Zimbra version
zimbra_version=$( zmcontrol -v | grep -Eo "[0-9][^\.]*" | head -n 1 )
if [ ${#zimbra_version} -lt 1 ]; then
 zimbra_version=0
fi

case $1 in
  start)
    #Check for processes from this script and also redolog replay. Don't count PID files older than uptime.
    if [ -f "$PID_FILE_REDO" ] && \
        [ $(( $( date +%s ) - $( stat -c '%Y' "$PID_FILE_REDO" ) )) -lt $( cat /proc/uptime | grep -Eo "[0-9]+" | head -n 1 ) ]; then
      pid_found="yes"
    fi
    if [ -f "$PID_FILE_LDAP" ] && \
        [ $(( $( date +%s ) - $( stat -c '%Y' "$PID_FILE_LDAP" ) )) -lt $( cat /proc/uptime | grep -Eo "[0-9]+" | head -n 1 ) ]; then
      pid_found="yes"
    fi
    if [ $pid_found ] || \
        ps aux | grep -E "zimbra.*java.*PlaybackUtil" | grep -v grep &>/dev/null; then
      echo "Proccess already running"
    else
      echo -n "Starting processes..."
      get_zimbra_config_globals
      echo "***************************************" >>"$LOG_FILE"
      logit 3 "Starting $( basename "$0" )" >>"$LOG_FILE"
      logit 3 "Incremental backups enabled : $incremental_backups" >>"$LOG_FILE"
      logit 3 "Convertd enabled : $convertd_enabled" >>"$LOG_FILE"

      ldap_live_sync >>"$LOG_FILE" 2>&1 &
      echo $! >"$PID_FILE_LDAP"
      redo_log_live_sync >>"$LOG_FILE" 2>&1 &
      echo $! >"$PID_FILE_REDO"
      echo "done"
    fi
    ;;
  stop)
    touch "$STOP_FILE"
    [ -d "$LOCK_STATE_DIR" ] && echo "Waiting for sync operations to complete..."
    while [ -d "$LOCK_STATE_DIR" ]; do
      sleep 5
    done
    rm -f "$STOP_FILE"
    replay_redo_logs
    kill_everything
    echo "done"
    ;;
  status)
    if ps aux | grep -E "zimbra.*java.*PlaybackUtil" | grep -v grep &>/dev/null; then
      echo "redolog is being replayed"
      replay_stat=0
    else
      replay_stat=3
    fi
    if [ -f  $PID_FILE_REDO ] && ps $( head -n 1 $PID_FILE_REDO 2>/dev/null ) &>/dev/null; then
      echo "redo log sync process OK"
      redo_stat=0
    else
      echo "redolog sync process stopped"
      redo_stat=3
    fi
    if [ -f  $PID_FILE_LDAP ] && ps $( head -n 1 $PID_FILE_LDAP 2>/dev/null ) &>/dev/null; then
      echo "ldap sync process OK"
      ldap_stat=0
    else
      echo "ldap sync process stopped"
      ldap_stat=3
    fi
    [ $ldap_stat == 3 ] && [ $redo_stat == 3 ] && [ $replay_stat == 3 ] && exit 3
    [ $ldap_stat == 0 ] && [ $redo_stat == 0 ] && exit 0
    exit 1
    ;;
  kill)
    kill_everything
    ;;
  *)
    trap quitting INT TERM SIGINT SIGTERM
    if ps aux | grep "redo_log_live_sync" | grep -v grep  &>/dev/null || \
        ps aux | grep "ldap_live_sync" | grep -v grep  &>/dev/null || \
        ps aux | grep -E "zimbra.*java.*PlaybackUtil" | grep -v grep &>/dev/null; then
      echo "Proccess already running"
    else
      echo "Starting processes in realtime"
      get_zimbra_config_globals
      logit 3 "Incremental backups enabled : $incremental_backups"
      logit 3 "Convertd enabled : $convertd_enabled"
      ldap_live_sync &
      echo $! >"$PID_FILE_LDAP"
      redo_log_live_sync &
      echo $! >"$PID_FILE_REDO"
      while [ 1 ]; do sleep 10; done
    fi
    ;;
esac
myzamri
Posts: 20
Joined: Tue Mar 28, 2017 2:46 am

Re: Updated live sync script

Post by myzamri »

maxxer wrote:
myzamri wrote:I increased the memory and the problem solved.
system memory or some zimbra param? thanks
system memory added. for virtualization environment, just up the memory size.
Post Reply