Another service.PROXY_ERROR HTTP/1.1 503 Service Unavailable Issue

[stevekk]

There appear to be a number of threads about this problem but so far I haven't found one that solves my issue.

I have a single CentOS 7 server running Zimbra for my email.

Code: Select all

$ zmcontrol -v
Release 8.7.9_GA_1794.RHEL7_64_20170505054622 RHEL7_64 FOSS edition.

All was fine until a reboot 3 days ago and since then I've been chasing issues within the install. One of the updates included the selinux policies and I have tried running the server with selinux in permissive mode, but that makes no difference.

At the moment I cannot launch the admin web page as that just loads a blank page: https://zimbra:7071/zimbraAdmin/

The mailboxes are also unable to be accessed, even though all the zimbra services show as running.

Code: Select all

$ zmcontrol status
Host xxx.xxx.co.uk
   amavis                  Running
   antispam                Running
   antivirus               Running
   dnscache                Running
   ldap                    Running
   logger                  Running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Running
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running


A simple query gets the dreaded 503 Service Unavailable error.

Code: Select all

$ zmprov gs
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)

In the trace_log file I see the following.

Code: Select all

https:https://localhost:7071/service/admin/soap/AuthRequest RESPONSE 503 text/html;charset=iso-8859-1

What is this authentication process that is failing and which credentials should I be checking?

Regards,
Steve

[axslingr]

All was fine until a reboot 3 days ago and since then I've been chasing issues within the install. One of the updates included the selinux policies and I have tried running the server with selinux in permissive mode, but that makes no difference.

Does it work with selinux disabled?

Lance

[stevekk]

Thanks Lance. I'd forgotten that permissive isn't quite the same as disabled. However setting to disabled in the config file and rebooting still doesn't help.

Code: Select all

# getenforce
Disabled
# su - zimbra
$ zmprov -v
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)
com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
ExceptionId:main:1549792750518:d923902bbc12f6d6
Code:service.PROXY_ERROR Arg:(url, STR, "https://localhost:7071/service/admin/soap/AuthRequest")
   at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:373)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:254)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:173)
   at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:419)
   at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:405)
   at com.zimbra.cs.account.soap.SoapProvisioning.invokeRequest(SoapProvisioning.java:449)
   at com.zimbra.cs.account.soap.SoapProvisioning.invoke(SoapProvisioning.java:457)
   at com.zimbra.cs.account.soap.SoapProvisioning.soapAdminAuthenticate(SoapProvisioning.java:401)
   at com.zimbra.cs.account.soap.SoapProvisioning.soapZimbraAdminAuthenticate(SoapProvisioning.java:429)
   at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1017)
   at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:3921)

Regards,
Steve

[axslingr]

You might try reconfiguring proxy with zmproxyconfig.

https://wiki.zimbra.com/wiki/Enabling_Z ... ng_servers

Lance

[stevekk]

Having checked the .bash_history I have configured the proxy in the past, so I've pushed this configuration again.

Code: Select all

$ /opt/zimbra/libexec/zmproxyconfig -e -w -o -a 8181:81:8444:444 -x https -H 'myhost.co.uk'

Restarting Zimbra still hasn't helped, so I did a bit of digging into the LDAP settings for ports and the proxy. In the LDAP I have spotted 2 potentially troublesome entries.

Code: Select all

zimbraReverseProxyAvailableLookupTargets: oldhost.co.uk
zimbraReverseProxyUpstreamLoginServers: oldhost.co.uk

So I try to update these settings, but get the following.

Code: Select all

$ zmprov ms zimbraReverseProxyAvailableLookupTargets 'myhost.co.uk'
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)
$ zmprov ms 'myhost.co.uk' zimbraReverseProxyAvailableLookupTargets 'myhost.co.uk'
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)
$ libexec/zmproxyconfgen -s 'myhost.co.uk'
[] INFO: Loading server object: myhost.co.uk
[] WARN: Invalid value found in 'zimbraReverseProxyAvailableLookupTargets': oldhost.co.uk
Please correct and run zmproxyconfgen again
[] WARN: No available nginx lookup handlers could be found
Exception in thread "main" java.lang.NullPointerException
   at com.zimbra.cs.util.ProxyConfVar.isValidUpstream(ProxyConfGen.java:310)
   at com.zimbra.cs.util.WebLoginSSLUpstreamServersVar.update(ProxyConfGen.java:1008)
   at com.zimbra.cs.util.ProxyConfGen.updateDefaultVars(ProxyConfGen.java:2705)
   at com.zimbra.cs.util.ProxyConfGen.createConf(ProxyConfGen.java:2875)
   at com.zimbra.cs.util.ProxyConfGen.main(ProxyConfGen.java:3070)

In the /etc/hosts file for myhost.co.uk I have oldhost.co.uk resolvable to the same ip address as myhost.co.uk, so I think the next question needs to be how can I update those entries in LDAP without using zmprov, or is there an alternative syntax that might work?

Regards,
Steve

[stevekk]

I worked out how I could correct those LDAP entries, although I can't necessarily recommend this method. Anyone following these steps needs to understand it could cause issues, so make sure you have backups in place. So I created a complete backup of /opt/zimbra after running zmcontrol stop. Then following instructions from the wiki I made backups of the LDAP data.

Code: Select all

$ /opt/zimbra/libexec/zmslapcat /data/backup/zimbra
$ /opt/zimbra/libexec/zmslapcat -c /data/backup/zimbra
$ /opt/zimbra/libexec/zmslapcat -a /data/backup/zimbra

The backup of the config and audit data wasn't actually necessary, but each was very quick to run. I then edited the file /data/backup/zimbra/ldap.bak and changed the 2 entries:

Code: Select all

zimbraReverseProxyAvailableLookupTargets: oldhost.co.uk
zimbraReverseProxyUpstreamLoginServers: oldhost.co.uk

To the correct server

Code: Select all

zimbraReverseProxyAvailableLookupTargets: myhost.co.uk
zimbraReverseProxyUpstreamLoginServers: myhost.co.uk

I then followed the instructions from the wiki to restore the database. Again I didn't use the config or access backups I'd created. After starting the zimbra services I was able to successfully run:

Code: Select all

$ libexec/zmproxyconfgen -s 'myhost.co.uk'

which generated the configuration files, so I restarted the Zimbra services, but still get the same error.

Code: Select all

$ zmprov -v
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)
com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
ExceptionId:main:1549890029342:62ba1a8f4ce2de7c
Code:service.PROXY_ERROR Arg:(url, STR, "https://localhost:7071/service/admin/soap/AuthRequest")
   at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:373)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:254)
   at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:173)
   at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:419)
   at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:405)
   at com.zimbra.cs.account.soap.SoapProvisioning.invokeRequest(SoapProvisioning.java:449)
   at com.zimbra.cs.account.soap.SoapProvisioning.invoke(SoapProvisioning.java:457)
   at com.zimbra.cs.account.soap.SoapProvisioning.soapAdminAuthenticate(SoapProvisioning.java:401)
   at com.zimbra.cs.account.soap.SoapProvisioning.soapZimbraAdminAuthenticate(SoapProvisioning.java:429)
   at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1017)
   at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:3921)


Running the following confirmed that the 2 entries have been corrected to the current host, although it is worth me noting that oldhost was migrated away from at least 18 months ago.

Code: Select all

$ /opt/zimbra/libexec/zmproxyconfgen  -D

So I am still stuck unable to access emails or to run most zmprov commands or access the WebAdmin console. Any help or ideas would be much appreciated.

Regards,
Steve

[stevekk]

Checking through the ports that my server is listening on shows that port 81 isn't, but I think that's not an issue as web.http.enabled = false in the config for my proxy - am I correct?

Should my server be listening on some other port(s) which could be the source of the error?

Code: Select all

# lsof -Pi :25,81,110,143,444,993,995,7071,7072,8181,8444,9071,11211
COMMAND    PID    USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
java       718  zimbra  103u  IPv4 14653683      0t0  TCP myhost.co.uk:8181 (LISTEN)
java       718  zimbra  105u  IPv4 14653686      0t0  TCP localhost:8181 (LISTEN)
java       718  zimbra  106u  IPv4 14653689      0t0  TCP myhost.co.uk:8444 (LISTEN)
java       718  zimbra  107u  IPv4 14653690      0t0  TCP myhost.co.uk:7071 (LISTEN)
java       718  zimbra  108u  IPv4 14653691      0t0  TCP localhost:7071 (LISTEN)
java       718  zimbra  113u  IPv4 14652730      0t0  TCP *:7072 (LISTEN)
memcached 1070  zimbra   26u  IPv4 14653049      0t0  TCP *:11211 (LISTEN)
memcached 1070  zimbra   27u  IPv6 14653050      0t0  TCP *:11211 (LISTEN)
memcached 1070  zimbra   28u  IPv4 14653053      0t0  UDP *:11211
memcached 1070  zimbra   29u  IPv4 14653053      0t0  UDP *:11211
memcached 1070  zimbra   30u  IPv4 14653053      0t0  UDP *:11211
memcached 1070  zimbra   31u  IPv4 14653053      0t0  UDP *:11211
memcached 1070  zimbra   32u  IPv6 14653054      0t0  UDP *:11211
memcached 1070  zimbra   33u  IPv6 14653054      0t0  UDP *:11211
memcached 1070  zimbra   34u  IPv6 14653054      0t0  UDP *:11211
memcached 1070  zimbra   35u  IPv6 14653054      0t0  UDP *:11211
memcached 1070  zimbra   36u  IPv4 14653380      0t0  TCP myhost.co.uk:11211->myhost.co.uk:48952 (ESTABLISHED)
memcached 1070  zimbra   37u  IPv4 14653381      0t0  TCP myhost.co.uk:11211->myhost.co.uk:48954 (ESTABLISHED)
memcached 1070  zimbra   38u  IPv4 14653382      0t0  TCP myhost.co.uk:11211->myhost.co.uk:48956 (ESTABLISHED)
memcached 1070  zimbra   39u  IPv4 14653383      0t0  TCP myhost.co.uk:11211->myhost.co.uk:48958 (ESTABLISHED)
nginx     1123  zimbra    6u  IPv4 14654593      0t0  TCP *:143 (LISTEN)
nginx     1123  zimbra    7u  IPv4 14654594      0t0  TCP *:993 (LISTEN)
nginx     1123  zimbra    8u  IPv4 14654595      0t0  TCP *:110 (LISTEN)
nginx     1123  zimbra    9u  IPv4 14654596      0t0  TCP *:995 (LISTEN)
nginx     1123  zimbra   10u  IPv4 14654597      0t0  TCP *:444 (LISTEN)
nginx     1125  zimbra    6u  IPv4 14654593      0t0  TCP *:143 (LISTEN)
nginx     1125  zimbra    7u  IPv4 14654594      0t0  TCP *:993 (LISTEN)
nginx     1125  zimbra    8u  IPv4 14654595      0t0  TCP *:110 (LISTEN)
nginx     1125  zimbra    9u  IPv4 14654596      0t0  TCP *:995 (LISTEN)
nginx     1125  zimbra   10u  IPv4 14654597      0t0  TCP *:444 (LISTEN)
nginx     1125  zimbra   13u  IPv4 14654027      0t0  TCP myhost.co.uk:48952->myhost.co.uk:11211 (ESTABLISHED)
nginx     1126  zimbra    6u  IPv4 14654593      0t0  TCP *:143 (LISTEN)
nginx     1126  zimbra    7u  IPv4 14654594      0t0  TCP *:993 (LISTEN)
nginx     1126  zimbra    8u  IPv4 14654595      0t0  TCP *:110 (LISTEN)
nginx     1126  zimbra    9u  IPv4 14654596      0t0  TCP *:995 (LISTEN)
nginx     1126  zimbra   10u  IPv4 14654597      0t0  TCP *:444 (LISTEN)
nginx     1126  zimbra   15u  IPv4 14654611      0t0  TCP myhost.co.uk:48956->myhost.co.uk:11211 (ESTABLISHED)
nginx     1127  zimbra    6u  IPv4 14654593      0t0  TCP *:143 (LISTEN)
nginx     1127  zimbra    7u  IPv4 14654594      0t0  TCP *:993 (LISTEN)
nginx     1127  zimbra    8u  IPv4 14654595      0t0  TCP *:110 (LISTEN)
nginx     1127  zimbra    9u  IPv4 14654596      0t0  TCP *:995 (LISTEN)
nginx     1127  zimbra   10u  IPv4 14654597      0t0  TCP *:444 (LISTEN)
nginx     1127  zimbra   17u  IPv4 14652096      0t0  TCP myhost.co.uk:48954->myhost.co.uk:11211 (ESTABLISHED)
nginx     1128  zimbra    6u  IPv4 14654593      0t0  TCP *:143 (LISTEN)
nginx     1128  zimbra    7u  IPv4 14654594      0t0  TCP *:993 (LISTEN)
nginx     1128  zimbra    8u  IPv4 14654595      0t0  TCP *:110 (LISTEN)
nginx     1128  zimbra    9u  IPv4 14654596      0t0  TCP *:995 (LISTEN)
nginx     1128  zimbra   10u  IPv4 14654597      0t0  TCP *:444 (LISTEN)
nginx     1128  zimbra   19u  IPv4 14652098      0t0  TCP myhost.co.uk:48958->myhost.co.uk:11211 (ESTABLISHED)
master    2727    root   13u  IPv4 14661535      0t0  TCP *:25 (LISTEN)
postscree 3422 postfix    6u  IPv4 14661535      0t0  TCP *:25 (LISTEN)

Regards,
Steve

[stevekk]

I'm getting a bit desperate now. Hoping it might be a bug in release 8.7.9GA after taking a full backup of my Zimbra folder I was able to successfully upgrade to 8.7.11GA. See the below for the log of the upgrade process.

Code: Select all

# ./install.sh

Operations logged to /tmp/install.log.RwBlT4dT
Checking for existing installation...
    zimbra-chat...NOT FOUND
    zimbra-drive...NOT FOUND
    zimbra-suiteplus...NOT FOUND
    zimbra-ldap...FOUND zimbra-ldap-8.7.9_GA_1794
    zimbra-logger...FOUND zimbra-logger-8.7.9_GA_1794
    zimbra-mta...FOUND zimbra-mta-8.7.9_GA_1794
    zimbra-dnscache...FOUND zimbra-dnscache-8.7.9_GA_1794
    zimbra-snmp...FOUND zimbra-snmp-8.7.9_GA_1794
    zimbra-store...FOUND zimbra-store-8.7.9_GA_1794
    zimbra-apache...FOUND zimbra-apache-8.7.9_GA_1794
    zimbra-spell...FOUND zimbra-spell-8.7.9_GA_1794
    zimbra-convertd...NOT FOUND
    zimbra-memcached...FOUND zimbra-memcached-1.4.37-2
    zimbra-proxy...FOUND zimbra-proxy-8.7.9_GA_1794
    zimbra-archiving...NOT FOUND
    zimbra-core...FOUND zimbra-core-8.7.9_GA_1794
ZCS upgrade from 8.7.9 to 8.7.11 will be performed.
Validating ldap configuration
LDAP validation succeeded.  Continuing.


----------------------------------------------------------------------
PLEASE READ THIS AGREEMENT CAREFULLY BEFORE USING THE SOFTWARE.
SYNACOR, INC. ("SYNACOR") WILL ONLY LICENSE THIS SOFTWARE TO YOU IF YOU
FIRST ACCEPT THE TERMS OF THIS AGREEMENT. BY DOWNLOADING OR INSTALLING
THE SOFTWARE, OR USING THE PRODUCT, YOU ARE CONSENTING TO BE BOUND BY
THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS OF THIS
AGREEMENT, THEN DO NOT DOWNLOAD, INSTALL OR USE THE PRODUCT.

License Terms for this Zimbra Collaboration Suite Software:
https://www.zimbra.com/license/zimbra-public-eula-2-6.html
----------------------------------------------------------------------



Do you agree with the terms of the software license agreement? [N] y



Checking current number of databases...

Do you want to verify message store database integrity? [Y]
Verifying integrity of message store databases.  This may take a while.
Starting mysqld...done.
mysqld is alive
No errors found
mysqld is alive
Stopping mysqld... done.


Use Zimbra's package repository [Y]


Configuring package repository

Checking for installable packages

Found zimbra-core (local)
Found zimbra-ldap (local)
Found zimbra-logger (local)
Found zimbra-mta (local)
Found zimbra-dnscache (local)
Found zimbra-snmp (local)
Found zimbra-store (local)
Found zimbra-apache (local)
Found zimbra-spell (local)
Found zimbra-memcached (repo)
Found zimbra-proxy (local)
Found zimbra-chat (repo)
Found zimbra-drive (repo)


The Zimbra Collaboration Server appears to already be installed.
It can be upgraded with no effect on existing accounts,
or the current installation can be completely removed prior
to installation for a clean install.

Do you wish to upgrade? [Y]

Scanning for any new or additional packages available for installation
Existing packages will be upgraded
    Upgrading zimbra-core
    Upgrading zimbra-ldap
    Upgrading zimbra-logger
    Upgrading zimbra-mta
    Upgrading zimbra-dnscache
    Upgrading zimbra-snmp
    Upgrading zimbra-store
    Upgrading zimbra-apache
    Upgrading zimbra-spell
    Upgrading zimbra-memcached
    Upgrading zimbra-proxy

Install zimbra-chat [N]

Install zimbra-drive [N]
Checking required space for zimbra-core
Checking space for zimbra-store
Checking required packages for zimbra-store
zimbra-store package check complete.

Installing:
    zimbra-core
    zimbra-ldap
    zimbra-logger
    zimbra-mta
    zimbra-dnscache
    zimbra-snmp
    zimbra-store
    zimbra-apache
    zimbra-spell
    zimbra-memcached
    zimbra-proxy

The system will be modified.  Continue? [N] y

Beginning Installation - see /tmp/install.log.RwBlT4dT for details...

                 zimbra-core will be installed.
      zimbra-core-components will be downloaded and installed.
                 zimbra-ldap will be installed.
      zimbra-ldap-components will be downloaded and installed.
               zimbra-logger will be installed.
                  zimbra-mta will be installed.
       zimbra-mta-components will be downloaded and installed.
             zimbra-dnscache will be installed.
  zimbra-dnscache-components will be downloaded and installed.
                 zimbra-snmp will be installed.
      zimbra-snmp-components will be downloaded and installed.
                zimbra-store will be installed.
     zimbra-store-components will be downloaded and installed.
               zimbra-apache will be installed.
    zimbra-apache-components will be downloaded and installed.
                zimbra-spell will be installed.
     zimbra-spell-components will be downloaded and installed.
            zimbra-memcached will be downloaded and installed.
                zimbra-proxy will be installed.
     zimbra-proxy-components will be downloaded and installed.

Downloading packages (10):
   zimbra-core-components
   zimbra-ldap-components
   zimbra-mta-components
   zimbra-dnscache-components
   zimbra-snmp-components
   zimbra-store-components
   zimbra-apache-components
   zimbra-spell-components
   zimbra-memcached
   zimbra-proxy-components
      ...done

Saving existing configuration file to /opt/zimbra/.saveconfig

Shutting down zimbra mail

Backing up the ldap database...done.
Upgrading the remote packages

Removing deployed webapp directories

Installing repo packages (10):
   zimbra-core-components
   zimbra-ldap-components
   zimbra-mta-components
   zimbra-dnscache-components
   zimbra-snmp-components
   zimbra-store-components
   zimbra-apache-components
   zimbra-spell-components
   zimbra-memcached
   zimbra-proxy-components
      ...done

Installing local packages (10):
   zimbra-core
   zimbra-ldap
   zimbra-logger
   zimbra-mta
   zimbra-dnscache
   zimbra-snmp
   zimbra-store
   zimbra-apache
   zimbra-spell
   zimbra-proxy
      ...done

Running Post Installation Configuration:

Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save
   HOSTNAME=myhost.co.uk
   LDAPHOST=myhost.co.uk
   LDAPPORT=636
   SNMPTRAPHOST=myhost.co.uk
   SMTPSOURCE=admin@myhost.co.uk
   SMTPDEST=steve@myhost.co.uk
   SNMPNOTIFY=yes
   SMTPNOTIFY=yes
   LDAPROOTPW=*
   LDAPZIMBRAPW=*
   LDAPPOSTPW=*
   LDAPREPPW=*
   LDAPAMAVISPW=*
   LDAPNGINXPW=*
Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml...done
Operations logged to /tmp/zmsetup.20190213-093937.log
Adding /opt/zimbra/conf/ca/ca.pem to cacerts
Upgrading from 8.7.9_GA_1794 to 8.7.11_GA_1854
Stopping zimbra services...done.
This appears to be 8.7.9_GA
Starting mysql...done.
Checking ldap status...not running.
Checking ldap status...not running.
Starting ldap...done.
Checking ldap status...already running.
Stopping mysql...done.
Updating global config and COS's with attributes introduced after 8.7.9_GA...done.
Stopping ldap...done.
Upgrade complete.

Checking ldap status....not running.
Starting ldap...done.
Setting defaults...done.
Setting defaults from existing config...done.
Checking for port conflicts
Port conflict detected: 80 (zimbra-store)
Port conflict detected: 53 (zimbra-dnscache)
Port conflict detected: 53 (zimbra-dnscache)
Port conflict detected: 443 (zimbra-store)
Port conflicts detected! - Press Enter/Return key to continue
Setting defaults from ldap...done.
Saving config in /opt/zimbra/config.15683...done.
Operations logged to /tmp/zmsetup.20190213-093937.log
Setting local config values...done.
Initializing core config...Setting up CA...done.
Deploying CA to /opt/zimbra/conf/ca ...done.
Setting replication password...done.
Setting Postfix password...done.
Setting amavis password...done.
Setting nginx password...done.
Creating server entry for myhost.co.uk...already exists.
Setting Zimbra IP Mode...done.
Saving CA in ldap...done.
Saving SSL Certificate in ldap...done.
Setting spell check URL...done.
Setting service ports on myhost.co.uk...done.
Setting Keyboard Shortcut Preferences...done.
Setting zimbraFeatureTasksEnabled=TRUE...done.
Setting zimbraFeatureBriefcasesEnabled=TRUE...done.
Setting Master DNS IP address(es)...done.
Setting DNS cache tcp lookup preference...done.
Setting DNS cache udp lookup preference...done.
Setting DNS tcp upstream preference...done.
Setting TimeZone Preference...done.
Initializing mta config...done.
Setting services on myhost.co.uk...done.
Adding myhost.co.uk to zimbraMailHostPool in default COS...done.
Creating user spam.wz5d5gc7u@myhost.co.uk...already exists.
Creating user ham.j1zhbwtb@myhost.co.uk...already exists.
Creating user virus-quarantine.yu6uezo_@garage.myhost.co.uk...failed.
Setting spam training and Anti-virus quarantine accounts...done.
Configuring SNMP...done.
Setting up syslog.conf...done.
Starting servers...done.
Checking for deprecated zimlets...done.
Checking for network zimlets in LDAP...done.
Removing network zimlets...
Finished removing network zimlets.
Installing common zimlets...
   com_zimbra_adminversioncheck...done.
   com_zimbra_attachcontacts...done.
   com_zimbra_attachmail...done.
   com_zimbra_bulkprovision...done.
   com_zimbra_cert_manager...done.
   com_zimbra_clientuploader...done.
   com_zimbra_date...done.
   com_zimbra_email...done.
   com_zimbra_mailarchive...done.
   com_zimbra_phone...done.
   com_zimbra_proxy_config...done.
   com_zimbra_srchhighlighter...done.
   com_zimbra_tooltip...done.
   com_zimbra_url...done.
   com_zimbra_viewmail...done.
   com_zimbra_webex...done.
   com_zimbra_ymemoticons...done.
Finished installing common zimlets.
Getting list of all zimlets...done.
Updating non-standard zimlets...
Finished updating non-standard zimlets.
Restarting mailboxd...done.
Skipping creation of default domain GAL sync account - existing install detected.

You have the option of notifying Zimbra of your installation.
This helps us to track the uptake of the Zimbra Collaboration Server.
The only information that will be transmitted is:
   The VERSION of zcs installed (8.7.11_GA_1854_RHEL7_64)
   The ADMIN EMAIL ADDRESS created (admin@myhost.co.uk)

Notify Zimbra of your installation? [Yes] no
Notification skipped
Setting up zimbra crontab...done.


Moving /tmp/zmsetup.20190213-093937.log to /opt/zimbra/log


Configuration complete - press return to exit

The only potential issues there are the following, but I do not believe this would be causing my issues as it was working for a couple of years with these ports in use by Apache and Bind and the Zimbra ports changed from the defaults.

Code: Select all

Port conflict detected: 80 (zimbra-store)
Port conflict detected: 53 (zimbra-dnscache)
Port conflict detected: 53 (zimbra-dnscache)
Port conflict detected: 443 (zimbra-store)

The server appears to be working as before but I still have access issues where I cannot connect an email client or log into the WebAdmin interface.

Code: Select all

$ zmprov gaa
ERROR: service.PROXY_ERROR (error while proxying request to target server: HTTP/1.1 503 Service Unavailable)
$ zmprov -l gaa
admin@myhost.co.uk
spam.wz5d5gc7u@myhost.co.uk
ham.j1zhbwtb@myhost.co.uk
virus-quarantine.yu6uezo_@myhost.co.uk
galsync.yt1qfxhi@myhost.co.uk
steve@myhost.co.uk

I still cannot work out why the proxy is not working.

I have a fresh Zimbra install waiting for my data once I can work out how to extract it from the current environment. This fresh install will only run Zimbra, therefore all the default ports can be utilised. Steps outlined in the Wiki - How to move ZCS to another server do not help as the issue moves to the new server, therefore it surely must be a configuration issue?

Regards,
Steve

[stevekk]

I'm not going to mark this thread as solved as I never managed to fix the original server. I do however have a working Zimbra environment again.

In the end I followed the instructions for ZCS to ZCS rsync Migration on a fresh VM I created, as the existing server was running other services, such as Apache HTTPD websites. This wasn't quite as easy as I hoped since I was performing this remotely from the original host and had to change the ip address of the new server.

After following all the steps I had the environment migrated, but could not get the mailbox service to start. By looking at all the logs it appeared there was an issue with the Jetty JavaVM failing to start, so I spent an age remotely troubleshooting that. It wasn't until I was able to get onsite and change the ip address of the original server and then use that old ip address on the new server that I finally managed to get everything running again.

Because the old server needed to return to it's original ip address I followed the inshavetructions to change the server name, which went through without issue. I was able to clean up and remove the old server name from the configuration. One point to add to that wiki document is when running zmloggerhostmap make sure you don't have a localhost entry, otherwise you might still have the old server listed in the WebAdmin interface, so I had to run:

Code: Select all

zmloggerhostmap -d localhost localhost

Next I changed the ip address of my new server, rebooted and I was back to having the mailbox service fail to start. There does not appear to be that much documentation to help changing the ip address of a server, as everything tends to say make sure you have the /etc/hosts file updated and the DNS entries, which I did. In the end I got lucky by running:

Code: Select all

zmprov -l gs myhost.co.uk | grep <oldIP>

This returned 4 entires which I updated as follows:

Code: Select all

zmprov -l ms myhost.co.uk zimbraAdminBindAddress <newIP>
zmprov -l ms myhost.co.uk zimbraMailBindAddress <newIP>
zmprov -l ms myhost.co.uk zimbraMailSSLBindAddress <newIP>
zmprov -l ms myhost.co.uk zimbraMailSSLClientCertBindAddress <newIP>

Finally after restarting the Zimbra services I had a working environment with a new host name and ip address, so the original server could be brought back online with it's origianl name and ip address.

Hopefully the above will be of use to someone.

Regards,
Steve