Zimbra 10.1.16 Released - Please Post Your Patch/Upgrade Results Here

[btrent98]

Hello,
We’re also experiencing the same backup errors as Thomas after updating to 10.1.16 on an Ubuntu 20.04 host:

Code: Select all

[zimbra@server]$ /opt/zimbra/bin/zmbackup -f -a all                                  
Error occurred: Cannot invoke "String.equals(Object)" because the return value of "com.zimbra.cs.account.Server.getAttr(String)" is null

Looking through the previously linked page for “Transitioning Scheduler Configuration” for Backup Engine v2, we also encountered two discrepancies:
1) On the web admin console GUI, there is not an option for “Enable deduplication”
2) When trying to run

Code: Select all

zmprov ms <server_name> zimbraBackupDeduplication nodedupe 

we get the error:

Code: Select all

ERROR: account.INVALID_ATTR_NAME (invalid attr name: invalid attr name - unable to modify attributes: ldap host=<hostname>.com:389: zimbraBackupDeduplication: attribute type undefined)

Like Thomas, there is nothing in our logs about failing to run the backup. Our system reports (zmcontrol -v) that we are running: Release 10.1.16.GA.4633.UBUNTU20_64 NETWORK edition.
We are a long standing server (8+ years) and haven’t previously had backup issues. I wonder if the error indicates we are missing some configuration item?

[rnilam24]

Hi all,

after updating to 10.1.16 zmbackup seems broken:

Code: Select all

[zimbra@server]$ /opt/zimbra/bin/zmbackup -f -a all                                  
Error occurred: Cannot invoke "String.equals(Object)" because the return value of "com.zimbra.cs.account.Server.getAttr(String)" is null

No matter how I try to reconfigure the backup schedule, this error persists (--zip, --zipStore, --noZIp, --zstd, --csd, --csds, standard or autogrouped backup, everything yields the same error message). I even tried removing the old backups or to only backup one account -> no luck....

Any hints?

Regards
Thomas

Can you please provide below details to debug this issue further:
From the mailbox node, can you provide values for below attributes:

Code: Select all

zmprov gs zmhostname zimbraBackupDeduplication
zmprov gs zmhostname zimbraBackupDedupeCompressionType
zmprov gs zmhostname zimbraBackupCSDReset

Also, can you give patch versions from:
LDAP node:

Code: Select all

dpkg -l | grep zimbra-ldap-patch

Mailbox node:

Code: Select all

dpkg -l | grep zimbra-patch

[thomas.klaube]

Looking through the previously linked page for “Transitioning Scheduler Configuration” for Backup Engine v2, we also encountered two discrepancies:
1) On the web admin console GUI, there is not an option for “Enable deduplication”

I have this option, but only in the global config section, not in the server section of the admin UI. But checking/unchecking this option seemed to made no difference for me...

2) When trying to run

Code: Select all

zmprov ms <server_name> zimbraBackupDeduplication nodedupe 

we get the error:

Code: Select all

ERROR: account.INVALID_ATTR_NAME (invalid attr name: invalid attr name - unable to modify attributes: ldap host=<hostname>.com:389: zimbraBackupDeduplication: attribute type undefined)

This is different for me! I can set this option in the CLI on the server without warning/error. After I set this option the backup seems to be working for me. So I presume that this value (zimbraBackupDeduplication) was for some reason non existent after the patch/upgrade which made the backup fail and only after setting this value the backup works again.

If you can not set this value on your server, there could be a ldap issue. Maybe some ldap scheme changes were not applied correctly.

Regards
Thomas

[thomas.klaube]

Can you please provide below details to debug this issue further:
From the mailbox node, can you provide values for below attributes:

Code: Select all

zmprov gs zmhostname zimbraBackupDeduplication
zmprov gs zmhostname zimbraBackupDedupeCompressionType
zmprov gs zmhostname zimbraBackupCSDReset

Also, can you give patch versions from:
LDAP node:

Code: Select all

dpkg -l | grep zimbra-ldap-patch

Mailbox node:

Code: Select all

dpkg -l | grep zimbra-patch

zimbra-ldap-patch-10.1.16.1767758060-1.r9.x86_64
zimbra-patch-10.1.16.1770142646-2.r9.x86_64

zimbraBackupDeduplication: nodedupe <--- after setting this value in the CLI to "nodedupe" the backup was working again. So I presume this value was unset before and caused the error
zimbraBackupDedupeCompressionType: nocompression
zimbraBackupCSDReset: TRUE

Regards
Thomas

[adrian.gibanel.btactic]

Hi all,

after updating to 10.1.16 zmbackup seems broken:

Code: Select all

[zimbra@server]$ /opt/zimbra/bin/zmbackup -f -a all                                  
Error occurred: Cannot invoke "String.equals(Object)" because the return value of "com.zimbra.cs.account.Server.getAttr(String)" is null

For the record when upgrading from ZCS NE 10.1.15 to ZCS NE 10.1.16 this didn't happen to us. The backup with the v1 configuration was done properly.

I want to clarify what previous statement.
We didn't actually test the zmbackup -f -a all invocation in the CLI but the scheduled regular NE backup.

After the ZCS NE 10.1.15 to ZCS NE 10.1.16 upgrade was done the regular scheduled backups (no upgrade to v2) were done properly. Those were both full and incremental.

We have a custom zimbraBackupTarget set at global level just in case it matters.

More technical details as requested:

Code: Select all

sudo su - zimbra
zimbra@mail:~$ zmprov gs $(zmhostname) zimbraBackupDeduplication
# name mail.example.net
zimbraBackupDeduplication: dedupe

zimbra@mail:~$ zmprov gs $(zmhostname) zimbraBackupDedupeCompressionType
# name mail.example.net
zimbraBackupDedupeCompressionType: zstd

zimbra@mail:~$ zmprov gs $(zmhostname) zimbraBackupCSDReset
# name mail.example.net
zimbraBackupCSDReset: TRUE

zimbra@mail:~$ dpkg -l | grep zimbra-ldap-patch
ii  zimbra-ldap-patch                          10.1.16.1767758060-1.u22                                            amd64        Zimbra ldap Patch
zimbra@mail:~$ dpkg -l | grep zimbra-patch
ii  zimbra-patch                               10.1.16.1770142646-2.u22                                            amd64        Zimbra Network Patch

[adrian.gibanel.btactic]

zimbraBackupDeduplication: nodedupe <--- after setting this value in the CLI to "nodedupe" the backup was working again. So I presume this value was unset before and caused the error

Regards
Thomas

That might be it.

Did you actually restart all of the Zimbra services in all of your Zimbra nodes after applying the package upgrade?

[thomas.klaube]

That might be it.

Did you actually restart all of the Zimbra services in all of your Zimbra nodes after applying the package upgrade?

Yes I did. I am very sure on this one... We even restarted the VM after applying pending OS updates. The only thing that might be a bit special on this machine is, that (before patch 10.1.16 was applied) the previously configured backup schedule was "autogrouped" and not "standard".

Regards
Thomas

[btrent98]

Code: Select all

root@newzmoby:~# su - zimbra
zimbra@newzmoby:~$ zmprov gs $(zmhostname) zimbraBackupDeduplication
# name newzmoby.atcorp.com

zimbra@newzmoby:~$ zmprov gs $(zmhostname) zimbraBackupDedupeCompressionType
# name newzmoby.atcorp.com

zimbra@newzmoby:~$ zmprov gs $(zmhostname) zimbraBackupCSDReset             
# name newzmoby.atcorp.com

zimbra@newzmoby:~$ dpkg -l | grep zimbra-ldap-patch
ii  zimbra-ldap-patch                          10.1.16.1767758060-1.u20                                            amd64        Zimbra ldap Patch
zimbra@newzmoby:~$ dpkg -l | grep zimbra-patch
ii  zimbra-patch                               10.1.16.1770142646-2.u20                                            amd64        Zimbra Network Patch

For whatever reason, it looks like we don't have any of those 3 config variables defined?

[thomas.klaube]

For whatever reason, it looks like we don't have any of those 3 config variables defined?

Could you please check, if these values exist in the global config?

Code: Select all

zmprov gacf zimbraBackupDeduplication
zmprov gacf zimbraBackupDedupeCompressionType
zmprov gacf zimbraBackupCSDReset 

I suspect that these values were not set after installing the patch on the global and/or server level...

Regards
Thomas

[Lostaff]

Code: Select all

root@newzmoby:~# su - zimbra
zimbra@newzmoby:~$ zmprov gs $(zmhostname) zimbraBackupDeduplication
# name newzmoby.atcorp.com

zimbra@newzmoby:~$ zmprov gs $(zmhostname) zimbraBackupDedupeCompressionType
# name newzmoby.atcorp.com

zimbra@newzmoby:~$ zmprov gs $(zmhostname) zimbraBackupCSDReset             
# name newzmoby.atcorp.com

zimbra@newzmoby:~$ dpkg -l | grep zimbra-ldap-patch
ii  zimbra-ldap-patch                          10.1.16.1767758060-1.u20                                            amd64        Zimbra ldap Patch
zimbra@newzmoby:~$ dpkg -l | grep zimbra-patch
ii  zimbra-patch                               10.1.16.1770142646-2.u20                                            amd64        Zimbra Network Patch

For whatever reason, it looks like we don't have any of those 3 config variables defined?

We had the same problem.

The error:

ERROR: account.INVALID_ATTR_NAME (invalid attr name: invalid attr name - unable to modify attributes: ldap host=<hostname>.com:389: zimbraBackupDeduplication: attribute type undefined)

means that the zimbraBackupDeduplication attribute is missing from the zimbra schema.

It is missing because the LDAP server update was most likely performed with Zimbra services turned off.

With services turned off, the LDAP schema zimbra.ldif is not updated.

To resolve the issue:
we ran the following script: /opt/zimbra/lib/patches/ldap/ldap_schema_upgrade.sh
and after zmcontrol restart, this updates the Zimbra schema.

Remember to always take a snapshot of the VM first.

Best regards