New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Ask questions about your setup or get help installing ZCS server (ZD section below).
halfgaar
Advanced member
Advanced member
Posts: 181
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by halfgaar »

zmcontrol wrote: Thu Mar 07, 2024 12:08 am Edit ldap dump to remove all instances of development attributes.

Can I ask what you mean by that exactly? I diff'ed a list of attributes and I'm going through ldap dumps, but these files are quite messy. The attribute names appear all over the file, even as broken strings (broken with newlines).

They appear as 'olcObjectIdentifier', 'olcAttributeTypes' and 'olcObjectClasses'. The last one is especially messy.

I'm not sure what exactly to remove, let alone write a script.

Or do you mean only the file 'ldap.bak' it creates? Then what about the 'ldap-config.bak'?
Consider seriously: because of the history of exploits: block Zimbra web interface with VPN, firewall or HTTP proxy.
zmcontrol
Posts: 34
Joined: Fri Jul 24, 2020 12:43 am

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by zmcontrol »

halfgaar wrote: Mon May 20, 2024 12:58 am Or do you mean only the file 'ldap.bak' it creates? Then what about the 'ldap-config.bak'?
halfgaar,

Yes, I only edited the ldap dump not the config.

One way to compare and edit:
Unzip the attached attr.txt file.

Compare with grep to check if any unknown attributes are present.

Code: Select all

grep -f attr.txt ldap.bak
Remove unknown attributes with sed.

Code: Select all

for i in `cat attr.txt`; do sed -i '/'$i'/d' ldap.bak; done
The attached attribute list was created on March 3rd, so you will need to check when your build was made.
Attachments
attr.zip
(1.01 KiB) Downloaded 41 times
halfgaar
Advanced member
Advanced member
Posts: 181
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by halfgaar »

zmcontrol wrote: Tue May 21, 2024 1:04 am
halfgaar wrote: Mon May 20, 2024 12:58 am Or do you mean only the file 'ldap.bak' it creates? Then what about the 'ldap-config.bak'?
halfgaar,

Yes, I only edited the ldap dump not the config.

One way to compare and edit:
Unzip the attached attr.txt file.

Compare with grep to check if any unknown attributes are present.

Code: Select all

grep -f attr.txt ldap.bak
Remove unknown attributes with sed.

Code: Select all

for i in `cat attr.txt`; do sed -i '/'$i'/d' ldap.bak; done
The attached attribute list was created on March 3rd, so you will need to check when your build was made.
Ah, ok, that makes it easier. When I only compare ldap.bak, the NEW attributes only appear in predictable places: config definitions, and for users who have a config value there. I will give it a go.

As a side note, I'm attaching an updated list. The filename shows between which versions. It contains about 8 or so more than yours.
Attachments
zimbra-attrs-10.0.6-to-dev-a60eb55a-2024-04-20-diff.zip
(1.21 KiB) Downloaded 74 times
Consider seriously: because of the history of exploits: block Zimbra web interface with VPN, firewall or HTTP proxy.
nguyennv94
Posts: 1
Joined: Tue Nov 20, 2018 10:20 am

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by nguyennv94 »

I have tried and removed some of the following attibutes.
And I upgraded from dev builds 10.0.7 to tag builds 10.0.8. Every Ok.
Backup your server before upgrade.
My flow

Code: Select all

su - zimbra
source ~/bin/zmshutil
zmsetvars
ldapmodify -x -H $ldap_master_url -D "uid=zimbra,cn=admins,cn=zimbra" -w $zimbra_ldap_password
dn: cn=config,cn=zimbra
changetype: modify
delete: zimbraDomainLoginPageEnabled
attibutes need to remove.

Code: Select all

###################################
zimbraDomainLoginPageEnabled
zimbraCountAccountsEnabled
zimbraEventIndexReplicationFactor
zimbraFeatureMailRecallEnabled
zimbraFeatureMailRecallTime
zimbraMaxSolrBatchDeletionSize
zimbraModernWebClientDisabled
zimbraSolrBatchDeletionInterval
zimbraTwoFactorAuthEmailCodeLength
Last edited by nguyennv94 on Tue May 21, 2024 4:45 am, edited 1 time in total.
halfgaar
Advanced member
Advanced member
Posts: 181
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by halfgaar »

nguyennv94 wrote: Tue May 21, 2024 4:44 am I have tried and removed some of the following attibutes.
And I upgraded from dev builds 10.0.7 to tag builds 10.0.8. Every Ok.
Backup your server before upgrade.
My flow
...
Brilliant. I was already looking into a way of using ldapmodify and your trick with the shell helpers was the missing piece. Here is a script that fixes my Zimbra's LDAP, by removing the attributes and setting the ldap schema version. I also attached it, for portability.

It currently hardcodes the LDAP schema version. Change to your needs.

And I can (almost) confirm it works. The upgrade to Maldua's 10.0.8 worked on a test copy of the server. That is, it finished. Then I shut off the server.

Code: Select all

#!/bin/bash
#
# Remove attributes from Zimbra's ldap from a supplied list, to bring it back
# to a tagged version. Example list is:
# zimbra-attrs-10.0.6-to-dev-a60eb55a-2024-04-20-diff.txt
#
# Author: wiebe@halfgaar.net

if [[ "$USER" != "zimbra" ]]; then
  echo "Run this as user zimbra"
  exit 1
fi

ATTR_LIST="$1"
RESET_LDAP_SCHEMA_VERSION="1673397105"

if [[ ! -f "$ATTR_LIST" ]]; then
  echo "You need to supply the list with zimbra attributes as first argument"
  exit 1
fi

source ~/bin/zmshutil
zmsetvars

ok_count=0
fail_count=0

while read -r attr; do
  echo -n "Doing $attr..."
  echo "dn: cn=config,cn=zimbra
changetype: modify
delete: $attr
" | ldapmodify -x -H "$ldap_master_url" -D "uid=zimbra,cn=admins,cn=zimbra" -w "$zimbra_ldap_password" &> /dev/null
  result="$?"

  if [[ "$result" -eq 0 ]]; then
    echo -e "\033[01;32mOK\033[00m"
    ok_count=$(( ok_count + 1 ))
  else
    echo -e "\033[01;33mFail - or attribute not found\033[00m"
    fail_count=$(( fail_count + 1 ))
  fi
done < "$ATTR_LIST"

echo

echo "Setting zimbraLDAPSchemaVersion to $RESET_LDAP_SCHEMA_VERSION"

echo "dn: cn=config,cn=zimbra
changetype: modify
replace: zimbraLDAPSchemaVersion
zimbraLDAPSchemaVersion: $RESET_LDAP_SCHEMA_VERSION" | ldapmodify -x -H "$ldap_master_url" -D "uid=zimbra,cn=admins,cn=zimbra" -w "$zimbra_ldap_password" &> /dev/null

echo
echo "Done. OK=$ok_count. Fail=$fail_count"
Attachments
script-and-attr-list-for-10.0.6.zip
(2.08 KiB) Downloaded 69 times
Consider seriously: because of the history of exploits: block Zimbra web interface with VPN, firewall or HTTP proxy.
User avatar
adrian.gibanel.btactic
Advanced member
Advanced member
Posts: 161
Joined: Thu Jan 30, 2014 11:13 am

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by adrian.gibanel.btactic »

zmcontrol wrote: Thu Mar 07, 2024 12:08 am Edit ldap dump, change attribute 'zimbraLDAPSchemaVersion' value to the 10.0.6/7 version '1673397105'.
Do you actually need to update 'zimbraLDAPSchemaVersion' ?
I know that zmsetup.pl is going to set the proper 'zimbraLDAPSchemaVersion' in the end so there would be no need to change that value prior to the installation.
Unless... you aware this attribute value being checked in the installation?

So... Is it checked somewhere or you changed that value just in case?
zmcontrol
Posts: 34
Joined: Fri Jul 24, 2020 12:43 am

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by zmcontrol »

adrian.gibanel.btactic wrote: Thu May 23, 2024 4:04 pm So... Is it checked somewhere or you changed that value just in case?
adrian.gibanel.btactic,

I changed it on general principle, however I didn't verify if the setup script checks for downgrading.
I did confirm that the upgrade process overwrites zimbra-attrs.xml, zimbra-attrs-schema, and the ldap config.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 910
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P40 NETWORK Edition

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by JDunphy »

I had looked into where the value for /opt/zimbra/conf/zimbra-attrs.xml was coming from and they are building it in the tarball with git syntax.
Ref: viewtopic.php?p=313213#p313213

I speculated that adding the correct value into that file would probably do the downgrade. Didn't test but the logic looked like they would open the file and set the value based on that... so if we are building based on the tag level then cd into zm-mailbox and issue this command.

Code: Select all

% pwd
/home/jad/build-zimbra/zmbuild/my-automated-build/zm-mailbox
%  git log -1 --pretty='format:%at' store/conf/attrs/zimbra-attrs.xml
1673397105
Put that value into /opt/zimbra/conf/zimbra-attrs.xml

They have this logic in zmsetup.pl:

Code: Select all

# zmsetup.pl (code of interest)
chomp (my $ldapSchemaVersion = do {
    local $/ = undef;
    open my $fh, "<", "/opt/zimbra/conf/zimbra-attrs-schema"
        or die "could not open /opt/zimbra/conf/zimbra-attrs-schema: $!";
    <$fh>;
});
sub configLDAPSchemaVersion {
  return if ($haveSetLdapSchemaVersion);
  if (isEnabled("zimbra-ldap")) {
    progress ("Updating zimbraLDAPSchemaVersion to version '$ldapSchemaVersion'\n");
    setLdapGlobalConfig('zimbraLDAPSchemaVersion', $ldapSchemaVersion);
    $haveSetLdapSchemaVersion = 1;
  }
}

    
Nice work in solving the downgrade. We should be able to automate it completely... I feel another option going into my build script. :-) ;-)

Jim
halfgaar
Advanced member
Advanced member
Posts: 181
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by halfgaar »

JDunphy wrote: Sat May 25, 2024 12:40 am I had looked into where the value for /opt/zimbra/conf/zimbra-attrs.xml was coming from and they are building it in the tarball with git syntax.
Ref: viewtopic.php?p=313213#p313213

I speculated that adding the correct value into that file would probably do the downgrade. Didn't test but the logic looked like they would open the file and set the value based on that... so if we are building based on the tag level then cd into zm-mailbox and issue this command.

Code: Select all

% pwd
/home/jad/build-zimbra/zmbuild/my-automated-build/zm-mailbox
%  git log -1 --pretty='format:%at' store/conf/attrs/zimbra-attrs.xml
1673397105
Put that value into /opt/zimbra/conf/zimbra-attrs.xml

They have this logic in zmsetup.pl:

Code: Select all

# zmsetup.pl (code of interest)
chomp (my $ldapSchemaVersion = do {
    local $/ = undef;
    open my $fh, "<", "/opt/zimbra/conf/zimbra-attrs-schema"
        or die "could not open /opt/zimbra/conf/zimbra-attrs-schema: $!";
    <$fh>;
});
sub configLDAPSchemaVersion {
  return if ($haveSetLdapSchemaVersion);
  if (isEnabled("zimbra-ldap")) {
    progress ("Updating zimbraLDAPSchemaVersion to version '$ldapSchemaVersion'\n");
    setLdapGlobalConfig('zimbraLDAPSchemaVersion', $ldapSchemaVersion);
    $haveSetLdapSchemaVersion = 1;
  }
}

    
Nice work in solving the downgrade. We should be able to automate it completely... I feel another option going into my build script. :-) ;-)

Jim
I doubt downgrade logic will have been implemented. Reversible migrations are tricky.

I suspect '/opt/zimbra/conf/zimbra-attrs-schema' is merely there to protect against running against the wrong LDAPA schema? Zimbra may verify it with 'zimbraLDAPSchemaVersion' and error out when they don't match? I can try easily, later.
Consider seriously: because of the history of exploits: block Zimbra web interface with VPN, firewall or HTTP proxy.
zmcontrol
Posts: 34
Joined: Fri Jul 24, 2020 12:43 am

Re: New Patch Releases 10.0.7 and 9.0.0 Patch 39 - Installation Results

Post by zmcontrol »

zmcontrol wrote: Tue May 21, 2024 1:04 am Remove unknown attributes with sed.

Code: Select all

for i in `cat attr.txt`; do sed -i '/'$i'/d' ldap.bak; done
Goes without saying, but please use the above sed command with caution when testing!

Be aware that if an attribute value exceeds the LDIF limit of the number of columns it 'soft' line breaks.
The newline is indented but belongs to the same value.

For example when testing, if after checking for unknown attribues any unknown values exceed more than one line.
Replace the '\n ' with a random marker for example '%M@rk3R!#', so all values are on one line.

Code: Select all

sed ':a;N;$!ba;s/\n /%M@rk3R!#/g' ldap.bak > ldap.nobreaks.bak
Remove unknown attributes/values with sed.

Code: Select all

for i in `cat attr.txt`; do sed -i '/'$i'/d' ldap.nobreaks.bak; done
Replace the 'marker' with the required '\n ' .

Code: Select all

sed 's/%M@rk3R!#/\n /g' ldap.nobreaks.bak > ldap.downgrade.bak
However in my testing no unknown attribute values exceeded the column limit.
Post Reply