zmprov Scripting performance tips

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
keros01
Posts: 3
Joined: Fri Jan 11, 2019 2:34 pm

zmprov Scripting performance tips

Post by keros01 »

Hi

I currently write a script for zimbra to get some informations from all accounts.
The script is realy small but takes forever because I can't find a good way to interact with zmprov

Code: Select all

# get all accounts
all_accounts=$(zmprov -l gaa)
echo "$all_accounts" > all_accounts.txt

for account in $all_accounts
do
    zmprov ga "$account" > ${account}.cfg
done

# this takes about 5 seconds for every account
# for about 130 accounts this takes ~11 minutes :(
I know that zmprov is much faster if I create a "zmprov" file an pass a patch of commands to zmprov.
Like this:

Code: Select all

echo "ga user01@mydomain.com" >> zmprov_command.txt
echo "ga user02@mydomain.com" >> zmprov_command.txt
echo "ga user03@mydomain.com" >> zmprov_command.txt
echo "ga user04@mydomain.com" >> zmprov_command.txt
...
zmprov -f zmprov_command.txt > some_output_file.txt

# this takes about 15 seconds for ~130 accounts
# much faster
But I need every user in its own output file.
Also "getaccount (ga)" is not the only command I need (I also want the cos by name for every account, ...).
I also have tried a ldapsearch, this is fast but also not a good solution for the things I wan't to do.

Has someone an idea how I can speed things up here and don't lose the benefits of using single "zmprov" commands?
I can live with a patch solution but I need a good way to keep the output seperated.

keros
User avatar
ccelis5215
Outstanding Member
Outstanding Member
Posts: 632
Joined: Sat Sep 13, 2014 2:04 am
Location: Caracas - Venezuela
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 P12

Re: zmprov Scripting performance tips

Post by ccelis5215 »

Hi keros,

Just split the output by pattern.

For example:

Code: Select all

awk '/prov> # name/{c++} {print > FILENAME"_"c }' some_output_file.txt
will generate a file per account.

ccelis
User avatar
ccelis5215
Outstanding Member
Outstanding Member
Posts: 632
Joined: Sat Sep 13, 2014 2:04 am
Location: Caracas - Venezuela
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 P12

Re: zmprov Scripting performance tips

Post by ccelis5215 »

After a google search found the complete solution:

Code: Select all

for file in some_output_file_*; 
do   
   acc_name=$(grep  "prov" "$file" |  awk '{print $4}');   
   new_filename="${acc_name}.cfg";   
   mv "$file" "$new_filename"; 
done
ccelis
keros01
Posts: 3
Joined: Fri Jan 11, 2019 2:34 pm

Re: zmprov Scripting performance tips

Post by keros01 »

I really hoped that I don't need to split the output myself. But ok.
I also need other data like all folders per account from zmmailbox so I created a function for splitting the data.
This takes ~15 seconds for ~130 Accounts.
Maybe there is a more elegant way for this. But it works so I'm happy.

I found the "zmpython" command, but without docu I stay with bash and the documented zmprov/zmmailbox/...

Code: Select all

function splitData()
{
        data="$1"
        split_string="$2"
        split_pos="$3"
        data_file_name="$4"
        data_base_fodler="$5"

        data_file_path=""
        while read line
        do
                # search  for the splitstring
                if [[ "$line" == "$split_string"* ]]
                then
                        data_name="$(echo "$line" | awk '{print $'$split_pos'}')"
                        data_folder="$data_base_fodler/$data_name"
                        data_file_path="$data_folder/$data_file_name"
                        mkdir "$data_folder"
                        touch "$data_file_path"
                fi

                # make sure we have a file we can write to
                if [ -z "$data_file_path" ]
                then
                        echo "Error while splitting the output into files"
                        echo "No account name found"
                        exit 1
                fi

                echo "$line" >> "$data_file_path"
        done < <(echo "$data")
}


function splitAccountData()
{
        data="$1"
        splitData "$data" "# name " "3" "account_config.cfg" "/tmp/zimbra_accounts/"
}

all_accounts="$(ssh root@$zimbra_server sudo -u zimbra $zmprov -l gaa -v "$domain")"
splitAccountData "$all_accounts"
keros01
Posts: 3
Joined: Fri Jan 11, 2019 2:34 pm

Re: zmprov Scripting performance tips

Post by keros01 »

I found a much better solution.
If I use named pipes I can create a daemon like zmpov command

Code: Select all

mkfifo /tmp/zmprov_pipe
touch /tmp/zmprov_out.txt

tail -f /tmp/zmprov_pipe > >(zmprov > /tmp/zmprov_out.txt) &
zmprov_pid=$1

echo "PID: $zmprov_pid"
wait "$zmprov_pid"
Now I can send single commands to zmprov without the starting overhead

Code: Select all

echo "ga user01@mydomain.com" > /tmp/zmprov_pipe
# here you need to wait some time (I have made function that checks the filesize)
head -n -1 "/tmp/zmprov_out.txt" | sed 's/[\x0]//g'
> "/tmp/zmprov_out.txt"

echo "ga user01@mydomain.com" > /tmp/zmprov_pipe
# here you need to wait some time (I have made function that checks the filesize)
head -n -1 "/tmp/zmprov_out.txt" | sed 's/[\x0]//g'
> "/tmp/zmprov_out.txt"

echo "ga user01@mydomain.com" > /tmp/zmprov_pipe
# here you need to wait some time (I have made function that checks the filesize)
head -n -1 "/tmp/zmprov_out.txt" | sed 's/[\x0]//g'     # remove the last line and remove all null bytes (I have currently not found a way to reset the offset of the file descriptor in the first script)
> "/tmp/zmprov_out.txt"
...

# To stop kill the pid from the first script
kill PID_FROM_FIRST_SCRIPT
I have now written something like a zmprov wrapper daemon which I can start and stop and my own zmprov_query command which puts the command to the pipe and returns the output of the file
Post Reply