occasional imapsync invalid date format error

Looking to migrate to ZCS? Ask here. Got a great tip or script that helped you migrate? Post it here.
oayfer
Posts: 31
Joined: Fri Sep 12, 2014 11:24 pm

occasional imapsync invalid date format error

Postby oayfer » Sun Mar 16, 2008 2:38 pm

Hello,
I am testing migration from a Solaris 9/dovecot system to Zimbra 5.0 on RHEL5. imapsync seems to work very well except for some occasional errors like:
Use of uninitialized value in concatenation (.) or string at /usr/bin/imapsync line 1340.

10791 BAD parse error: invalid date format

at /usr/bin/imapsync line 1358

Couldn't append msg #28587 (Subject:[something]) to folder INBOX: 10791 BAD parse error: invalid date format

By occasional, I mean a few hundred such erroneous e-mails for the entire migration of 300000+ e-mails.
I tried finding the bad date headers, but can't see anyhting wrong with them. Here are two examples:
Date: Sun, 31 Dec 2006 17:18:22 -0800
Date: Mon, 31 Dec 2007 16:04:21 -0800

This led me to check the dates associated with more of the errors, and they are all December 31st!
Any ideas what is going on here?
Thanks.


oayfer
Posts: 31
Joined: Fri Sep 12, 2014 11:24 pm

occasional imapsync invalid date format error

Postby oayfer » Wed Mar 19, 2008 12:28 am

Running with the --debug flag produces these two interesting lines:
internal date from 1: [31-Dec-2007 16:04:27 +4000]

internal date from 1: [""] (fixed)

compared to a good e-mail that produces:
internal date from 1: [18-Mar-2008 18:34:21 -0700]

internal date from 1: ["18-Mar-2008 18:34:21 -0700"] (fixed)

+4000 doesn't make sense as a time offset, and I still don't know where that comes from.
In my understanding, the internal dates as derived from an mbox file come from the very first line of an e-mail that begins with "From ". In this example, those lines are as follows:
broken:

From user@domain.com  Mon Dec 31 16:04:27 2007

not broken:

From user@domain.com  Tue Mar 18 18:34:21 2008

On the same system where I'm running imapsync, I have tried this tiny perl program:



#!/usr/bin/perl
require Date::Manip;

Date::Manip->import(qw(ParseDate Date_Cmp UnixDate));
$a = 'Mon Dec 31 16:04:27 2007';
$d = ParseDate($a);
$dd = UnixDate($d, "%d-%b-%Y %H:%M:%S %z");
print $dd;

print "

";

which returns:



31-Dec-2007 16:04:27 -0700

Which is the correct answer (note -0700 vs +4000). So the date manipulations involved in imapsync seem to be working correctly.
This is the last roadblock I have come across in the process of migrating to Zimbra. Any help would be greatly appreciated.
Thanks.
bthom73
Posts: 18
Joined: Fri Sep 12, 2014 11:59 pm

occasional imapsync invalid date format error

Postby bthom73 » Mon Mar 28, 2011 10:39 pm

Hi Oayfer,
Oddly enough, I just ran into the same thing. Any luck figuring out what was causing it?
The user in question here is migrating from Dovecot as well, so maybe it's somehow related to Dovecot? Or maybe a bug in imapsync?
Thanks,

Brian
oayfer
Posts: 31
Joined: Fri Sep 12, 2014 11:24 pm

occasional imapsync invalid date format error

Postby oayfer » Mon Mar 28, 2011 11:53 pm

Over 3 years later I'm having trouble remembering the details but I think we ended up simply isolating the handful of messages among hundreds of thousands and dealt with them individually. I will look through my old notes when I get a chance and will post again if I find any further clues.
bthom73
Posts: 18
Joined: Fri Sep 12, 2014 11:59 pm

occasional imapsync invalid date format error

Postby bthom73 » Tue Mar 29, 2011 6:17 am

Thanks for the quick reply. I also noticed that the troublesome messages are within hour range of the new year, accounting for timezone offset. We're in the Eastern timezone (UTC-5 in December) and all of the troublesome messages are within 5 hours of midnight on Dec 31st. So, I'm assuming it's some sort of new year rollover issue.
Although our Zimbra server is running Ubuntu 8.04LTS (and that's where I was running imapsync from), as a test I just did a fresh install of Ubuntu 10.04.1LTS on a separate machine and tried to do the imapsync again from there. Same issue. So, if the issue is a bug in imapsync or a bug in PERL's Date::Manip (called by imapsync), it still seems to be broken.
If you run into any notes, please let me know.
Thanks again,

Brian
oayfer
Posts: 31
Joined: Fri Sep 12, 2014 11:24 pm

occasional imapsync invalid date format error

Postby oayfer » Wed Mar 30, 2011 12:27 am

Found some notes. It seems we thought this was a Solaris problem at the time, but you have proven this to be wrong. Perhaps the bug is in the Perl libraries. In any case, since this affected so few messages, we fixed this using the following addition (in Perl) to our migration scripts:
[QUOTE] $d = $f_idate;

$debug and print "internal date from 1: [$d]
";
# this is a horrible TERRIBLE workaround for dovecot/solaris9/mbox

# returning +4000 as the tz offset for dates near a year boundary

# like dec31 and jan1. don't do this. this really is terrible.

# it may make some internaldates wrong by 1hr or something.

#

if ($d =~ /+4000$/) {

$d =~ s/+4000$/-0700/;

$debug and print "internal date from 1: [$d] FIXED solaris9 bug
";

}
require Date::Manip;

Date::Manip->import(qw(ParseDate Date_Cmp UnixDate));[/QUOTE]

Hope this helps. Feel free to reach out with additional questions, I try to help the community as much as I can.

Return to “Migration”

Who is online

Users browsing this forum: No registered users and 2 guests