Microsoft Outlook – duplicate email messages Part II
This is part II, click here for part one of the saga…
Okay, so I revisit to install Microsoft Office 2003 SP2 and hope that solves it. Somewhere along this time, I had also found a suggestion of removing the entire mail PROFILE and creating a new one. *(The theory being that previous account information may still be in a profile in spite of the mail account being deleted.) Again two or three days go by and just when you thought it was fixed, duplicates AGAIN…. This time I had log information though, finally a second piece of log information to work with.
It took several passes over it to catch what was happening but here’s how I laid it out to the web host…
For example, here’s a successful check… the server greets at login
with OK user has 49 visible messages…, then client STAT’s and receives
the appropriate response….
Authorizing to server
2005.10.26 18:03:41 POP3 (mail.site.net): [tx] USER user@site.net
2005.10.26 18:03:41 POP3 (mail.site.net): <rx>
+OK Password required for
user.
2005.10.26 18:03:41 POP3 (mail.site.net): [tx] PASS *****
2005.10.26 18:03:41 POP3 (mail.site.net): <rx>
+OK user has 49 visible
messages (0 hidden) in 397827 octets.
2005.10.26 18:03:41 POP3 (mail.site.net): Authorized to host
2005.10.26 18:03:41 POP3 (mail.site.net): Connected to host
2005.10.26 18:03:41 POP3 (mail.site.net): [tx] STAT
2005.10.26 18:03:41 POP3 (mail.site.net): <rx>
+OK 49 397827
2005.10.26 18:03:41 POP3 (mail.site.net): [tx] UIDL
2005.10.26 18:03:41 POP3 (mail.site.net): <rx>
+OK uidl command
accepted.
Here’s the connection where Outlook decides nothing’s there…
Authorizing to server
2005.10.26 18:27:47 POP3 (mail.site.net): [tx] USER user@site.net
2005.10.26 18:27:47 POP3 (mail.site.net): <rx>
+OK Qpopper (version
4.0.4, modified by Sphera Technologies) at site.net starting.
<13150.1130365667@sphera>
2005.10.26 18:27:47 POP3 (mail.site.net): [tx] PASS *****
2005.10.26 18:27:47 POP3 (mail.site.net): <rx>
+OK Password required for
user.
2005.10.26 18:27:47 POP3 (mail.site.net): Authorized to host
2005.10.26 18:27:47 POP3 (mail.site.net): Connected to host
2005.10.26 18:27:47 POP3 (mail.site.net): [tx] STAT
2005.10.26 18:27:47 POP3 (mail.site.net): <rx>
+OK user has 49 visible
messages (0 hidden) in 397827 octets.
The process seems ok – but on a closer look the server is one step behind? Is Outlook not waiting long enough???
Here’s another example…
2005.10.19 13:57:47 POP3 (mail.domain.com): Connected to host
2005.10.19 13:57:47 POP3 (mail.domain.com): <rx>
+OK Qpopper
(version 4.0.4, modified by Sphera Technologies) ready.
<28442.1129744666@sphera>
2005.10.19 13:57:47 POP3 (mail.domain.com): Authorizing to server
2005.10.19 13:57:47 POP3 (mail.domain.com): [tx] USER
mailuser@domain.com
2005.10.19 13:57:47 POP3 (mail.domain.com): <rx>
+OK Qpopper
(version 4.0.4, modified by Sphera Technologies) at domain.com
starting. <28442.1129744666@sphera>
2005.10.19 13:57:47 POP3 (mail.domain.com): [tx] PASS *****
2005.10.19 13:57:47 POP3 (mail.domain.com): <rx>
+OK Password
required for mailuser.
2005.10.19 13:57:47 POP3 (mail.domain.com): Authorized to host
2005.10.19 13:57:47 POP3 (mail.domain.com): Connected to host
2005.10.19 13:57:47 POP3 (mail.domain.com): [tx] STAT
2005.10.19 13:57:48 POP3 (mail.domain.com): <rx>
+OK mailuser has
8 visible messages (0 hidden) in 251220 octets.
The server’s giving TWO GREETINGS?
2005.10.19 13:57:47 POP3 (mail.domain.com): <rx>
+OK Qpopper
(version 4.0.4, modified by Sphera Technologies) ready.
and then
2005.10.19 13:57:47 POP3 (mail.domain.com): <rx>
+OK Qpopper
(version 4.0.4, modified by Sphera Technologies) at domain.com
starting. <28442.1129744666@sphera>
Okay – time for some background….
This webhost uses two kinds of Domain hosting in VPS’s (Virtual Private Server’s)… One type of account is a Name-based account, where several domain names share a single IP address and so for logins, like email you have to pass user@domain.com instead of the traditional user… .in other words the server has to be told which domain you’re logging in as. The other type of account is an IP based account where the domain has a unique IP address. Outside of the need to clarify which domain you’re logging in for there should be no real difference (except you need IP based to anonymous ftp).
In comparison an IP based qpopper session looks like this…
+OK Qpopper (version 4.0.4, modified by Sphera Technologies) at domain.com starting. <22317.1133325894@domain.com>
user mailuser
+OK Password required for mailuser.
pass *****
+OK mailuser has 0 visible messages (0 hidden) in 0 octets.
So, it goes ahead and announces “Qpopper (version 4.0.4, modified by Sphera Technologies) at domain.com starting. “, instead of “+OK Qpopper
(version 4.0.4, modified by Sphera Technologies) ready.” It’s almost as if in high load periods the lag is long enough that Qpopper has time to announce a second time that it’s starting for a specific domain. So, if this is correct a switch to an IP-based account will work as there’s no “figure out what domain I am” step.
I approach the web tech support with this log analysis. They’re skeptical if I can duplicate the results. I tried with telnet to the POP server (as they did) and I speculated that I just could throw the data fast enough to catch the server welcoming twice, so I set up an older version of Outlook (98) to test from another network. Sure enough I did manage to catch it happening (checked mail every 3 minutes…) and saw it 2 or three times over several hours.
It’s interesting to note that Outlook 98 didn’t duplicate messages when this happened. As I understand it when Outlook XP came along there was a major rewrite of “already seen message” handling and that code was just revised for Outlook 2003, so they both should treat messages similarly when checking to see if it’s been seen before.
That was enough evidence for the tech support to suspect I was right in moving to an IP based account although they couldn’t guarantee it. They essentially asked me to let them know so they could pass word along to their techs if it solved the issue.
It has now been a month and the issue appears to be resolved. Ultimately I am not 100% certain that the duplicates will never recurr because of the way Outlook 2003 deals with the issue. Basically IF Outlook thinks it has a good (successful- i.e. no errors) transaction then it uses that as a baseline for what messages it’s seen. The next transaction if there’s something it didn’t see on it’s last check, it will download. Now, IF for some reason Outlook is “flustered” into thinking the transaction is well when, in fact, there were errors… it may assume that there were no messages and on the next event “wow, new messages!!!”
All in all, it’s the mail clients responsiblity to have a good plan for preventing duplicates. Pure and simple. POP3 was never really meant to be used that way anyway. IMAP is a much better choice if it’s available and anyone struggling with this kind of issue, I would recommend you look into IMAP if at all possible. Was it a server problem… in many ways yes it was a problem there too…. Two welcome messages is certainly a non-standard implementation of the POP protocol. What’s interesting though is that I tested several other mail clients and NONE other than Outlook seemed to fall prey to making duplicate messages as a result of duplicate welcome messages.
So, in summary, if you use either Outlook XP or Outlook 2003 AND your mail is hosted on a Name-based VPS (Virtual Private Server) (with Qpopper as the pop server) AND you have “leave a copy of messages on server” checked, then you MIGHT get duplicate messages occasionally. This machine was on 24/7 checking mail every 5 minutes, so it happened 0.3% of the mail transactions. and when it was checking every 12 minutes, it was about 0.277% of the time… so if you only open Outlook to check mail once a day it might happen once or twice a year, if that… maybe once a week if you leave Outlook open just 8 hours a day… (checking every 5 minutes)
Obscure huh??? What makes it trickier is that for any other mail client the “double welcome” transactions get shrugged off with no error message, so it’s not an issue. If you know of another mail client that get’s duplicates from this same exact scenario (double mailserver welcome in a Name-based VPS (Virtual Private server) account) I’d love to know….