[TUHS] ML archive

[Random832]

On Sat, Jul 2, 2016, at 07:00, Michael Welle wrote:
> Hello,
> 
> I want to complete my local ML archive (I deleted a few emails and I
> wasn't subscribed before 2001 or so I think). After downloading the
> archives and hitting them a few times to get somewhat importable mboxes,
> I ended with 8699 emails in a maildir (in theory that should be a
> superset of the 5027 emails in my regular TUHS maildir. I will merge
> them next.). Two dozens mails are obviously defective (can be repaired
> manually maybe) and some more might be defective (needs deeper
> checking). So, has anybody more ;)?

Gah, the archive files from before 2002 are a nightmare.

My best guess on the proper message count is 8675. This is the number of
blocks of non-blank lines which either start with "From " (the easy
case) or, for the hard case, meet the following conditions:

In a file dated 2001 or earlier
First line contains a colon
Contains at least one line starting with "Received:"

There might still be a handful of messages this fails to split out. I
would recommend interpreting files dated October 2001 or later as a
strict MBOXO archive, and only doing special processing to files dated
September 2001 or earlier (I haven't factored out my own script to be
able to do anything other than count messages)