Best way to dedupe my Thunderbird addressbook?

I've got a ton of duplicate contacts in my Thunderbird addressbook. I don't want to share them with Google, but I do want to find a way to de-dupe them. There was a duplicate contact manager plug in, but it seems to have been abandoned.

Can I do this at the commandline?

Solution 1:

I had the same problem several years ago, and wrote a very small python script to unify the LDIF export of a Thunderbird addressbook:

1. export addressbook as LDIF to e.g. abook.ldif
2. run cat abook.ldif | unify_ldif.py > abook_new.ldif
3. import abook_new.ldif again (maybe rename old addressbook before)

The script currently matches duplicate entries on email address and identical name, but this can of course be adapted (in function find_existing_entry). Does this work for you?

The program is here (EDIT: you need the python-ldap package):

#!/usr/bin/env python

import sys
from ldif import LDIFParser, LDIFWriter

def find_existing_entry(ldif_entries, ldif_entry):
    for dn, entry in ldif_entries.items():
        if 'mail' in ldif_entry and 'mail' in entry:
            for mail in ldif_entry['mail']:
                if 'mail' in entry and mail in entry['mail']:
                    return dn
        if 'cn' in ldif_entry and 'cn' in entry and ldif_entry['cn'][0] in entry['cn']:
            return dn
        if 'sn' in ldif_entry and 'sn' in entry and 'givenName' in ldif_entry and 'givenName' in entry and ldif_entry['sn'][0] in entry['sn'] and ldif_entry['givenName'][0] in entry['givenName']:
            return dn
    return ''

class MyLDIF(LDIFParser):
    def __init__(self, input, output):
        LDIFParser.__init__(self, input)
        self.writer = LDIFWriter(output)
        self.entries = {}
    def merge(self, dn, entry):
        if 'mail' in entry.keys():
            if 'mail' in self.entries[dn].keys():
                for mail in entry['mail']:
                    if mail not in self.entries[dn]['mail']:
                        self.entries[dn]['mail'].append(mail)
            else:
                self.entries[dn]['mail'] = entry['mail']
        for key in entry.keys():
            if key not in self.entries[dn].keys():
                self.entries[dn][key] = entry[key]

    def handle(self, dn, entry):
        if dn in self.entries.keys():
            self.merge(dn, entry)
        else:
            found_dn = find_existing_entry(self.entries, entry)
            if found_dn != '':
                self.merge(found_dn, entry)
            else:
                self.entries[dn] = entry
    def output(self):
        for dn, entry in self.entries.items():
            self.writer.unparse(dn, entry)

parser = MyLDIF(sys.stdin, sys.stdout)
parser.parse()
parser.output()