Data Cleaning and Deduping

Hi...we are looking for a very EXPERIENCED person or company who has software such as Trillium that can help us clean 150,000 Business Listings. Cleaning includes: 1) Looking for fuzzy duplicates based on several queries (matches by address, phone, biz name, etc.) and carefully delete listings which seem duplicates..this process may start with highest level matches, manual review by me to make sure we are not deleting good data, then a 2nd level match where not all fields are alike, again a manual review by me before we auto-delete, etc...so will involve some back and forth. We also might need some customized coding to tell the database which listing to delete, because we have ranked the data according to the source: first we prefer to keep User-entered data and delete others, next is recently pulled data, 3rd is old pulled data, etc. Also if 2 listings are alike but one has more info than the other (email, website address, or 4 lines of business description) we need to merge the listings and fill the maximum fields before deleting the extra one. 2) Filling in all missing zip codes, states, cities and area codes - we have the US Post Office database with all 4 of these fields, so just have to match it against the database. 100% accuracy not needed... 3) Converting misspelled or lower-caps or ALL CAPS to proper formatting (I think many softwares have this feature). We are not expecting manual cleaning, automated only, all manual will be done by me. You will get our database of 150,000 listings, and the US Postal database, and basically we will both work together and try to clean the database again and again until we can clean no more. The format is in Excel right now, we can convert it to Access if needed. If you can do a good job for us, you will have my complete loyalty for all future projects in this area as we have ongoing data cleaning throughout the year. ## Deliverables 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables): a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment. b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request. 3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement). * * *This broadcast message was sent to all bidders on Wednesday Oct 4, 2006 5:08:53 PM: Hi - Just wanted to give an update. I have found several advanced softwares that already have the artificial intelligence programmed in to really find fuzzy duplicates of all kinds. Coding the intelligence from scratch seems unnecessary as there are at least 10 prize-winning softwares available that do this already. I am going through demos of these softwares and researching through next week...meanwhile I will need help making the final decision on what software to buy and then help to use it to clean our data and add custom queries...so will still need you, but in a different way, less coding and more data manipulation. Please give me a few days, will be in touch again soon. ## Platform Excel or Access is fine