file wrangling help requested

footfootfoot · 02-23-2011, 11:40 AM

I've got a problem with one of my back up drives due to poor computer hygiene.

There are about 50,000 files on it, of which a huge number are duplicates residing in a series of nested folders.

For example,
L:\music\elvis costello\my aim is true has tracks 1,2,4,5,7,8,9

and
L:\music\itunes music\~E\elvis costello\my aim is true has tracks 1,2,3,6,10,11

and
L:\music\MISC Vinyl\elvis costello\my aim is true has a live version of track 3 with the same file name as track 3 above, but a different bit rate and time.

How can I get all these organized, deleting the dupes and not deleting the similarly named songs.

I've tried using digital volcano's Duplicate Cleaner with some success, but it still misses dupes when using MD5 or byte by byte, and it comes up with false positives when there are multiple versions of songs, especially a problem with greatest hits and compilation discs.

Any suggestions?

glatt · 02-23-2011, 11:49 AM

You need an intern. Post an opening at your local college. Spring semester is internship time.

Perry Winkle · 02-23-2011, 12:00 PM

I don't know of any out of the box software.

If you search around for a Python or Ruby script, I'm sure you could find one.

Basically you want something that will create an index of all of your music based on something like an md5 checksum of the file (the filename doesn't matter). Then it should remove duplicates, and maybe move them all to a consistent location.

Perry Winkle · 02-23-2011, 12:02 PM

This will give you a list of all of the duplicates.

footfootfoot · 02-23-2011, 12:34 PM

OK, I will get an intern to run that script for me.

Gravdigr · 03-01-2011, 01:38 AM

Hah!

02-23-2011, 11:40 AM	#1
footfootfoot To shreds, you say? Join Date: Aug 2004 Location: in the house and on the street-how many, many feet we meet! Posts: 18,449	file wrangling help requested I've got a problem with one of my back up drives due to poor computer hygiene. There are about 50,000 files on it, of which a huge number are duplicates residing in a series of nested folders. For example, *L:\music\elvis costello\my aim is true* has tracks 1,2,4,5,7,8,9 and *L:\music\itunes music\~E\elvis costello\my aim is true* has tracks 1,2,3,6,10,11 and *L:\music\MISC Vinyl\elvis costello\my aim is true* has a live version of track 3 with the same file name as track 3 above, but a different bit rate and time. How can I get all these organized, deleting the dupes and not deleting the similarly named songs. I've tried using digital volcano's Duplicate Cleaner with some success, but it still misses dupes when using MD5 or byte by byte, and it comes up with false positives when there are multiple versions of songs, especially a problem with greatest hits and compilation discs. Any suggestions? __________________ The internet is a hateful stew of vomit you can never take completely seriously. - Her Fobs

02-23-2011, 12:34 PM	#5
footfootfoot To shreds, you say? Join Date: Aug 2004 Location: in the house and on the street-how many, many feet we meet! Posts: 18,449	OK, I will get an intern to run that script for me. __________________ The internet is a hateful stew of vomit you can never take completely seriously. - Her Fobs

03-01-2011, 01:38 AM	#6
Gravdigr The Un-Tuckian Join Date: Apr 2007 Location: South Central...KY that is Posts: 39,517	Hah! __________________ These statements have not been evaluated by the FDA, EPA, FBI, DEA, CDC, or FDIC. These statements are not intended to diagnose, cause, treat, cure, or prevent any disease. If you feel you have been harmed/offended by, or, disagree with any of the above statements or images, please feel free to fuck right off.

02-23-2011, 11:49 AM	#2
glatt ™ Join Date: Jul 2003 Location: Arlington, VA Posts: 27,717	You need an intern. Post an opening at your local college. Spring semester is internship time.

02-23-2011, 12:00 PM	#3
Perry Winkle Esnohplad Semaj Ton Join Date: Feb 2005 Location: A little south of sanity Posts: 2,259	I don't know of any out of the box software. If you search around for a Python or Ruby script, I'm sure you could find one. Basically you want something that will create an index of all of your music based on something like an md5 checksum of the file (the filename doesn't matter). Then it should remove duplicates, and maybe move them all to a consistent location.

02-23-2011, 12:02 PM	#4
Perry Winkle Esnohplad Semaj Ton Join Date: Feb 2005 Location: A little south of sanity Posts: 2,259	This will give you a list of all of the duplicates.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)