Why does a disk get split into subdirs?

[wolfer]

I have a big old library that’s seen many prior attempts to clean. After trying a Cleanup & Delete Dupes, I end up with a lot of new folders where the release is split across them.

Artist\Album (1)
Artist\Album (2)

How can I get songkong to consolidate these instead?

Sometimes these folders just contain a jpg. Is there any value to deleting ALL of the JPG’s in the whole library and just letting songkong recreate them? I don’t really care about the pictures other than it identifies the release in some way.

[toshni]

SongKong saves duplicate files in numbered directories (“Album (1), Album (2), …”).
The album covers are only saved (the file not the tag) if the option for this is selected.
I think removing duplicates only affects the audio files, not the cover images. There is a post of mine on removing duplicate files.
Also give a separate directory for the unmatched files in case they are not tagged correctly.
I hope I could help a bit and the online translator works well as I speak German.

[toshni]

I forgot: I wish you a Merry Christmas and a Happy New Year despite or because of these hard times.

[paultaylor]

Clean Up (Fix Songs) would most likely create two folders with same name except for number if you had rename enabled and multiple copies of the same file. SongKong is attempting to create multiple albums folders for the same album because that generally makes more sense than having all in one folder.

But it would be helpful if you could run Create Support Files then I can see if there is a problem somewhere.

Currently Delete Duplicates only deletes the audio files itself not the enclosing folder, but I agree if the folder now contains nothing except the cover art it would make sense to delete it, I will consider this as an enhancement request.emphasized text

[wolfer]

OK sending support files now.

Question - I’m sorry I ever started splitting apart compilations into separate folders. And from reading a diff thread, it sounds like one way to fix that is to put all the files in one directory and let songkong recreate the folder structures. That sounds promising if I could fix how I messed up all my compilations.

Years ago I started downloading SXSW torrents, which were 1000 songs each year, one for each band. Breaking those, and all the garage rock and psych compilations I’ve bought, into thousands of separate directories with one song has only been a pain in the ass.

Maybe that’s the next step I could try?

Thx

[paultaylor]

I’m not not quite sure what your problems are and what you are asking since it is different from your original question. Can you give me a more concrete example that I can work with please.

I have looked at your latest reports FixSongs0029 and FixSongs0034 and the results look good to me with matches for most songs.

and I cannot see any problem with Delete Duplicates results either.

[wolfer]

This is the only the latest run. And good that it’s cleaned up file names.

My mess evolved over years itunes issues and many attempts to fix using other apps and then songkong, which I think i bought more than 10 years ago. But at a certain point it seemed to make more sense to re-rip my whole collection to FLAC and then salvage what I can out of the older MP3’s and switch to Roon and never touch itunes again!

So now I’m just trying to lessen the mess of the parts I want to keep - mainly dupes and songs fragmented across copies of directories. This is an extreme example:

As a test, I’m trying to see what happens if I put all the recently cleaned songs (46,000+) into flat folder and then allow songkong to recreate the folders and download new album art.

Unfortunately even after a couple of days, Songkong seems to be stuck on the progress screen showing 0 songs processed. I’ve rebooted and tried a couple of times. Any suggestion?

Should I try to break the songs into smaller batches?

[paultaylor]

@wolfer

So now I’m just trying to lessen the mess of the parts I want to keep - mainly dupes and songs fragmented across copies of directories. This is an extreme example:

Not clear if the album was split over multiple subfolders, or if many of those subfolders was now empty and just needed deleting. Its seems from looking at your last report
that actually there were just a few folders containing songs, and you had a couple of copies of some discs.

If by cleaned you mean removed all the metadata then by removing the folder structure as well you have given nothing for SongKong to go on apart from running Acoustids and looking them up but this takes at least a second per song. SongKong works folder by folder, although there are some shortcuts to try and deal with massive folders it seems like in this case they haven’t kicked in and it possible trying to run acoustid on every song first before doing anything else.

Would have been better not to have done this rather drastic step, but since you have done it are you seeing the count for songs fingerprinted going up. As long as you are not running in preview mode then fingerprints should be saved so when you restarted it could skip over them but if you are running in preview mode then nothing is saved to file and so each time you run it would be like starting again.

[wolfer]

Thx for your help!

I am running it in preview mode and after two days all the progress indicators just said 0. Even stopping it is slow. I killed the task the other day but now I’m just waiting to see if it stops on its own. Probably I should kill it again as it’s preview mode anyway.

I do have copies of the whole thing before I flattened the files - but that was after I ran a full songkong clean.

The duplicate folders were a pain - some were empty or just had a jpg, but many contain a few of the songs split across them or duplicates. I tried last year to manually reconcile them but it was a lot of work so I only went so far.

So this seemed like the extreme approach was worth trying. I could go back to the folders version if you recommend that. Or I could split things into smaller batches

[paultaylor]

Hi
In report00034 you had configured all songs matched to release to be moved to C:|temp\match out and all songs not matched to be put in C:Temp\match fail folder

So this should mean all the original folders still in C:\Temp\New Hive of iTunes are now devoid of music files and can be deleted, doesn’t that solve the problem?

So do you still have that, if not retry same approach with the folders version.

[wolfer]

Most of the dirs are empty but it does leave behind about 250 music files. I could deal with them manually as some are Wavs or Home made stuff. However the Match Out still has lots of duplicate directories.

So I did a fresh test with about 14,000 files in a flat structure. And it also created a lot of duplicate directories

Here’s an example.

There are 3 identical copies of this song in the starting folder but they have unique names.

After processing songkong created 3 directories

Each with one of the identical songs.

And a more extreme version is from this 3 cd Box set. There are 135 files in 8 diff directories which ends up being a lot to clean up.

How do I screen these out?

[paultaylor]

@wolfer

Most of the dirs are empty but it does leave behind about 250 music files. I could deal with them manually as some are Wavs or Home made stuff. However the Match Out still has lots of duplicate directories.

Right okay, the trouble is you are confusing different things.

Your original problem was that you had directories with no music files inside them, and it was difficult to delete them. My solution for this works except for a few music files that could not be loaded/processed so I think that can be handled manually as you say. COuld be useful if folders were deleted if only contain artwork as files are moved though - https://jthink.atlassian.net/browse/SONGKONG-2129

This new issue with Hello Is this thing On is simply that you have three copies of the same song and therefore SongKong sensibly put them in three different folders (i.e imagine you have three copies of the same album) rather than lumping them inside one folder.

Now Fix Songs does alot of things but it doesnt find duplicates but if you ran Delete Duplicates it would probably delete two of the files (depending on what you have configured for duplicates), your problem could be you are left with two folders containing just a folder.jpg file. Now if instead when you ran Fix Songs you set Artwork:Save cover art to filesystem to No the artwork file would not be created (artwork would still be embedded in the audio file) and therefore you would just be left with two empty folders. And then I think you can use another tool to just delete empty folders (cant think of what at moment), but it would be nice if Delete Duplicates deleted empty folders as duplicates were deleted, I have raised an issue - https://jthink.atlassian.net/browse/SONGKONG-2130

I think the other example is just a more complex version of the same thing.

[wolfer]

Thx - I realize my q’s morphed somewhat but the main issue’s always been extra directories after running Songkong and duplicates being hard to get rid of. And after consolidating various half sorted out messes, especially after a few years of trying to get around itunes or windows File Path length issues, there are even more duplicate files and directories.

I can deal with the empty Dirs or the ones with just jpgs. Now that I figured it out, deleting empty folders is fairly easy with a batch file. Although this command using Robocopy looks easiest - I haven’t tried it yet: ROBOCOPY folder1 folder1 /S /MOVE

But the dirs with fragments of a release are confusing. I get what you’re saying about putting found dupes in folders but I’d rather it didn’t do that! If it was possible, deleting dupes should be part of cleaning. Just put everything thats questionable, like a diff bit rate or file size in a “to be reviewed” folder and make a large scale cleanup as easy as possible.

I’m sure it’s not that simple. Compilations must make it a particular problem.

So at this point, now that I’ve cleaned about 45,000 songs. Should I just run dedupe on the flattened directory, and then Songkong yet again to put them back into folders?

[paultaylor]

@wolfer

But the dirs with fragments of a release are confusing. I get what you’re saying about putting found dupes in folders but I’d rather it didn’t do that! If it was possible

Its worth noting that it only does this because you have Rename filename from metadata enabled, if you set it back to No then it would not happen.

@wolfer

deleting dupes should be part of cleaning

It isn’t, and it never will be since Fix Songs already does an awful lot. In fact we plan to move file renaming options into a separate File Renamer task.

@wolfer

Compilations must make it a particular problem.

For one thing, exactly what is aduplicate needs to be decided upon by user in Delete Duplicates options.

@wolfer

So at this point, now that I’ve cleaned about 45,000 songs. Should I just run dedupe on the flattened directory, and then Songkong yet again to put them back into folders?

I thought SongKong was failed on this single folder so not sure that is an option, personally I would stick with the original folder per album approach.

[wolfer]

@paultaylor

Rename filename from metadata=No

But if the dupes exist because of many attempts to clean over time with diff settings, don’t they need to be renamed as a first step to standardize them? Also, I’m not sure why renaming also means dupe directories get created - its a little confusing.

So I’m trying with a small subset of 352 frank zappa files to get it right. It seems like I get better results (less dupe folders and songs split across folders) if I scan for duplicates first and then let songkong reorganize them. But this works only if the musicbrainz id exists.

So probably my best bet is to:

• scan my flattend library and Force Acoustic fingerprint even if already matched but don’t rename, move or get art.
• Delete Duplicates based on Musizbrainz and AcoustID (maybe even sounds same?)
• Rerun songkong to create folders and get jpgs.

songkong wouldn’t run on the giant folder of 46K songs in one shot but I can try it in several chunks.

Much thx for your help

[paultaylor]

@wolfer

But if the dupes exist because of many attempts to clean over time with diff settings, don’t they need to be renamed as a first step to standardize them? Also, I’m not sure why renaming also means dupe directories get created - its a little confusing.

Not really, Delete Duplicates works on the metadata that exists within them, so the filename doesnt matter to Delete Duplicates. Fix Songs is concerned with identifying songs rather than deleting them. However I do have a solution for you if you keep Rename from Metadata enabled but enable Filenaming:Rename filename part only as well then it will only rename the file part instead of creating subfolders when duplicate songs exist.

@wolfer

So I’m trying with a small subset of 352 frank zappa files to get it right. It seems like I get better results (less dupe folders and songs split across folders) if I scan for duplicates first and then let songkong reorganize them. But this works only if the musicbrainz id exists.

I disagree, you need to run Fix Songs first to get the musicbrainzids ( if you already have the musicbrainz ids then you have in effecct already run Fix Songs). Set the Rename filename part only option and Fix Songs should not create these extra folders, or disable Rename Filename from Metadata entirely then no empty folders wil be left either, then run Delete Duplicates, having gone rid of duplicate files then rerun Fix Songs with rename metadata enabled to further organize the remaining files and folders.

[wolfer]

Makes sense. I think it only worked as I first mentioned because many of the files already have musicbrainz id. But to make 100% sure, I reprocessed them all with Fix Songs, so now I’ve got a flatfile folder of 46,000 songs all tagged with musibrainz ids.

But the Delete function seems to be failing. There’s no error, it just says 0 songs loaded or processed and ends after a few minutes. If I use a smaller subset of files it works, so I don’t think it’s the settings or the files themselves.

[paultaylor]

@wolfer

But the Delete function seems to be failing. There’s no error, it just says 0 songs loaded or processed and ends after a few minutes. If I use a smaller subset of files it works, so I don’t think it’s the settings or the files themselves.

Well its probably due to this massive single folder problem, I don’t think I have tested Delete Duplicates on such a folder but we already know that Fix Songs doesn’t like it so not that surprising that Delete Duplicates doesn’t either, but if you resend support files I can check the logs.

But then, how did you reprocess them all with Fix Songs?

[wolfer]

Good question. I did clean in 3 separate chunks of around 15K files. Trying same now for delete.

I’m assuming delete dupes only checks against the group of files it loads?

[paultaylor]

Looked at the logs, it counts the files to check but then doesnt seem to load them, not sure why. if it was me I would rerun Fix Songs as suggested above to recreate a suitable folder structure then I dont think you’l have any issue with Delete Duplicates.