Songkong matches albums and only do move a few tracks

[meaning]

Hi there,

This morning, I detected that since this morning, almost all the tracks that were moved by songkong generates “incomplete” albums.

So basically, matched or not (I move matched albums to /processed and unmatched to /unmatched) I end up now with albums that have a few tracks, and missing all the other ones.

is it due to the last option listed here ?

Basic

• Ignore songs previously checked that could not be matched: Yes
• For songs already fully matched : Ignore
• Rename files based on metadata: Yes if has metadata
• Update Artwork: Yes
• Update Genres: Yes
• Update Mood and other acoustic attributes such as BPM (Pro or Melco license only): Yes
• Only allow match if all songs in grouping match to one album: Yes
  Only allow match if all tracks in album were matched: No

one example between a lot of others :

I cancelled the job as it seems it’s a mess right now, and super strange thing is my tail -f running shows songkong is still running :

I will wait for a while and see if these fils eventually get copied after I cancelled the job. Any clue why this is happening ? As songkong keep running even when I cancelled the job in the wueui, could it be it will process rest of the files later on ? Is it usually moviing an album once it is totally finished, or could it be it did not totally process these folders yet ?

one more time, this happens on both matched, and unmated tracks (as I do know I have correct metadata in 99% of my files, I decided to rename and move the unmatched releases to unmatched, and also there, I only have a handful of tracks in last folders, instead of all of them)

I checked the logs for one of the moved files and here is what happened :

scary…

So, I guess this happened as I did not check the “only allow match if all tracks in albums were matched”.

Time now to understand how to reverse this. Can I reverse only last job moves ? or will songkong try to revert everything that was moved till now ?

[paultaylor]

No they would only be allowed to split up if uncheck Only allow match if all songs in grouping match to one album which you havent.

Now I would have suggested don’t rename your files to be more cautious in first instance but you had already run SongKong over a significant number of files and no complaints about the file renaming so it seems weird for there suddenly to be a problem. So I would suggest that you resend your support files before you run anything else so I can have a look at it

@meaning

I will wait for a while and see if these fils eventually get copied after I cancelled the job. Any clue why this is happening ? As songkong keep running even when I cancelled the job in the wueui, could it be it will process rest of the files later on ? Is it usually moviing an album once it is totally finished, or could it be it did not totally process these folders yet ?

When you cancel the job doesnt terminate immediately, its need to shutdown the process and then create the report so I think you just have to be patient. And yes it doesnt rename files (not copied) until has matched all songs to one album

@meaning

one more time, this happens on both matched, and unmated tracks (as I do know I have correct metadata in 99% of my files

My experience is that actually user added metadata is not as correct and certainly not complete as maybe think

There is the Undo Changes task, but really you need to let me look at the results before you do anything else, otherwise you will likely may make more work for yourself.

[paultaylor]

I did have a quick look at one of your earlier reports

and it looked fine to me, are you happy with these results or was there a problem at this stage ?

[meaning]

@meaning

Only allow match if all tracks in album were matched: No

I believe this thing was checked for previous runs :

Only allow match if all tracks in album were matched: YES

So, it looks like without this, songkong is actually moving files, even if it only detects a few tracks of an album.

What seems less logical to me is why it also only did copy single track(s) in my “unmatched” folder. as the album was not matched at all, why would it only have copied this or that track instead of all tracks present in the non-identified folder ?

[paultaylor]

@meaning

Only allow match if all songs in grouping match to one album: Yes

I check the reports I have for you, and it was checked for reports 16 and 17 but not report 26, do you know why you switched it off ?

However if your songs are already organized one folder = one album it should not make much difference because the songs will be grouped to start with by folder and will only be matched to the album if all songs in folder could be matched to album. The option you switched off it more for the case that you only have part of the album to start with and then if enabled prevents album match…

So if you start with a folder containing only two songs of a ten song album then default option Only allow match if all songs in grouping match to one album will only allow them to be marked as matched to an album if both could be matched to same album. But you are not restricted to having all songs of album unless you also have Only allow match if all tracks in album were matched enabled, this is unchecked by default because usually you would not want Songkong to prevent you matching your songs to albums just because yo don’t have all the songs.

If songs are matched to an album then they a MusicBrainz_Release_Id or Discogs_Release_Url field set, and the Matched to Release option of file renaming checks this to decide if to rename file.

@meaning

What seems less logical to me is why it also only did copy single track(s) in my “unmatched” folder. as the album was not matched at all, why would it only have copied this or that track instead of all tracks present in the non-identified folder ?

I need to see your support files and example to give you an answer on this.

[meaning]

Well why it switched to NO is a total mystery to be honest.

No worries, I simply did switch it back to yes, I only have a few albums impacted, and I will simply make sure to fix these manually if needed.

[meaning]

So, I’ve ran another fix tracks job yesterday.

I confirm it keeps moving a few tracks per album only.

The above screenshot comes from the unmatched folder, where I’d expect songkong to move the files, all files, as the release was not matched at all. why is it only moving random tracks from these unmatched folders?

here are the options I used :

Basic

• Ignore songs previously checked that could not be matched: Yes
• For songs already fully matched : Ignore
• Rename files based on metadata: Yes if has metadata
• Update Artwork: Yes
• Update Genres: Yes
• Update Mood and other acoustic attributes such as BPM (Pro or Melco license only): Yes
• Only allow match if all songs in grouping match to one album: Yes
• Only allow match if all tracks in album were matched: Yes

I upload the support files so you can have a look into this. But with the last YES present in there, I would expect :

1. the processed folder only do contain fully matched albums, with all tracks inside the folders
2. the unmatched folder contains all the folders of releases that were not matched, and that they contain all tracks, simply renamed based on their metadata

[meaning]

Even more scary, please do have a look at cat songkong_debug0-5.log.
For some reason, songkong tried to copy a lot of folder located in 03-2021 to travis scott’s artist folder.

just one line I’ve taken from there :

11/10/2022 00.17.25:CEST:FileRenamer:moveEverything:WARNING: Unable to move subfolder from /music/03-2021/Retroactive - Nothing Rhymes with Orange (2013) - 16 Bit FLAC to /music/processed/Travis Scott feat. Future, Young Thug & M.I.A/FRANCHISE (REMIX) because contains other audio files

This happened to several hundred folders. And the album folders are now full of “empty” folders that were not renamed at all.

look, it simply tried to copy all the foldes of the monthly folder to that album folder :

11/10/2022 00.17.22:CEST:FileRenamer:renameSubFolderAndFilenameFromMask:WARNING: New SubFolder/Filename Path is(1):Travis Scott feat. Future, Young Thug & M.I.A/FRANCHISE (REMIX)/01 - FRANCHISE (REMIX).flac
11/10/2022 00.17.22:CEST:SongSaver:moveAssociatedFiles:SEVERE: Moving files from /music/03-2021 to /music/processed/Travis Scott/HIGHEST IN THE ROOM

I believe something happens as soon you pick “rename if has metadata”. Everything was perfect before I started using this. So I will for now keep everything unmatched in it’s original folder and only move totally matched to a release stuff in /processed.

[paultaylor]

@meaning

Rename files based on metadata: Yes if has metadata

Because of this I expect, I explained to you earlier it should be set to Yes, if matched to Release and I thought you had changed it.

[meaning]

I was just replying, see above. In my mind, if has metadata should move the matched stuff in processed. while unmatched stuff should be kept intact -> and moved to /unmatched (but renamed based on tracks existing metadata).

But as I said, I’m going back to “yes if matched to release” only.

[paultaylor]

@meaning

Even more scary, please do have a look at cat songkong_debug0-5.log.
For some reason, songkong tried to copy a lot of folder located in 03-2021 to travis scott’s artist folder.

Hi, okay there is an associated error in the report at this time as well

This does look like a bug, and I think its along the lines of doing a rename because you have metadata, but not creating the patrh correctly because yo dont have the metadata to create the folder structure. I will raise a bug for this.

[paultaylor]

@meaning

I confirm it keeps moving a few tracks per album only.

The above screenshot comes from the unmatched folder, where I’d expect songkong to move the files, all files, as the release was not matched at all. why is it only moving random tracks from these unmatched folders?

Its not moving these files as such, they are not being moved from /music to /music/unprocessed or /music/processed and they are just being renamed (as far as what SongKong means by rename), but because the rename mask you use is causing the new filename to be the same (because tracks have the same title) it has to create folder(1), folder(2). This is usually preferable to putting them all in the same folder because if you have duplicate copies of album you want them in own folder. This is good example of why using Rename if have Metadata is not usually a good idea.

What I don’t understand is why they were moved to /music/unmatched but I wonder if it is because /music/unmatched is a subfolder of /music and therefore may be confusing SongKong into reprocessing the file after renamed. I will investigate but think it would be a good idea to change your folder structure to /music, /musicunmatched, /musicprocessed and see if that helps.

[meaning]

here is what I will do for now to secure all that :

1. I am currently moving /music/processed/ to /
   This will make sure the processed folder is located OUTSIDE the /music folder I am actually processing.

2. I changed back to “yes if matched to release” to avoid weird things (as stated above)

3. I will not ask songkong to move unmatched tracks anymore! it means that if anything was not perfectly matched to a release as configured at #2 will remain in its original folder (under /music)

I hope this will bring things back to normal.

I also think I should run these fix jobs in batches (one month folder after another) as I do notice that the scan starts from the very beginning each time, making me loose a lot of time watching songkong skipping already processed tracks. (see report pasted above).

Do that seem OK to you ?

[paultaylor]

@meaning

I am currently moving /music/processed/ to /

I think moving to /processed would be better than dumping music files directly into /

I agree with 2 & 3

@meaning

I also think I should run these fix jobs in batches (one month folder after another)

I wouldn’t bother, you may cause other another issue such as some confusion with base folder/sub folder split.

[paultaylor]

Okay raised https://jthink.atlassian.net/browse/SONGKONG-2343 and https://jthink.atlassian.net/browse/SONGKONG-2344

Good news is no subfolders were incorrectly moved under just the file being processed.

but the problem is this file was directly underneath /music/03-2021, and then there were other artist folder in same folder and move everything option tried to move all these thinking they were part of the album , i.e thought that /music/03-2021 was an album folder.

Thankfully this is not the usual case, but I will fix the issue.

[meaning]

Just to be sure, should I get rid of any flac file present on one of the “non album” folders ? to avoid this to happen in next runs ?

[paultaylor]

Could do though as I say there are checks to actually prevent those subfolders incorrectly being moved, so I dont think any real harm was done.

[meaning]

Sure, still, I ended up with hundreds of folders in /processed/artist/album(s)/

So, the content of the folders were not moved, but I do have a lot of empty folders in there which are not wanted.

unfortunately, after doing what I listed above (move the processed folder out of the main music folder) I’ve ran a test and songkong starts to do this again

It creates several copies of the same tracks in single albums occurences.

I believe songkong db do not understand that these test albums were already copied to “another” location (/music/processed/ instead of /music_processed/ now) and therefore re-moves the same files or something.

[paultaylor]

Did you move from /music/processed to /processed or just copy the folders within to new folder /processed, clearly if you just copied them you are going to get duplicates of your files.

If you didn’t do that then I need your support files again.

@meaning

I believe songkong db do not understand that these test albums were already copied to “another” location (/music/processed/ instead of /music_processed/ now) and therefore re-moves the same files or something.

Sounds like you just copied them, how would SongKong know this. There seems to be a fundamental confusion on your part on how move matched option works. To reiterate, if songs in your start folder are matched to an album then the basefolder of these files is moved from its current location to the matched folder, if you also have Rename files from metadata set to Yes, if Matched to Release the subfolder and filename part of the original file path will be adjusted based on the rename mask. When you run a Fix Songs tasks it processes the files based on your options, any previous runs or processing are not taken into consideration. Even Ignore songs previously checked that could not be matched only references previous runs indirectly by checking for a metadata field in your files, but it doesn’t analyse previous runs.

[paultaylor]

@meaning

Sure, still, I ended up with hundreds of folders in /processed/artist/album(s)/

So, the content of the folders were not moved, but I do have a lot of empty folders in there which are not wanted.

What do you mean, can you give example of this as I have missed this detail when I looked at the reports.