Loaded Songs Count dont match when run FixSongs from Cmdline

[meaning]

ok I confirm it works !

I checked the logs and the fix task did use my “other runs” profile as it should.

the reported loaded tracks is now correct, which means the notification my script pushes at the end of a task is correct.

For this value as an example : Songs loaded 12863:Fingerprinted 12618: MusicBrainz 146: Discogs 6736: Saved 82

As you can see the ETA estimation for next folder still needs some work, but rest of the information is now right.

It’s also right for the delete duplicates task:

(this folder was already processed, therefore no more duplicates found, I’ll get back to you begging for bandcamp integration, don’t worry about that, as I can definitely say the remaining unmatched releases comes from this source…)

and here is the confirmation that rename is not crashing anymore :

Executing the rename command: docker run --rm --name songkong_rename_01_1804 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -f /music/Music_dump/01-1804

Now I need to add the reporting for rename tasks, but this is not a great deal

[paultaylor]

Great, also raised this issue for another time

[meaning]

@meaning

docker run --rm --name songkong_rename_01_1804 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -f /music/Music_dump/01-1804

and here we go for the Rename report. All done. Of course, some improvements will become, I guess. But it does the job now!

[paultaylor]

Would be interested in seeing FixSongsReport00202 just because so few songs matched to musicbrainz.

[meaning]

Indeed, I did notice this too.

The script makes the access to reports a little complicated (they are not appearing in the reports list for some reason)

so let me do the following. for these two tasks :

that was followed by this task :

I am sending you a download link by email.

[paultaylor]

Okay there are a few possibilities here for Report213

• You have enabled the quite restrictive Only allow match if all tracks in album were matched option. So for example maybe you have an EP with 4 tracks, and MusicBrainz only has the 5 track EP version then it will not be matched to because can only match 4 or the 5 tracks
• Mostly E.Ps and singles, MusicBrainz coverage of albums is better then singles, Discogs certainly better for singles because more collectibles
• Alot from 2023, albunack database not been updated since may 2023 because recent update had to be rolledback because a problem with Disocgs data downloads, should be resolved soon

Now, I think the most significant issue is the Only allow match if all tracks in album were matched option for two reasons:

• Prevents matches to good but not perfect matches
• Encountering an unknown issue that is preventing some complete valid matches because option enabled.

What would be really good is if you could rerun against this folder with the Only allow match if all tracks in album were matched option and preview enabled so that we can compare the results.

[paultaylor]

The reports are not displayed in the reports list because you are deleting the database and the database contains details of the tasks run and hence reports created. But you can still create the report independently by opening the starting report html file e.g FixSongsReport00001.html

[meaning]

Do we agree that even with this setting " Only allow match if all tracks in album were matched" disabled, The rename feature will keep moving full albums / EPs, …, if 4 out of the 5 tracks of an EP were fully matched ? I’m a little concerned about that.

I know I’m taking th erestrictive path, but I’d rather not face another situation where I’ll end up with a ton of incomplete albums / folders.

so this is what you recommend ?

Anyway, I run it now in preview mode, lets see how it behaves.

[paultaylor]

We have had this discussion before and I’m not sure if we understood each other, but I try again. If you uncheck the Only allow match if all tracks in album were matched option but keep the Only allow match if all songs in grouping match to one album option enabled which is the default this is what happens

• If you have a folder A with 4 songs in it and Fix Songs cannot find 4 track release but finds a potential release with 5 tracks and can match the 4 tracks to 4 of the tracks on the release then it will.

• But if you another folder B with 4 songs in it and Fix Songs cannot find 4 track release but finds a potential release with only 3 tracks on it then even if it can match 3 of its tracks to the 3 tracks on the album it will not.

• When you run Rename Files and have Move Folder enabled it will move files that have a MusicBrainzReleaseId or DiscogsReleaseId. Therefore it will move the folder A but not folder B so move will not split a folder but isn’t necessarily only move complete albums

If you check the Only allow match if all tracks in album were matched option then Fix Songs is unable to match Folder A (because the 4 song album not in database) so no songs matched to album, and hence when run Rename Files no songs are moved.

[paultaylor]

@meaning

Anyway, I run it now in preview mode, lets see how it behaves.

No, thats wrong way round, you should have Only allow match if all songs in grouping match to one album enabled and Only allow match if all tracks in album were matched disabled

[meaning]

ok, it runs, I’ll keep you posted.

files are loaded, and it is now checking their fingerprints, we can see the number of MB releases is still pretty low, but the musicbrainz song only is already at 3200 (which is still low, but better than the previous run)

now I have one question, is there any risk for me to keep these settings ? what would be the impact ? One more time, I really, really want to avoid to end up with incomplete albums once i’ll have run the deletes duplicates and rename tasks for that folder (and the other ones).

[paultaylor]

Please make sure you send me the report afterwards because I want to see if now matching MusicBrainz releases that it should have been able to match first time round even with the more stringent setting enabled.

@meaning

I really, really want to avoid to end up with incomplete albums once i’ll have run the deletes duplicates and rename tasks for that folder (and the other ones).

Im not totally clear what you mean by incomplete albums but I explained the situation with those options and renames in some detail in above post, so are you happy with that or not ?

Regarding Delete Duplicates the key to not splitting albums is the order of your Preferred Deletion Criteria. The best thing to do is run Delete Duplicates in preview mode and check the results to see if they are how you want them.

[meaning]

I’ve sent the support files.

Ok now let me try to contextualize what I am fearing. Here is a random album that was processed using the preview mode:

All the tracks in this list do have the right artist and album name.

using this option you mentioned, I have one out of 13 tracks that was actually updated as it was found as a “musicbrainz song only”.

If I open it, here is what I can see :

So, what happens when my script runs the delete duplicates using these settings ?

And even more scray is the following :
As I have one of these tracks tagged as “Classical” now and not the others, what happens when I rename this album ? Will that single track end up in /Classical/Artists ? And the other ones in ArtistLetter/Artist Name/Album name/ ?

As a reminder, here is the seting I’m using:

Wont such “per track” retags make me take the risk of seeing my files been spread here and there when I’ll run the rename tasks ? This is my question. It might sound dumb, maybe, but it is still unclear for me and I have no more security net.

[paultaylor]

Hi, you make a good point here that I hadn’t considered before actually.

The point of MusicBrainz Song Only match is that SongKong has correctly identified the song but not the correct album (because it could not find an album that all the songs in the grouping could match to) so it only updates song info and not album info. So all songs have been matched to a Discogs album, but only one song has been matched to a MusicBrainz song so for that for that song we do not modify album fields such as album, albumArtist and trackno therefore if these songs were renamed using a standard rename mask they would not be split up.

However, when I wrote the MusicBrainz song only code it didn’t occur to me that users may organize the music into classical and non -classical, the classical rename masks were added much later on. So it checks to see if the album the song has been found on is a classical album, if it is it set to IS_CLASSICAL to 1, if not it sets it to 0. So in your particular example the song is not set to classical so not a problem if you were to rename this folder, but you are right that you could have circumstance where MusicBrainz Song Only matches to Classical and sets to 1, and the whole album is matched to Discogs and the rest of the songs are not marked as Classical. Then if renamed (as it would because both Move Folder and Rename files based on metadata is set to Yes if Matched to Release includes Discogs release only matches) and mask uses IsClassical it would be split.

So MusicBrainz Song Only matches should not set the IsClassical field, raised https://jthink.atlassian.net/browse/SONGKONG-2503

Now, this is not related to the Only allow match if all tracks in album were matched option because that controls if group are songs are matched to an album, it doesn’t affect if songs are matched song only. So this problem could occur with or without this setting being enabled. I think this song was updated because second time round there was more information in the songs that allowed a few more to be matched, it would probably have been updated if you had run again without changing any options. You could test this out if you want by running again in Preview mode with option re-enabled.

Although in normal circumstances it is rare for this problem to occur because it needs SongKong to match songs to Discogs album, but fail to match songs to MusicBrainz album, but be able to match some songs to MusicBrainz and at least one of those songs need to be on an album that is considered classical.

Regarding duplicates Same MusicBrainz song and same album refers to MusicBrainz album, since it has not be matched to MusicBrainz album it would not be considered as a potential duplicate, we dont currently consider Discogs matches for Delete Duplicates task.

[meaning]

@paultaylor

So MusicBrainz Song Only matches should not set the IsClassical field, raised https://jthink.atlassian.net/browse/SONGKONG-2503

Cool! in the meanwhile I’ll keep the super restrictive settings, it will already help me move some fully identified albums, and next time, when this will be fixed, there will be less tracks to analyse. Please remember we discussed Classical (is classical) but that I am also using the “Is compilation” tag to put VA’s in /Compilation. I guess this will also be problematic and needs to be considered aside “Is classical”

@paultaylor

we dont currently consider Discogs matches for Delete Duplicates task.

Is there a security reason therefore ? Are Discogs matches not considered as “full albums” ? I paste this again, just to understand, last column definitely indicates all these tracks were matched on discogs, and have the very same discogs release ID.

@paultaylor

Regarding duplicates Same MusicBrainz song and same album refers to MusicBrainz album, since it has not be matched to MusicBrainz album it would not be considered as a potential duplicate, we dont currently consider Discogs matches for Delete Duplicates task.

This is reassuring.

[paultaylor]

@meaning

Cool! in the meanwhile I’ll keep the super restrictive settings, it will already help me move some fully identified albums, and next time, when this will be fixed, there will be less tracks to analyse

Okay, but as I said above this doesnt prevent this problem occurring.

I would recommend you hold of running the automated rename task until we have resolved all Fix Songs issues, Im going to review your latest run in a bit.

@meaning

but that I am also using the “Is compilation” tag to put VA’s in /Compilation. I guess this will also be problematic and needs to be considered aside “Is classical”

No, I have checked this and this is okay, we don’t set this field for a MusicBrainz Song Only match

@meaning

Is there a security reason therefore ? Are Discogs matches not considered as “full albums” ? I paste this again, just to understand, last column definitely indicates all these tracks were matched on discogs, and have the very same discogs release ID.

No it just that MusicBrainz is our primary database and haven’t had time to add Discogs. We could certainly do it but with Discogs we could only add the Same Discogs song and same Album (specific version) because Discogs doesn’t have as advanced relationships between albums as MusicBrainz does,

Discogs album is considered a matched release in the context of Fix Songs task

[meaning]

@paultaylor

Okay, but as I said above this doesnt prevent this problem occurring.

OK so you recommend me to keep settings like this, right ?

and by the way, since this night, around 2am I’d say, Jthinkserver times out:

22/09/2023 08.04.39:GMT:JthinkSearchServer:doQuery:SEVERE: Read timed out
java.net.SocketTimeoutException: Read timed out

This happens when all tracks has been loaded and songkong starts to query your server. Might be a good idea to have a look at it

[paultaylor]

@meaning

OK so you recommend me to keep settings like this, right ?

I’m not necessarily recommending it just point out to you that keeping the more restrictive criteria will not prevent the isclassical problem. However, yes I would keep the less restrictive settings because if all songs in your grouping are matched to an MusicBrainz album that is better them not being matched to an album. And if they are all matched that should mean it is indeed the correct album and you just have an incomplete rip, or you have matched to the right album/release just not to the right version (because the exact version you have is not in database)

But before that what would be really good is once server fixed is you reran against same folder with more restrictive setting because would be good to compare how many of the extra matches second time round were because less restrictive setting and how many were just because found some more matches because of metadata added first time.

I’m going to check the results in FixSongsReport213 and FixSongsReport216 now because there were a couple of things in your original FixSongsReport0213 that didn’t seem quite right.

@meaning

and by the way, since this night, around 2am I’d say, Jthinkserver times out:

You are correct, never seen this error before seems to be issue with Apache server at front end which is not something i usually have to deal with

https://jthink.atlassian.net/browse/SONGKONG-2504

Im starting new server, takes a while to initalize though, should take about 20 minutes.

[paultaylor]

Server is working okay now.