Running songkong cmdline in script in Docker environment

[meaning]

it is running for 5 days now, it ate a lot of CPU and 40GB of ram while fingerprinting files.

but now, I have another “issue”. Since it stopped fingerprinting files, I noticed the CPU usage dropped to almost nothing, and RAM is only used at 8GB still.

yesterday noon (12h30 to be perfectly clear) I was here :

this morning, 8:52am, here :

so it only did save 17913 tracks in 20h of run time. Does that seems possible for you ? what could be slowing down things this way ?

The very weird thing is it actually saved files way faster during the fingerprinting / files check process. Once it was done and fingerprinting didn’t raise anymore, things did slow down. It’s like if SK did stop using the CPU’s somehow.

long story short, I saved approx 300k files in 4 days, while the fingerprinting checks were operated, and now, I calculated it runs at approximately 35k files saved per day.

any clue ?

[paultaylor]

Fingerprinting is very cpu intensive, it maybe if no fingerprinting to do the delay/bottleneck is getting results from server instead since we have to limit how many calls can be made to server by single instance of application

@meaning

long story short, I saved approx 300k files in 4 days, while the fingerprinting checks were operated, and now, I calculated it runs at approximately 35k files saved per day.

But also SongKong is only going to actually save the file if changes made. So if file already had fingerprint and was already matched and nothing changed in match then the file doesn’t need to be saved. So what is key is the value of the Completed bar not the Saved bar.

But I thought the idea of the script was to work on one folder at a time, and then stop container and restart because we know SongKong seems to slow down over time with very large number of files.

But this screenshots seem to indicate you are running SongKong in one go on the whole thing again ?

[meaning]

@paultaylor

But I thought the idea of the script was to work on one folder at a time, and then stop container and restart because we know SongKong seems to slow down over time with very large number of files.

But this screenshots seem to indicate you are running SongKong in one go on the whole thing again ?

Not really, I have one of my “monthly” folders that contains a big amount of data. The numeber you see represents half of my library and is the “big piece”. but it’s still a monthly folder. all the other ones (next ones) are way smaller.

the script I wrote is identifying folders that have a date format, and will process each of them in a newly started songkong instance. once songkong finished its task, the script detects it, stops the docker container, and goes to the next folder, and start again.

And indeed, it looks like songkong starts to slow down at some point. Question is…why. as you can see it literally ain’t doing anything atm.

Now, I could eventually use the rename feature of songkong, in order to move the files that do have fingerprints and were correctly identified in another folder, that might reduce the monthly folder size upfront next run. as only fully identified albums will be moved and renamed, I guess it might be helpfull. right ?

Can you eventually let me know if these settings seems to be allright ?

I selected FirstLetterOfartists in order to get “smaller” folders, instead of ONE gigantic folder containing thousand of artists.

Now, regarding compilations / various artists, I’d like to get them all in a specific folder called “Various”, the way to get this done is still unclear to me.

in short here is the structure I would like to get :

Music/Releases/Artists/ArtistLetters/Artists/Album/…
Music/Releases/Classical/ArtistLetters/Artists/Album/…
Music/Releases/Compilations/ArtistLetters/Artists/Album/…

[paultaylor]

@meaning

Not really, I have one of my “monthly” folders that contains a big amount of data. The numeber you see represents half of my library and is the “big piece”. but it’s still a monthly folder. all the other ones (next ones) are way smaller.

Okay it s not really a monthly folder in any sensible way if half your collection is in one folder, since the whole point of the script was to break up your huge collection into more manageable chunks.

@meaning

And indeed, it looks like songkong starts to slow down at some point. Question is…why. as you can see it literally ain’t doing anything atm.

I have no idea, it would be helpful if you could send the logs they may give an answer. But I suspect it is to do with something getting too large, therefore the best thing you could do is break up this massive folder to allow you to complete processing.

I hesitate at this stage to get RenameFiles involved, could you just use the linux cmds to break this folder into subfolders of approximately 100,000 songs each ?

[meaning]

@paultaylor

I hesitate at this stage to get RenameFiles involved, could you just use the linux cmds to break this folder into subfolders of approximately 100,000 songs each ?

well, this specific folder /was/is a mess. I decided to start a rename task in order to move the already identified albums (if all matched to a release).

I’ve told the renamer to move the “marched” items to music_matched and the unmatched items to music_unmatched.

But damn, look what happens now :

boxsets are merged like you can see above.

some fully matched releases only do have a single track:

and the good old flac and mp3 mixes are back :

and I also do have strange rename behavior like this :

is it me, of should I normally ONLY see perfectly organized and renamed files in there as all these files are supposed to be part of a unique release ? with its unique discogs / musicbrainz tag ?

Now, I do undertstand the mp3 / flacs laying in the same folder issue, as these are actually identified as the same album.

But why do I have an album, matched, with a single track which is track5 ?
Why can I see an album in last example where I have twice the same track, once identified as track 09 from disk one, while it is also identified as track 3 of a single disk release ?

why are albums sometimes put in two separate folders,and sometimes not ?

why do I clearly have fully matched albums that are moved, that misses most of their tracks ?

Why are two totally different releases identified as a single one, and put in a single folder as we can see here ?

these are just a few examples as I am browsing the folders songkong is currently moving.

I believe it’s no big deal as I plan to let it run till the end, and I’m only moving files, therefore, I can rollback using the rollback feature of Songkong, right ? But generally, I’m not super comfy with what I’m seing happening.

[paultaylor]

Hmm, well I was hoping you were going to send me the logs and I did warn against using Rename Files at this point because you should only be renaming files if you have checked and are happy they are matched correctly, that is why we split Rename Files from Fix Songs task There are alot of recent improvements in the FixSongs report to allow you to check things like this quickly.

Having said that if you only have Rename Files set to Move files then the folder structure and filename should remain the same, only the base folder should change. But from what you say above it looks like you have renaming configured as well .

Regarding Box sets It looks like maybe you are using a rename mask that doesn’t use DiscNo to separate discs

In cases where you have only matched one track to release it is not clear if you only have one track of the release, or tracks from a folder have been incorrectly matched to multiple releases. This is easy for you to determine using the Fix Songs report, Browse by Folder section.

If you are renaming and you have multiple versions of the same album then you should probably make audioformat part of the folder structure.

@meaning

I believe it’s no big deal as I plan to let it run till the end, and I’m only moving files, therefore, I can rollback using the rollback feature of Songkong, right ? But generally, I’m not super comfy with what I’m seing happening.

Yes, but it will rollback all changes to the selected folders since the songs were loaded into current SongKong database, so this could include matching/fingerprinting as well as renaming.

Really to help you further I need your support files, and I hope it has a valid FixSongs report and RenameFiles report because these reports contain all the information needed to work out what SongKong has done and are much more useful to then the screenshots posted. The screenshots do not explain why something has happened only the end result.

[meaning]

@paultaylor

so this could include matching/fingerprinting as well as renaming.

nice

Support files are on their way

one thing worth mentioning is that I selected to put my artists inside their own Letter (to reduce the size of my final rename folder, and keep it as “separated” as possible.

Worth to mention as well that aside my LETTER folders, I have tons of folders that are totally not put in a letter folder, sorry one more screenshot needed :

Alltogether, I’d like to be able to trust songkong and not have to go throug this process again and again. it should simply move the albums that were fully matched, without any issue. as this is the meaning of it.

[meaning]

by the way I analyzed the Drukqs folder with picard. the promo (5 tracks) and vinyl versions of the album were merged in a single folder.

here is how it looks like once you import it in a media manager (here, navirome) :

another very straightforward example is OK computer, you can see 3 versions where merged in a single folder by songkong :

1. the OK/NOTOK version which includes an extra disc
2. an old mp3 version of the regular release
3. the flac version of the regular release

here is how it looks like FOLDER wise :

But this time, once imported in navidrome, you can see the album is displayed as 3 distinct releases ! I’m missing something…

Obviously, I’d rather see the Druqks release and its promo been displayed as two separate releases !

[paultaylor]

@meaning

Alltogether, I’d like to be able to trust songkong and not have to go throug this process again and again. it should simply move the albums that were fully matched, without any issue. as this is the meaning of it.

Sure, but you are not simply moving the files you also are renaming them and my suspicion is your rename mask is not correct for what you want to do, anyway will find out when look at reports.

Also you say simply move albums that were full matched, when you say album do you mean the collection of songs in a folder on your hard drive or do you mean the MusicBrainz/Discogs album the songs have been matched to?

By album i mean the MusicBrainz/Discogs album. So SongKong will move songs that have been matched to an album (not just a song), and this should usually mean that all songs in folder have to be matched to the same album and therefore all these songs will get moved to same location. But if you only have some of the songs from the album then all songs files can be matched to the album but the album can not be fully matched.

[meaning]

@paultaylor

By album i mean the MusicBrainz/Discogs album. So SongKong will move songs that have been matched to an album (not just a song),

exactly. I selected to move all matched albums (only full albums, fully matched to musicbrainz/discogs) to the matched folder. And to move all the non-matched ones to the unmatched folder.

most f the time (I am manually checking stuff while we speak) I have perfectly fine albums in the letter/artists/album/ folder structure.

but all these other folders that are aside the letter folders, are a mess. and I can find specific examples of mixed/merged albums (like the druqks example I shown to you). which are “problematic” but not the en dof the world.

At the end of the day, I’m currently seeing a proper structure been built, but all these albums that were moved to the “matched” folder because considered as “fully matched / full album only” remains an issue.

Also it would be nice to figure out why specifics like Druqks did happen.

We’ll figure it out !

by the way, if analysing these two folders (Druqks and OK Computer) using picard, see, the Druqks folder once imported correctly do display the vinyl rip of the album + the promo separately !

[paultaylor]

@meaning

exactly. I selected to move all matched albums (only full albums, fully matched to musicbrainz/discogs) to the matched folder. And to move all the non-matched ones to the unmatched folder.

Thats still unclear to me because there is no option to move only full albums

[meaning]

Well, regarding the fix task(s), you got the logs.

Now, regarding the ongoing rename task :

As stated in your tutorial : " Often the best choice is Yes if matched to a release this ensures that songs are only renamed if matched to a MusicBrainz or Discogs release, this in turn ensures that complete albums of songs are renamed and therefore kept together."

This is what I’ve picked, as it seems the safest option. This is what currently runs.

[paultaylor]

Okay that is the best option, but just to be clear if you have 5 songs in a folder matched to a 6 song release then they will all be renamed, so the songs are together but it is not a full album.

So I have extracted your support files but it only contains

DeleteDuplicates106
StatusReport103
StatusReport105

Do you know why that is ?

Do you have a recent FixSongsReport ?

Also it seems RenameSongs is still running ?

[paultaylor]

Anyway, regarding albums not being listed under letter this is because only your rename mask is set to use a letter, but your compilation rename mask is set to AlbumArtist - Album (Year)/Track - Artist - Title, this mask doesn’t group by letter nor does it put multidisc albums into separate folders for each disc.

ifnotempty(albumartist,' - ')
+ ifnotempty(album,' ')
+ '(' + (albumyear.length>4 ? albumyear.substring(0,4):albumyear) +')/'
+ ifnotempty(pad(trackno,2),' - ')
+ ifnotempty(artist,' - ')
+ title

So this explains why you are getting songs not within letter folder and not grouped into discs.

[paultaylor]

It would be so much easier to explain these problems if you could post screenshots from the songkong report itself rather than random music players. But looking at the Radiohead example it seems there is no problem, it looks like SongKong has identified the three separate releases correctly with Fix Songs but the rename mask you have chosen doesn’t include anything to distinguish these releases so Rename Files put them in same folder. You could either use the Add EP, Single, Compilation, Live and Remix release types to release title or Add Audio Format to release title options on Album Format tab to modify what SongKong stores as albumname so that when rename files the three albums will have different values for Album and thefore be stored in different folders

Although we don’t currently have option to add disambugation name to the album field, which might be useful to you https://jthink.atlassian.net/jira/software/c/projects/SONGKONG/issues/SONGKONG-1835

Or you could modify rename mask to use extra information to change the filename without changing the metadata, the albumversion field is useful this stores the album disambuguation if it has one in MusicBrainz.

With Aphex Twin example you just seem to be asking why its isnt displaying Promo at the end of the track title, but why would it the track title doesnt include Promo at end. Or are you just saying why isnt it displaying as two albums like it did for Radiohead ?

Well because I cannot see the metadata in the files (because report missing) and because I have never used navidrome I cannot give you an answer on that.

[paultaylor]

@meaning

But why do I have an album, matched, with a single track which is track5 ?

When I did some improvements for boxset there was a few bugs that allowed incorrect single song matching such as https://jthink.atlassian.net/browse/SONGKONG-2472 when they should not have. This was only released as part of SongKong 9.4 Raise so this issue may be a bug you encountered in matching algorithm when you ran against these songs that is now fixed.

[paultaylor]

So since you have gone down the renaming path I would do the folowing

1. Fix Compilation Rename mask setting
2. Modify both Compilation and Rename mask to include albumversion in mask -see Tutorial 41: Modifying Rename Masks
3. Rerun Rename Files again, it will only have to rename a small amount of the files so this is the quickest approach.

This should resolve most issues.

[meaning]

ideally, here is what I would like to achieve for this next run :

1. add the EP, single, etc. seems to be as simple as a single click before running the rename task again. Great !
2. Modify the mask(s), I’m afraid I’ll need a little bit of your help here, I understand I can clone an existing mask, and I’d like to do that, but could you eventually share the right masks I should use ?

Thanks again paul for your always great support.

And yes, I decided to let the current renaming go, and start a new one on the processed folder afterwards, this should only remane the albums/files for which a mask change is needed, right ? and keep the already renamed ones (matching the settings) in place, is this correct ?

[paultaylor]

@meaning

add the EP, single, etc. seems to be as simple as a single click before running the rename task again. Great !

Well, not quite because it is modifying metadata and hence is part of FixSongs task. So would have to run Fix Songs again to change the album metadata field. So I would suggest we leave that for when you next run Fix Songs and just make the changes to the masks for now and rerun Rename Files.

@meaning

Modify the mask(s), I’m afraid I’ll need a little bit of your help here, I understand I can clone an existing mask, and I’d like to do that, but could you eventually share the right masks I should use ?

Okay I will do that now.

@meaning

And yes, I decided to let the current renaming go, and start a new one on the processed folder afterwards, this should only remane the albums/files for which a mask change is needed, right ? and keep the already renamed ones (matching the settings) in place, is this correct ?

Correct.

But do you know why there is no FixSongsReport ?

[paultaylor]

Okay go to Preferences and then Filename masks tab

Select FirstLetterOfArtist/AlbumArtistorArtist/Album/AlbumArtistorArtist - Album - Track - Title from filename mask dropdopwn and selecte Clone,

Give it a new name e.g FirstLetterOfArtist with AlbumVersion

Modify Mask from

ifnotempty2(albumartist.substring(0,1),artist.substring(0,1),'/')
+ ifnotempty2(albumartist,artist,'/')
+ ifnotempty(album,'/')
+ ifnotempty2(albumartist,artist,' - ')
+ ifnotempty(album,' - ')
+ ifmultidisc(ifnotempty(pad(discno,2),' - '))
+ ifnotempty(pad(trackno,2),' - ')
+ title

to

ifnotempty2(albumartist.substring(0,1),artist.substring(0,1),'/')
+ ifnotempty2(albumartist,artist,'/')
+ ifnotempty(album +  (albumversion.length > 0 ? ' (' + albumversion + ')'  : ''),'/')
+ ifnotempty2(albumartist,artist,' - ')
+ ifnotempty(album,' - ')
+ ifmultidisc(ifnotempty(pad(discno,2),' - '))
+ ifnotempty(pad(trackno,2),' - ')
+ titleSelect *Save*

Then when run Rename Files task set Rename Mask and Compilation Mask to new name e.g FirstLetterOfArtist with AlbumVersion

Will now include album disambuguation in brackets after album folder

However, not many albums actually have this and it wont fix the multiple audio formats problem or single with same name as album problem. So we could also add in audiofornat and mbreleasetype bearing in mind if at a later stage you make the change to add them to the album metadata field, the information would be duplicated.

So we could change mask to

ifnotempty2(albumartist.substring(0,1),artist.substring(0,1),'/')
+ ifnotempty2(albumartist,artist,'/')
+ ifnotempty(album + ' (' + audioformat + ',  ' + mbreleasetype +') '  
+ (albumversion.length > 0 ? ' (' + albumversion + ')'  : ''),
'/')
+ ifnotempty2(albumartist,artist,' - ')
+ ifnotempty(album,' - ')
+ ifmultidisc(ifnotempty(pad(discno,2),' - '))
+ ifnotempty(pad(trackno,2),' - ')
+ title

This would give us

Obviously you can make adjustments to the mask for the exact formatting you would like.