Use cache (nvme) / re-use multi threading to speedup song fixing?

[meaning]

Hi @paultaylor,

I was pretty silent last 6 weeks, as I basically was re-downloading all my music library from my cloud storage (remember I fucked up everything using the duplicate finder tool? ^^).

As a matter of fact, I think I’ll start a new fix files job as soon I’ll have re-downloaded everything, and my SK database still existing, I guess it will be way faster this time.

But still, I have some questions on how I could speed all my future fixes and renames:

1. do you know if there could be a way to move files to a temporary cache on a NVME, in the background, right before songkong would start fixing the files? would this make sens I/O wise ?
2. What is the current status of multi-threading? are all CPU threads used again during the fix task ?

Just currious

Nice to post here again by the way ! and congrats for the new release, I am really looking forward to test it !

[paultaylor]

@meaning

As a matter of fact, I think I’ll start a new fix files job as soon I’ll have re-downloaded everything, and my SK database still existing, I guess it will be way faster this time.

When you upgrade to a new release of SongKong the database is recreated, but it should still be faster than before because of the fixes and improvements we have made.

@meaning

1. do you know if there could be a way to move files to a temporary cache on a NVME, in the background, right before songkong would start fixing the files? would this make sens I/O wise ?

No, I don’t think so, remember that SongKong is a Java program available on a number of different operating systems. therefore we stay away from OS specific optimizations, if they are not done automatically by the Java engine then we will not be doing them. Anyway it would not make much difference because as long as the files are local (not networked) then reading/writing from files will be quicker than looking up MusicBrainz releases and matching them.

@meaning

2. What is the current status of multi-threading? are all CPU threads used again during the fix task ?

Yes, that was fixed in earlier release and has not changed.

@meaning

Nice to post here again by the way ! and congrats for the new release, I am really looking forward to test it !

Welcome back !

[meaning]

thanks !

can you confirm my “new” strategy is allright ?

Music_new = the dump where new music enters

1. music enters the dump in monthly folder/daily folder/ -> therefore, after the first initial match and duplicates job task ends, it would be nice to allow SK to monitor this folder, and run each night to identify the newly arrived music in there (cron task?) -> doable ?

Music_matched = the folder where albums that are matched on MB are stored -> for organizational reasons I want SK to :

1. NOT create one single artist folder containing all the artists (showed it would grow to much, making plex uncomfortable adding new content when new scans happens) -> how to do this?
2. FOR THE FIRST RUN OF THE FIX TASK : SK will create one prefix folder (A, B, C, D, …) and put the respective artists in their respective prefix folder -> how to do this?
3. REGARDING CLASSICAL RELEASES -> all classical “artists” folder should go under a general “Classical” folder -> how to do this?
4. REGARDING VARIOUS ARTISTS AND COMPILATION releases -> all these releases should go under a general “Various Artists” or “Compilation” folder -> how to do this?
5. This folder is NOT added to plex as we want to avoid dupes!!! therefore a duplicates job need to be run first.

Music_unmatched = the folder where albums that were not matched on MB are moved from Music_new
I wonder if I better keep these albums names as they are initially, or if I use embedded tags to already name them “sort of” ?

1. All the albums that are not matched by SK goes here, they are simply moved from the music_new folder to here
2. This folder should be reprocessed from time to time
3. This folder is added to plex as “Music (unmatched)”

Music_processed = the folder that contains the matched and duplicates free albums / va / compilations

1. This is the folder where duplicates task moves the previously matched albums once it has a “winner”, only best available copy (best quality / most tracks) remains here.
2. One copy per album version = we keep the best version quality wise of each album version (Eg: Initial release, release with bonus tracks…)
3. This folder is added to plex, as music in there is dupe free and contains the best quality versions of each album/comp/va

Music_to_delete = the folder where we move the albums identified as “loosers” by the duplicate task

1. This folder is here as a security net, but will be emptied after checking the duplicates job was correctly run (no silly deletion of random tracks…souvenir…)

Extras :

1. I need to figure out a way to inform plex about the content that gets pushed to Music_processed and Music_unmatched on the fly. This to avoid plex to -re-scan the entire library again and again

[paultaylor]

Hi, sorry there is so much in this post Im struggling to take it all in, so can I suggest we break it down a bit.

Now, before worrying about your daily updates you need to reprocess your existing library so lets deal with this first. Now in this new version filenaming is no longer part of Fix Songs (you can still move songs with Watch Songs) so I would suggest basic approach would be

• Fix Songs
• Delete Duplicates
• Rename Files

By running Delete Duplicates before Rename Files we can deal with duplicated files in their original folders and we reduce the chances of Rename Files clashing on filenames due to duplicate songs since most duplicates should now be removed.

We have existing rename masks that have a top level A/B/C/D folder

e.g FirstLetterOfArtist/AlbumArtistorArtist/Album/AlbumArtistorArtist - Album - Track - Title

and put Classical in different location

e.g IsClassical/HD/AlbumArtistorArtist/Album/DiscIfMultiDisc/Artist - Album - Track - Title

and you can use separate mask for compilations to non compilations

So you should be able to devise a mask from these that does exactly what you want.

So think best to get this done first before worrying about music updates

[meaning]

OK I finally have a look at this now.

First quesiton would be, as I now totally got rid of Roon since multiroom is a thing with plexamp -> should I not prefer to use the pelx music scanner format ? It looks like this is best practice to get plex read and organize well, right ?

Now, what do I exactly have to do in order to put classical and compilations / VA releases in a separate folder ? This is a little unclear.

Also since the GUI was changed, the job/task names are not displayed in the UI, making it mandatory to make a mouse hover to understand what section you are heading to.

First screen of fix songs now asks me the profile I want to use, I guess I better take default in there ? (I rather do a full fix of metadata AND images, so is default the right choice?)

anyway can you confirm the following settings please ?

as you can see I try to avoid multiple artists folders as much as possible by selecting single artists and avoid featured artists. Also I want to only display Composer as artists for classical releases (so if the london symphony plays Berlioz, I want this album to be put in Berlioz folder, and not Berlioz, the london symphony…).

does these settings seem fine for you ? for the fix job task ?

[paultaylor]

@meaning

First quesiton would be, as I now totally got rid of Roon since multiroom is a thing with plexamp -> should I not prefer to use the pelx music scanner format ? It looks like this is best practice to get plex read and organize well, right ?

That would seem sensible, recommend you try it out since I havent used Plex myself for some time.

The Plex Rename mask is

ifnotempty(albumartist,'/')
+ ifnotempty(album,'/')
+ ifmultidisc(discno)
+ ifnotempty(pad(trackno,2),' - ')
+ title

So if you want classical in separate folder you could simply amend to

   addClassicalOnly()()
    + ifnotempty(albumartist,'/')
    + ifnotempty(album,'/')
    + ifmultidisc(discno)
    + ifnotempty(pad(trackno,2),' - ')
    + title

this now adds a top level Classical folder for Classical (Is Classical field set),

function addClassicalOnly()
{
 return isclassical =='1' ? 'Classical/' : '';
}

If you want a top level folder for non classical as well you could use addClassical() this adds a PopRock folder for everything else, if this isnt quite what you want modify the addClassical Javascript function, it is currently

function addClassical()
{
    return isclassical =='1' ? 'Classical/' : 'PopRock/';
}

You can create similar function for Is Compilation.

@meaning

Also since the GUI was changed, the job/task names are not displayed in the UI, making it mandatory to make a mouse hover to understand what section you are heading to.

I dont understand what you mean, please explain further.

@meaning

anyway can you confirm the following settings please ?

Looks okay except:
I wouldn’t set Multi Disc Releases to Add Disc No to the release title if disc has title because this means each disc has different name, so plex may think different albums, also it means the foldername will differ at album level but it should be same, and the plex mask handles multi discs with a DiscNo subfolder. Just set it to Never add disc no to release title

You have slight problem with Classical Album artist because you can only set the Track Artist to composer, and you want to set Album Artist to composer, I think we discussed this before. Now its important to remember that Album Artist needs to be same for all songs in album, otherwise music players will split album into multiple albums and file renaming using the Album Artist field will split into multiple folders, and you will certainly have some classical albums that have different composers for different tracks.

Now if you are happy for the album artist to contain the main performers (Conductor/Orchestra) but you just want the Album Artist folder to contain composer(s) then I have one idea. The album title field usually contains a list of the Composers , then a ‘:’ then the album name. And even if originally missing then SongKongs tries to achieve this with the Add Composers to start of Album Title option, so in your rename mask you could add new function

function addClassicalAlbumArtist()
{
     if(isclassical=='1')
     {
          if(album.contains(':'))
          {
                return  album.substring(0, album.indexOf(':')) + '/';
          }
          else
          {
                return  ifnotempty(albumartist,'/');
          }
    }
    else
    {
          return ifnotempty(albumartist,'/');		
    }
}

and use it like this in your rename mask

addClassical()
+ addClassicalAlbumArtist()
+ ifnotempty(album,'/')
+ ifmultidisc(discno)
+ ifnotempty(pad(trackno,2),' - ')
+ title

So for Classical albums, if it can find composers in album title then it uses this instead of album artist for the folder above the album folder.

[meaning]

long question -> long answer. I will read you and take the time to process. thanks !

I just wanted to let you know what I meant regarding the Gui change :

Basically, you can’t see which icon relates to which task without putting your pointer on top of the icon.

[paultaylor]

But isn’t that the point of icons, they are instead of text. I thought this is how it worked before anyway, I don’t recall them having visible labels?

[meaning]

not really, modernising the UI is one thing, but even with a flat design, a description of the tasks would be welcome. Think about new users, or people who did not use snkong in a long time. It is now neede to hover to remember / understand which icon leads you here or there. Just my UI/UX two cents

[paultaylor]

Okay, but to clarify you are not saying it used to have descriptions, just that because icons have changed would be useful to have descriptions displayed the whole time to make it easier for existing users to migrate to new version?

I don’t think there is space for permanent descriptions, in fact on my mac laptop in desktop mode (which does show descriptions to match mac guidelines) there will not be enough room if I add any more tasks.

[meaning]

Ok well, there we go. I started the song fix task about an hour ago.

Fingers crossed.

I already can see RAM is not saturated as it was last time, nor the CPU.

I hope this time, I won’t encounter any timeouts killing the task anymore (but that was fixed after I reported the bug, right?)

I’m running a little tail -f on the songkong_user0-0.log and I’m already concerned by something :

19/06/2023 20.53.42:CEST:SEVERE: --Submitting 11 Fingerprints
19/06/2023 20.53.42:CEST:SEVERE: --Acoustid Submission Failed Check Api Key:lj5NgggIwD

I’m getting shitloads of these.

More generally, is there somewhere I can seek for information about the logs ? What are the events I should look for ? I’d like to understand better what is happening this time.

Thanks @paultaylor

[paultaylor]

There was a change with Acoustid that broke acoustid submissions but that was fixed with https://community.jthink.net/t/songkong-9-1-juju-released-26th-of-may-2023/

So are you using latest version or not?

The logs are really for my benefit in case customers have issue. It is not really expected customers will look at the logs so currently no documentation on them.

[meaning]

The page you linked do not exist, but as I am auto-updating my docker containers, yes.

SongKong 9.1.1 Juju

Still, you can see lots of these api key issues.

[paultaylor]

Seems the issue has been fixed now, have the errors now stopped?

[meaning]

20/06/2023 02.08.10:CEST:AbstractAcoustidQuery:performBasicSubmissionQuery:SEVERE: Posting to url:http://api.acoustid.org/v2/user/lookup?format=xml&client=8XaBELgH&user=lj5NgggIwD
20/06/2023 02.08.10:CEST:AbstractAcoustidQuery:analyseErrors:WARNING: AcoustidError<?xml version='1.0' encoding='UTF-8'?>
<response><error><code>6</code><message>invalid user API key ("User with the API key does not exist")</message></error><status>error</status></response>

20/06/2023 02.08.10:CEST:AbstractAcoustidQuery:doPerformQuery:SEVERE: Acoustid Exception looking up from AcoustId:<?xml version='1.0' encoding='UTF-8'?>
<response><error><code>6</code><message>invalid user API key ("User with the API key does not exist")</message></error><status>error</status></response>

com.jthink.acoustid.exception.AcoustidException: <?xml version='1.0' encoding='UTF-8'?>
<response><error><code>6</code><message>invalid user API key ("User with the API key does not exist")</message></error><status>error</status></response>

	at com.jthink.acoustid.query.AbstractAcoustidQuery.doPerformQuery(AbstractAcoustidQuery.java:299)
	at com.jthink.acoustid.query.AbstractAcoustidQuery.performQuery(AbstractAcoustidQuery.java:227)
	at com.jthink.acoustid.query.AcoustIdQuery.lookupUser(AcoustIdQuery.java:157)
	at com.jthink.songkong.analyse.acoustid.AcoustId.isValidAcoustidUser(AcoustId.java:720)
	at com.jthink.songkong.cmdline.SongKong.setUserAgent(SongKong.java:688)
	at com.jthink.songkong.cmdline.SongKong.cmdlineOrRemoteModeStart(SongKong.java:1069)
	at com.jthink.songkong.cmdline.SongKong.finish(SongKong.java:1284)
	at com.jthink.songkong.cmdline.SongKong.main(SongKong.java:1314)
20/06/2023 02.08.10:CEST:AcoustId:isValidAcoustidUser:SEVERE: Problem connecting to AcoustId, assume current User okay
20/06/2023 02.08.10:CEST:SongKong:setUserAgent:WARNING: end

[paultaylor]

I think that is just a different error at startup which can be ignored, is the submission error still continuing?

[meaning]

I’m running :

root@Tower:/mnt/cache/appdata/songkong/Logs# tail -f songkong_debug0-0.log |grep "Submission Failed"

for a minute now, and so far so good, no error message is popping up. Did you change something ?

[paultaylor]

No, the problem is not with my code but the Acoustid server, and it appears to be fixed https://community.metabrainz.org/t/acoustid-server-replied-internal-server-error-for-submission/641670

[meaning]

Just to be sure, if I check “SaveComplete” events in the logs, these are the files that are done being fixed, right ? So I an consider all files that are NOT “SaveComplete” are the ones which are not matched with MB or Discogs ?

[paultaylor]

No, it just means the file has been saved.

In fact on first run all files should be saved because SongKong adds a SONGKONG_ID to every song so that later runs know if we have ever tried to match the song before (used by Ignore songs previously checked that could not be matched option)

Also, worth noting that songs can be matched to Acoustid Albums , and AcoustId Song Only as well.

You cannot rely on the logs for any kind of automated processing.