SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

Songkong keeps crashing randomly

#1

Hi @paultaylor,

FYI, I started to fix a medium size folder which contains approx 680k tracks.

As you can see, songkong keeps crashing randomly. This did happen not only to bigger folders, but also almost every time I ran a fix on smaller (monthly) folders. I had to start the job again and again.

On very strange thing I did notice is that between first crash, and the next run, songkong usually runs a 0 track analysis, like you can see it in the following screenshot :

image
image2886Ă—250 262 KB

As you can see, it firstly loaded 19047 tracks, then crashed.
Next time I ran it, same folder, it loaded 0 tracks
I has to start the job one more time in order to get it to start loading tracs again, and it crashed one moe time after loading 58141 tracks.

I can tell you this is no memory issue, nor is it a docker container remaining drive space issue, as I did allocate it 500GB.

I believe we could take this as an opportunity to dig the logs and try to fix this random crash issue once for all, make songkong more rock solid and stable :wink:

Thanks !

By the way, I like how the logs did change, they are more readable, and I also like how the general songkong.log (of the docker) displays the processed folders :

image
image2400Ă—1554 1.43 MB

I think this log should be used for the lambda users, it would be nice to add a few things here like:

Loaded folder /album_Folder/
Identified : /album_folder/
Unidentified : /Album_folder/
Rename and Moved to /Album_Folder/ -> Destination folder

Just my two cents :wink:

#2

I will look but is it possible to try on your Mac, because my feeling is that this is related to Docker in some way.

#3

I’d like to, but two problems.

  1. As I already ran the entirety of my fixes on my unraid server, I’d like to keep adding the info to the existing DB, as you know this is important for the next step(s) (Eg: duplicates finder)
  2. I only have one macbook, which I cannot really leave running for such long times, I use it for work and have to travel with it almost every day.

Only thing I would have considered aside the docker was a VM (windows). But you told me it would be even worse than the docker container :wink:

#4

Regarding step 1, everything is written to the files themselves. So the state of the database wouldn’t make any difference to the results you get with Delete Duplucates, what is important is that you have run Fix Songs first so they have musicbrainz ids ecetera, but it doesn’t need to be done on same computer.

The database is three things.

  • A cache of your music files, so we don’t have to go back to file each time.
  • A cache of data from Discogs/Musicbrainz so don’t have to retrieve from Internet each time.
  • A record of changes so can use Undo Changes to revert changes.

But it shouldn’t have any effect on results only performance.

#5

Okay it didn’t actually crash, a crash is when the program stops unexpectedly and throws a dump. It stopped,and it stopped because after making several attempts to contact MusicBrainz server it gave up

image
image764Ă—808 39.6 KB

I looked in the logs to see why report 61 did nothing, but there is nothing to indicate an error or any reason why it didnt load anything.I currently dont have an idea why this has happened, I thought maybe something wasnt reset because the previous run failed, but since it didnt crash and since we reset everything when we start task I cannot see a problem. I will add in a bit of extra debugging to see if that can help us.

Okay sorry to disappoint bu this just contains the output of anything written to System.out, this list is debugging information that I meant to remove and forgot. But a summary of what is being processed and the extra information you request is mainly in songkong_user-0-0.log

#6

haha ok, well, I ran a new job.

130k tracks processed after a little over 46 hours run time.

Patience is the key.

I’ll keep it running for now and will keep you posted. By the way, it’s not the first time I notice songkong stops due to musicbrainz being “offline”. My internet line never goes down, so I’m surprised this actually do happen.

#7

Logged as https://jthink.atlassian.net/browse/SONGKONG-2368

#8

great. FYI I just sent another set of support files.

I got a new musicbrainz access issue after just a few releases were checked this morning.

I’ve started monitoring my connection to musibrainz servers, and everything is allright on my end.

24/11/2022 08.45.41:CET:SEVERE: Adding Error:Problem accessing MusicBrainz Server Read timed out
java.lang.Exception
at com.jthink.songkong.analyse.general.Errors.addError(Errors.java:28)
at com.jthink.songkong.exception.ExceptionHandling.handleAlbunackNetworkRetriesExceeded(ExceptionHandling.java:200)
at com.jthink.songkong.analyse.analyser.AbstractMusicBrainzGroupMatcher.matchSongs(AbstractMusicBrainzGroupMatcher.java:160)
at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher.doTask(MusicBrainzSongGroupMatcher.java:454)
at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher.call(MusicBrainzSongGroupMatcher.java:485)
at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher2.call(MusicBrainzSongGroupMatcher2.java:58)
at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher2.call(MusicBrainzSongGroupMatcher2.java:16)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)

But as far as I remember you told me sk is actually connecting to a mirror of MB, right ? would it be a good idea to run a simple ping to this server and see if I get timeouts, from within the container ?

#9

Yes you are right it connects to a SongKong server, although this is not a simple mirror it is merged database of MusicBrainz/Acoustid and Discogs.

So you had problerm at 08:45 CET, this is 07:45 GMT and when I look at the server stats I can see there was a problem at this time with cpu at 100% and returning error codes.

image
image1920Ă—1040 163 KB

But it sorted itself out, maybe I need to upgrade server but most of the time cpu utilization is only about 20%

image
image1920Ă—1040 224 KB

Maybe I need to change to SongKong to wait longer for server to sort itself out before giving up.

#10

Bingo ! you spotted it !

The timeout delay could indeed be increased. But what seems weird in the graphs you share is how sudden this cpu usage grows.

It really looks like a sudden leak to me.

#11

Possibly, but for now…

I have just built a new version of SongKong 8.7 for Docker where the timeouts have been increased so the total timeout after ten retries increases from 3 minutes to 18 minutes, this should significantly reduce the chance of failure due to timeout due to temporary overload of server, also added a bit of debugging.

I also noticed that the desktop version pauses if hits this error and then gives you chance to continue or cancel, whereas the web version just cancels, so I will get this fixed when I can.

Not enough for a new official release, so just reget the latest image, it will still say 8.7, but if you select About it should say 22/11/2022 (whereas the original release would say 21/11/2022)

image
image624Ă—653 22.9 KB

1 Like
#12

I installed the 22now build.

unfortunately :

Nov 26, 2022, 7:46:02 PM

Problem accessing MusicBrainz Server Read timed out

It keeps stopping for the same reason.

I send the support files right away.

#13

Okay I see it failed again, have you retried since, I have made a few adjustments to the server configuration, lets see if that helps.

#14

yes and unfortunately :

Nov 29, 2022, 12:09:47 AM

Problem accessing MusicBrainz Server Read timed out

I’ve updated the support files again.

By the way I can confirm that each time it crashes, I have to start the job twice, as the job I start right after the crash simply don’t process anything (and reports nothing).

#15

Okay I have made some further adjustments, and for now just released the docker version for you to try.

If you reget the latest image, it will still say 8.7 , but if you select About it should now say 23/11/2022 (whereas the previous version would say 22/11/2022 )

#16

OK, I let it run for now and will update / test again with new version after next crash. :wink:

1 Like
#17

hard to say if it’s crazy stable or not as I had to stop a job yesterday due to a planned power outage, but I can tell you I just spent a few days fixing tracks without SK stopping due to musicbrainz timeouts.

#18

If you could send me the logs, I can see if it encountered temporary timeouts and dealt with them or not.

#19

I uploaded the support files.

#20

Okay no timeouts seem to have occurred, so no definitive that the change has fixed the issue or not, but it also shows the change hasn’t caused any problems either so thats good.