SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

SongKong stuck after 4037 files

Hi Paul
I have reinstalled SongKong today and bought a license for a year. I am using Version 8.1
I tried to scan a directory with 16/44.1kHz FLAC files which has 49k Songs. SongKong started scanning, but stopped at 4037 files. After five hours it is still at 4037
ASUS X370, Ryzen 1800X, Win 10 Pro

Hi, so I assume all 49K songs are directly within a single folder rather than artist/album folders under that?

If so this is probably the issue, usually customers have their songs organized in multiple folders usually with one folder = one album, but also some random folders as well. And in such cases SongKong works on multiple folders in parallel, it cannot really do this with just one massive folder, also one folder = one album helps with the correct album matching. Now with a one massive folder SongKong should be able to use existing metadata to split the folder into potential album groupings but maybe you have no existing metadata either?

Also I’m not clear has it taken a long time to get to 4000 files, or did it do that quite quickly but then not progressed further?

So I would suggest you cancel the task, press Stop and wait for it to create the report, then run Help:Create Support Files so I have better understanding of what is going on here. Once you have done that if you are anxious to get on with stuff you would get better results if you could split the 49k into some arbitrary 1K folders and then rerun (songs already matched in first run will be skipped over)

I will look at your support files, and I am also going to revisit the code for handling large number of files in a single folder because although not ideal starting point it is common enough that SongKong should deal with it better.

Thanks Paul
The files are not all in the same folder. They are organized in an Artist / Album folder structure
Album folders will have the numbers of tracks an album usually has

Okay should be no problem then, so did it get stuck ?

Just ran the scan again. This time it took around 5 minutes to load 4335 files and then stopped.
I have created the support files using rudolf.fischer@beyuls.ch

Okay I cant see the first report because you recreated the database after running it ?

I can see the second report, but it looks like you just cancelled it after 7 minutes, from the logs and the report it doesn’t look like it had stopped to me until you cancelled it, why do you think it had stopped ?

I think you just looking at the Loaded bar, now of course it quicker to load files into SongKong rather than identify and fix them so SongKong doesnt continually load files once it has substantial queue of work to do because that will just use up valuable memory resource, so in your second report it had loaded 4335 songs, but had only completed processing of 2177 files.

So please run again but this time just leave it alone for a few hours and I think it will continue, I notice the folder referenced is W: drive so I assume this is a remote folder and may well slow things down significantly depending on your home network

W is a local drive pointing to the F:\music, so that should not be an issue. F:\ sits on a RAID0 array of 6x6TB on a Broadcom 9361-8i controller
This is what the scan looked like after around 30 minutes, appearing to have stopped:

This morning after around 8 hours the scan is continuing:

I will let it continue until it has completed or I am sure it has stopped.
One possibility is that previously the PC went to sleep. I have now started caffeine to keep the PC awake.

ok, SongKong finished with an unreported error:

This is the report SongKong created:
Song Matching Summary
49,778 songs checked against MusicBrainz and Discogs before task cancelled by SongKong because of unreported error
Errors and Warnings:31
Selected Folder :W:\2a 44.1_16
Base Folder:W:\2a 44.1_16

Unfortunately I did not create the diagnostic reports as I thought SongKong had finished normally. I will run the whole thing again and then do the extended reports

Hi, Im a little confused in that it only done 18,000 then shortly afterwards done 49,000, did you just need to refresh the screen or something ?

Now the good news is because you didn’t cancel the task I think it had create the report so you just need to run Create Support Files and it would send me the report. Also SongKong makes changes as it goes along so even though it may have reported an error that wouldn’t prevent the matching and changes taking place. But Forty Nine Thousand files is quite alot of files to process in one go its not that surprising that some error occurred but it probably had minimal effect on the overall process.

Now for possibly the bad news, do you realize you have it running in Preview Mode so nothing will actually be modified, so unless you meant to do this you would need to rerun but ensure you uncheck the Preview Only option on the Basic tab

I have investigated further:
Thinking that the issue might be that the music files resided on a RAID0 array, I have not put the test music folders on a simple HDD. I then ran SongKong across that test set.
After a while (between 1 and three hours, can’t tell exactly as I was out of the house), SongKong stalled again

The two SongKong windows are unresponsive. When I move the mouse there, it turns into a blue circle. In Task Manager, SongKong is shown as “suspended”.

As SongKong is unresponsive, I cannot retrieve logs. I will now restart the machine and then send the logs.

I did notice one error where it was struggling to work out if drive was a local or remote drive https://jthink.atlassian.net/browse/SONGKONG-2289 but didnt think it was relevant, but may it is. I couldn’t replicate issue because I don’t understand how do you map a drive letter to a local drive I thought you could only map to a network drive ?

Could try setting Limit File Path to Windows Explorer 259 Character Limit to No as that should avoid the problematic check.

Going to your previous run, Im still not clear how it jumped from 18,000 files to suddenly having completed ?

The drive in the last test set is a straight Drive D:, no mapping at all. The scan directory was D:\music_test

Would I do this by disabling “Enable Win32 long paths”?
win32longpath-min

Re the previous run: When I looked, the count of files loaded was at 18k and when I looked an hour later it was at 49k. I have no idea what happened in between.

This is how to map a local folder to a drive letter. Very helpful to shorten path names under Windows

Okay the path checking idea is probably a false start, thanks for the link

I have your latest support files, this shows report 2 and report 3 both complete but if you go to the Errors and Warnings tab both had very similar errors, some groups timed out (which then caused the could not prepare statement further down), the groups seemed to all me classical but other than that the groups were different.

So I think what has happened is SongKong started and was working fine, it works on multiple groupings (usually but not always folders) in parallel (depending on how many cpus your computer has) . Then we got to the situation that most groups had completed but a number of groups were not progressing so looks like it stalled, then these eventually timed out and the task completed.

I haven’t worked out why yo are getting this but I have an easy enough workaround for you.

Run again (ensure Preview Only unchecked this time), most groups will be processed and I I expect some will timeout again and then SongKong will eventually complete.

Then run again, but set For songs already matched to Ignore so SongKong will skip over the songs already identified and just concentrate on matching the songs failed to match first time. Some/All of these will work this time

Then may have to repeat once more.

I will look at your log files in more detail to see if I can work out why this is happening.

I have run SongKong on the whole set several times with the option to ignore already scanned files. It alway aborts, the fourth time at around 16k (which would be around 40% of the total number of files). So that workaround does not seem to solve the issue.

I have now let SongKong loose on a folder of 3322 DSF files. That scan seems to have run to completion, but in the log SongKong still reports an “unreported error”. I have sent the logs.

ok, next try. I have now tried scanning a small subset of around 230 m4a files.
I did this twice. The first time SongKong aborted at 220 files, the second time (after clearing the library) at 170 files. I have sent the log of the second scan.
The issue does not seem to have anything to do with the number of files included in the scan. Also it does not seem to abort at the same file. This would indicate the files are not the cause.

My hardware is as follows:
ASUS X370 Pro, Ryzen 1800X, 16GB EEC RAM

Okay are these files on your RAID0 array of 6x6TB on a Broadcom 9361-8i controller or usual drive ?

I have moved all my testing over from the Broadcom controller to a single WD Red 6TB drive to eliminate issues with the RAID0 and the folder mapping. The current test drive is not mapped, the path to the directory for the last two tests is D:\music_test\m4a\ . That directory contains around 230 m4a files in Artist\Album folders.

Okay do I have these tests in the last support files you have sent over ?

You have the last one that stopped at 170 files. I deleted the database and logs before that to start from a clean slate

Interesting that it is failing on such a small set of files, unfortunately the last report I have is report 5 but because you then cleared the logs (which i didnt ask you to do) I only have logs for after that so I cant see what caused the problem in report 5, and there is no report 6 because it was aborted.

  • Report 5 started at Mar 23, 2022, 9:51:37 AM
  • No changes were made
  • Report 5 finished at Mar 23, 2022, 9:54:46 AM

and no report 6 was created.

So sorry this is no help to me, so I suggest you do the following:

Help:Delete Reports
File:Empty Database
Now run your small test
Help:Create Support FIles