SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

Running out of memory even though increased the memory available ?

#1

There actually two type of memory used by Jaikoz, heap memory and permanent memory. The instructions only explain how to increase heap memory but it is possible to run outof permanent memory (which doesnt normally happen) . If you are running out of memory after loading a mosdest amount ofg files please try increasing your permananent memory from the default of 64mb to 150 mb as follows.

Add -XX:MaxPermSize=150m as an additional option

OSX:Info.plist
-Xms150m -Xmx300m
becomes
-Xms150m -Xmx300m -XX:MaxPermSize=150m

linux:jaikoz.sh
java -Xms150m -Xmx300m -jar lib/jaikoz.jar -l2 -m2 -f
becomes
java -Xms150m -Xmx300m -XX:MaxPermSize=150m -jar lib/jaikoz.jar -l2 -m2 -f

windows:jaikoz.bat
java -Xms150m -Xmx300m lib\jaikoz.jar -l2 -m2 -f
becomes
java -Xms150m -Xmx300m -XX:MaxPermSize=150m -jar lib\jaikoz.jar -l2 -m2 -f

windows:jaikoz.exe
here the bad news, its not possible to change this value for .exe file so if you are running into this problem you will have to use jaikoz.bat instead for the moment.

#2

cool, thanks paul. helps somewhat. i’ve just run a test with:

-d64 -Xms1250m -Xmx2300m -XX:MaxPermSize=2300m

on 12,000 tracks (about 7% of my library) through autocorrect. (i have 32GB ram in my desktop, so this is not unreasonable). I’ll report on higher numbers when i let this test run out in its entirety, but this is the first time i’ve been able to get it to be stable much over 4k.

of course, there seem to be major opportunities for optimization and parallellization, and you start to really feel those when running over a very large library. retrieve acoustic ids runs extremely slowly (~1 track per 30 seconds) with all cpus at a max of 8% utilized, seems to be only using one thread most of the time. loading songs initially is also another opportunity, its not iobound, something else is making it just crawl through the initial reads. i’ll be more than happy to help you run some diagnostics offline if it will help.

#3

[quote=chroma]retrieve acoustic ids runs extremely slowly (~1 track per 30 seconds) with all cpus at a max of 8% utilized, seems to be only using one thread most of the time.
[/quote]
Hmm, shoudld take about 5 seconds per track, and Jaikoz utilizes a seperate thread for each cpu when retrieving acoustic ids so if youve 4 cpus it should process 4 in parallel. If the file are on a network try moving them to local fs that should improve it.

[quote=chroma]
loading songs initially is also another opportunity, its not iobound, something else is making it just crawl through the initial reads. i’ll be more than happy to help you run some diagnostics offline if it will help.[/quote]
This cant easily be parallelised, are you reading the files over a network that slows down stuff alot.

#4

no, the filesystem is local, and very performant.

it ‘feels’ like a concurrence locking problem.

#5

also, i have eight very fast cores.

occasionally, but rarely, it will suddenly move fast for a few tracks and will utilize all cores. but this is by far the exception. this is why i say it ‘feels’ like some sort of threadlocking issue.

#6

How large can the permanent memory be set to?
My library is over 100K songs. Jaikoz only seems to be able to load about a third of that at a time. Is it possible to get everything loaded if I bump up the memory a whole lot?

#7

Have you increased Heap memory this is the most important memory to increase, but Jaikoz will probably run a bit slow if you do load all 100,000 tracks , I would recommend working with smaller sets.

#8

What do I do add

java -Xms150m -Xmx300m -XX:MaxPermSize=150m -jar lib\jaikoz.jar -l2 -m2 -f

jaikoz.exe

to the bat file from there? Its telling me

‘java’ is not recognized as an internal or external command,
operable program or batch file.

do I now have java installed correctly?

I’m recovering from a hard drive crash and my backups weren’t tagged ironically I thin the drive failed because I was tagging so I could back it up. I’m doing about 12k in files. I have 4 gig of ram. How do I give it like a gig of ram?

#9

Oh yeah a copy of my path

%CommonProgramFiles%\Microsoft Shared\Windows Live%c:\Program Files (x86)\Java\jre6\bin%

#10

You either edit jaikoz.bat and run Jaikoz.bar or you edit the properties on Jaikoz.exe and run Jaikoz.exe

#11

There should not be a % at the end of your path. The error you were getting is because your Windows system cannot find Java.

#12

What is the status of this information with regards to the new release as of a couple days ago - and especially with regards to both the java and .exe versions for Windows?

[quote=chroma]cool, thanks paul. helps somewhat. i’ve just run a test with:

-d64 -Xms1250m -Xmx2300m -XX:MaxPermSize=2300m

on 12,000 tracks (about 7% of my library) through autocorrect. (i have 32GB ram in my desktop, so this is not unreasonable). I’ll report on higher numbers when i let this test run out in its entirety, but this is the first time i’ve been able to get it to be stable much over 4k.

of course, there seem to be major opportunities for optimization and parallellization, and you start to really feel those when running over a very large library. retrieve acoustic ids runs extremely slowly (~1 track per 30 seconds) with all cpus at a max of 8% utilized, seems to be only using one thread most of the time. loading songs initially is also another opportunity, its not iobound, something else is making it just crawl through the initial reads. i’ll be more than happy to help you run some diagnostics offline if it will help.[/quote]

#13

I believe you still have to use the bat file on windows.

As for the rest of the information, still seems valid, although I note the beta defaults the MaxPerm to 200 now instead of 150.

I just opened 100K+ files on my 6GB machine today with these settings

-Xms300m -Xmx4200m -XX:MaxPermSize=300m

and was running a few other apps and still had over a GB of RAM left. It was still fairly reactive (a little delay but that is expected) and I had it chugging on a couple of thousand files. (autocorrecting)

(This is OSX)

#14

[quote=chroma]no, the filesystem is local, and very performant.

it ‘feels’ like a concurrence locking problem.[/quote]

Yep, it’s locking.

Whenever I run Jaikoz over more than the recommended number of files*, my harddisks die.

Absolutely no access to anything on them, everything stalls etc…

  • The recommended number of files, according to Paul, is “less” than the number of files which reveal the programming errors in Jaikoz, being ONE file at a time as nothing is less than that and two is more.

Thanks Paul!

#15

Sounds like you are jumping to invalid conclusions here have you actually tried increasing memory allocated ?

To summarise, with Java you cannot automatically use available memory on your system unless you have configured Java application accordingly, and this cannot be done by the installer because only the user can know what is an acceptable upper memory limit to use.

Jaikoz memory usage is proportional to how many files are opened in Jaikoz, because the metadata is stored in memory, and this is the bulk of the memory usage. It was not really envisaged that users would load their whole large collection into Jaikoz in one go, in which case there is no problem. This is a problem which affects many taggers not just Jaikoz.

With hindsight it may have been better to store details in a database and only have a few screens worth of the songs metadata held in memory at one time, but this is not without its drawbacks and its very difficult to retrofit this into the existing application, but Im still hoping to do that.

#16

[quote=paultaylor]Sounds like you are jumping to invalid conclusions here have you actually tried increasing memory allocated ?
[/quote]

Memory has nothing to do with locks.

These are my settings in jaikoz.bat:

java -Xms300m -Xmx1024m -XX:MaxPermSize=300m -jar lib\\jaikoz.jar -l2 -m2 -f

I consistently experience huge delays and inresponsiveness in other applications using the harddisk, when Jaikoz is running, REGARDLESS of the number of files loaded in Jaikoz or the number of files being autocorrected, with ONE exception.

When I use automation software to fix one song at a time.

There is something very wrong with how Jaikoz interacts with the files it is working on.

EDIT: …it doesn’t leave thousands of files open and locked until it has completed auto-correcting all of them.

Right?

RIGHT?

[color=darkred][size=24]RIGHT?[/size][/color]

#17

Ok, this discussion thread was about memory usage which is why I thought you were talking about that. Im not ware of any locking problems using Jaikoz, in fact its only if you have a Save Task as a part of the Autocorrecter, or you have the Retrieve Acoustic Ids tasks configured to save after adding an acoustic id that the files are even accessed.

I will check, but you could try disabling [b]Preferences/Musicbrainz/AmpliFIND/Save Acoustic Ids as Retrieved/b] option and any Save tasks to see if it makes any difference

How do you know its not just problem with cpu contention, most of the autocorrecter tasks are multithreaded and will take advanatge of multiple cpus if your computer has them.

#18

Ok, I did that and ran autocorrecter with a set of 6500 files.
I clicked cancel after some ~50 files, and tried closing jaikoz when it asked to save files, which I then chose.

When the autocorrecter was running, I observed a negligible queue length (concurrent disk accesses), working set less than assigned for the memory, and very little cpu utilisation.

My setup is:

  • win7 64 bit ultimate, updated
  • java 64 bit
  • i920
  • 6 GiB RAM
  • 4 TiB WD HDs in a volume

When Jaikoz was saving the files, I observed a sky-high disk queue (100+).
Referring to the resource-monitor, the response-time for the various jaikoz-related threads jumped to between 400 and 500 ms.
Before saving, the response-time was between 5 and 15.
Nothing else utilized the disk.

This indicates to me that Jaikoz is trying to save all files at once.

I can help you out with more information or testing if you like.

EDIT: I just checked with “Save changes” activated as last task in “Manipulators”.
These are the problems:

  1. Files are not saved as they are changed, but at the end of all other operations
    This leads to two problems:
    A) Memory bloat, to the point of causing application crash
    B) The application is flooding the disk with (concurrent?) write operations (as many as running threads? I saw a very consistent average of 8 with an 8-core CPU) as the final task of correcting many files.

If the writes are in fact concurrent, the whole process of writing out the changes to the files will take much, much longer than if these writes were sequential.
I assume each thread will have the same priority to disk and that this causes contention.

#19

Sort of, its actually allocating a thread per cpu for saving files, so if you have 4 cpus there will be 4 threads trying to save files at once. Im not sure why I have done this because there is a comment in the code stating that we should use a maximum of two threads when saving because of file contention. I think it is better to use two threads rather than one in order to allow one thread to do preparation for saving (which doesnt involve file i/o) whilst other is doing I/O

Yes, Save Changes as a task in the Autocorrecter uses the same code, so will use one thread per cpu for saving files. I guess because you have an eight cpu machine you are seeing this problem to a greater degree than most, I only have a quad processor machine.

Files are saved at the end of the operation at the moment because Save Changes was envisaged as a separate task and for technical reasons to do with users marking files as deletions. It was an addition because I originally expected users would always want to check changes made by jaikoz before committing.

Memory bloat is due to the fact that the metadata for each file is loaded into memory when the files are opened in Jaikoz, and information can be viewed for any file in the main table. Running the autocorrecter itself only increases memory usage in the way that if it finds more data then there is more metadata to be stored. As discussed before I would like to move to this data being held in a database but this is a big reworking. However you could get round this with your 64bit machine by greatly increasing the maximum memory allocated to jaikoz.

I’m going to limit the cpu’s to two now, and upload as Jaikoz 3.7.0 Beta 3 for you to try out. But the Save And Move Autocorrecter task is not concurrent so Id be interested if you try that instead and see what results you get

The memory and Save part of task ideas are bigger problems, but Im already planning on changes to the way autocerrector works for Jaikoz 3.8.0 so the Save as part of task idea should make it that release.

#20

Jaikoz 3.7.0 Beta 3 now available with the threads limited to two when saving files to try out.