SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

Memory Issue w/ 2.9 : Please explain heap vs permanent

#1

Okay i downloaded v2.9 today and set it on a batch of about 3,500 files and fairly quickly got the dreaded out of memory message when “Correcting Tags from MusicBrainz”.

As it had been awhile since gotten such messages, I double checked the forum and the help for instructions to increase memory. It looks like the installation of 2.9 overwrote my previous edits to Info.plist (I am on a Mac). The MaxPermSize is an option I am not familiar with from the last time I edited that file.

Paul, you have a thread here (http://www.jthink.net/jaikozforum/posts/list/776.page) where you describe increasing MaxPermSize but I will admit I do not understand how the three parameters relate to each other and to the system as a whole.

I am on a machine with 6GB of memory. When I use Jaikoz, I usually run only Jaikoz in order to ensure as much memory is available as possible.

So if I wanted Jaikoz to grab a big chunk of memory from the get go…

This is the line I have currently in my Info.plist :

-Xms600m -Xmx1600m -XX:MaxPermSize=600m

(Note: I multiplied everything by 4 before I decided to make this post and ask to ensure I understood the ramifications).

Xms : What is this? (starting memory?)
Xmx : What is this? (maximum allowed memory?)
MaxPermSize : It looks like this is related to Xms. How is this different than Xmx?

Do these values need to change in a relationship/ratio to the other parameters or can they be changed independently from each other?

If I understand correctly, Jaikoz gets a full allocation of memory upon launch that it cannot increase. True? (doesn’t seem like it is true, because of the Xms and Xmx) If true, then if I wanted Jaikoz to use as much memory as it can grab (aside from the default system) which value should change?

#2

Also in this thread (http://www.jthink.net/jaikozforum/posts/list/459.page), a user suggested turning off the display of the column for artwork.

Is this true? If so, does it still get updated even if not displayed? If so, I am currently displaying all fields, should I not display the fields I do not plan to actively edit and inspect in order to save memory?

#3

[quote=wynlyndd]Okay i downloaded v2.9 today and set it on a batch of about 3,500 files and fairly quickly got the dreaded out of memory message when “Correcting Tags from MusicBrainz”.

As it had been awhile since gotten such messages, I double checked the forum and the help for instructions to increase memory. It looks like the installation of 2.9 overwrote my previous edits to Info.plist (I am on a Mac). The MaxPermSize is an option I am not familiar with from the last time I edited that file.
[/quote]
Short answer MaxPermSize is okay, max heap size needs to be increased.

Long answer, the memory settings (not being part of the setings.jai) file do get overwritten, perhaps I can do something cleverer to keep them. Yes MaxPermSize was new to me as well until recently, most objects you create go on the heap which uses up the heap memory (maximum defined by -Xmx) but interned strings use permanent memory , and this is defined by -XX. Jaikoz uses quite alot of interned strings so that the same value in two different fields is not stored more than once but I didnt realise they were stored in different memory. I might have to change this because I cant change the default on the Windows in Jaikoz.exe. But the MaxPermSize has already been increased from the Java default in the Jaikoz files so you shouldnt have to modify it.

With 32 bit JVM you can only address about 2GB, so if you want to use all the memory you have available you need to use the 64 JVM. On a Mac you can change this (for all Java applications) by going to Applications/Utilities/Java/Java Preferences and changing Java Applications versions so it has JS2E 5.0 64-bit at the top.

This will should allow you to increase -Xmx to 5000m , there is one gui bug with the 64bit version of Java, the menubar is the wrong colour.

Starting heap,Max Heap, Max Perm

Jaikoz will use the value set in Xms + MaxPermSize on start up, and then continue allocating more memory as required upto a maximum of Xmx + MaxPermSize.

I would recommend in your case trying

-Xms=500m -Xmx=5000 -XX:MaxPermSize=150m

wrt to you second post.

Yes Jaikoz updates all fields whether visible or not, so you dont have to diaply all fields to update them but hiding them will not reduce the memory fottprint that much, that applies to album artwork as well.

[/quote]

#4

Thank you for the quick reply, Paul. I believe I understand quite well now and hopefully this helped someone else. I shall bookmark this one so hopefully I won’t have to bother you ever again about memory (unless I forget…that’s a memory joke, folks!)

Thank you for adding the part about 64bit JVM. I was indeed already using in order :

Java SE 6 64bit
J2SE 5.0 64bit
J2SE 5.0 32bit
J2SE 1.4.2 32bit

Jaikoz doesn’t appear to have any issues with SE 6. (Well maybe one which I will make another post about) except I think I see the GUI issue you refer to (I didn’t even notice before).

#5

Ugh, sorry to bother you again, Paul.

2009…0324 4:29:27 PM [0x0-0x22022].com.jthink.jaikoz[185] 24/03/2009 16.29.27:com.jthink.jaikoz.monitoring.MemoryManager:addMemoryNotification:WARNING: Low memory limit is set for:Tenured Gen:1901454950

This was after rebooting the machine so almost no other memory was used.

Here’s the memory line from my Info.plist

-Xms500m -Xmx3200m -XX:MaxPermSize=150m

I was only opening a folder of 30 items. I had only opened the folder and had performed no other operations.

#6

Timely question, clear answer. Thanks to you both.

Some things I’ve noticed:

I could load ~50k items with 32-bit java, memory settings 150-2000-150

With 64-bit java, 150-3000-150, I could only load around 42k

With 64 bit java, 300-3000-300, I loaded 58k; however, at least one other program wouldn’t run under 64 bit java

With 32-bit java, 200-2000-200, Jaikoz wouldn’t load

So, I’m back to 32-bit java, 150-2000-150 as my standard setting. That lets me do other work while Jaikoz is running and is sufficient for everything but those monster sessions.

All these settings will vary according to your machine, memory and files (mine have embedded art & lyrics). For the record, I’m working on an iMac with 4gb total RAM.

#7

Jaikoz trys to anticipate when you are getting low on memory so it can take evasive action to prevent you actually running Out of Memory. Im not sure why its issueing a warning at less than 2GB but if it isnt actually reporting an error I dont think its a big issue.
[/quote]

#8

[quote=mcammer]Timely question, clear answer. Thanks to you both.

Some things I’ve noticed:

I could load ~50k items with 32-bit java, memory settings 150-2000-150

With 64-bit java, 150-3000-150, I could only load around 42k

With 64 bit java, 300-3000-300, I loaded 58k; however, at least one other program wouldn’t run under 64 bit java

With 32-bit java, 200-2000-200, Jaikoz wouldn’t load

So, I’m back to 32-bit java, 150-2000-150 as my standard setting. That lets me do other work while Jaikoz is running and is sufficient for everything but those monster sessions.

All these settings will vary according to your machine, memory and files (mine have embedded art & lyrics). For the record, I’m working on an iMac with 4gb total RAM.

[/quote]
I havent done tetsing with 64bit myself.
Not sure why you can load less tracks with 64bit,maybe some object take up more space in 64bit, are these test results consistent. The improvement from 42k to 58k when increasing permanent memory indicates that maybe you need more permananent memory after all when loading this many files.

#9

I tried loading w 64 bit twice, both times after restarting. I dumped out after a dialog box opened with a low memory message at about 42k files loaded. Since it took more than 20 minutes to load up that many files, I didn’t try it more than that.

However, there was a difference with how the program behaved before and that made me try adjusting the MaxPerm memory. I keep things like album reviews pasted in the lyrics section (the same 4 paras present in a dozen or more files). Based on what you said above, it seemed possible this was eating up Perm memory. So I bumped that up and was able to load more files (after restarting). I would’ve kept those settings but at least one program outside Jaikoz wouldn’t run until I went back to 32-bit java.

Anyway, I think it’s worth each user playing with these settings themselves. I could imagine someone with more RAM and sparsely tagged files loading up a lot more initially. (though they could run out of memory as Jaikoz fills in fields, right?). I would leave at least a gb of RAM free for background operations of the computer, even if you aren’t running another program. Jaikoz gave me an error message and stopped loading files one time right as Time Machine started preparing a backup of my hard drive. Now I turn that off before Jaikoz sessions.

#10

I haven’t tried loading up any truely massive batches yet (largest recently was 13,000 files roughly) but I will soon. (relaized that on several batches I didn’t run a discog check.)

But, in general things seem more stable and run more smoothly than ever before (now if we could wave a wand and improve the musicbrainz servers a bit…wish I had a local copy of it :):):slight_smile: )

Just eyeballing my memory I think I can start loading bigger batches.

mcammer, you mentioned you had an imac. Which processor is in your imac?

#11

2.93 C2D

#12

[quote=wynlyndd]now if we could wave a wand and improve the musicbrainz servers a bit…wish I had a local copy of it :):):slight_smile: )
[/quote]
I am just in the early stages of looking at this, it isnt that difficult to setup your own server and I would like to sketch out instructions on setting and remove the throttling if you are using your own local server.

#13

[quote=paultaylor][quote=wynlyndd]now if we could wave a wand and improve the musicbrainz servers a bit…wish I had a local copy of it :):):slight_smile: )
[/quote]
I am just in the early stages of looking at this, it isnt that difficult to setup your own server and I would like to sketch out instructions on setting and remove the throttling if you are using your own local server.[/quote]

I appreciate any pointers you have for this. I have spent only about a tiny amount of time looking at this. I need to rustle up a spare machine first (not too hard to do though)

So they package up the database in such a way we can get snapshots of it? I need to see if I can get rackspace at a friends colo.

#14

I wil publish something when I ve done the work but there are plaenty of documents on Musicbrainz Wiki explaining how to setup a server, and one option is to use the provided Virtaul machine that does half the work for you.

However if you the reason you want to do this is purely for speed it will not make much difference because the current version of Jaikoz has mandatory throttling imposed to only allow one query per second. Now this could be removed when using a local machine but I havent done made the relevent code changes yet.