SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

Memory

#1

No matter how much memory I provide to Jaikoz, I still run out of memory. So let’s talk tuning; I have approximately 45,000 songs in my library. I have a 3.02 Ghz 4 core cpu. 20GB memory. 16 TB of SASS Raid 5 disk space. Sync 10MB/sec link. Jaikoz is the only application on the Windows 2008 R2 (fully patched) system.

Currently the only way to have Jaikoz re-tag my library is to select a couple of hundred songs and perform the re-tag. This is #1 not efficient and #2 does not allow for full visibility of all of the music for performing a “remove duplicates” or other global operation.

So my question is: How much memory does Jaikoz need for each song? If we assume that everything has to be performed, how much memory would each song have to have. Now once we have that number, how does that equate to the ?Xms/x and PermMemory parameters?

I am also having an issue writing the songs to disk (open by another file error) even though this is the only application with access to the file.

#2

Dang, I wish I had that type of power :slight_smile: I have about a quarter of that on between my windows home server as well as on my win7 64bit PC. I keep my 300k mp3s on the NAS and edit them from my pc where i have jaikoz installed. Other than some slowness from running it across my old network it isn’t to bad, and my lame 2gb capped ram on my NAS. I find I can load about 10k to 15k files fairly easily without having to modify any of the memory settings. Even after modifying the batch file, I typically try not to load over 25k files as it seems to bog the software down, at least on my systems. I have found that even on my friends gaming system which has specs fairly maxed out, that anything above 25k can possibly give me trouble if im working on a lot of tasks within the app.

#3

There are some benefits to growing older ? better hardware! The reason I purchased the software was to reconcile my complete library, not chunks of it at a time. I am very perplexed as the software is slow at performing many different aspects of the process. I attribute the majority to this being in JAVA (note: I?m not a fan), but that is a technical discussion which will only devolve the base issues. My network is first a class symmetric 10MB link yet my lookup are sloooow. My hardware is RAID 5 on fiber channel with 15k spinning drives on the far end. My processors are 1 generation back chips so still capable chips.

I have tried tuning the java parameters however it is better to tune the system by knowing hard facts. I could spend some time reverse-engineering the code however it that is only for apps which there is no discernible owner. We know the engineer who created this and this request is on his forum. He knows how large his objects are, he knows how they are moved in and out of memory. He knows how much is left for JAVA to cleanup and how much he is cleaning up himself. The bottom line is only he truly knows how to tune the system for the best performance.

When we are talking memory exhaustion it is from multiple things; bad programming, bad collectors and occasionally bad memory managers. Since the system runs along happily for 10,000 songs or so I would point the finger at either not cleaning up after one?s self, or really crappy garbage collecting. I do not know the programmer so I can only guess and that?s not polite to either party. So let?s concentrate on the JAVA parameters themselves. Any help would be greatly appreciated on this subject.

As a note: I moved this to a larger server 4 chip, 24 cores, 256GB memory, 10TB direct 15k SASS drives and 4 1GB network connections with direct access to the Cisco router. I moved the music library to the main drives and let loose the Jaikoz software; little to know difference? it?s the software or the tuning parameters. One note on this system is that I set Xms=150m Xmx=1024m and the jvm complained it could not acquire the memory for the heap?I love JAVA!

#4

I use Windows 7 64bit and 8GByte of RAM (Java-runtime: jre-6u29-windows-i586 and jre-6u29-windows-x64 both installed). No NAS, files are on an internally installed but removable disc.
I can load up to 100000 mp3-files (never tried more), but it takes more than 24 hours to load them :frowning: :frowning: :frowning: . After that, I can fairly fast sort the files in the Edit-window, jump around and edit the meta-data :slight_smile: .
You can find the settings of my jaikoz.bat file elsewhere here in the forum.

#5

I too can load many files, 44k and my load is also slow. However, once you have loaded the files try to do something other than sort them. I first attempting to fix the files with “AutoCorrect”; which fails after ~4k files. Then I am attempting to Acoustic ID the files; which fails at ~7k. So loading the files is trivial, manipulating the files is where this software fails. My assumptions are that as ?new? information is received from the service provider, that new information is placed into a structure/linked list/some other storage mechanism. As the number of items which are modified increases, so does the need to store all that ?new? information. It?s this expansion of storage that is killing the applications (I assume).

Now if the system would detect that memory is running short, offer the user the opportunity to save the changes and continue, or better yet allow the user to say ?auto save all changes?, then the need for memory would drop as you could write out the changes as you received them (use threading so things don?t slow down).

But I am not attempting to offer suggestions to the developer?I am attempting to understand how the parameters are ?best? tuned. I have reviewed the forum and not found anything substantial.

#6

Have you actually increased the max memory available to Jaikoz, by default it will never use more than 800MB you have to chnage the memory settings to allow it to use more. Also could you please send me your support files so I can try and work out why it is running so slow for you.

#7

@UnixMover,
I can’t agree with you.
Last week I loaded 84562 files into Jaikoz and did “Retrieving Acoustic IDs using Acoustic Sever” and “Submit MusicBrainz/AcousticID pair” and about metadata of 55000 files were submitted. See http://acoustid.org/stats

#8

I have been working with some friends from Oracle (Sun) on how to tune this app. I have set the parameters as they have suggested and loaded 8,249 files. I then set the “Auto” feature to pull and update acoustic id, tags from the net, and then modify the tags on the files themselves, and finally save the information. I did all of this on a ?normal? computer 6GB memory, 4TB disks space, 4 core 2.6GHz proc, and a 1GB net card; the process has been running for over two days now and only now on the ?Correct from MusicBrainz? phase. I will allow the process to complete and then send all of the logs, except the binary one to you (I do not allow binary files to leave my system).

#9

Another question WRT folders. My music is normally filed in “C:\Music\Artist\Album” there are some that go by years so the folders would look like “C:\Music\year\Artist\Album”. The questions is what is meant by “base folder”? Do I set the base folder to 3 if the majority of the files conform to the “artist/album”?

#10

The value just defines how we split a folder path between base folder and subfolder, this is only useful if we are renaming subfolders based on metadata, it counts upwards from the full pathname to towards the root.

But yes, this is a problem if some songs are stored using a shorter path then others, as you say the sensible thing to do is set the length which works for the majority, then you’ll have to use File and Folder Correct/Shift Sub Folder to Base Folder for the others.

With this in mind I’m thinking of changing how this works so that it counts the base folder from the root , so in your case the setting would just be 1 which would be equal to C:\Music. Also I’m wondering whether to instead of letting you specify it in the folder dialog I should remove the option there, try and guess the correct split and then display this to allow you to override if you want, what do you think ?

This helps with another common case, when multidisc releases are stored one subfolder per disc they have an extra level of folder compared to single disc releases.

#11

I like your idea of auto-magically doing the work. Since you are the one reading in the files you know how many levels each directory has. With this a priori knowledge you adjust each folder with the known level is at. This way you can handle many different levels in the same tree.

#12

Who is h21.zqv.me (46.4.69.238)?? Either JAVA or your application make a connection to that http site. I get the stimpy.musibrainz.com but the other… not sure. The network is registered to RIPE which is a known hackers network. This concerns me!

#13

Bad form to reply to ones on msg; it appears to be liked to acoustid.org.

#14

Yes, it must be acoustid.org I dont connect to a site using the name h21.zqv.me

#15

I agree. I am very diligent in monitoring my network and sometimes the littlest things get the hair on the back of neck up.

WRT the memory issue: The app finally died during a 19.5k run (note it took 4 days before it died). I then implemented the Sun recommendations for the JAVA engine however to no avail, the system died during a 18k data run. My assumption is that when you download metadata you store that in an object/structure of some type; do you release those memory areas when the user implements a save? I have taken to saving after every operation (see next paragraph for issues related to this trick) I perform on the data. When I work in small bits, things are ok. When I work in 15k and greater chunks the application dies.

Despite the application failure (due to memory issues), the one thing that I have to mention is that the system is very slow with large datasets. Especially in file operations. I have taken to watching the console to see if the system is performing any data saves as this seems to be a time-consuming process. There also seems to be an issue with the owner of the file and how the threads are spawned in the application.

I am currently running on a Windows 2003 system and implement file security. There are few users on the system and the majority of the users come from external. The administrator & the administrators group have full permissions. The users have read only permission (774 Un*X style). The issues manifest themselves when you write a file. Sometimes the files write out with no issues, other times they complain that someone has a lock on the file (no one else is accessing the system at these times). And still other times the console states that it cannot find the file and a ?Null Pointer? error has been thrown. If I set the permission to wide open (777), your app does not complain. Not sure and I can set the permission set to open when there are no users and then switch it when the first user logs in; so I can work around the issue.

#16

[quote=UnixMover]The app finally died during a 19.5k run (note it took 4 days before it died). I then implemented the Sun recommendations for the JAVA engine however to no avail, the system died during a 18k data run.
[/quote]
That does sound very slow, what do you have heap set to and what other modifications are you making.

Yes, but the majority of the memory is held onto by the metadata in the files. i.e if you just load 20,000 songs into Jaikoz (and do nothing else) that will use alot more memory than if you load 5,000 songs.

More memory is used when you autocorrect, but most of that should be freed once the autocorrect has completed. Saving changes means that there is no difference in the metadata in the song on disk and the song in Jaikoz, that will freeup some more memory, however to reiterate the majority of the memory is used up loading songs into Jaikoz

Im well aware of this problem and have a solution to do this, essentially alll the metadata for your loaded songs will be held in database, and only the subset currently displayed will need to be held in memory. but this is
a major change that I am having to implement outside of Jaikoz and then merge back in when I am happy with it.

You need to increase your max heap, you can see what this is set to in Jaikoz by looking the first line in the Jaikoz console, the default value is approx 800 mb.

It is rather difficult for a Java program to get a comprehensive view of the security setup on Windows, but the situation improves somewhat with Java 7. Id need to see the log files to find out if there are cases where Jaikoz is unable to write when it should be able to write but you may be encountering the following:

Sometimes Jaikoz has to create a new file in a folder, write to that file and delete your original file rather than just write directly to your file. So for that reason you need permissions to create files in the parent folder rather than just be able to edit the files. Whether or not this is required depends upon whether the original file has a metadata tag large enough for the new metadata, and the audio format you are writing to.

Would be keen to see the stack trace of this exception, should be in Jaikozdebug0-0.log

#17

Paul, while I truly appreciate all of the effort that has gone into this application-- I just cannot get reliable performance – so it is with a heavy heart that I thank you for what you have created, and wish you the best of luck in the future. I will check back from time to time to see what progress you have made, however I need to located an application which can perform on “my” dataset of 40k music.

I hope your holiday season is fruitful. I also hope your 2012 finds you prosperous.

Respectfully,

Thomas Klinect
SDVOSB
TPI Inc.
619-270-9533 (V) | 619-342-2684 (F)

#18

[quote=UnixMover]Paul, while I truly appreciate all of the effort that has gone into this application-- I just cannot get reliable performance – so it is with a heavy heart that I thank you for what you have created, and wish you the best of luck in the future. I will check back from time to time to see what progress you have made, however I need to located an application which can perform on “my” dataset of 40k music.
[/quote]
Hi, when originally developed it was originally envisaged that customers would not want to load their whole library in every time, I assumed they would just load in their incorrect files, fix them, and then load a new batch next time. Additionally the average size of music collections is steadily increasing.

Having said that I have been moving towards making it easier to load even very large sets of files. Stage 1 is that you should be able to load and process very large collections as long as you have a decent machine and allocate enough memory, Stage 2 is to allow large colections to be processed with just the default memory settings and a standard machine.

I’m currently working on both problems, I have one customer who sucessfully loads and processes 250,000 songs in one go, however with this many files there are some problems that can occur during file loading, file processing and file saving. I’m working through these and should have a new release in January, your 40,000 song collection should then work as you expected.

Stage 2 is a longer term, and I’m working on a prototype to solve this problem.

but stage 2 is longer term.