SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

How to process large number of files?

#1

MAC OS X 10.5.2 w/ 4 GB RAM
Jaikoz v2.4.0

What is the best method of consolidating a large number of files, approximately 25k (60 GB)?

What I have done is consolidated all of my PC and MAC HDD on to one 1 TB HDD. I know that I might have a lot of duplicate files, so what I was thinking about is setting up the manipulators, run Jaikoz against the entire drive and then filter out all duplicates, correct the file names and then sub folders. But if I load more than 10k I get an memory low error. Any thoughts?

Is there a way to deal with the wma files also?

Thanks

Chuck

#2

[quote=chuckkyle] But if I load more than 10k I get an memory low error. Any thoughts?
[/quote]
With the default memory settings you can only load about 10,000 files, and Jaikoz uses upto 300 MB of memory. If your machine has a lot of memory it is not possible for Jaikoz to use the memory automatically but you can increase this quite easily as described here
http://www.jthink.net/jaikoz/jsp/help/osx/help.html#d0e309

If you dont have this luxury then I would suggest you work in batches, maybe split the files up alphabetically.

Not yet but it is something I have been working on and I expect it to be in the 2.5.0 release.

#3

Hi. Have read the manual on upping memory allocation. However, I have searched my Mac and find no Info.plist associated with Jaikoz. What am I missing?

#4

I seem to have messed up the online help file, it should say

Go to where you installed Jaikoz
Show popup, and select ‘Show Package Contents’
Go into Contents
Select popup on Info.plist and choose Open With Other…
Select Application/TextEdit
Find the line that says -Xms64m -Xms300m
Increase the value of Xms to the value you require

does that make things clearer.

#5

I seem to be also having some “difficulties” with large numbers of files. I upped my memory to 600MB, and still could not load all of the 25000 files I have. I’ve upped it to 800MB to see if that will fix it. I’m using the “trial” license right now just to look at it, and it seems to me that with this many files things are VERY slow. I tried the “auto” button, and it jumped right into doing the Music IP ID grabbing after it did a number of other updates. This was doing probably no more than one file every 45 seconds. I stopped that and did a smaller subset, and that seems better.

My concern of course, is that although I’m sure I have a fair number of “duplicates”, we’re also still only about 1/2 way through our music ripping process. Then again, once we get the library “in order” maybe we won’t really even “need” the product any longer, but I have teenagers involved in this project too, and my guess is they will not be that consistent over time .

Will this product really be able to handle our volume?

Thanks.

D.

#6

[quote=danitaz]I seem to be also having some “difficulties” with large numbers of files. I upped my memory to 600MB, and still could not load all of the 25000 files I have. I’ve upped it to 800MB to see if that will fix it. I’m using the “trial” license right now just to look at it, and it seems to me that with this many files things are VERY slow. I tried the “auto” button, and it jumped right into doing the Music IP ID grabbing after it did a number of other updates. This was doing probably no more than one file every 45 seconds. I stopped that and did a smaller subset, and that seems better.

My concern of course, is that although I’m sure I have a fair number of “duplicates”, we’re also still only about 1/2 way through our music ripping process. Then again, once we get the library “in order” maybe we won’t really even “need” the product any longer, but I have teenagers involved in this project too, and my guess is they will not be that consistent over time .

Will this product really be able to handle our volume?

Thanks.

D.
[/quote]
Hi with 800mb you should be able to load all 25,000 files, if you cannot and the number of files being loaded is not increasing anymore this points to there being a problem file which is consuming remaining memory - if this appears to be the case please send me your log files.

When increasing your memory please allow for the memory use of other programs and the physical memory of your computer or things will get very slow.

Things to go get slower as you load more files because there are more items to manage, this is the case with all computer programs. For this reason I would usually recommend working with no more than 10,000 records at a time, it was not really intended that would customers would load all their files in one single go.

One customer with a very large collection settled on batches of about 20,000 and was happy with the results, see this post http://www.jthink.net/jaikozforum/posts/list/459.page

Acoustic id creation is normally about 5 seconds a file, if processing many files I would recommend running this overnight.

#7

I may do this as well. I have some other questions, but they aren’t related to “large number of files” - so I’ll post a new thread to avoid confusiong :slight_smile:

Thanks.

Danita