SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

3.70 beta 4

#1

Couple of observations.

  • read settings from previous installation if exists
  • for each installation, it shows the choice “Norwegian” as selected, not the choice for my existing installation (this “translatian” seems to have been done by an automatic translator by the way, and worth it be not have more but less makes it so)
  • all choices for manipulators are overwritten with jaikoz defaults

  • searching in offline help doesn’t work

  • default choices for jaikoz manipulators seems to be weird and the documentation is less than clarifying

  • what is the difference between local correct and correct
  • which tasks can run file by file
  • what does cluster albums do (please update the documentation at http://www.jthink.net/jaikoz/jsp/help/windows/help.html#ClusterAlbums to reflect what “groups them by artist and album and tries to reduce the number of release ids the tracks are spread over” actually means to the user and why he would want to chose this)
  • in what order should manipulators be chosen, so that running task 1-5 and task 3-7 and task 9-10000 actually makes sense or conveys any meaning to the user
  • can’t run more than one instance of jaikoz at a time
#2

[quote=MAzE5h1p69wB]Couple of observations.

  • read settings from previous installation if exists
    [/quote]
    Generally it does convert the preferences from the existing preferences, however some preferences are on purposely not updated sometimes because of new changes in the way Jaikoz works. For example the Autocorrecter used to have ‘Correct from Musicbrainz’ and ‘Update from Discogs’ as default tasks, but now that Correct from Musicbrainz automatically updates the information from Discogs when it has a link to Discogs there is not normally any point having Update from Discogs as a default task. Also There is now a ‘Correct from Discogs’ task which is make sense to include. So the preferences for the Autocorrecter have not been preserved for this release.

Yes, the installer just tries to use the default language of the computer, I don’t develop the installer not sure if this can be changed, but within Jaikoz itself your Language and Preferred Country are preserved between versions.

This is now fixed.

Local correct does not requite Internet Access, Correct is really Remote Correct.

This is all going to get an overhaul and will probably change in the next release

You have ten tracks, five linked to release with name ‘fred’ and release id 1, and another five to another release also called ‘fred’ but with release id 2. Cluster Albums tries to match all tracks to either release 1 or release 2.

Cluster Albums may well become defunct once I have changed Correct from Muscibrainz to be more release orientated,

I don’t understand your question but as I say I will be working on changing this, if you have any ideas how you think it should work please let me know.

That is correct, I cannot see why you would want to do this.

#3

Well, I find the manipulators and what they do, confusing.

Perhaps a tree-view, grouping related tasks, would make things a bit clearer?

Such as grouping together all tasks having to do with correcting metadata.

For example:

  • Correct tags
    ** Correct artist (checked by default)
    ** Correct album (checked by default)

And also, perhaps grouping:

  • Fetch metadata
    ** Generate acustic
    *** Fetch from musicbrainz
    *** Fetch from discogs
#4

Hmm, almost everything is about correcting metadata really, but somethings also have to find a matching track (i.e Correct Metadata from Musicbrainz’ whilst others already know the matching track (i.e Update Metadata from existing Musicbrainz Id’ , and I think different people would group things differently (i.e Remote versus Local).

What is confusing is the ‘Fix Song by Song’ option, I added this because a common problem was that a users autocorrecter might consist of Retrieve Acoustic Ids and then Correct Tags from Musicbrainz but they didn’t want to wait for Retrieve Acoustic Ids to be run against all songs before Correct tags from Musicbrainz was started so they could see some results quicker.

Both of these tasks can be run Fix Song by Song, so if you just had these two in your Autocorrecter, Jaikoz would Retrivee Acoustic Id for Song 1, then AutoCorrect Metadata from Musicbrainz for Song 1 , then Retrieve Acoustic Id for Song 2 and so on. But the AutoCorrect Metadata from Discogs task does not work song by song, it essentially fixes songs folder by folder/album by album, so if you add this as your third task to the Autocorrecter then Jaikoz would complete Task 1 and Task 2 before starting Task 3. The trouble is that Autocorrect Metadata from Musicbrainz is also going to be more release orientated in the next release so ‘Fix song by Song’ will not work fot this either, then we are back at the situation we had before ‘Fix Song By Song’ was added !

My current thinking is to drop ‘Fix Song by Song’ to ‘Fix Folder by Folder’

#5

New bug:

Constant disk queue of 50, 8 instances of genpuid.exe with delta I/O reads between 52 and 128 each.

sigh

EDIT: I set the affinity for java.exe to use one core only, and the queue dropped to 10. Still seeing 8 instances of genpuid, but I assume that is because jaikoz was running and changing the affinity doesn’t kill threads.

#6

Does disabling ‘Musicbrainz/AmpliFIND/Save Acoustic Ids as Retrieved’ resolve this ?

#7

Yes.

With that disabled, there is no disk queue.

Nor am I seeing any genpuid.exe subprocesses of java.exe, and running jaikoz.bat with retrieve acustic ids as a manipulator, reports warning createmusicipacusticidworkerthreadN ended.

#8

warning createmusicipacusticidworkerthreadN should be logLevel config and shouldnt show up, ignore this.

If it is creating acoustic ids it is working properly.

So the solution is just to limit the saving of acoustic id to be single threaded, but still allow 8 threads for an 8 cpu machine, or is there a need to limit threads to a lower number because even with this option disabled there is disc read i/O involved in generating the acoustic ids ?

Edit:I cant have use multiple threads for generating acoustic ids but force any read I/O to go through a single thread because genpuid is a non-Java application provided by AmpliFIND. So the choice is either to have one Genpuid working one file at a time, or have multiple threads each running Genpuid and consequently concurrent disc reads. But the saving of acoustic ids is under my control and that can be made single threaded whilst still having multliple Create Acoustic Id threads.

#9

[quote=paultaylor]

That is correct, I cannot see why you would want to do this.[/quote]

I would want to have multiple instances of Jaikoz running, so that they can all use a separate disk and be limited to using a separate core.

That way I hope to quadruple the throughput, seeing as Jaikoz handles disk-operations in a manner which causes acustic id-generation to fail and alse slows things down enormously.

#10

[quote=paultaylor]…or is there a need to limit threads to a lower number because even with this option disabled there is disc read i/O involved in generating the acoustic ids ?

b[/b]…multiple threads each running Genpuid and consequently concurrent disc reads. But the b[/b]saving of acoustic ids is under my control and that can be made single threaded whilst still having multliple Create Acoustic Id threads. [/quote]

For 1), I’m seeing a queue-length of 50 (maximum allowed, I guess), which causes both huge delays and also fails, as mentioned.

For 2), saving doesn’t fail but with a queue-length of 8 I would at a wild guess say saving takes two magnitudes time longer than a sequential save would.

Either way, with non-SSD disks, file-operations should be limited to one read / write at a time.

EDIT: I’ll see if I can compare this time required, with the same set of 1000 files, for a non-SSD disk and an SSD raid-0.

#11

Multiple instance of jaikoz running is not the correct solution for a number of reasons but Im going to make a couple of changes for you now which should solve/ease the issue.

  1. Save Acoustic Ids will now be sequential write.

  2. Max threads for retrieving acoustic ids will be limited to 4 even if you have more physical cpus (of course if you only have one physical cpu you are only allowed one thread)

#12

Are you using Windows ?

If so Jaikoz 3.7.0 final is now available for download in the beta page For Windows Only

If you run Retrieve Acoustic Ids standalone or as part of the Autocorrecter with Fix Song by Song unchecked it will use a max of four threads for generating acoustic ids. If you run it as part of the Autocorrecter with Fix Song by Song enabled then it will still use 8 threads, however depending on what your autocorrecter tasks are not all threads will be working on Retrieve Acoustic Ids at the same time, (some will be working on the next task) so the resultant file i/o should be similar. In both case file write I/O is constrained to be single threaded.

Give it a go, and let me know

Edit:Now available for Linux and OSX as well

#13

[quote=MAzE5h1p69wB]
EDIT: I’ll see if I can compare this time required, with the same set of 1000 files, for a non-SSD disk and an SSD raid-0.[/quote]

I ran a test-case of 1020 files.

Fragmentation was negligible, load from other processes was negligible.

For the RAID-0 with two Intel X25-M G2, the batch took 33m48s.
For the single regular harddisk, it took 34m34s.

Nothing new here, as we have come to the conclusion that what slows things down are concurrent reads and writes.

What I do find strange, though, is that the load was so very, very small, and yet it took so long.

Almost no CPU, net, or IO-load, and yet it took 2 seconds per file.

I think it may be possible to get this down to 200ms per file, if not even less. :slight_smile:

#14

Is this is with 3.7 Beta or 3.7 Final ?

Is this running just Retrieve Acoustic Ids or running AutoCorrect , if running AutoCorrect comprises what tasks ?

#15

[quote=paultaylor]Is this is with 3.7 Beta or 3.7 Final ?

Is this running just Retrieve Acoustic Ids or running AutoCorrect , if running AutoCorrect comprises what tasks ?[/quote]

Beta 4.

Uploaded with ImageShack.us

#16

Well hangon, consider you are retrieving data from AmpliFind, Musicbrainz and Lyrics Fly, and the last two impose a webservice limit of 1 request per second and you are saving changes to your files I think 2 seconds per song is not too bad.

Generally when you are correcting lots of songs its expected that you will just leave Jaikoz to do its thing.

#17

[quote=paultaylor]… a webservice limit of 1 request per second and you are saving changes to your files I think 2 seconds per song is not too bad.
[/quote]

Well hokay, I did not know that. Mystery solved, I guess.

Then I have really nothing to complain about, except I’d like to just grab ahold of this for a minute:

[quote=paultaylor]
Generally when you are correcting lots of songs its expected that you will just leave Jaikoz to do its thing.[/quote]

Ideally, I’d like to see Jaikoz running in the background, as a service, monitoring a folder, continuously processing files.

There are applications which do that, very successfully.
Such as for example utorrent and sabnzbd.

#18

[quote=MAzE5h1p69wB]
Ideally, I’d like to see Jaikoz running in the background, as a service, monitoring a folder, continuously processing files.
.[/quote]

I have thought about doing this, but it would probably be another program because you get plenty for your money in Jaikoz, and its complex enough as it is without adding a completely new level of functionality.

So it might happen, but I dont want to spread myself too thin.

EDIT:It would be interesting if you could run your test again with jaikoz 3.7.0 to see how much improvment there is with the I/O threading changes.

#19

Would you like me to compare beta 4 with 3.7 or 3.6 to 3.7?

#20

Beta 4 to 3.7, so just run 3.7 and see if my changes have resulted in an improvement for you.