I’ve implemented a 10 retry mechanism with increasing delays between each retry and a few other adjustments and tested it against 10,000 files without it failing so this should solve the problem, will get this release rolled out within two weeks.
If anyone is still running into problems I will add the option to retry indefinitently.
Maybe usefull is check http header
X-Rate-Limited: 600.5 600.0 10, where 600.5 means actual request, 600 is limit, 10 is 10 second. If you reach 600+ requests per 10 seconds youll be temporatily “banned” you can test it with multiple opened links from firefox for example.
– Edit.
In firefox when you reach limit ( 10 for 10 secs ) youll get
Your requests are exceeding the allowable rate limit (11.8, limit is 11.0 per 10 seconds)
Please slow down then try again.
Jaikoz has inbuilt throttler which prevents more than one query a second, and Im sure its not brokem.
And is some testcase how to track this error ? What you want track paul ? some headers or jaikoz debug files ?
Fix now available in Jaikoz 2.8.2
Paul,
sorry, no joy here. Issue still exists with BM Server busy.
Here is the spacing between requests that failed:
Feb 10, 2009 12:15:24 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 1
Feb 10, 2009 12:15:25 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 2
Feb 10, 2009 12:15:27 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 3
Feb 10, 2009 12:15:30 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 4
Feb 10, 2009 12:15:34 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 5
Feb 10, 2009 12:15:39 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 6
Feb 10, 2009 12:15:45 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 7
Feb 10, 2009 12:15:52 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 8
Feb 10, 2009 12:16:00 AM: WARNING: MiniProcess 0 having problems connecting to MusicBrainz Service, may be down or just busy, retry attempt 9
I thought there were supposed to have a more spaced out starting interval. This starts at 1 second and then increments by one (your new algo I guess). But sometimes when it failed in the past requests were spaced by 5 sec or 3 sec or 7 sec (has always done this). Is this design intent to make initial retries a random starting interval?
I still think the issue is environmental or an uninitialized (or corrupted) variable. It can fail consitantly on a specific file in one period of time but allowing it to sit for say an hour and then retrying will often have that file succeed. Other files around it have no problem at either point in time.
Is there a simple way to trace the exact request and response between Jaikoz and MB? This would at least allow us to isolate whether it is local or an MB interaction.
Christian
Yes previously there was only a 1 second delay between each retry.
If you set logging to -l7 it will log all calls made to Musicbrainz in the debug log, perhaps you could try this and send me the logs with Action/Create Support FIles
I also need to take this up futher with Musicbrainz, because we rarely used to hit this issue.
[quote=pschonmann]Maybe usefull is check http header
X-Rate-Limited: 600.5 600.0 10, where 600.5 means actual request, 600 is limit, 10 is 10 second. If you reach 600+ requests per 10 seconds youll be temporatily “banned” you can test it with multiple opened links from firefox for example.
[/quote]
Sounds like you could be right this information is now logged in my dev server and Im taking to Musicbrainz about this.
I think I figured it out!
A number of users are going through an http proxy and cache either in their own environment or their ISP’s environment.
When MB fails with 503 (service unavailable) the proxy caches this as a “negative hit” and gives it a Time To Live (TTL) before it will pass a new request. In my case I use squid which defaults to a TTL of 5 minutes.
So once you hit a 503, the cache will continue to fail your requests within the TTL window.
You should be able to override this in the HTTP request within Jaikoz as in your case your subsequent requests are in fact meant to bypass any caches.
I verified this by waiting for a 503 with retries then disabled the proxy while it was retrying which then allowed Jaikoz to proceed properly.
If I disable the proxy all together and run Jaikoz I get the occasional MB busy message but it works fine on the first retry.
Given many people won’t have control over their cache and proxy, it is best to have Jaikoz force the proxy not to cache.
If you wish, I can test a fix as I do have control on my cache.
Christian
I set the negative_ttl=0 on squid.
Just ran >5000 files and it completely successfully.
MB returned 503 a number of times and the most retried I needed was 2.
Good Work Christian, I will try and set the relevent header in Jaikoz to ignore caching and hopefully you can retest for me.
Ive added the following for all url connections
uc.setRequestProperty("Cache-Control", "no-store, no-cache");
uc.setRequestProperty("Pragma","no-cache");
[quote=paultaylor]Ive added the following for all url connections
[/quote]
ok. if you want me to test it ahead of release, feel free to send a patch via email. You should have my email address.
Just to confirm 2.8.4 did the trick. Confirmed on the most problematic sets I had with over 7500 files.
I think there is still something wrong with MB as it kicks up quite a few 503s. One for about every 250 files.
It would be good to see if it resolved other people’s problems too.
Christian
As pleased as I am that this has been addressed, I still will not be satisfied until the final “Unable to contact muscibrainz” failure message, after all automated retries, has a manual retry button as well as the okay button.
With 125,000 tracks, there is sufficient time for things like transient service provider problems and WAP freezes, both of which I experienced over my last two attempts to query for my entire library. I was left twice with, after many hours of successes and a few minutes of problems, the unfortunate need to start over again. That seems unnecessary and silly.
Thanks
OK I see your point,
And I assume if muicbrainz was being run as part of the autocorrecter you expect autocorrecter to halt at this stage waiting for your decision or carry on to the next task.
And would you want this behaviour for other processes that could time out such as UpdatefromDiscogs or Update Lyrics
Just so this is not dropped before the next release pending a response from me:
Paul, that sounds reasonable.