Bandcamp support for SongKong

[earnmyturns]

Uploaded the support files. First time I did it, I hope it worked.

[paultaylor]

Looking at the logs it seems that you have 11 songs, but the bandcamp album only has 10 tracks so a match cannot be made, is that correct?

21/06/2024 21.19.31:PDT:MetadataGatherer:groupByMetadataAndFileDataForSongsWithoutMetadata:WARNING: :songs:11:artists:1:trackArtists:1:releases:1:simplereleases:1:songswithoutartists: 0:songswithoutrelease: 0:songsWithNoReleaseOrArtist:0
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateReleases:WARNING: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:Candidate Releases names:2
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateReleases:SEVERE: G2:Foldername:Abiding Memory:isPartOfMultiDisc:false::abiding memory
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateReleases:SEVERE: G2:Foldername:Abiding Memory:isPartOfMultiDisc:false::Abiding Memory
21/06/2024 21.19.31:PDT:MetadataGatherer:groupByMetadataAndFileDataForSongsWithoutMetadata:WARNING: :songs:11:artists:1:trackArtists:1:releases:1:simplereleases:1:songswithoutartists: 0:songswithoutrelease: 0:songsWithNoReleaseOrArtist:0
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateArtists:WARNING: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:Candidate Artist names:1
21/06/2024 21.19.36:PDT:Matrix:checkResultsAndCreateMapping:SEVERE: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:null:Best match gives no possible match for row 10: so ignoring this release
21/06/2024 21.19.42:PDT:Matrix:checkResultsAndCreateMapping:SEVERE: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:null:Best match gives no possible match for row 10: so ignoring this release

[meaning]

I am fixing my script, and you were right— it no longer handles the task types correctly. I will update you with the fixed script as soon as possible.

In the meantime, I’m running a Bandcamp task (the first one, as I hadn’t had time to check this earlier):

root@Murray:/mnt/cache/appdata/songkong/Prefs# docker run --rm --name songkong_bandcamp_-mnt-user-MURRAY-Music-Music_dump-06-2024 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -e /music/Music_dump/06-2024/ -p songkong_bandcamp.properties
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log
Using Supplied Profile: Default: songkong_bandcamp.properties
Start Bandcamp Matcher
Songs Loaded 983: Bandcamp Matched 535: Saved 300: Done 813

It’s ongoing, and this folder has approximately 5,000 files to match. It’s been running for 2 hours and has loaded 983 files so far.

I searched for Bandcamp API usage limits but couldn’t find any specific information in their documentation.

Question: Is the slowness of the Bandcamp task due to hitting the Bandcamp API rate limits? What are the limits?

22/06/2024 07.46.55:GMT:CONFIG: Bandcamp Http:0:https://duststoredigital.com/album/other-like-me
22/06/2024 07.46.56:GMT:SEVERE: Bandcamp HttpError:0:https://duststoredigital.com/album/other-like-me:429
22/06/2024 07.46.58:GMT:CONFIG: Bandcamp Http:1:https://duststoredigital.com/album/other-like-me
22/06/2024 07.46.59:GMT:CONFIG: Bandcamp Http:0:https://oscarmulero-oficial.bandcamp.com/album/sesiones-mercurio-1-ep-polegroup-63
22/06/2024 07.47.00:GMT:CONFIG: Bandcamp Http:0:https://enniomorriconemusic.bandcamp.com/album/gli-intoccabili-machine-gun-mccain-original-motion-picture-soundtrack
22/06/2024 07.47.01:GMT:CONFIG: Bandcamp Http:0:https://duststoredigital.com/album/the-white-ep
22/06/2024 07.47.01:GMT:SEVERE: Bandcamp HttpError:0:https://duststoredigital.com/album/the-white-ep:429
22/06/2024 07.47.03:GMT:CONFIG: Bandcamp Http:1:https://duststoredigital.com/album/the-white-ep
...

It looks like we are hitting the API limit quite quickly. Look at all these HTTP 429 (Too Many Requests) errors. Given the initial speed of the match and the fact that we are already hitting the API limits after such a small batch of files has been matched, I don’t expect this task to be useful for people with large libraries. Damn… Bandcamp…

Is there no way to have a locally hosted bandcamp DB like you do for Musicbrainz ?

[paultaylor]

@meaning

It looks like we are hitting the API limit quite quickly. Look at all these HTTP 429 (Too Many Requests) errors. Given the initial speed of the match and the fact that we are already hitting the API limits after such a small batch of files has been matched, I don’t expect this task to be useful for people with large libraries. Damn… Bandcamp…

I couldn’t find official limits but testing seems to indicate it is one query per second, if we hit the limit we wait a couple of seconds and try again, if that fails we wait 4 seconds and try again, we repeat this doubling the delay each time until we get to ten retries and then give up. Looking at the logs it only had to retry once andf then i t was okay, so just because you have large libraries it should not fail it should continue to run okay with just a few retries.

But yes it is slower than the MusicBrainz/Discogs task, that is one reason we put it as separate task, and also provided the option to skip albums already matched to MusicBrainz/Discogs.

@meaning

Is there no way to have a locally hosted bandcamp DB like you do for Musicbrainz ?

Well yes there is and I have thought about this.

It would involve using my api to lookup every artist/album in bandcamp and then building a search index based on the data. So it would take some time to get the data to build the index and take some effort to then write the code to create the index. And then there is the issue of keeping it topped up with new data.

I may do it, but so far haven’t seen that much interest in the new Bandcamp task so far so it is not top priority.

[meaning]

ok the approx. 5K files task finalized. For you to have a good overview of ho it ran, I just sent the report.

it seems to work, but it indeed is slow as hell.

good point is that it ain’t using any resources compared to musicbrainz match task, so I guess we could leave it running in the background without sucking too much power.

[paultaylor]

Thanks, looking at the logs doesnt appear to be hitting the rate limit to much so I think I have it about right.

There a few errors to investigate.

Unfortunately there is a minor bug, I forgot to add bandcamp reports to the list of items to be sent over in support files so this means the actual bandcamp report is not available to me. Would be useful if you could send the report over.

@meaning

good point is that it ain’t using any resources compared to musicbrainz match task, so I guess we could leave it running in the background without sucking too much power.

Yes because there is no fingerprinting, and only one query being sent per second is the limiting factor meaning there is not that much for the computer to do.

[earnmyturns]

Thanks, good catch. The digital download has an extra track, an alternate take of the title track, relative to the CD and what’s on the album page on Bandcamp. Normal “Fix Songs” does not require full album match if that option is not checked, would that option make sense for Bandcamp match? This kind of situation where digital downloads have bonus tracks relative to the CD is not common in Bandcamp but I’ve seen it before.

[earnmyturns]

For album https://andymilnepiano.bandcamp.com/album/time-will-tell, none of the performer data is extracted and added to the album’s metadata. Creating this post to include in support file upload.

[paultaylor]

Hi, there is a small bug with Create Support Files I forgot to add code to add Bandcamp reports to support files when I added task, so I’m not getting the report only your logs.

Perhaps you could post a screenshot showing what is being added to the Performer and Credits fields.

[earnmyturns]

Does this help?

[paultaylor]

It hasn’t actually matched to the album, there is no Bandcamp Release Url field, I’ll have a look at the log file tommorrow.

But because the Bandcamp search is quite limited the matching we can do is much more limited than what we do for MusicBrainz and Discogs matching so it us not going to work 100%.

You can look yourself in the logs folder, songkong-user0-0.log summarizes what SongKong is doing.

[earnmyturns]

Looked at the log file, I think I found the problem. Album artist is “Andy Milne and Unison” so SongKong searches Bandcamp for “andy milne unison” which only retrieves a previous album by the same artists (I checked on Bandcamp). The “Time Will Tell” album only shows up when searching “andy milne” even though the album page itself refers to “Andy Milne & Unison”. Looks like a bug in Bandcamp’s search function. Thank you for your help!

[paultaylor]

Good work, if you manually enter the Bandcamp Artist Url and Bandcamp Release Url then you should be able to rerun Bandcamp task against it and it should update the metadata from the album.

[paultaylor]

Normal “Fix Songs” does not require full album match if that option is not checked, would that option make sense for Bandcamp match?

With defaults FixSongs works the other way only, you don’t need to match all songs on an album, but you do need to match all the songs in the folder to one album.

You can disable this option to allow songs to matched individually but not usually recommended. It would not be helpful for Bandcamp matching with limited api they offer it would just increase the chances of matching to the wrong album.

[earnmyturns]

@paultaylor

if you manually enter the Bandcamp Artist Url and Bandcamp Release Url

Looked around at the documentation and forum posts, I must have missed how to do this in SongKong. More generally, I see how to edit tag values, but not how to add tags.

[paultaylor]

Okay you can edit fields using Manual Edit task but I forgot to add the Bandcamp Artist Url and Bandcamp Release url fields to this, have added for next release, but in meantime you would have to use a different tag editor to add them in.

We dont have a feature to add arbitary fields, we focus on providing a large set of fields from the start that can mostly be populated automatically, we think this is better than letting users create their own fields because this usually leads to compatability or consistency problems.

[meaning]

🎵 SongKong Bandcamp Summary for folder: /mnt/user/MURRAY/Music/Music_dump/01-1823 🎵

Report Created:http://192.168.3.2/BandcampReport00099/BandcampReport00099.html (0.00%)
Songs loaded:8562 (100.00%)
Songs matched to Bandcamp:2793 (32.62%)
Songs saved:1643 (19.19%)
Completed:8562 (100.00%)
Errors and Warnings:0 (0.00%)

⏱ Total Time Taken: 3:37:54.007960 ⏱

📊 Overall Progress: 48/149 folders processed (32.21%). 101 folders remaining.
🔗 Report URL: http://192.168.3.2

🔍 Estimated Time for Next Folder: -1 day, 23:40:39.621704
❗ Error Count: 1
📊 Efficiency: 0.65 songs/second
💾 Disk Usage: Total: 288057660344, Used: 258489595820, Free: 29553016284

32%, not bad at all ! <3

[paultaylor]

@paultaylor

Okay you can edit fields using Manual Edit task but I forgot to add the Bandcamp Artist Url and Bandcamp Release url fields to this, have added for next release, but in meantime you would have to use a different tag editor to add them in.

Fixed in

SongKong 10.8 Parade released 10th July 2024

We have a new release of SongKong with new Fix Charset Encoding task Fix Charset Encoding When metadata is saved to our music files can be encoded according to a charset, historically different charsets have been designed for encoding particular languages, for example the ISO-8859-1 charset can encode English and some Western European Languages but is no use for different scripts such as used in Japanese, Russian (e.g Cyrillic) and Chinese. These days we usually use UTF-8 because this can sto…