Tomorrow, I’ll adapt the autokong script, for the sake of making it run per folder, each folder been divided in 500gb parts.
Bandcamp support for SongKong
Sorry I don’t get why you can’t leave a tab on Web browser open?
But you normally run from cmd line anyway.
by the way, here :
songkong:server unable to recreate start pagenull
>>>>>/start.task
songkong:server unable to recreate start pagenull
I just did try to start songkong using the cli, it looks like I have to use specific options to run this or that task. I’ll dig it.
root@Murray:/mnt/cache/appdata/songkong/Prefs# docker run --rm --name songkong_bandcamp_manual -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -p songkong_deleteduplicates2.properties
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log
Unrecognised Options for SongKong
Options for SongKong are as follows:
Usage: songkong [options] file1 file2 file3 ...
Options:
-a
auto edit
Default: false
-b
remove specified fields with metagrater
Default: false
-c
match one album, must also specify release id using -o
musicbrainzOrDiscogsId=url
Default: false
-d
delete duplicates in specified files
Default: false
-e
bandcamp matcher for specified files
Default: false
-f
rename or move files
Default: false
-g
run in gui
Default: false
-h
show usage information
Default: false
-m
fix songs in specified files
Default: false
-n
import naim metadata files for specified files
Default: false
-o
overrideOption1=value1, overideOption2=value2
Default: []
-p
profile
-r
run in remote mode, access via browser (i.e http://localhost:4567)
Default: false
-s
create status report for specified files
Default: false
-u
undo any changes made to songs in specified files by SongKong
Default: false
-w
watch folder in specified folder
Default: false
root@Murray:~# docker run --rm --name songkong_bandcamp_manual -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -e /music/Music_dump/01-1800/
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log
Using Current Profile:Default:songkong_bandcamp.properties
Start Bandcamp Matcher
Songs Loaded 3506: Bandcamp Matched 0: Saved 0: Done 0
lets see. But for now it seems I cannot pass a profile anymore
Looking at first line it looks that you specified a profile but not a task. You always have to specify a task and you can optionally specify a profile as well.
I can tell you that my script that was working fine before the update now also do report these options errors. so it definitely looks like the syntax changed. here is what happens if I run it manually :
root@Murray:/mnt/cache/appdata/scripts# python3 Autokong_nightly_process.py
Starting processing for folder: /mnt/user/MURRAY/Music/Music_dump/06-2024/
Executing the musicbrainz command: docker run --rm --name songkong_musicbrainz_mnt-user-MURRAY-Music-Music_dump-06-2024 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -p songkong_musicbrainz.properties
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log
Usage: songkong [options] file1 file2 file3 ...
Options:
-a
auto edit
Default: false
-b
remove specified fields with metagrater
Default: false
-c
match one album, must also specify release id using -o
musicbrainzOrDiscogsId=url
Default: false
-d
delete duplicates in specified files
Default: false
-e
bandcamp matcher for specified files
Default: false
-f
rename or move files
Default: false
-g
run in gui
Default: false
-h
show usage information
-n
import naim metadata files for specified files
Default: false
-o
overrideOption1=value1, overideOption2=value2
Default: []
-p
profile
-r
run in remote mode, access via browser (i.e http://localhost:4567)
Default: false
-s
create status report for specified files
Default: false
-u
undo any changes made to songs in specified files by SongKong
Default: false
-w
watch folder in specified folder
Default: false
Sending Pushover notification: 🎵 SongKong Musicbrainz Summary for folder: /mnt/user/MURRAY/Music/Music_dump/06-2024/ 🎵
⏱ Total Time Taken: 0:00:09.912550 ⏱
📊 Overall Progress: 1/10 folders processed (10.00%). 9 folders remaining.
🔗 Report URL: http://192.168.3.2:4569
🔍 Estimated Time for Next Folder: -1 day, 23:59:59.954847
❗ Error Count: 0
📊 Efficiency: 0.00 songs/second
💾 Disk Usage: Total: 288057660344, Used: 250415157168, Free: 37626034456
Executing the bandcamp command: docker run --rm --name songkong_bandcamp_mnt-user-MURRAY-Music-Music_dump-06-2024 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -p songkong_bandcamp.properties
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log
auto edit
Default: false
-b
remove specified fields with metagrater
Default: false
-c
match one album, must also specify release id using -o
musicbrainzOrDiscogsId=url
Default: false
-d
delete duplicates in specified files
Default: false
-e
bandcamp matcher for specified files
Default: false
-f
rename or move files
Default: false
-g
run in gui
Default: false
-h
show usage information
Default: false
-m
fix songs in specified files
Default: false
-n
import naim metadata files for specified files
Default: false
-o
overrideOption1=value1, overideOption2=value2
Default: []
-p
profile
-r
run in remote mode, access via browser (i.e http://localhost:4567)
Default: false
-s
create status report for specified files
Default: false
-u
undo any changes made to songs in specified files by SongKong
Default: false
-w
watch folder in specified folder
Default: false
Sending Pushover notification: 🎵 SongKong Musicbrainz Summary for folder: /mnt/user/MURRAY/Music/Music_dump/06-2024/ 🎵
⏱ Total Time Taken: 0:00:09.912550 ⏱
📊 Overall Progress: 1/10 folders processed (10.00%). 9 folders remaining.
🔗 Report URL: http://192.168.3.2:4569
🔍 Estimated Time for Next Folder: -1 day, 23:59:59.954847
❗ Error Count: 0
📊 Efficiency: 0.00 songs/second
💾 Disk Usage: Total: 288057660344, Used: 250415157168, Free: 37626034456
Executing the bandcamp command: docker run --rm --name songkong_bandcamp_mnt-user-MURRAY-Music-Music_dump-06-2024 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -p songkong_bandcamp.properties
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log
Usage: songkong [options] file1 file2 file3 ...
Options:
-a
auto edit
Default: false
-b
remove specified fields with metagrater
Default: false
-c
match one album, must also specify release id using -o
musicbrainzOrDiscogsId=url
Default: false
-d
delete duplicates in specified files
Default: false
-e
bandcamp matcher for specified files
Default: false
-f
rename or move files
Default: false
-g
run in gui
Default: false
-h
show usage information
Default: false
-m
fix songs in specified files
Default: false
-n
import naim metadata files for specified files
Default: false
-o
overrideOption1=value1, overideOption2=value2
Default: []
-p
profile
-r
run in remote mode, access via browser (i.e http://localhost:4567)
Default: false
-s
create status report for specified files
Default: false
-u
undo any changes made to songs in specified files by SongKong
Default: false
-w
watch folder in specified folder
Default: false
Sending Pushover notification: 🎵 SongKong Bandcamp Summary for folder: /mnt/user/MURRAY/Music/Music_dump/06-2024/ 🎵
⏱ Total Time Taken: 0:00:05.271638 ⏱
📊 Overall Progress: 2/10 folders processed (20.00%). 8 folders remaining.
🔗 Report URL: http://192.168.3.2:4569
🔍 Estimated Time for Next Folder: 0:00:02.266810
❗ Error Count: 0
📊 Efficiency: 0.00 songs/second
💾 Disk Usage: Total: 288057660344, Used: 250415037808, Free: 37626153560
^CTraceback (most recent call last):
File "/mnt/cache/appdata/scripts/Autokong_nightly_process.py", line 277, in <module>
run_songkong_task(HOST_FOLDER, "songkong_bandcamp.properties", "bandcamp") # Bandcamp verification task
File "/mnt/cache/appdata/scripts/Autokong_nightly_process.py", line 216, in run_songkong_task
send_pushover_notification(summary_message)
File "/mnt/cache/appdata/scripts/Autokong_nightly_process.py", line 87, in send_pushover_notification
requests.post(url, data=data)
File "/usr/lib64/python3.9/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "/usr/lib64/python3.9/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib64/python3.9/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib64/python3.9/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/lib64/python3.9/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/usr/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 537, in _make_request
response = conn.getresponse()
File "/usr/lib64/python3.9/site-packages/urllib3/connection.py", line 466, in getresponse
httplib_response = super().getresponse()
File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse
response.begin()
File "/usr/lib64/python3.9/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.9/ssl.py", line 1242, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.9/ssl.py", line 1100, in read
return self._sslobj.read(len, buffer)
KeyboardInterrupt
FYI, there is no implication of the bandcamp task at all in this script for now, it is just the same script running just like it did for months before this update
See this thread you’ll see the default docker settings ran -r for remote mode but you had to modify to -m to run Fix Songs
Also see Tutorial: SongKong Command Line
I dont know the situation regarding your script but SongKong has always required a task to be passed to it not just a profile.
I’m underwater work wise, but I will try to find a spot to debug this, I’ll get back to you as soon it is done !
Thing is, I am running the exact same script as I was before. The files load, but are not been processed.
Not sure I already shared my nightly process script with you by the way:
import os
import subprocess
import re
from datetime import datetime, timedelta
import time
import threading
import shutil
try:
import requests
except ImportError:
import subprocess
subprocess.check_call(["pip3", "install", "requests"])
import requests
# Calculate the date of the previous day
yesterday = datetime.now() - timedelta(days=1)
day = yesterday.strftime("%d")
month_year = yesterday.strftime("%m-%Y")
day_month = yesterday.strftime("%d-%m")
# Configuration
HOST_FOLDER = f"/mnt/user/MURRAY/Music/Music_dump/{month_year}/"
DOCKER_FOLDER = f"/music/Music_dump/{month_year}/"
DOCKER_IMAGE_NAME = "songkong/songkong"
pushover_user_key = "u8pztbghz47d689h8nwsctga1jp7z1"
pushover_api_token = "ajnkq8s9f9ggwg5ooyq9zppxi2z2is"
NOTIFICATION_INTERVAL_MINUTES = 60
FULL_REPORTING = True # Change to True for full reporting
SEND_INTERMEDIATE_NOTIFICATIONS = False # Set to False to disable intermediate notifications
RUN_FOLDERS_SPLIT_BASH_SCRIPT = False # Change to False if you do not want to run the Bash script
SERVER_IP = "192.168.3.2" # Your usual SongKong server IP
SERVER_PORT = "4569" # Your SongKong port
# Delete the database before starting the processing
shutil.rmtree("/mnt/cache/appdata/songkong/Prefs/Database", ignore_errors=True)
# Run the Bash script before starting processing, if configured to do so
if RUN_FOLDERS_SPLIT_BASH_SCRIPT:
bash_script_path = "split_script.sh"
subprocess.run(["bash", bash_script_path], check=True)
# Global variable to store past processing times and processed folders
past_processing_times = []
processed_folders = []
# Add the new global variable here
processing_complete = False
def get_server_url():
"""
Returns the server URL based on the SERVER_IP and SERVER_PORT configuration.
"""
if SERVER_PORT:
return f"http://{SERVER_IP}:{SERVER_PORT}"
return f"http://{SERVER_IP}"
def initialize_processed_folders():
global processed_folders
if os.path.exists("songkong_log.txt"):
with open("songkong_log.txt", "r") as log_file:
processed_folders = [line.strip() for line in log_file.readlines()]
def calculate_eta(start_time):
if not past_processing_times:
return "Unknown"
average_time = sum(past_processing_times) / len(past_processing_times)
remaining_time = average_time - (datetime.now() - start_time).total_seconds()
return str(timedelta(seconds=remaining_time))
def get_progress_summary():
total_folders = len(find_all_subfolders())
processed_count = len(processed_folders)
remaining_count = total_folders - processed_count
progress_percentage = (processed_count / total_folders) * 100
return f"📊 Overall Progress: {processed_count}/{total_folders} folders processed ({progress_percentage:.2f}%). {remaining_count} folders remaining."
def send_pushover_notification(message):
log_action(f"Sending Pushover notification: {message}")
if pushover_user_key and pushover_api_token:
url = "https://api.pushover.net/1/messages.json"
data = {
"token": pushover_api_token,
"user": pushover_user_key,
"message": message
}
requests.post(url, data=data)
def log_action(action):
print(action)
with open("action_log.txt", "a") as log_file:
log_file.write(f"{datetime.now()} - {action}\n")
def was_folder_processed(folder):
print(f"Checking if folder {folder} was processed...")
if not os.path.exists("songkong_log.txt"):
return False
with open("songkong_log.txt", "r") as log_file:
logs = log_file.read()
print(f"Complete Log Contents:\n{logs}\n") # Add this line for debugging
if folder in logs:
print(f"Folder {folder} was found in the log.")
return True
return False
def move_logs_to_backup(relative_path, end_time):
log_folder = "/mnt/cache/appdata/songkong/Logs/"
backup_folder = "/mnt/cache/appdata/songkong/Logs_backup/"
date_time_suffix = end_time.strftime("%Y%m%d_%H%M%S")
for log_file in os.listdir(log_folder):
new_log_file = f"{relative_path}_{date_time_suffix}_{log_file}"
shutil.move(os.path.join(log_folder, log_file), os.path.join(backup_folder, new_log_file))
def extract_value(pattern, text):
match = re.search(pattern, text)
if match:
return int(match.group(1).replace(',', ''))
return 0
def send_periodic_notification(process_output, start_time):
global processing_complete
if processing_complete:
return
if process_output:
last_line = process_output[-1]
loaded = extract_value(r'Songs loaded ([\d,]+)', last_line)
fingerprinted = extract_value(r'Fingerprinted ([\d,]+)', last_line)
musicbrainz = extract_value(r'MusicBrainz ([\d,]+)', last_line)
discogs = extract_value(r'Discogs ([\d,]+)', last_line)
saved = extract_value(r'Saved ([\d,]+)', last_line)
if not (loaded or fingerprinted or musicbrainz or discogs or saved):
return
fingerprinted_percent = (fingerprinted / loaded) * 100 if loaded else 0
musicbrainz_percent = (musicbrainz / loaded) * 100 if loaded else 0
discogs_percent = (discogs / loaded) * 100 if loaded else 0
saved_percent = (saved / loaded) * 100 if loaded else 0
elapsed_time = str(datetime.now() - start_time).split('.')[0]
notification_message = f"🎵 SongKong Processing Summary for folder: {current_folder} 🎵\n\n"
notification_message += f"Songs loaded: {loaded} (100.00%)\n"
notification_message += f"Songs fingerprinted: {fingerprinted} ({fingerprinted_percent:.2f}%)\n"
notification_message += f"Songs matched to MusicBrainz: {musicbrainz} ({musicbrainz_percent:.2f}%)\n"
notification_message += f"Songs matched to Discogs: {discogs} ({discogs_percent:.2f}%)\n"
notification_message += f"Songs saved: {saved} ({saved_percent:.2f}%)\n"
notification_message += f"⏱ Total Time Taken: {elapsed_time} ⏱\n\n"
notification_message += get_progress_summary()
send_pushover_notification(notification_message)
threading.Timer(NOTIFICATION_INTERVAL_MINUTES * 60, send_periodic_notification, args=[process_output, start_time]).start()
def run_songkong_task(relative_path, task_properties, task_name):
global current_folder, processing_complete
processing_complete = False
start_time = datetime.now()
current_folder = relative_path.replace('/', '-').strip('-')
container_name = f"songkong_{task_name}_{current_folder}"
cmd = f'docker run --rm --name {container_name} -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music {DOCKER_IMAGE_NAME} -p {task_properties}'
retry_count = 0
MAX_RETRIES = 2
database_corrupt_error = "Database /songkong/Prefs/Database appears corrupt"
while retry_count <= MAX_RETRIES:
log_action(f"Executing the {task_name} command: {cmd}")
process_output = []
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
error_detected = False
for line in iter(process.stdout.readline, ''):
print(line.strip())
process_output.append(line.strip())
if database_corrupt_error in line:
error_detected = True
process.terminate()
log_action(f"Error detected: {database_corrupt_error}. Deleting database and retrying...")
shutil.rmtree("/mnt/cache/appdata/songkong/Prefs/Database", ignore_errors=True)
retry_count += 1
break
if not error_detected:
break
end_time = datetime.now()
time_taken = (end_time - start_time).total_seconds()
past_processing_times.append(time_taken)
processed_folders.append(relative_path)
summary_message = f"🎵 SongKong {task_name.capitalize()} Summary for folder: {relative_path} 🎵\n\n"
total_songs = 0
error_count = 0
for line in process_output:
if "Songs loaded:" in line:
total_songs = int(re.search(r'\d+', line).group())
if "Error" in line:
error_count += 1
if any(keyword in line for keyword in ["Songs loaded:", "Songs fingerprinted:", "Songs matched to MusicBrainz", "Songs matched to Discogs", "Songs matched to Bandcamp", "Songs saved", "Completed", "Errors and Warnings", "Reports"]):
value = int(re.search(r'\d+', line).group())
percent = (value / total_songs) * 100 if total_songs else 0
summary_message += f"{line} ({percent:.2f}%)\n"
summary_message += f"\n⏱ Total Time Taken: {str(timedelta(seconds=time_taken))} ⏱\n\n"
summary_message += get_progress_summary()
if FULL_REPORTING:
summary_message += f"\n🔗 Report URL: {get_server_url()}\n"
summary_message += f"\n🔍 Estimated Time for Next Folder: {calculate_eta(start_time)}"
summary_message += f"\n❗ Error Count: {error_count}"
summary_message += f"\n📊 Efficiency: {total_songs/time_taken:.2f} songs/second"
total_space, used_space, free_space = os.popen('df /mnt/user/MURRAY/Music').read().split("\n")[1].split()[1:4]
summary_message += f"\n💾 Disk Usage: Total: {total_space}, Used: {used_space}, Free: {free_space}"
summary_message = summary_message.replace("/songkong/Reports", get_server_url())
processing_complete = True
send_pushover_notification(summary_message)
move_logs_to_backup(relative_path, end_time)
def extract_delete_duplicates_summary(output):
summary_pattern = re.compile(r"(Processing:\d+|Songs loaded:\d+|Duplicate groups found :\d+|Duplicate songs deleted:\d+|Errors and Warnings:\d+)")
summary = "\n".join(summary_pattern.findall(output))
return summary
def send_delete_duplicates_notification(relative_path, summary):
notification_message = f"🎵 SongKong Delete Duplicates Summary for folder: {relative_path} 🎵\n\n"
notification_message += summary
send_pushover_notification(notification_message)
def run_songkong_delete_duplicates(relative_path):
current_folder = relative_path.replace('/', '-').strip('-') # Formatting for container name
container_name = f"songkong_delete_{current_folder}"
cmd = f'docker run --rm --name {container_name} -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music {DOCKER_IMAGE_NAME} -d "{DOCKER_FOLDER}" -p songkong_deleteduplicates2.properties'
time.sleep(60)
log_action(f"Executing the delete duplicates command: {cmd}")
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
output, _ = process.communicate()
summary = extract_delete_duplicates_summary(output)
send_delete_duplicates_notification(relative_path, summary)
def extract_rename_summary(output):
summary_pattern = re.compile(r"Songs Report is: (.+)|Songs loaded:(\d+)|Songs renamed:(\d+)|Completed:(\d+)|Errors and Warnings:(\d+)|Report Creation:(\d+)")
summary = "\n".join(summary_pattern.findall(output))
return summary
def run_songkong_rename(relative_path):
current_folder = relative_path.replace('/', '-').strip('-') # Formatting for container name
container_name = f"songkong_rename_{current_folder}"
cmd = f'docker run --rm --name {container_name} -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music {DOCKER_IMAGE_NAME} -f "{DOCKER_FOLDER}"'
time.sleep(60)
log_action(f"Executing the rename command: {cmd}")
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
output, _ = process.communicate()
rename_summary = extract_rename_summary(output)
send_pushover_notification(rename_summary)
def find_all_subfolders():
return [folder for folder in os.listdir(HOST_FOLDER) if os.path.isdir(os.path.join(HOST_FOLDER, folder))]
def is_folder_empty(folder_path):
return not os.listdir(folder_path)
def delete_folder(folder_path):
os.rmdir(folder_path)
log_action(f"Deleted empty folder: {folder_path}")
if __name__ == "__main__":
if RUN_FOLDERS_SPLIT_BASH_SCRIPT:
bash_script_path = "split_script.sh"
subprocess.run(["bash", bash_script_path], check=True)
print(f"\n\nStarting processing for folder: {HOST_FOLDER}")
run_songkong_task(HOST_FOLDER, "songkong_musicbrainz.properties", "musicbrainz") # Correction task
run_songkong_task(HOST_FOLDER, "songkong_bandcamp.properties", "bandcamp") # Bandcamp verification task
run_songkong_delete_duplicates(HOST_FOLDER) # Duplicate deletion task
run_songkong_rename(HOST_FOLDER) # Renaming task
send_pushover_notification("🎉 Script has finished processing the folder. 🎉")
One thing ain’t clear :
fix songs (-m) is the fix songs task, but how do we call the bandcamp task using the cli ? I can’t see any parameter allowing to start this specific task type.
I have been running for a few days and it’s stalled about 80k out of 90k songs. Matched 6k, which is great. Think maybe the API rate limited me.
Not sure how I exit w/o undoing all the matches.
Thoughts?
Folders are saved as they are matched, so stopping will not undo any changes made so there is no issue.
-e runs the bandcamp task
Tried Bandcamp match on https://phillipgolub.bandcamp.com/album/abiding-memory-2 (FLAC digital download). No additional information (performers, genre, …) was added to any tracks. Using the Mac app. Is there a way to get more information on detailed matching successes or failures? I can also try on Linux, if that would give more information. Thank you.
Please run Create Support Files so I get your report and logs then I can look into it.
Uploaded the support files. First time I did it, I hope it worked.
Looking at the logs it seems that you have 11 songs, but the bandcamp album only has 10 tracks so a match cannot be made, is that correct?
21/06/2024 21.19.31:PDT:MetadataGatherer:groupByMetadataAndFileDataForSongsWithoutMetadata:WARNING: :songs:11:artists:1:trackArtists:1:releases:1:simplereleases:1:songswithoutartists: 0:songswithoutrelease: 0:songsWithNoReleaseOrArtist:0
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateReleases:WARNING: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:Candidate Releases names:2
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateReleases:SEVERE: G2:Foldername:Abiding Memory:isPartOfMultiDisc:false::abiding memory
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateReleases:SEVERE: G2:Foldername:Abiding Memory:isPartOfMultiDisc:false::Abiding Memory
21/06/2024 21.19.31:PDT:MetadataGatherer:groupByMetadataAndFileDataForSongsWithoutMetadata:WARNING: :songs:11:artists:1:trackArtists:1:releases:1:simplereleases:1:songswithoutartists: 0:songswithoutrelease: 0:songsWithNoReleaseOrArtist:0
21/06/2024 21.19.31:PDT:FindReleaseCandidateNames:queryForCandidateArtists:WARNING: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:Candidate Artist names:1
21/06/2024 21.19.36:PDT:Matrix:checkResultsAndCreateMapping:SEVERE: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:null:Best match gives no possible match for row 10: so ignoring this release
21/06/2024 21.19.42:PDT:Matrix:checkResultsAndCreateMapping:SEVERE: G2:/Users/pereira/Downloads/Abiding Memory:false:false:false:false:false:null:Best match gives no possible match for row 10: so ignoring this releaseI am fixing my script, and you were right— it no longer handles the task types correctly. I will update you with the fixed script as soon as possible.
In the meantime, I’m running a Bandcamp task (the first one, as I hadn’t had time to check this earlier):
root@Murray:/mnt/cache/appdata/songkong/Prefs# docker run --rm --name songkong_bandcamp_-mnt-user-MURRAY-Music-Music_dump-06-2024 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -e /music/Music_dump/06-2024/ -p songkong_bandcamp.properties
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log
Using Supplied Profile: Default: songkong_bandcamp.properties
Start Bandcamp Matcher
Songs Loaded 983: Bandcamp Matched 535: Saved 300: Done 813
It’s ongoing, and this folder has approximately 5,000 files to match. It’s been running for 2 hours and has loaded 983 files so far.
I searched for Bandcamp API usage limits but couldn’t find any specific information in their documentation.
Question: Is the slowness of the Bandcamp task due to hitting the Bandcamp API rate limits? What are the limits?
22/06/2024 07.46.55:GMT:CONFIG: Bandcamp Http:0:https://duststoredigital.com/album/other-like-me
22/06/2024 07.46.56:GMT:SEVERE: Bandcamp HttpError:0:https://duststoredigital.com/album/other-like-me:429
22/06/2024 07.46.58:GMT:CONFIG: Bandcamp Http:1:https://duststoredigital.com/album/other-like-me
22/06/2024 07.46.59:GMT:CONFIG: Bandcamp Http:0:https://oscarmulero-oficial.bandcamp.com/album/sesiones-mercurio-1-ep-polegroup-63
22/06/2024 07.47.00:GMT:CONFIG: Bandcamp Http:0:https://enniomorriconemusic.bandcamp.com/album/gli-intoccabili-machine-gun-mccain-original-motion-picture-soundtrack
22/06/2024 07.47.01:GMT:CONFIG: Bandcamp Http:0:https://duststoredigital.com/album/the-white-ep
22/06/2024 07.47.01:GMT:SEVERE: Bandcamp HttpError:0:https://duststoredigital.com/album/the-white-ep:429
22/06/2024 07.47.03:GMT:CONFIG: Bandcamp Http:1:https://duststoredigital.com/album/the-white-ep
...
It looks like we are hitting the API limit quite quickly. Look at all these HTTP 429 (Too Many Requests) errors. Given the initial speed of the match and the fact that we are already hitting the API limits after such a small batch of files has been matched, I don’t expect this task to be useful for people with large libraries. Damn… Bandcamp…
Is there no way to have a locally hosted bandcamp DB like you do for Musicbrainz ?
I couldn’t find official limits but testing seems to indicate it is one query per second, if we hit the limit we wait a couple of seconds and try again, if that fails we wait 4 seconds and try again, we repeat this doubling the delay each time until we get to ten retries and then give up. Looking at the logs it only had to retry once andf then i t was okay, so just because you have large libraries it should not fail it should continue to run okay with just a few retries.
But yes it is slower than the MusicBrainz/Discogs task, that is one reason we put it as separate task, and also provided the option to skip albums already matched to MusicBrainz/Discogs.
Well yes there is and I have thought about this.
It would involve using my api to lookup every artist/album in bandcamp and then building a search index based on the data. So it would take some time to get the data to build the index and take some effort to then write the code to create the index. And then there is the issue of keeping it topped up with new data.
I may do it, but so far haven’t seen that much interest in the new Bandcamp task so far so it is not top priority.
ok the approx. 5K files task finalized. For you to have a good overview of ho it ran, I just sent the report.
it seems to work, but it indeed is slow as hell.
good point is that it ain’t using any resources compared to musicbrainz match task, so I guess we could leave it running in the background without sucking too much power.
Thanks, looking at the logs doesnt appear to be hitting the rate limit to much so I think I have it about right.
There a few errors to investigate.
Unfortunately there is a minor bug, I forgot to add bandcamp reports to the list of items to be sent over in support files so this means the actual bandcamp report is not available to me. Would be useful if you could send the report over.
Yes because there is no fingerprinting, and only one query being sent per second is the limiting factor meaning there is not that much for the computer to do.