SongKong and Jaikoz Music Tagger Community Forum

meaning · 19 September 2023 14:42

Hi again,

I forgot to mention the my script already backups the logs crated by songkong at the end of each folder it processes :

def move_logs_to_backup(relative_path, end_time):
    log_folder = "/mnt/cache/appdata/songkong/Logs/"
    backup_folder = "/mnt/cache/appdata/songkong/Logs_backup/"
    date_time_suffix = end_time.strftime("%Y%m%d_%H%M%S")
    for log_file in os.listdir(log_folder):
        new_log_file = f"{relative_path}_{date_time_suffix}_{log_file}"
        os.rename(os.path.join(log_folder, log_file), os.path.join(backup_folder, new_log_file))

this way, only the "running logs will be visible in the Logs folder, and the older ones are moved to the “Logs_backup” folder and dated.

Regarding the DB, if it is only needed during the process of the task(s), I’ll make sure to delete it manually ach time a monthly folder process ended.this way, the database will exist during the process of the folder, which goes like explained above, and a new, fresh db will be created each time anew folder gets processed.

And no, I do like the reports Paul, but they take a LOT of space due to my library size. So, if they are not really needed, I might want to get rid of all the reports, except the latest XX for each task.

Regarding the rename task, nope, it is not starting using the cli and I’ll have to wait for you to fix the code. But as it simply reports an error, and then goes to the next folder, it’s safe to run the script with this line uncommented :

 run_songkong_rename(target_folder)  # Rename task

by the way paul, I’ve noticed a little bug in the end of fix task report of songkong (using the cli) :

Songs loaded 96,512:Fingerprinted 74,107: MusicBrainz 68,111: Discogs 73,807: Saved 94,281
Songs loaded 96,512:Fingerprinted 74,107: MusicBrainz 68,111: Discogs 73,807: Saved 94,281
Songs loaded 96,512:Fingerprinted 74,107: MusicBrainz 68,111: Discogs 73,807: Saved 94,281
Songs loaded 96,512:Fingerprinted 74,107: MusicBrainz 68,111: Discogs 73,807: Saved 94,281
Songs Report is:/songkong/Reports/FixSongsReport00179/FixSongsReport00179.html
Processing:0
Songs loaded:10297
Songs fingerprinted:74107
Songs ignored because already matched:0
Songs matched to MusicBrainz release:7011
Songs matched to MusicBrainz song only:15261
Songs matched to AcoustId release:14878
Songs matched to Acoustid song only:15346
Songs matched to Discogs release:7886
Songs matched with artwork:75258
Songs matched to AcousticBrainz:35061
Songs saved (if not preview):94281
Completed:96495
Errors and Warnings:311
Reports:66
Sending Pushover notification: 🎵 SongKong Processing Summary for folder: 01-1901 🎵

Songs Report is:/songkong/Reports/FixSongsReport00179/FixSongsReport00179.html (0.00%)
Songs loaded:10297 (100.00%)
Songs fingerprinted:74107 (719.70%)
Songs matched to MusicBrainz release:7011 (68.09%)
Songs matched to MusicBrainz song only:15261 (148.21%)
Songs matched to Discogs release:7886 (76.59%)
Songs saved (if not preview):94281 (915.62%)
Completed:96495 (937.12%)
Errors and Warnings:311 (3.02%)
Reports:66 (0.64%)

⏱ Total Time Taken: 10:24:46.132377 ⏱

📊 Overall Progress: 10/164 folders processed (6.10%). 154 folders remaining.
🔍 Estimated Time for Next Folder: -1 day, 15:49:57.731217
❗ Error Count: 1
📊 Efficiency: 0.27 songs/second
💾 Disk Usage: Total: 183574745720, Used: 145218838340, Free: 38355907380

Feel free to create a separate forum topic for that one, but look at the songs loaded during the process : 96,512, while the final report announces 10297. This is obviously totally messing with the percentagesthat gets calculated at the end of the run.

I’ve seen the same exact behavious again and again when running fix task using cli. Not a BIG issue, as in the end, files are been processed anyway, but worth mentioning as it messes with the final report.

As you can see, it is also incorrectly reported once the “remove duplicates” task starts :

Autokong
🎵 SongKong Delete Duplicates Summary for folder: 01-1901 🎵

Processing:0
Songs loaded:10297
Duplicate groups found :4437
Duplicate songs deleted:4437
Errors and Warnings:0

Actually, if I manually go to the UI, here is what I can see :

Loaded 96512 songs for duplicate checking

4437 Duplicates found and moved in 41 minutes 5 seconds

paultaylor · 19 September 2023 14:42

Okay I can replicate the issue, (not as easy to solve as you may think though)

meaning · 19 September 2023 15:21

and, as I am currious, why is the reported amount different ?

paultaylor · 19 September 2023 15:44

Hi, okay unfortunately I have found a more serious bug in the cmdline, because of a silly bug instead of using the last profile you had used in gui it ignores that and just uses a random from the list using a consistent but incorrect ordering - https://jthink.atlassian.net/browse/SONGKONG-2496

So this means (for me when using English and default installation) it uses the Default profile except for Fix Songs task where it uses the Add new metadata only profile.

I noticed your log output it says

Songs saved (if not preview):94281 (915.62%)

Implying running in preview mode, and I didn’t think you were running in preview so unfortunately got me wondering if using the wrong profile, you can check by looking at the Options tab of FixSongsReport00179.html

Although not many customers using cmdline I may have to release a bug fix release early to resolve this.

meaning · 19 September 2023 15:51

Are you saying that songkong messed with my files again ? what is this Add new metadata only profile ? it will only add missing metadata and ignore the rest ?

paultaylor · 19 September 2023 15:54

It will identify the files okay, but it means it will only add metadata for fields that dont have any metadata, it will not replace existing metadata.

Has it been using that ?

meaning · 19 September 2023 16:13

I was able to open the report :

Options

Current Profile: Add new metadata only

DAMN.

Why am I always falling on such bugs?

Now, it is not catastrophic, but it puts on the ground the whole idea of having the album titles changed with specific information like is it an EP, a single, …

I simply ran my script for nothing since two days

OK I stop it and will start it again once :

right profile(s) are selected based on last one used for each type of task ran in songkong
the rename task will be usable from the CLI

Shortly, it is simportant that songkong applies the user selected options when the user runs a task.

another option for bullet 1. would be to add a way to call a specific profile from the command line.

paultaylor · 19 September 2023 16:25

Because you are a trailblazer, few people using the cmdline, and this bug was recently introduced, it used to work when we only had one profile type that was shared between different tasks

Well, yes and it my fault the bug is there, but that is why it is good to check the reports then you may have noticed this earlier.

I can see that, but until now not had much demand for cmdline, I have added a line that now outputs what profile is being used.

meaning · 19 September 2023 18:10

Well, I’m afraid a line telling me a random profile is used instead of the ones I defined won’t do the trick. Could I at least get songkong to start the last profile I used for each type of task ? running songkong using the command line is useless if it doesn’t update all the metadata.

since the “split” I have 142 folders to process. You understand it is not an option for me to process them one after the other, going back to my UI every hour to see how it goes and if the previous task is done

I really need to get these few bugs fixed in order to be able to fix the files, get rid of duplicates, and rename (move) the fully matched albums specific versions in a 3 steps process, per monthly folder.

paultaylor · 19 September 2023 22:02

Well yes obviously that is the bug im fixing.

Of course, but it would be worth doing for the first folder processed.

Yes, that is why I have said I may do a bug fix release before the major release I was working on.

meaning · 19 September 2023 18:27

Thanks Paul !

I’m quite sure the script I worked on might become handy for some power users around here. I might want to share it once fully tested, the features so far are:

Configuration Settings :

Customizable paths for processing folders.
Option to use a specific Docker image for SongKong.
Ability to send notifications using Pushover service.
Options to control detailed reporting, intermediate notifications, and running a Bash script.

Bash Script Execution :

Ability to run an external Bash script before processing. (Eg: a script to organize the music folder upfront songkong processing)

Music Folder Management :

Identifies target folders with a specific date pattern ( MM-YYYY ).
Deletes empty folders in the target list (as I noticed fully empty folders were making songkong hang!)
Checks and keeps track of previously processed folders to avoid redundancy. songkong can resume where it left.

SongKong Music Processing :

Processes music folders using SongKong with retry functionality for certain errors.
Calculates and reports processing progress and efficiency metrics.
Sends detailed and periodic Pushover notifications on processing progress. (Can be enabled or disabled in the configuration)
Backs up processing logs with a timestamp and folder identifier.

Duplicate Song Management :

Executes SongKong’s delete duplicates task and notifies about the summary.

Music Renaming Task (Disabled as currently broken in songkong):

Has a capability (though commented out as the rename feature fails to start using songkong cli) to rename songs using SongKong

Logging :

Logs various actions in action_log.txt .
Keeps a log of processed folders in songkong_log.txt .

Final Notification :

Sends a completion notification once all folders are processed.

I’m looking forward to testing this script with an optimized version of songkong

meaning · 19 September 2023 18:49

I just ran a fix, remove duplicates and rename task on folder 01-1800 throught the webui. everything went fine. So there is nothgin bad on the webui side. I can confirm !

thanks again for your help Paul ! I’ll have a look here waiting for the bug fixes

paultaylor · 20 September 2023 13:59

Okay, SongKong 9.5.1 Jazz is now available for Docker and Melco versions, will be available for other platforms later today.

This fixes the Rename Files not starting from cmdline, counts not correct form cmdline, and tasks not using the last profile run from gui issues.

meaning · 20 September 2023 15:57

ok I confirm it works !

I checked the logs and the fix task did use my “other runs” profile as it should.

the reported loaded tracks is now correct, which means the notification my script pushes at the end of a task is correct.

For this value as an example : Songs loaded 12863:Fingerprinted 12618: MusicBrainz 146: Discogs 6736: Saved 82

As you can see the ETA estimation for next folder still needs some work, but rest of the information is now right.

It’s also right for the delete duplicates task:

(this folder was already processed, therefore no more duplicates found, I’ll get back to you begging for bandcamp integration, don’t worry about that, as I can definitely say the remaining unmatched releases comes from this source…)

and here is the confirmation that rename is not crashing anymore :

Executing the rename command: docker run --rm --name songkong_rename_01_1804 -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music songkong/songkong -f /music/Music_dump/01-1804

Now I need to add the reporting for rename tasks, but this is not a great deal

paultaylor · 20 September 2023 15:02

Great, also raised this issue for another time

meaning · 20 September 2023 16:19

and here we go for the Rename report. All done. Of course, some improvements will become, I guess. But it does the job now!

paultaylor · 20 September 2023 21:38

Would be interested in seeing FixSongsReport00202 just because so few songs matched to musicbrainz.

meaning · 21 September 2023 10:50

Indeed, I did notice this too.

The script makes the access to reports a little complicated (they are not appearing in the reports list for some reason)

so let me do the following. for these two tasks :

that was followed by this task :

I am sending you a download link by email.

paultaylor · 21 September 2023 11:57

Okay there are a few possibilities here for Report213

You have enabled the quite restrictive Only allow match if all tracks in album were matched option. So for example maybe you have an EP with 4 tracks, and MusicBrainz only has the 5 track EP version then it will not be matched to because can only match 4 or the 5 tracks
Mostly E.Ps and singles, MusicBrainz coverage of albums is better then singles, Discogs certainly better for singles because more collectibles
Alot from 2023, albunack database not been updated since may 2023 because recent update had to be rolledback because a problem with Disocgs data downloads, should be resolved soon

Now, I think the most significant issue is the Only allow match if all tracks in album were matched option for two reasons:

Prevents matches to good but not perfect matches
Encountering an unknown issue that is preventing some complete valid matches because option enabled.

What would be really good is if you could rerun against this folder with the Only allow match if all tracks in album were matched option and preview enabled so that we can compare the results.

paultaylor · 21 September 2023 12:03

The reports are not displayed in the reports list because you are deleting the database and the database contains details of the tasks run and hence reports created. But you can still create the report independently by opening the starting report html file e.g FixSongsReport00001.html

Loaded Songs Count dont match when run FixSongs from Cmdline

Options

Current Profile: Add new metadata only