SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

Running songkong cmdline in script in Docker environment

As the dupes finder reached 2 millins tracks (and therefore, was not finding any dupes anymore, as previous fix tracks task crashed at that point), I decided to properly cancel the task.

I updated the SK docker container, restarted it.

The quoted method you describe above is indeed something I already was thinking about. Especially because once I’ll have fixed and dupe checked the majority of my library, I’ll have to deadl with daily folders. It will therefore make sense to run songkong daily and pass it a specific daily folder at each task run.

While it would be great to figure out a way for songkong itself to do this, I guess I’ll have to call it using some sort of a script for now. (As you easily understand I won’t start 40+ tasks one by one, manually). :wink:

My guess is I’ll have to call songkong using the cli, with a sublte combination of docker exec. I could then schedule the runs using cron, but this would not allow me to know if the previously running tasks ended or not.

Therefore, I’d need to query songkong status upfront running it with the next folder.

In an ideal world (and I know I am a niche user), there would be a feature in songkong allowing me to “process each subfolder as a dedicated task”. This would be a checkbox to check, that would start the task, and would process each subfolder by starting a new task, ending it, then start processing the next subfolder.

I believe it would be a nice feature to get, but, can we brainstorm together on what would be the current, best practice, to achieve an automated way to process each of my monthly folders, one after the other?

My guess is I could:

  1. build a python script that will take care of calling songkong from inside the docker container. the script will also be able to “loop” between subfolders conained inside my music_dump folder
  2. query songkong to see if it is still running a task using docker exec (I have no clue how to do this)
  3. if no task is running, the python script will run docker exec, and feed songkong with the next monthly folder. start the next fix task. as an example, my fix tracks profile is now called “Other runs”, could you help me writing the right syntax sonkong waits ?

regarding the python script, I think it is easy to get working, a sublt combination of folder management and docker exec commands, with some sub folders in this kind of format : music_library/$(/bin/date +’%m-%Y’)/$(/bin/date +’%d-%m’) to handle both monthly and daily folders, should do the trick.

So adding a process each subfolder as dedicated task option would not make sense because it is important in your case to restart SongKong between each folder to clear all resource use.

Don’t complicate things by trying to think of a solution for daily and monthly folders, they are different problems. Finish processing of what is in your current library is the current problem.

As you already know the name of the folders, and have no reason to expect SongKong to fail when processing one folder a time wouldn’t a simple python script that calls SongKong once for each folder work?

I’m on it, mthe script is already starting a non running SK docker container, and is correctly parsing the monhtly and daily folders.

May I ask you to remember me the right sonkong.sh syntax to use to start a fix songs task ?

fyi here is how the script loos like for now :slight_smile:

import os
import subprocess
import time
from datetime import datetime
import requests

# Configuration for the folders
HOST_FOLDER = "/mnt/user/MURRAY/Music/Music_dump"
DOCKER_CONTAINER_NAME = "songkong"
DOCKER_FOLDER = "/music/Music_dump"

# Pushover connection information
pushover_user_key = "xxxx"
pushover_api_token = "xxxx"

def send_pushover_notification(message):
    log_action("Sending Pushover notification: {}".format(message))
    if pushover_user_key and pushover_api_token:
        url = "https://api.pushover.net/1/messages.json"
        data = {
            "token": pushover_api_token,
            "user": pushover_user_key,
            "message": message
        }
        response = requests.post(url, data=data)
        return response.json()
    return None

def log_action(action):
    print(action)
    with open("action_log.txt", "a") as log_file:
        log_file.write("{} - {}\n".format(datetime.now(), action))

def was_folder_processed(folder):
    if not os.path.exists("songkong_log.txt"):
        return False
    with open("songkong_log.txt", "r") as log_file:
        logs = log_file.read()
        return folder in logs

def start_songkong_docker():
    cmd = "docker start {}".format(DOCKER_CONTAINER_NAME)
    subprocess.call(cmd.split())

def stop_songkong_docker():
    cmd = "docker stop {}".format(DOCKER_CONTAINER_NAME)
    subprocess.call(cmd.split())

def run_songkong(relative_path):
    # Execute SongKong in Docker
    cmd = "docker exec {} /opt/songkong/songkong.sh -m fixsongs {}".format(DOCKER_CONTAINER_NAME, os.path.join(DOCKER_FOLDER, relative_path))
    log_action("Executing the command: {}".format(cmd))
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    output, error = process.communicate()

    # Check if SongKong started the task successfully
    if process.returncode == 0:
        log_action("SongKong successfully processed folder: {}".format(relative_path))
    else:
        log_action("SongKong failed to process folder: {}. Error: {}".format(relative_path, error.decode("utf-8")))
        send_pushover_notification("SongKong failed to process folder: {}. Check logs for more details.".format(relative_path))

def find_date_folders():
    folders = [folder for folder in os.listdir(HOST_FOLDER) if os.path.isdir(os.path.join(HOST_FOLDER, folder))]
    folders.sort(key=lambda date: datetime.strptime(date, '%m-%Y') if '-' in date else datetime.strptime('01-' + date, '%d-%m-%Y'))
    return folders

if __name__ == "__main__":
    target_folders = find_date_folders()
    log_action("Target folders identified: {}".format(target_folders))

    for target_folder in target_folders:
        if not was_folder_processed(target_folder):
            start_songkong_docker()  # Start SongKong Docker
            print("Attempting to run SongKong on folder: {}".format(os.path.join(DOCKER_FOLDER, target_folder)))
            run_songkong(target_folder)
            with open("songkong_log.txt", "a") as log_file:
                log_file.write("Processed folder: {}\n".format(target_folder))
            stop_songkong_docker()  # Stop SongKong Docker

    send_pushover_notification("Script has finished processing folders: {}".format(target_folders))

And here is what I can see in songkong logs :

14/08/2023 09.30.11:CEST:StartPage:setFolderInputField:SEVERE: -- GET:/music/Music_dump
14/08/2023 09.30.11:CEST:CmdRemote:lambda$start$180:SEVERE: >>>>>/foldertree
14/08/2023 09.30.11:CEST:CmdRemote:lambda$start$180:SEVERE: >>>>>/style/fontawesome/webfonts/fa-light-300.woff2
14/08/2023 09.30.11:CEST:CmdRemote:lambda$start$180:SEVERE: >>>>>/style/fontawesome/webfonts/custom-icons.woff2
14/08/2023 09.32.59:CEST:CmdRemote:lambda$start$180:SEVERE: >>>>>/start.task
14/08/2023 09.32.59:CEST:BaseFolderGuesser$FindFile:visitFile:WARNING: BaseFolderGuesser checking:01-Exhaust-A History Of Guerrilla Warfare.flac
14/08/2023 09.32.59:CEST:CmdRemote:lambda$start$180:SEVERE: >>>>>/fixsongs.select_profile
14/08/2023 09.32.59:CEST:CmdRemote:lambda$start$180:SEVERE: >>>>>/fixsongs.select_profile
14/08/2023 09.32.59:CEST:ServerProfile:selectProfile:SEVERE: selectProfile:/songkong/Prefs/songkong_fixsongs4.properties
14/08/2023 09.32.59:CEST:ServerProfile:selectProfile:SEVERE: selectProfileDone

But it gets stuck there it seems… it is running till 10 minutes+ and there is nothing more ingoing in the debug log file.

here is te output of the script:

Target folders identified: ['01-2015', '01-2016', '01-2017', '09-2019', '10-2019', '11-2019', '12-2019', '01-2020', '02-2020', '03-2020', '04-2020', '05-2020', '06-2020', '07-2020', '08-2020', '09-2020', '10-2020', '11-2020', '12-2020', '01-2021', '02-2021', '03-2021', '04-2021', '05-2021', '06-2021', '07-2021', '08-2021', '09-2021', '10-2021', '11-2021', '12-2021', '01-2022', '02-2022', '03-2022', '04-2022', '05-2022', '06-2022', '07-2022', '08-2022', '09-2022', '10-2022', '11-2022', '12-2022', '01-2023', '02-2023', '03-2023', '04-2023', '05-2023']
songkong
Attempting to run SongKong on folder: /music/Music_dump/01-2015
Executing the command: docker exec songkong /opt/songkong/songkong.sh -m fixsongs /music/Music_dump/01-2015

after quite some time, the sk logs will eventually state this

14/08/2023 10.34.21:CEST:WARNING: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@64ba3208 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.

It is important to say that even running the following command directly from the docker console also seems to “hang” :

/opt/songkong # ./songkong.sh -m /music/Music_dump/03-2023
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log

So there must b something wrong with the syntax or something like that.

There was issue with original version of 9.4 and cmdline see Songkong Aerial - Command Line not working anymore
so please try getting image again to see if that fixes it.

I snatched it this morning. Also I’m not getting any error message.

Running it directly from the docker container outputs this :

/opt/songkong # ./songkong.sh -m /music/Music_dump/03-2023
debuglogfile is:/songkong/Logs/songkong_debug%u-%g.log
userlogfile is:/songkong/Logs/songkong_user%u-%g.log

But for some reason, nothing happens. No cores are used, and the logs only states the job starts. But nothing gets processed.

in first state, a debug log gets written, stating some DB lock issue :

14/08/2023 14.49.58:CEST:WARNING: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@299321e2 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.

If I run it again, then I can see two logs are renamed as .lck.

If I kill the cli command in the docker container, then I can tail the log again and see this :

14/08/2023 14.53.12:CEST:SongKong:finish:WARNING: finish
14/08/2023 14.53.12:CEST:SongKong:cmdlineOrRemoteModeStart:WARNING: start
14/08/2023 14.53.12:CEST:SongKong:cmdCheckDatabase:WARNING: start
14/08/2023 14.53.12:CEST:SongKong:cmdCheckDatabase:WARNING: deletingDbLock
14/08/2023 14.53.12:CEST:SongKong:cmdCheckDatabase:WARNING: end
14/08/2023 14.53.13:CEST:SongKongDatabase:checkDatabaseCmdLine:WARNING: Setting Db Folder:/songkong/Prefs/Database
14/08/2023 14.53.13:CEST:SongKongDatabase:checkDatabaseCmdLine:WARNING: Lock File remaining from previous, deleting lock
14/08/2023 14.53.13:CEST:HibernateUtil:createFactory:SEVERE: ----Initilizing Hibernate Session factory
14/08/2023 14.53.13:CEST:INFO: Initializing c3p0-0.9.2.1 [built 20-March-2013 10:47:27 +0000; debug? true; trace: 10]
14/08/2023 14.53.13:CEST:INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@6b154ffd [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@f3ce6138 [ acquireIncrement -> 3, acquireRetryAttempts -> 10, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> true, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1bqq1i2ax1vqbmfcv4ujbm|1b955cac, idleConnectionTestPeriod -> 3000, initialPoolSize -> 1, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 3600, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 200, maxStatements -> 3000, maxStatementsPerConnection -> 50, minPoolSize -> 1, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@ef917735 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 1bqq1i2ax1vqbmfcv4ujbm|57c03d88, jdbcUrl -> jdbc:h2:async:/songkong/Prefs/Database/Database;FILE_LOCK=SOCKET;MVCC=TRUE;DB_CLOSE_ON_EXIT=FALSE;CACHE_SIZE=50000;, properties -> {password=******, user=******} ], preferredTestQuery -> null, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 3300, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 1bqq1i2ax1vqbmfcv4ujbm|e19bb76, numHelperThreads -> 3 ]

But nothing happens. the debug 0_0 log never starts to get written and songkong doesn’t seem to do anything.

I’ll start a regular task from the webui for now I guess. Impossible to get songkong to start from the cli at the moment.

@paultaylor back to my previous question about the DB and reports, there is one thing I keep not getting.

After an update, the DB gets deleted. Allright. But the reports remains :

image

And I guess the reports are actually listed in the DB, as they are not showing in the webui anymore after an update !

So, is it safe to assume that reports are also totally useless after an update ?? And that songkong will actually have to fingerprint each file again ? This is pretty confusing. :slight_smile:

Reports are just reports, they are html webpages that show results of SongKong tasks. After deleting the database the references to these reports no longer exist so they are not listed in the report list page. However the reports are still valid and can be opened to look at them as you require.

SongKong doesn’t have to re-fingerprint songs as fingerprint are stored in the files themselves. There is potentially a performance hit when delete database just because have to read file metdata from files themselves instead of database, and have to get MusicBrainz/Discogs releases from server because not cached locally.

I’m away from office until Thursday so cannot investigate the cli issue until then.

OK this makes full sense, especially if yo uhave to use another instance of SK to process the same files in the future. Clear now. :slight_smile: I can see (thanks to the “ignore because already matched” line, the files that are being skipped.

Enjoy your days off :wink: I’ve ran a regular fix task using the webui for now.

Could the problem be described in this article

In my DockerFile the last lines are

ENTRYPOINT ["/opt/songkong/songkong.sh"]
CMD ["-r"]

So having built SongKong it starts it in remote mode, which is usual behaviour that most customers want.

I was assuming you would build your own DockerFile to build own image that doesn’t actually start SongKong , and then use the Docker terminal to pass the correct options. But it looks like you are using the original Docker image and then trying to start SongKong with over-ridden options and the override isnt working ?

For my part on Synology I seem only to be able access Terminal for container that is already running, so since that means SongKong already running there is not an obvious way for me to try and run it another way.

The article I point to indicates different behaviour for ENTRYPOINT and CMD, but also suggests you use one or the other so perhaps I have done this wrong ?

Or maybe from terminal can just do

docker run containername -m /music/datefolder

to run the container with -m /music/datefolder part overriding the -r part

indeed, I am using the same docker instance and try to call the bash script from the host server.

So I’m doing two things:

  1. I start the docker container
  2. I call sonkong.sh from inside the now running docker container by using docker exec command

If I do that, then songkong wont process any of my files. and display the logs I pasted earlier.

I will try to bypass the -r command as you suggested by calling docker run and the command combined.

That isn’t going to work because songkong already running if using my docker image.

I have only ever used docker for the purposes of creating a version of SongKong that can run on docker environments, I dont use for any other purposes so Im not very knowledgeable about it. But Im going to try setup docker on my linux box to see if can get it running

No, the script starts the container, runs the script, analyses the process, and will send a notification once the process of the folder is done (using pushover as this is the notification agent I usually use).

Once done, the script stops the docker container, and starts over (skipping to the next folder)

But doesn’t starting the container automatically start SongKong, as it is started at end of the Dockerfile?

fair point. I create a dedicated DockerFile now and will run it as Songkong-cli ^^

ok @paultaylor now have a dedicated docker running called songkong-cli, it doesn’t start songkong.sh at all at start. I can try to run my script again BUT this is a fresh install, and I have no profile configured.

this is the profile songkong seems to use (in the "regular docker) :

#Tue Aug 15 09:27:25 CEST 2023
acousticFingerprint=false
actionIfSongAlreadyMatched=0
addNonAlbumReleaseTypesToTitle=false
classicalFieldsModifyIfEmpty=
classicalFieldsNeverModify=
classicalTrackArtist=4
discSubTitleInTitle=0
discogsGenreFromOption=0
discogsGenreGroupingFromOption=0
discogsGenreGroupingMaxFieldsOption=1
discogsGenreGroupingOverwriteOption=3
discogsGenreMaxFieldsOption=1
discogsGenreOverwriteOption=0
embedArtwork=2
featuredAlbumArtists=1
featuredArtists=1
fieldsModifyIfEmpty=BPM,KEY,MOOD,
fsArtwork=1
identifyClassical=0
ignoreFolderMetadataWhenSongMatching=false
isAddArtistArtwork=false
isAddAudioEncodingToReleaseTitle=false
isAddComposerToAlbumTitle=true
isAddComposerToMinimGroup=false
isAddComposerToMinimWork=false
isAddHdToReleaseTitle=false
isAlwaysUseArtistType=false
isApplyClassical=true
isDefault=false
isFindBackCover=false
isFindFrontCoverart=true
isIgnorePreviouslyCheckedFiles=false
isModifyIfSongOnlyMatch=true
isRemoveClassicalAlbumArtist=false
isSetBoxsetRoonAlbumTag=false
matchAllSongsOnReleaseOnly=true
matchByFolderOnly=true
maxImageSize=1200
minImageSize=200
modifyAcousticAnalysis=false
modifyArtwork=true
modifyGenres=true
movementToTitle=0
noRandomFolders=false
numberPadding=1
preferredMedia=278
preferredReleaseDateType=2
preview=false
profileName=Other runs
putOriginalReleaseYearIntoYear=false
saveArtworkFilename=folder
saveOperaFormat=0
searchDiscogs=true
searchMusicBrainz=true
storeYearOnlyInDateFields=true
translateForeignArtists=true
useArtistName=true
useArtistNameFromRecording=false
useDiscogs=true
useRecordingTitle=false
useReleaseGroupTitle=true
workToGroup=2

how do I make sure I create this profile and keep all its settings before I invoque songkong.sh in the newly cretaed docker container ?

EDIT: it turns out I cannot start java from inside the container I just built.

How does your DockerFile differ from mine ?

I’m asking mysef the same exact question. here is the modified docker file content :

FROM adoptopenjdk/openjdk14:alpine AS build

RUN /opt/java/openjdk/bin/jlink --module-path=/opt/java/openjdk/jmods \
--add-modules java.desktop,java.datatransfer,java.logging,java.management,java.naming,java.net.http,java.prefs,java.scripting,java.sql,jdk.management,jdk.unsupported,jdk.jcmd,jdk.crypto.ec,jdk.dynalink \
--output /opt/songkong/jre

RUN apk --no-cache add \
      curl \
      tini

RUN mkdir -p /opt \
 && curl http://www.jthink.net/songkong/downloads/current/songkong-linux-docker.tgz?val=11522| tar -C /opt -xzf - \
&& find /opt/songkong -perm /u+x -type f -print0 | xargs -0 chmod a+x

RUN rm -fr /opt/java

FROM alpine:3.11

ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'

# Votre code pour la configuration d'Alpine, l'installation de GLIBC, etc...

RUN mkdir -p /opt

COPY --from=build /opt/songkong /opt/songkong

RUN apk --no-cache add \
      ca-certificates \
      curl \
      fontconfig \
      msttcorefonts-installer \
      tini \
 && update-ms-fonts \
 && fc-cache -f

EXPOSE 4567

# Config, License, Logs, Reports and Internal Database
VOLUME /songkong

# Music folder should be mounted here
VOLUME /music

WORKDIR /opt/songkong

ENTRYPOINT ["/bin/sh"]

Hi, took me a while to remember how docker works but okay its simpler that than after all.

This is all I did on Synology

Launched standard songkong/songkong Image but as well as configuring /music and /songkong folders I went to the Environment tab and modified the Command option from

-r

to

-m /music

Then once it started, i selected Details on the container and then Terminal and could see it had started correctly

confirmed this, by going to Log tab

and then once completed processing it automatically terminates

So from command line I think you can use standard image and just need to run something like

docker run -v /CONFIGFOLDER:/songkong -v /MUSICFOLDER:/music songkong/songkong -m /music/DATEFOLDER 

replacing CONFIGFOLDER, MUSICFOLDER and DATEFOLDER with actual values.

Note when running SongKong task from command line it will use the profile last selected for the task. So if you want to use a different profile, you would need to first start task with that profile in the WebUI and start task (and then cancel if you wish since profile will now be set for that task).

This is a task for tomorrow morning ! thanks for helping here ! :slight_smile:

problem using docker run is that each time we run docker run, a new ocntainer is created. therefore, it is impossible to use any existing profile.

also, running docker run again and again leaves a ton of unused containers on the server.

I’me trying to run the task using docker exec instead :

Starting to process folder: /music/Music_dump/03-2023/01-03
Sending Pushover notification: Starting to process folder: /music/Music_dump/03-2023/01-03
Executing the command: docker exec f8ccfc3f1ad4 /opt/songkong/songkong.sh -m /music/Music_dump/03-2023/01-03

but this leads to the same behavior as the previous time. The process is simply not starting. I think something needs to be changed inside songkong code in order to allow to start a task using docker xec command.

so here is how I fixed this (somehow). I’ve use the following function to actually run a docker, that gets deleted once the process ends. this to avoid having multiple stopped docker containers piling on my server :

def run_songkong(relative_path):
    # Use docker run with --rm to start SongKong in a new container for each folder and delete the container afterwards
    container_name = "songkong_{}".format(relative_path.replace('-', '_'))  # Naming the container for clarity
    cmd = "docker run --rm --name {} -v /mnt/cache/appdata/songkong:/songkong -v /mnt/user/MURRAY/Music:/music {} -m {}".format(container_name, DOCKER_IMAGE_NAME, os.path.join(DOCKER_FOLDER, relative_path))
    log_action("Executing the command: {}".format(cmd))
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    
    while True:
        line = process.stdout.readline().decode("utf-8").strip()
        if "Reports:" in line:
            log_action("SongKong finished processing for folder: {}".format(relative_path))
            break

    output, error = process.communicate()

now this has one major issue. as the docker container is pulled each time, I cannot keep any preferences. It means I have to use the default profile to fix my tracks, as it is setup when you inisially install songkong.

If you have a good idea on how to pass the settings through this command paul, I’m taking your advices ! :slight_smile:

here is how it looks like in the terminal :

So, it works. it starts a container, runs he task, sends pushover notifications, and stops the container / remove it, then goes to the next folder, and so on. I simply need to figure out with you how to make the profile settings persistant. For the fix songs task, it is not super important. But once i’ll move to the duplicates mover / finder, ti will become really important. :slight_smile:

of course, as it uses the “default” songkong folders (Eg: it creates the reports in the Report folder and. so on), I might want to simply remove all the profiles, and only keep one. That might do the trick, as songkong won’t have any other profile to use ! I’ll test that later on.