SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

SK crashes. Memory issue?

#61

Hi, received your latest report, showing Fix Songs running for 10 days and eventually failing after 10 days after successfully processing 680, 971 music files

image
image1434×969 55.1 KB

I think you would agree that is a substantial number of files to process and you are pushing SongKong to the limits. Processing is done folder by folder and only the details of a few folders are held in memory at any one time, however there are some global variables that hold minimal information, but because you have processed so many files may be larger than usual. So you are probably right when you say SongKong gets slower and slower over time. With this is mind maybe it is better to process the top level date folders one at a time as you previously suggested.

There was some timeout errors, but there was no memory errors. However if memory was getting low it could be that SongKong was spending too much time recovering memory from heap . For the docker version of SongKong the image runs the songkong.sh file, and this contains

#!/bin/sh
umask 000
./jre/bin/java  -XX:MaxRAMPercentage=60 -XX:MetaspaceSize=45 -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dorg.jboss.logging.provider=jdk -Djava.util.logging.config.class=com.jthink.songkong.logging.StandardLogging -Dhttps.protocols=TLSv1.1,TLSv1.2 --add-opens java.base/java.lang=ALL-UNNAMED -jar lib/songkong-8.5.jar "$@"

Note the option -XX:MaxRAMPercentage=60 , this means the max heap that can be used is 60% of the amount of memory you have allocated to the container - https://www.baeldung.com/java-jvm-parameters-rampercentage

#62

Also I note your last support files was 2.2 GB, this is rather large, and I guess inconvenient for you to upload. Create Support Files sends the last five reports for each report type (Fix Songs, Delete Duplicates ectera), and because you have some very large reports for both Fix Songs and Delete Duplicates the support file is large. If you no longer need these reports you could run Delete Reports from the Admin menu , then next time you run Create Support Files it will be much quicker.

#63

Hi Paul,

Sorry for my late reply.

I deleted previous reports.

RAMpercentage was already adapted, and I actually fixed the memory leak by reserving the RAM at docker level a long time ago (Remember?).

Still, it do slow down and I agree this is a challenge for SK (processing such a big queue of files).

What I believe would be the best practice for a next version is the implementation of an option that would process the folders in batch.

As an example, I am currently manually processing each monthly folder one by one. It would be convenient if SK was able to processes the folder, then finish its job, and automatically starts processing the next one. one after each other.

This leads me to a workaround for now.

As I know I can run SK in CLI mode, I could make use of a simple bash script to achieve this, right ? Could you help me out with that ?

#64

Yes, you are not actually running out of memory, but I just wonder if the memory usage is increasing over time and eventually SongKong gets to the point where it has to do alot of garbage collection which is a load on the cpu and hence slows down processing. With the desktop gui you can see memory usage in the bottom right hand corner, so in this screenshot after just starting Songkong we see it is just using 44mb of heap memory (but is allowed to use upto 18,353 Mb), but this is currently missing from the web display.

image
image761×173 18.3 KB

So you mean it does the folder, including creating report then once done it starts a new task. If so I think it may work better if SongKong is actually restarted after finished each folder, that way we ensure any memory still held by SongKong is fully released.

You can run songkong in command line mode and specify folder with the -m option, but Im not sure the best way to do this in a docker environment, my docker experience is limited to creation of the songkong docker image

#65

Well, it would need a proper openssh server + screen included in the docker container I guess. This would allow people to connect the docker remotely, start a screen, and let it run in the background.

Without this, I’m afraid a docker exec session will kill the running process as soon the shell window will be closed by the user.

#66

As I understand it there is nothing to stop you building an image based on the SongKong docker image with that feature, I’m sorry but I don’t have the time to work on this for you.

#67

Totally understood and agreed. This is a niche thing. :slight_smile:

May I eventually ask you to help me out with the exact command to use ?

I am still fixing songs for now. Based on the settings I am using, what would be the command line to use ?

In this phase (initial fixing of all existing folders), I want to create a command line command that uses these options :

Basic

  • Ignore songs previously checked that could not be matched: Yes
  • For songs already fully matched : Ignore
  • Rename files based on metadata: Yes if matched to a release
  • Update Artwork: Yes
  • Update Genres: Yes
  • Update Mood and other acoustic attributes such as BPM (Pro or Melco license only): Yes
  • Only allow match if all songs in grouping match to one album: Yes
  • Only allow match if all tracks in album were matched: Yes

Match

  • Force Acoustic fingerprints even if already matched: No
  • For Naim wav files read the accompanying metadata files (Melco license only): No
  • Search for a MusicBrainz match: Yes
  • Update from Discogs: Yes
  • Search for a Discogs match: Yes
  • Ignore existing metadata when matching individual songs: No
  • Ignore metadata derived from filename when matching individual songs: No
  • All existing folders represent a single album: No
  • Preferred media formats:
  • Preferred Release Date: Earliest Release Date
  • Preferred Release Countries:

Album Artwork

  • Ignore artwork smaller than this (pixels): 200
  • Resize artwork if dimensions larger than (pixels): 1200
  • Find Front Cover Artwork: Yes
  • Save front cover art embedded within song file: Replace if empty
  • Save front cover art to filesystem: Yes, but do not overwrite existing artwork files in folder
  • Saved front cover art filename: folder
  • Find Back Cover Artwork: No

Other Artwork

  • Other Artwork Options (Pro or Melco license only): Yes
  • Ignore artwork smaller than this (pixels): 200
  • Resize artwork if dimensions larger than (pixels): 800
  • Find artwork for these types: Album Artist, Artist, Composer
  • Save album artist artwork to artist folders in filesystem: Yes, and overwrite existing artwork files in folder
  • Add all artwork to MinimServer IndexArtwork folder: No

Genres

  • Genre: Replace if empty From Discogs Style , Max no 1
  • Grouping: Never Replace From Discogs Style , Max no 1

Format

  • Never modify or add these fields:
  • Only modify these fields if empty: BPM, Key, Mood
  • Delete all metadata from these fields:
  • Allow changes to songs existing metadata fields if Song Only match: Yes
  • Romanize non-Latin script artist names wherever possible: Yes
  • Use standard Artist name instead of name displayed on cover: Yes
  • Use standard Track title instead of title displayed on cover: No
  • Use standard Release title instead of title displayed on cover: Yes
  • Use Original Release Date: No
  • Use Year instead of full dates for Date fields: No
  • Add EP, Single, Compilation, Live and Remix release types to release title: No
  • Add Audio Format to release title: No
  • Add [HD] to album title for High Definition albums (Pro or Melco license only): No
  • Add RoonAlbumTag to albums identified as box sets: No
  • When tracks contains featured artists: Add all contributing artists to the artist field
  • When albums contains featured album artists: Add all contributing album artists to the album artist field
  • Multi Disc Releases : Add Disc No to the release title if disc has title

Classical

  • Identify Classical releases: Classical albums identified by SongKong

  • Apply these options to releases identified as Classical: Yes

  • Add Composers to start of Album Title : Yes

  • Remove Composer from Album Artist : No

  • Add Composer to start of Overall Work, this is used by MinimServer for indexing Classical Works: No

  • Add Composer to start of MinimServer Group, this is used by MinimServer for playing Classical Works: No

  • Use only Artist Type to categorise groups as ensembles, choirs or orchestras: No

  • Shorten Song Title to the Movement: No

  • Copy Work to Grouping field: MP3 and AIF (iTunes)

  • Opera Work format: Use MinimServer format (Work/Overall Work)

  • Track Artist: Composer

  • Never modify or add these fields:

  • Only modify these fields if empty:

File Naming

  • Simplify non-standard characters: Yes
  • Rename filename part only: No
  • Limit File Path to Windows Explorer 259 Character limit: Apply to internal and locally connected NTFS/FAT32 drives only
  • Use Windows compatible characters in filenames: Yes
  • Enable Maximum File Path Length: No
  • Move Folder: Yes
  • Move Folder: /music_processed/
  • Move Unmatched Folder: No
  • Rename mask: [ Plex Music Scanner ] AlbumArtist/Album/Track - Title
  • Compilation rename mask: [ Plex Music Scanner ] AlbumArtist/Album/Track - Title

Save

  • If song moved to another folder move : Everything
  • MP3 Metatag Version: Same as or v23
  • Disc / Track number padding: Pad with up to one zero
  • Save Vorbis/Flac AlbumArtist as: ALBUMARTIST
  • After SongKong has finished processing songs: Always open Report in browser
  • Add technical roles with own field to InvolvedPeople field:
  • Save multiple values as separate fields:
  • Save songs so they work best with iTunes: No

Nxt phase would be to process the daily folders, each day. This means I would use the same exact params, but will then use music_library/$(/bin/date +’%m-%Y’)/$(/bin/date +’%d-%m’) as a param, therefore pointing to the path of the day.

#68

When you start SongKong in gui mode it shows the currently selected profile, when you run in command line mode it uses the current profile. So use gui mode to get profile correct, and then just run with command line with -m foldername to fix songs using options from that profile.

2 Likes