SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

Limit memory usage within docker

I am trying to limit the maximum memory usage. In my Docker installation Songkong reports

20/10/2024 23.51.08:CEST:WARNING: SongKong has been configured with minimum heap memory of 1,008 mb, maximum heap memory of 38,584 mb and maximum permanent memory of -32 mb

I am using JAVA_OPTS="-Xmx2g -Xms1g" without any positive result. Am I missing something obvious?

A rename job on my library consumes more than 20GB and crashes the server eventually…

Looks as if the startup script does not obey JAVA_OPTS. I starts with

./jre/bin/java -XX:MaxRAMPercentage=60

but is not using -XX:+UseContainerSupport which would allow java in the container to see the memory limit imposed on the container with --memory=xG. Without it it allows Songkong to use 60% of the memory of the base system regardless of the maximum memory assigned to the docker. @paultaylor is this something for you?

Moreover the startup script should honor the JAVA_OPTS environment variable so one can change these options with docker environment variables.

The script contents are

umask 000
./jre/bin/java  -XX:MaxRAMPercentage=60 -XX:MetaspaceSize=45 -Ddocker=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dorg.jboss.logging.provider=jdk -Djava.util.logging.config.class=com.jthink.songkong.logging.StandardLogging -Dhttps.protocols=TLSv1.1,TLSv1.2 --add-opens java.base/java.lang=ALL-UNNAMED -jar lib/songkong-11.0.jar "$@"

So if I change it to

umask 000
./jre/bin/java  -XX:+UseContainerSupport -XX:MaxRAMPercentage=60 -XX:MetaspaceSize=45 -Ddocker=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dorg.jboss.logging.provider=jdk -Djava.util.logging.config.class=com.jthink.songkong.logging.StandardLogging -Dhttps.protocols=TLSv1.1,TLSv1.2 --add-opens java.base/java.lang=ALL-UNNAMED -jar lib/songkong-11.0.jar "$@"

that would solve first problem right ?

And if I then changed it to

umask 000
./jre/bin/java JAVA_OPTS -XX:+UseContainerSupport -XX:MaxRAMPercentage=60 -XX:MetaspaceSize=45 -Ddocker=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dorg.jboss.logging.provider=jdk -Djava.util.logging.config.class=com.jthink.songkong.logging.StandardLogging -Dhttps.protocols=TLSv1.1,TLSv1.2 --add-opens java.base/java.lang=ALL-UNNAMED -jar lib/songkong-11.0.jar "$@"

would that solve second ?

According to what I read yesterday: Yes. It would not make it configurable but at least would tell Java (in theory) to obey the memory restrictions of the container. I would love to test this ASAP and give you feedback. If you could provide a package with that change or a way for me to git clone/pull and experiment on my own I might be able to assist.

I believe it should be “java $JAVA_OPTS” (missing the $) so that when whoever starts the docker sets the environment variable JAVA_OPTs this would be part of the java call. I am however unsure what happens if you set options in JAVA_OPTs that you then also set later in the same line (which one takes precedence). Would have to analyze that a bit further.

It is not my strong suit but I can take a look.

The DockerFile is publicably available here

I think what you would have to do is after this section

RUN mkdir -p /opt \
&& curl http://www.jthink.net/songkong/downloads/current/songkong-linux-docker.tgz?val=1175.1| tar -C /opt -xzf - \
&& find /opt/songkong -perm /u+x -type f -print0 | xargs -0 chmod a+x

is overwrite the /opt/songkong/songkong.sh extracted with your own version with the changes outlined above.

So if you understand Docker this should be simple enough, if not I can raise an issue for this but it is not something I can work on immediately.

I am working on it. The problem appears to be the openjdk version since the version and base image you are using in Dockerfile is not correctly handling container constraints. I am working with

FROM eclipse-temurin:17-jdk-alpine AS build

now and after setting the additional -XX:+UseContainerSupport it now shows

SongKong has been configured with minimum heap memory of 320 mb, maximum heap memory of 12,288 mb and maximum permanent memory of -32 mb

when the container is started with --memory=20G. So this now looks correct. I am currently in the process of running a rename on my large library and will let you know the result. Would a change of the JRE version (and a new base image) be ok with you? That would solve some problems. So far all looks fine!

In case that is ok I will try to come up with a modification to the startup script obeying additional JAVA options in the environment-file. Question is then: How can I contribute these to the project as the only one I have access per your link is the Dockerfile and not the process creating the tar you are referring to (and downloading).

Using Java 17 instead of Java 14 is fine, so changing

FROM adoptopenjdk/openjdk14:alpine AS build

to

FROM eclipse-temurin:17-jdk-alpine AS build

is fine from that p.o.v

But then notice I use

FROM alpine:3.11

because in the first stage i use jlink to build a jre that only contains the packages I require, and then I run it on alpine:3.11 because this is much smaller than using a base build which already includes Java. So does it run okay using this alpine:3.11 base ?

Are you saying by simply upgrading jre it automatically respects the container options or do we need to add -XX:+UseContainerSupport to script ?

If you have new script for JAVA_OPTS you can modify docker file to use that one, and if it works I can incorporate into the tar buld as part of next release.

Hi,

all I did (so far) was to exchange the FROM line and change the call in songkong.sh. Build went fine, image fires up and I was able to use Songkong and do the rename successfully! Looking good. Let me check/work on how to best integrate the JAVA_OPTS and I will come back with the diffs. How would you like them? here? Mail? git something voodoo magic?

Thanks, just post here is fine.

Proposal: songkong.sh

#!/bin/sh
umask 000

# Set default Java runtime options (can be overridden by JAVA_OPTS)
DEFAULT_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=60 -XX:MetaspaceSize=45"

# Set default Java system properties (always included)
DEFAULT_PROPS="-Ddocker=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dorg.jboss.logging.provider=jdk -Djava.util.logging.config.class=com.jthink.songkong.logging.StandardLogging -Dhttps.protocols=TLSv1.1,TLSv1.2 --add-opens java.base/java.lang=ALL-UNNAMED"

# Check if JAVA_OPTS is provided; if not, use DEFAULT_OPTS, otherwise, use JAVA_OPTS
if [ -z "$JAVA_OPTS" ]; then
    # No JAVA_OPTS provided, use the default runtime options
    JAVA_OPTS_FINAL="$DEFAULT_OPTS"
else
    # JAVA_OPTS is provided, override DEFAULT_OPTS
    JAVA_OPTS_FINAL="$JAVA_OPTS"
fi

# Always append default properties (DEFAULT_PROPS) to the final options
JAVA_OPTS_FINAL="$JAVA_OPTS_FINAL $DEFAULT_PROPS"

# Run Java with the final options
./jre/bin/java $JAVA_OPTS_FINAL -jar lib/songkong-11.0.jar "$@"

And Dockerfile:

FROM eclipse-temurin:17-jdk-alpine AS build

RUN /opt/java/openjdk/bin/jlink --module-path=/opt/java/openjdk/jmods \
--add-modules java.desktop,java.datatransfer,java.logging,java.management,java.naming,java.net.http,java.prefs,java.scripting,java.sql,jdk.management,jdk.unsupported,jdk.jcmd,jdk.crypto.ec,jdk.dynalink \
--output /opt/songkong/jre

RUN apk --no-cache add \
      curl \
      tini

RUN mkdir -p /opt \
 && curl http://www.jthink.net/songkong/downloads/current/songkong-linux-docker.tgz?val=1175.1| tar -C /opt -xzf - \
&& find /opt/songkong -perm /u+x -type f -print0 | xargs -0 chmod a+x

RUN rm -fr /opt/java

FROM alpine:3.11

ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'

RUN apk add --no-cache --virtual .build-deps curl binutils \
    && GLIBC_VER="2.31-r0" \
    && ALPINE_GLIBC_REPO="https://github.com/sgerrand/alpine-pkg-glibc/releases/download" \
    && GCC_LIBS_URL="https://archive.archlinux.org/packages/g/gcc-libs/gcc-libs-9.1.0-2-x86_64.pkg.tar.xz" \
    && GCC_LIBS_SHA256="91dba90f3c20d32fcf7f1dbe91523653018aa0b8d2230b00f822f6722804cf08" \
    && ZLIB_URL="https://archive.archlinux.org/packages/z/zlib/zlib-1%3A1.2.11-4-x86_64.pkg.tar.xz" \
    && ZLIB_SHA256=17aede0b9f8baa789c5aa3f358fbf8c68a5f1228c5e6cba1a5dd34102ef4d4e5 \
    && curl -LfsS https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
    && SGERRAND_RSA_SHA256="823b54589c93b02497f1ba4dc622eaef9c813e6b0f0ebbb2f771e32adf9f4ef2" \
    && echo "${SGERRAND_RSA_SHA256} */etc/apk/keys/sgerrand.rsa.pub" | sha256sum -c - \
    && curl -LfsS ${ALPINE_GLIBC_REPO}/${GLIBC_VER}/glibc-${GLIBC_VER}.apk > /tmp/glibc-${GLIBC_VER}.apk \
    && apk add --no-cache /tmp/glibc-${GLIBC_VER}.apk \
    && curl -LfsS ${ALPINE_GLIBC_REPO}/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk > /tmp/glibc-bin-${GLIBC_VER}.apk \
    && apk add --no-cache /tmp/glibc-bin-${GLIBC_VER}.apk \
    && curl -Ls ${ALPINE_GLIBC_REPO}/${GLIBC_VER}/glibc-i18n-${GLIBC_VER}.apk > /tmp/glibc-i18n-${GLIBC_VER}.apk \
    && apk add --no-cache /tmp/glibc-i18n-${GLIBC_VER}.apk \
    && /usr/glibc-compat/bin/localedef --force --inputfile POSIX --charmap UTF-8 "$LANG" || true \
    && echo "export LANG=$LANG" > /etc/profile.d/locale.sh \
    && curl -LfsS ${GCC_LIBS_URL} -o /tmp/gcc-libs.tar.xz \
    && echo "${GCC_LIBS_SHA256} */tmp/gcc-libs.tar.xz" | sha256sum -c - \
    && mkdir /tmp/gcc \
    && tar -xf /tmp/gcc-libs.tar.xz -C /tmp/gcc \
    && mv /tmp/gcc/usr/lib/libgcc* /tmp/gcc/usr/lib/libstdc++* /usr/glibc-compat/lib \
    && strip /usr/glibc-compat/lib/libgcc_s.so.* /usr/glibc-compat/lib/libstdc++.so* \
    && curl  -LfsS ${ZLIB_URL} -o /tmp/libz.tar.xz \
    && mkdir /tmp/libz \
    && tar -xf /tmp/libz.tar.xz -C /tmp/libz \
    && mv /tmp/libz/usr/lib/libz.so* /usr/glibc-compat/lib \
    && apk del --purge .build-deps glibc-i18n \
    && rm -rf /tmp/*.apk /tmp/gcc /tmp/gcc-libs.tar.xz /tmp/libz /tmp/libz.tar.xz /var/cache/apk/*

RUN mkdir -p /opt

COPY --from=build /opt/songkong /opt/songkong

RUN apk --no-cache add \
      ca-certificates \
      curl \
      fontconfig \
      msttcorefonts-installer \
      tini \
 && update-ms-fonts \
 && fc-cache -f

EXPOSE 4567

# Config, License, Logs, Reports and Internal Database
VOLUME /songkong

# Music folder should be mounted here
VOLUME /music

WORKDIR /opt/songkong

ENTRYPOINT ["/opt/songkong/songkong.sh"]
CMD ["-r"]

Working like a charm here. If no JAVA_OPTS are presented via Environment it will use

-XX:+UseContainerSupport -XX:MaxRAMPercentage=60 -XX:MetaspaceSize=45

which is what you used before plus the UseContainerSupport. If you put JAVA_OPTS in the environment these three parameters will be dropped and you can (and must) handle them yourself.

Let me know if this is ok with you and once there is a new docker-version on docker hub. I hope it was not too much trouble and thank you very much for all your efforts!

Hi, will be done for next release but is not simple to do now without introducing the work for next release so would prefer to do as part of 11.1

Raised https://jthink.atlassian.net/browse/SONGKONG-2652 and https://jthink.atlassian.net/browse/SONGKONG-2653 and

Update

I have rebuilt the Docker Image to use Java 17, so if you download the image it should be using Java17. Now having read a bit more about UseContainerSupport option it should be enabled by default on a JVM that supports it so there is no need to add the -XX+UseContainerSupport option.

So can you test it please and see if that issue is solved simply by updating to Java 17. Because if so I would prefer not to add -XX:+UseContainerSupport becuase this option is not supported by Java on the current arm32 version of Docker and would cause java to fail - https://bitbucket.org/ijabz/songkongdockerarm32/src/master/Dockerfile and I havent found a simple Java 17 replacement for this architecture.

I verified that with Java17 it is detecting the necessary things without that option so omitting it will not cause a problem. BUT please make the changes to the songkong.sh (just get rid of that one option). If someone is not using JAVA_OPTs no harm is done. And if someone wants to use JAVA_OPTs he/she has full control over the options. Win win for everyone and no trouble for the arm32 version.

Thanks for confirming, yes I will make that change as part of next release.

Thank you very much!

Now fixed in SongKong 11.1 Absolutely released 30th October 2024