Okay the problem is as follows:
jthinksearch as used by SongKong, Jaikoz and Albunack runs on Amazon Web Services (AWS) using ElasticBeanstalk, the application is installed on an Amazon Linux virtual machine that runs Java and Apache Tomcat. This application failed yesterday, my suspicion being that there was a small memory leak that over time had more effect until the application ran out of memory. The application is periodically redeployed with a new MusicBrainz/Discogs index, this is meant to occur every month but actually it is 5 months since the last redeployment which is why we haven’t had this memory leak issue before. Apologies for that, in fact I was rebuild the indexes from the latest MusicBrainz/Discogs data ready for deployment when the problem occurred.
Then because it failed AWS tried to redeploy the application, as part of this redeployment I have written a number of scripts (ebextensions) and one of these uses and installs the AWS-CLI tool which is required to run another script. But yesterday AWS updated the AWS-CLI tool so that it requires Python 3.6, before that Python 2.7 was only required. But the linux virtual machine that jthinksearch uses only supports Python 2.7 so the script failed and deployment stopped.
I didn’t understand the error message at first because I don’t have the script with the name shown, but the name shown was a virtual taskname within another script. I tried using a newer Linux virtual machine but that had newer version of Tomcat and Java that was not compatible with the code base.
The solution (for now) is to modify my script so that it gets an older version of the AWS-CLI tool that is still compatible with Python 2.7, the problem was my script was just getting the current version and when Amazon changed it to require 3.6 that broke my script.
So I am working on that now.