SongKong Jaikoz

SongKong and Jaikoz Music Tagger Community Forum

Delete Duplicates not finding all Duplicates

#1

in the meanwhile, I decided to fine tune my fix task settings, as you remember I did not add the “release type” first time I ran it.

I decided to head to my Aphex Twin folder, as this is an artist I had a crazy amount of dupes for.

I ran a new fix task, and a new remove duplicates, and a new rename. And even after this, look :

image
image2466×1888 418 KB

it’s funny, because this is happening to almost all the albums / EP’s I do have from Richard D James.

other examples :

image
image2586×1642 377 KB

Let me share my fix task settings :

Basic

  • Ignore songs previously checked that could not be matched: No
  • For songs already fully matched : Update Metadata only
  • Update Artwork: Yes
  • Update Genres: Yes
  • Update Mood and other acoustic attributes such as BPM: No
  • Only allow match if all songs in grouping match to one album: Yes
  • Only allow match if all tracks in album were matched: No

Match

  • Force Acoustic fingerprints even if already matched: No
  • Search for a MusicBrainz match: Yes
  • Update from Discogs: Yes
  • Search for a Discogs match: Yes
  • Ignore metadata derived from filename when matching individual songs: No
  • All existing folders represent a single album: No
  • Preferred media formats: CD,Digital Media,
  • Preferred Release Date: Latest Release Date
  • Preferred Release Countries:

Album Artwork

  • Ignore artwork smaller than this (pixels): 200
  • Resize artwork if dimensions larger than (pixels): 1200
  • Find Front Cover Artwork: Yes
  • Save front cover art embedded within song file: Replace if empty
  • Save front cover art to filesystem: Yes, and overwrite existing artwork files in folder
  • Saved front cover art filename: folder
  • Find Back Cover Artwork: No

Other Artwork

  • Other Artwork Options: No
  • Ignore artwork smaller than this (pixels): 200
  • Resize artwork if dimensions larger than (pixels): 800

Genres

  • Genre: Always replace values From Discogs Style , Max no 1
  • Grouping: Never Replace From Discogs Style , Max no 1

Format

  • Never modify or add these fields:
  • Only modify these fields if empty: BPM, Key, Mood
  • Allow changes to songs existing metadata fields if Song Only match: Yes
  • Romanize non-Latin script artist names wherever possible: Yes
  • Use standard Artist name instead of name displayed on cover: Yes
  • Use Recording Artist instead of Track Artist: No
  • Use standard song title instead of title displayed on his release: No
  • When tracks contains featured artists: Only use main artist in the artist field and discard other artists
  • When albums contains featured album artists: Only use main artist in the album artist field and discard other artists

Album Format

  • Use standard Release title instead of title displayed on this release: Yes
  • Use Original Release Date: Yes
  • Use Year instead of full dates for Date fields: Yes
  • Add EP, Single, Compilation, Live and Remix release types to release title: Yes
  • Add Audio Format to release title: No
  • Add [HD] to album title for High Definition albums: No
  • Add RoonAlbumTag to albums identified as box sets: No
  • Multi Disc Releases : Never add Disc No information to the release title

Classical

  • Identify Classical releases: Classical albums identified by SongKong

  • Apply these options to releases identified as Classical: Yes

  • Add Composers to start of Album Title : Yes

  • Remove Composer from Album Artist : No

  • Add Composer to start of Overall Work, this is used by MinimServer for indexing Classical Works: No

  • Add Composer to start of MinimServer Group, this is used by MinimServer for playing Classical Works: No

  • Use only Artist Type to categorise groups as ensembles, choirs or orchestras: No

  • Shorten Song Title to the Movement: No

  • Copy Work to Grouping field: MP3 and AIF (iTunes)

  • Opera Work format: Use MinimServer format (Work/Overall Work)

  • Track Artist: Composer

  • Never modify or add these fields:

  • Only modify these fields if empty:

Save

  • MP3 Metatag Version: Same as or v24
  • Disc / Track number padding: Pad with up to one zero
  • Save Vorbis/Flac AlbumArtist as: ALBUMARTIST
  • After SongKong has finished processing songs: Always open Report in browser
  • Add technical roles with own field to InvolvedPeople field:
  • Save multiple values as separate fields:
  • Save songs so they work best with iTunes: No

and my duplicates settings :

Options

Current Profile: Default

Basic

  • Find duplicates within same audio format only: No
  • Song is a duplicate if has same: Same MusicBrainz song and same album (specific version e.g. same country/date)
  • When you have duplicate songs: Move duplicate to Duplicates folder: /music/Music_dupes
  • When find duplicates delete folder and any sub folders if empty or only contains the following files: Yes

Advanced

  • Preferred Deletion Criteria, Keep Song that Matches Criteria:
    • Highest Quality Audio Format
    • Highest Bit Depth
    • Album with Most Tracks
    • Most Songs in Same Folder
    • Not Compilation Album
    • Earliest Folder Modified Date
    • Earliest File Modified Date
    • Original Filename
    • Highest Bitrate
    • Longest Track Length
    • Latest Folder Modified Date
    • Latest File Modified Date
    • Latest Folder Creation Date
    • Latest File Creation Date
  • Highest Quality Audio Format:
    • dsf
    • aif
    • aiff
    • wav
    • flac
    • m4a
    • mp4
    • m4b
    • m4p
    • aifc
    • ogg
    • mp3
    • wma

and my rename settings:

Basic

  • If song moved to another folder move : Everything
  • After SongKong has finished processing songs: Always open Report in browser
  • Limit File Path to Windows Explorer 259 Character limit: Apply to internal and locally connected NTFS/FAT32 drives only
  • Use Windows compatible characters in filenames: No
  • Enable Maximum File Path Length: No

Rename Files

  • Rename files based on metadata: Yes if matched to a release
  • Simplify non-standard characters: Yes
  • Rename filename part only: No
  • Rename mask: Rename Mask (albums)
  • Compilation rename mask: Rename mask (Compilations)

Rename Files

  • Move Folder: Yes
  • Move Folder: /music/Music_matched
  • Move Unmatched Folder: No

and as you can see, I still do have separate folders like this :

image

image
image706×176 32.4 KB

and they are still merged as a single album in my music library manager.

running additional delete duplicates tasks are not identifying any dupe. This is a real headache ! :smiley:

As I selected I want to keep best possible audio format for each of these albums, I really wonder why it keeps mp3 versions of an album detected as the same.

or keeps a version of I care because you do for whatever reason, as both versions are FLAC, but I still can see some tracks has a slight leght difference :

image
image2492×1832 421 KB

but again, the * Longest Track Length deletion criteria should get rid of the other track then.

fine tuning is a hell. And I’d really like to figure out the right settings to get rid of all these albums that ends up displaying the same track again and again, as many times I have a dupe. This makes it impossible to listen to anything, as the same track will play several times in a row.

Do you have ideas on how I could, once for all, manage this ?

In the meanwhile, I’m running the fix over my previously matched items, as it DID help, removing a lot of dupes. I was only wishing they would all be gone :wink:

I send the report files.

Loaded Songs Count dont match when run FixSongs from Cmdline
#2

Just quickly if you are asking me why are more files not deleted is it because you have

So it isnt going to delete any song that has not been matched to MusicBrainz, could that be issue ?

If they are matched to MusicBrainz then are they matched to exact same album or not you can easily see this within FixSongs report, if they are not then they wont be considered duplicates.

Also, adding album type/format to foldername but not album title itself is unlikely to affect what your music player shows because it will be looking at metadata not filename.

#3

Well, as you know, Aphex is far of an “obscure” artist.
Therefore every single track was matched to a musicbrainz release.

image
image4046×850 472 KB

and here is how the musibrainz matches looks like (I’ve put some identical album matches in RED) :

image
image3926×2030 1.9 MB

and here is how they end in my music manager :

image
image2724×1902 512 KB

Sooooo, I believe this is due to the fact my music manager only uses the artist and album name, and not fancy tags like the MusicBrainzID, to generate the albums. I did fix part of this by adding the files format to the tags (mp3, flac). This helps, as it allows me to keep a mp3 version and a flac version separated (Eg: if you have a close look "Richard D James album, you’ll notice there are actually two versions of the album, one with 5 extra tracks).
But this wont help when the SAME album gets detected, same file format (Eg:flac), and same track count gets detected. At that point, the only way I could find to differentiate the albums, is to head to picard, where I then can see the MusicBrainzID tag is actually different.

So these albums are perfectly similar, but do have another MB_ID tag.

So, wouldn’t it be nice to have a way to get rid of perfectly similar albums if they simply do have another MusicBrainzID but all the rest is purely identical ?

#4

Sure you cannot see the difference in your music player, that is limitation of your music player, but you can see the difference in SongKong report as I said.

You can also make use of Album Format options to add differentiators to album name

image
image942×865 36.3 KB

I also have issue to allow disambuguation comment to be added to album name - https://jthink.atlassian.net/jira/software/c/projects/SONGKONG/issues/SONGKONG-1835

You can delete different versions of the same album by using Same MusicBrainz song and same album (any version) or Same MusicBrainz song and same album (any version) and sounds the same

#5

I did this already :slight_smile: I still end up with albums that has several occurence of all their tracks.

Wait, so, this setting WILL get rid of the double albums, same amount of tracks, same titles, that do have different musicBrainzID only ?

If I pick this setting, what happens with an album that has a version of 10 tracks, another one with 15 tracks, and a third one with 10 tracks as well but one out of the 10 is different ? which one remains ?

I will run this on a copy of my already processed Aphex twin folder, as I like to see how it works.

EDIT: it is indeed WAY BETTER using (any version) ! I still have one of the EP’s that has doubled tracks…though. but most of the albums are now clean.

#6

So technically the way its works is that in MusicBrainz releases are part of a releasegroup, so if have different versions of the same album should have same MbReleaseGroupId but different MbReleaseId so picking this option in SongKong uses the MbReleaseGroupId and MbRecordingId instead of MbReleaseId and MbRecordingId.

So with this option songs that are in different versions of the same release group can be deleted, even if the track listing is not identical.

Deletion depends on preferred deletion criteria. So usual type of scenario you have a 10 track album and 15 track album and you want to keep the 15 track album. So assume same audio format , with your preferred deletion criteria it is going to delete al the tracks in the 10 track version because when songs get to 3rd criteria, the 15 track album is always preferred to 10 track album

  • Highest Quality Audio Format
  • Highest Bit Depth
  • Album with Most Tracks
  • Most Songs in Same Folder

However, if you have a 10 track Flac album, and 15 track mp3 album what do you want it to do now ?
With current settings because Highest Quality Audio Format is first criteria it would keep the 10 track flac album, delete the 10 tracks for the mp3 that matched the 10 tracks for flac, and keep the remaining 5 mp3 tracks. I think this is the best result in this case but maybe you want something else.

There is no option to say

Same MusicBrainz song and same album (any version) as long as they have same number of tracks

#7

This is exaactly what happened with that 10/15 album. It kept the “original” album I have in flac format, and the 5 exra tracks (I only have them in mp3 for now), remains in a separate release.

I agree this is best way to proceed, and I’m now thinking I FINALLY reached the profile settings I was looking for. In a few days I managedto get music classified in folders as I wanted, and finally get rid of the dupes the way I want.

Thank you so much for your help !!!

next step will be to build a script to compare folder names (they now include the album name + format, and to detect which ones needs to be resnatched in flac format. :slight_smile: