SBE fix gives different result with shntool & flac front-end?

[fatoldpig]

here's the shntool info on original files:
length expanded size cdr WAVE problems filename
8:51.48 93781908 -b- -- ---xx Track01.flac
14:12.74 150467836 -b- -- ---xx Track02.flac
11:51.70 125585712 -b- -- ---xx Track03.flac
17:21.18 183674548 -b- -- ---xx Track04.flac
52:17.61 553510004 B (totals for 4 files, 0.5096 overall compression ratio)

if i use shntool fix -o flac *.flac, i get:
Track01-fixed.flac:f1775ff933b949f297f0ff836d82bee2
Track02-fixed.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03-fixed.flac:2442d9a69bde6b6911eb1174547d83b7
Track04-fixed.flac:fce2288e1d15fd84d706d5b53defb01c

if i use flac > wav > flac (flac front-end with align on SBE), I get:
Track01.flac:f1775ff933b949f297f0ff836d82bee2
Track02.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03.flac:2442d9a69bde6b6911eb1174547d83b7
Track04.flac:a7e3d48aeefa4d2c5e5a4721063d2b5b

flac front-end log
options: --sector-align -P 4096 -b 4608 -m -l 8 -e -q 0 -r 0,6 -V
Track01.wav: Verify OK, wrote 50181676 bytes, ratio=0.535
Track01.wav: INFO: sector alignment causing 142 samples to be carried over
Track02.wav: Verify OK, wrote 78855773 bytes, ratio=0.524
Track02.wav: INFO: sector alignment causing 378 samples to be carried over
Track03.wav: Verify OK, wrote 61637536 bytes, ratio=0.491
Track03.wav: INFO: sector alignment causing 535 samples to be carried over
Track04.wav: Verify OK, wrote 91908702 bytes, ratio=0.500
Track04.wav: INFO: sector alignment causing 111 zero samples to be appended


notice, last track's ffp doesn't match. any idea why?

[TheMamba]

I'll let the true Techno experts give you a definitive answer but...the fact that it is on the last track only seems to mean that they pad/don't pad the last file differently. Being that it is the last file, it should be the last track you listen to and won't notice a SBE...

[4candles]

Originally Posted by TheMamba

I'll let the true Techno experts give you a definitive answer but...the fact that it is on the last track only seems to mean that they pad/don't pad the last file differently. Being that it is the last file, it should be the last track you listen to and won't notice a SBE...

That was my first guess, but at least in the version of shntool I have, the default action is to pad the last track. From the output posted, it seems that FLAC does the same thing.

Can you post the output of "shntool info" on those two versions of track 4?

[fatoldpig]

Originally Posted by 4candles

Can you post the output of "shntool info" on those two versions of track 4?

shntool version:
length expanded size cdr WAVE problems filename
8:51.48 93781340 --- -- ---xx Track01-fixed.flac
14:12.74 150466892 --- -- ---xx Track02-fixed.flac
11:51.70 125585084 --- -- ---xx Track03-fixed.flac
17:21.19 183677132 --- -- ---xx Track04-fixed.flac
52:17.61 553510448 B (totals for 4 files, 0.5111 overall compression ratio)
Track01-fixed.flac:f1775ff933b949f297f0ff836d82bee2
Track02-fixed.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03-fixed.flac:2442d9a69bde6b6911eb1174547d83b7
Track04-fixed.flac:fce2288e1d15fd84d706d5b53defb01c


flac front-end version:
length expanded size cdr WAVE problems filename
8:51.48 93781340 --- -- ---xx Track01.flac
14:12.74 150466892 --- -- ---xx Track02.flac
11:51.70 125585084 --- -- ---xx Track03.flac
17:21.19 183677132 --- -- ---xx Track04.flac
52:17.61 553510448 B (totals for 4 files, 0.5105 overall compression ratio)
Track01.flac:f1775ff933b949f297f0ff836d82bee2
Track02.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03.flac:2442d9a69bde6b6911eb1174547d83b7
Track04.flac:a7e3d48aeefa4d2c5e5a4721063d2b5b

shntool info seems identical

[Ted]

I don't know much about fixing tracks and the padding and such yet, but I do know that a single "bit" of data will make a completely different hash. How does the track sound? Is the FA and SA much different between the two? - or don't those matter in this case?

[AAR.oner]

SA/FA wouldn't matter in this...this isn't my area of expertise, but i'd try running it thru TLH's sbe fix and compare with the other two you've already run...

[TheMamba]

Originally Posted by AAR.oner

SA/FA wouldn't matter in this...this isn't my area of expertise, but i'd try running it thru TLH's sbe fix and compare with the other two you've already run...

That's a pretty good idea too. Where are Five and Raindawg?

[fatoldpig]

is there any way to know how many sample it's shifting/carrying over or padding during the shntool fix? that might give a clue what shntool is doing compare to flac front-end.

[4candles]

I undertook an experiment and created a set of 4 CD-quality WAV files that weren't cut on sector boundaries.

I then created a set of fixed WAV files using "shntool fix", and a set of fixed FLAC files used "flac --sector-align". I then compared the output of "shntool md5" and get different checksums for the final file. "shntool len" gives the same output for both sets of four files.

Using a hex editor I compared the two resulting WAV versions of track 04. The shntool-fixed version contains 448 zero bytes (112 samples). The FLAC-fixed version contains about 224 zero bytes, but then about 224 bytes containing data at the very end. So this seems to be a bug in FLAC.

The above tests were done using FLAC 1.1.1. I don't have the latest version installed (1.1.2) to test with, but the "changelog" for v1.1.2 doesn't mention any bugs regarding the --sector-align option.

Is anyone able to do a similar test using FLAC 1.1.2?

[fatoldpig]

Originally Posted by 4candles

I undertook an experiment and created a set of 4 CD-quality WAV files that weren't cut on sector boundaries.

I then created a set of fixed WAV files using "shntool fix", and a set of fixed FLAC files used "flac --sector-align". I then compared the output of "shntool md5" and get different checksums for the final file. "shntool len" gives the same output for both sets of four files.

Using a hex editor I compared the two resulting WAV versions of track 04. The shntool-fixed version contains 448 zero bytes (112 samples). The FLAC-fixed version contains about 224 zero bytes, but then about 224 bytes containing data at the very end. So this seems to be a bug in FLAC.

The above tests were done using FLAC 1.1.1. I don't have the latest version installed (1.1.2) to test with, but the "changelog" for v1.1.2 doesn't mention any bugs regarding the --sector-align option.

Is anyone able to do a similar test using FLAC 1.1.2?

good find 4candles. i'll do some test with flac 1.1.2 when i get home.

Originally Posted by fatoldpig

is there any way to know how many sample it's shifting/carrying over or padding during the shntool fix? that might give a clue what shntool is doing compare to flac front-end.

Yes. Shntool prints it to the screen while processing the files.

[4candles]

Originally Posted by fatoldpig

good find 4candles. i'll do some test with flac 1.1.2 when i get home.

There's no need - I can confirm the bug is there in 1.1.2 as well.

I've tracked down the bug in the source code and I'll let the developer know. The buffer used to pad the end of the data with zeros is not being entirely cleared - only the first half of the buffer is being cleared.

So until flac 1.1.3 (assuming the bug is fixed) is released I would strongly advise against using the --sector-align option in FLAC. In all cases, the second half the "zero padding" at the end of the last track is not guaranteed to contain zeros.

[Five]

I was fixing a similar set recently where it just needed padding on the last track. I was thinking that I could add the padding manually in CEP, so I decided to try it with FLAC frontend first and really zoom in and look at what it does. The last few samples aren't zero samples, they have values like 12, which is very very very close to zero. I remember something about how cd players don't like it when the final sample is a zero, it has to be a little different. So I'm thinking that maybe both FLAC frontend and SHNtool pad the last track with "almost zero" samples at the tail end, perhaps these samples are slightly different. I've also taken a track which was padded by FLAC frontend and mixpased the inverted original over it and came up with a perfect flat line, so the difference should be in the padding by my theory. I can't recall if I did this, but another test to do would be to pad the track, then open it with a wav editor and only remove the padding (remember to turn smoothing off!), resave then generate a SHNtool md5 and see if there is any variance.

Hopefully we can get to the bottom of this when I find a couple hours to test all these variations. I can't get to it tonight, if somebody wants to beat me to it I won't be offended.

[4candles]

Originally Posted by Five

I was fixing a similar set recently where it just needed padding on the last track. I was thinking that I could add the padding manually in CEP, so I decided to try it with FLAC frontend first and really zoom in and look at what it does. The last few samples aren't zero samples, they have values like 12, which is very very very close to zero. I remember something about how cd players don't like it when the final sample is a zero, it has to be a little different. So I'm thinking that maybe both FLAC frontend and SHNtool pad the last track with "almost zero" samples at the tail end, perhaps these samples are slightly different. I've also taken a track which was padded by FLAC frontend and mixpased the inverted original over it and came up with a perfect flat line, so the difference should be in the padding by my theory. I can't recall if I did this, but another test to do would be to pad the track, then open it with a wav editor and only remove the padding (remember to turn smoothing off!), resave then generate a SHNtool md5 and see if there is any variance.

Hopefully we can get to the bottom of this when I find a couple hours to test all these variations. I can't get to it tonight, if somebody wants to beat me to it I won't be offended.

I don't think it's a case of FLAC doing something clever - it's clear from looking at the source code that it's a bug.

If you use a hex editor to look at the WAV file "fixed" by shntool, you will see that it contains the correct number of zero-valued bytes at the end.

If you look at the WAV version of the last track "fixed" by FLAC, then you will see that the first half of the padding is zero, but the second half isn't. This is easier to see if the last track you are fixing doesn't already have silence at the end.

Originally Posted by Five

I remember something about how cd players don't like it when the final sample is a zero, it has to be a little different. So I'm thinking that maybe both FLAC frontend and SHNtool pad the last track with "almost zero" samples at the tail end, perhaps these samples are slightly different.

I've never heard this before. I'm skeptical. Even if it were true I seriously doubt that FLAC is randomly putting in a few non-zero samples to deal with it. There is no mention of this in the FLAC documentation. I think 4candles indentification of a bug in the source code sounds like the best explanation. Good catch, 4candle & pig. We need to put something about this in the FAQ and in the etree wiki.