Graphics Reference
In-Depth Information
a
b
First stage of forward transform
Second stage of forward transform
−
32768
−
32768
−
32768
−
32768
−
256
−
256
−
256
−
256
0
0
0
0
−
256
−
256
−
256
−
256
0
0
0
0
−
256
−
256
−
256
−
256
0
0
0
0
−
256
−
256
−
256
−
256
[]
[]
×
T
×
D
D
−
65536
−
65536
−
65536
−
65536
−
8388608
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
S
T
1
= 2
−(
B
+
M
−9)
= 2
−1
>>
>>
8
1
S
T
2
= 2
−(
M
+6)
= 2
−8
−
32768
−
32768
−
32768
−
32768
−
32768
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Fig. 6.4
Intermediate scaling factor determination for the forward transform so that the interme-
diate and output values fit within 16-bits.
B
is video bit depth and
M
log
2
(
N
)where
N
is the
transform size. Worst case bit-depth analysis is done assuming a residual block with all samples
having maximum amplitude equal to
D
2
B
8 is the video bit depth), as input to the first
stage of the forward transform. (
a
) First stage of the forward transform, (
b
) Second stage of the
forward transform. Reproduced with permission from [
6
]. © IEEE 2013
(where
B
D
In summary the constraints imposed in this section result in the following scale
factors after different transform stages:
After the first forward transform stage:
S
T
1
D
2
(
B
C
M
9)
After the second forward transform stage:
S
T
2
D
2
(
M
C 6)
After the first inverse transform stage:
S
IT
1
D
2
6
After the second inverse transform stage:
S
IT
2
D
2
(21
B
)
where
B
is the bit depth of the input/output signal (e.g. 8 bit) and
M
D
log
2
(
N
).
Without quantization/de-quantization, this choice of scale factors ensures a bit
depth of 16 bit after all transform stages. However, quantization errors introduced by
the quantization/de-quantization process might increase the dynamic range before
each inverse transform stage to more than 16 bit. For example, consider the situation
where
B
D
8 and all input samples to the forward transform are equal to 255. In this
case, the output of the forward transform will be a DC coefficient with value equal to
255 << 7
D
32640. For high QP values and with a quantizer rounding upwards, the
input to each inverse transform stage can easily exceed the allowed 16 bit dynamic
range of [
32768, 32767]. While clipping to 16 bit range was considered trivial
after the de-quantizer, it was considered undesirable after the first inverse transform
stage. In order to allow for quantization error of some reasonable magnitude and at