Game Development Reference
In-Depth Information
∼
And if the prediction mode is 0
23, the vertical transform will be done as
Y
4
×
4
=
S
4
×
Y
4
×
4
.
(5.21)
S
4
is defined as
⊡
⊣
⊤
⊦
.
123
−
35
−
8
−
3
−
32
−
120 30
10
S
4
=
14
25
123
−
22
8
13
19
126
4 intraprediction block, if the secondary transform is used, a combined
transform is done as
For 4
×
D
4
.
Y
4
×
4
=
D
4
×
X
×
(5.22)
D
4
is defined as
⊡
⊤
34 58
72
81
⊣
⊦
77 69
−
7
−
75
D
4
=
79
−
33
−
75 58
55
−
84 73
−
28
5.3.3 Quantization in AVS2
In AVS2, the quantization is combined together with the normalization of the trans-
form, which can be implemented by multiplication and right shift. Assuming the
transform coefficients matrix is
Y
M
1
×
M
2
, and the
BitDepth
is 8 the quantization is
done as,
YQ
[
i
][
j
]=
(((
Y
[
i
][
j
]×
W
[
i
][
j
]
>>
WqmShif t
)
×
Q
[
qp
]
)>>
4
+
(<< (
Quant Shi f t
−
1
))) >>
Quant Shi f t
(5.23)
where
W
is listed as Table
5.1
,
where qp is the quantization parameter.
QuantShift
which can be derived form slice
header or sequence header is dependent on the transform block size, sample bit-depth
BitDepth
, which is computed as,
[
i
][
j
]
is the weighted quantization matrix,
Q
[
qp
]
Quant Shi f t
=
ST
[
qp
]+
(
BitDepth
−
14
)
+
log
2
(
M
1
×
M
2
)/
2
(5.24)
where the ST indicates the shifting table which is listed in Table
5.1
.
