In-Loop Filter - Advanced Video Coding Systems - page 118

7.2.2.1 Filtering Processing

After filtering strength being determined, the different filters should be applied to
the block boundaries corresponding to different BS respectively. For each filtering
unit, the left vertical boundary is first filtered and then the up horizontal boundary
is filtered with the filtered samples in previous stage. For luminance component, at
most three samples are modified on each side of the filtering unit, and at most two
samples are modified for chroma components. When BS equals to 4, the strongest
filter is applied as follows:

P 0 = ( p 0 + (( p 0 + p 2 )<< 3 ) + p 2 + ( q 0 << 3 ) + ( q 2 << 2 ) + ( q 2 << 1 ) + 16 )>> 5 ,
(7.9)
P 1 = (( p 0 << 3 ) − p 0 + ( p 2 << 2 ) + ( p 2 << 1 ) + q 0 + ( q 0 << 1 ) + 8 )>> 4 , (7.10)
P 2 = (( p 0 << 2 ) + p 2 + ( p 2 << 1 ) + q 0 + 4 )>> 3 ,

(7.11)
Q 0 = ( q 0 + (( q 0 + q 2 )<< 3 ) + q 2 + ( p 0 << 3 ) + ( p 2 << 2 ) + ( p 2 << 1 ) + 16 )>> 5 ,
(7.12)

Q 1 = (( q 0 << 3 ) − q 0 + ( q 2 << 2 ) + ( q 2 << 1 ) + p 0 + ( p 0 << 1 ) + 8 )>> 4 ,

(7.13)

(7.14) Q 2 = (( q 0 << 2 ) + q 2 + ( q 2 << 1 ) + p 0 + 4 )>> 3 .

When BS equals to 3, the filtering operation can be expressed as,

P 0 = ( p 2 + ( p 1 <<

) + (

p 0 <<

) + (

p 0 <<

) + (

q 0 <<

) +

q 1 +

)>> ,

2 2 1 2 8 4

(7.15)

Q 0 = ( p 1 + ( p 0 << 2 ) + ( q 0 << 2 ) + ( q 0 << 1 ) + ( q 1 << 2 ) + q 2 + 8 )>> ,

(7.16)

4

P 1 = (( p 2 << 1 ) + p 2 + ( p 1 << 3 ) + ( p 0 << 2 ) + q 0 + 8 )>> 4 , (7.17)

Q 1 = (( q 2 << 1 ) + q 2 + ( q 1 << 3 ) + ( q 0 << 2 ) + p 0 + 8 )>> 4 . (7.18)

When BS equals to 2, the filtered pixels are calculated from the following equations.

P 0 = (( p 1 << 1 ) + p 1 + ( p 0 << 3 ) + ( p 0 << 1 ) + ( q 0 << 1 ) + q 0 + 8 )>> 4 ,

(7.19)

Q 0 = (( p 0 << 1 ) + p 0 + ( q 0 << 3 ) + ( q 0 << 1 ) + ( q 1 << 1 ) + q 1 + 8 )>> .

(7.20)

4

For BS equal to 1, the weakest filtering operation is utilized and only one pixel on
each side of the filter unit is modified.

P 0 = (( p 0 << 1 ) + p 0 + q 0 + 2 )>> 2 , (7.21)

Q 0 = (( q 0 << 1 ) + q 0 + p 0 + 2 )>> 2 . (7.22)