Stenciling - Introduction to 3D Game Programming with DirectX 9.0 - page 135

Note: If you do not understand this talk of bits and masking, it most
likely means that you need to brush up on your binary, hexadecimal,
and bit-wise operations.
8.1.3.3 Stencil Value
As stated previously, this is the value in the stencil buffer for the cur-
rent pixel that we are stencil testing. For example, if we are performing
the stencil test on the ij th pixel, then value will be the value in the ij th
entry of the stencil buffer. We cannot explicitly set individual stencil
values, but recall that we can clear the stencil buffer. In addition we can
use the stencil render states to control what gets written to the stencil
buffer. The stencil-related render states are covered shortly.
8.1.3.4 Comparison Operation
We can set the comparison operation by setting the D3DRS_STENCIL-
FUNC render state. The comparison operation can be any member of
the D3DCMPFUNC enumerated type:
typedef enum _D3DCMPFUNC {
D3DCMP_NEVER = 1,
D3DCMP_LESS = 2,
D3DCMP_EQUAL = 3,
D3DCMP_LESSEQUAL = 4,
D3DCMP_GREATER = 5,
D3DCMP_NOTEQUAL = 6,
D3DCMP_GREATEREQUAL = 7,
D3DCMP_ALWAYS = 8,
D3DCMP_FORCE_DWORD = 0x7fffffff
} D3DCMPFUNC;
D3DCMP_NEVER —The stencil test never succeeds.
D3DCMP_LESS —The stencil test succeeds if LHS < RHS .
D3DCMP_EQUAL —The stencil test succeeds if LHS = RHS .
D3DCMP_LESSEQUAL —The stencil test succeeds if LHS RHS .
D3DCMP_GREATER —The stencil test succeeds if LHS > RHS .
D3DCMP_NOTEQUAL —The stencil test succeeds if LHS RHS .
D3DCMP_GREATEREQUAL —The stencil test succeeds if LHS !
RHS .
D3DCMP_ALWAYS —The stencil test always succeeds.
8.1.3 Updating the Stencil Buffer
In addition to deciding whether to write or block a particular pixel from
being written to the back buffer, we can define how the stencil buffer
entry should be updated based on three possible cases: