Stenciling - Introduction to 3D Game Programming with DirectX 9.0 - page 136

The stencil test fails for the ij th pixel. We can define how to update
the ij th entry in the stencil buffer in response to this case by setting
the D3DRS_STENCILFAIL render state:
Device->SetRenderState(D3DRS_STENCILFAIL, StencilOperation);

The depth test fails for the ij th pixel. We can define how to update
the ij th entry in response to this case by setting the D3DRS_STEN-
CILZFAIL render state:
Device->SetRenderState(D3DRS_STENCILZFAIL, StencilOperation);

The depth test and stencil test succeed for the ij th pixel. We can
define how to update the ij th entry in response to this case by set-
ting the D3DRS_STENCILPASS render state:
Device->SetRenderState(D3DRS_STENCILPASS, StencilOperation);
where StencilOperation can be one of the following predefined
constants:

D3DSTENCILOP_KEEP —Specifies to not change the stencil
buffer (that is, keep the value currently there)

D3DSTENCILOP_ZERO —Specifies to set the stencil buffer
entry to zero

D3DSTENCILOP_REPLACE —Specifies to replace the stencil
buffer entry with the stencil reference value

D3DSTENCILOP_INCRSAT —Specifies to increment the stencil
buffer entry. If the incremented value exceeds the maximum
allowed value, we clamp the entry to that maximum.

D3DSTENCILOP_DECRSAT —Specifies to decrement the sten-
cil buffer entry. If the decremented value is less than zero, we
clamp the entry to zero.

D3DSTENCILOP_INVERT —Specifies to invert the bits of the
stencil buffer entry

D3DSTENCILOP_INCR —Specifies to increment the stencil
buffer entry. If the incremented value exceeds the maximum
allowed value, we wrap to zero.

D3DSTENCILOP_DECR —Specifies to decrement the stencil
buffer entry. If the decremented value is less than zero, we
wrap to the maximum allowed value.