Introduction to Vertex Shaders - Introduction to 3D Game Programming with DirectX 9.0 - page 309

If s < 0, that implies the angle between the light vector and vertex nor-
mal is greater than 90 degrees, which implies that the surface receives
no light. Therefore, if s <0,welet s =0.So s [0, 1].
Now, in the usual diffuse lighting model, we use s to scale our color
vector such that the vertex colors are darkened based on the amount of
light that they receive:
diffuseColor (, , , )

However, this will result in a smooth transition from light to dark
shades. This is the opposite of what we desire for cartoon shading. We
want an abrupt transition between a few different shades (around two
to four shades works well for cartoon rendering).
Instead of using s to scale the color vector, we are going to use it as
the u texture coordinate for the luminance texture that we spoke of
earlier—the one depicted in Figure 17.3.

s r g b a

Note: The scalar s is of course a valid texture coordinate since

s [0, 1], which is the usual texture coordinate interval.

In this way the vertices won't be shaded smoothly but rather abruptly.
For example, the luminance texture might be divided into three shades,
as Figure 17.4 shows.

Figure 17.4: The shade used
depends on the interval the tex-
ture coordinate falls in.

Then values of s [0, 0.33] are shaded using shade 0 , values of s (0.33,
0.66] are shaded using shade 1 , and values of s (0.66, 1] are shaded
using shade 2 . Of course, the transition from one of these shades to the
next is abrupt, giving us the desired effect.

Note: We turn off texture filtering for cartoon shading as well
because the filtering attempts to smooth out the shade transitions. This
is undesirable since we want abrupt transitions.

17.5.2 The Cartoon Shading Vertex Shader Code
We now present the vertex shader for cartoon shading. The primary
task of the shader is merely to compute and set the texture coordinate
based on s LN . Observe that in the output structure, we have added

a data member to store the computed texture coordinate. Also note that
we still output the color of the vertex, though we don't modify it, and