Mathematical Topics from 3D Graphics - 3D Math Primer for Graphics and Game Development - page 455

The bottlenecks are different in software rendering compared to hard-
ware rendering (notably, the bandwidth required for raw delivery of data to
the hardware), and in software, backface culling is usually done in 3D. The
basic idea with the 3D backfacing test is to determine whether the camera
position is on the front side of the triangle's plane. To make this deter-
mination quickly, we store a precomputed triangle normal. This is shown
in Figure 10.42, in which the backfacing triangles that could be culled are
drawn in gray. Notice that backface culling doesn't depend on whether a
triangle is inside or outside the view frustum. In fact, it doesn't depend
on the orientation of the camera at all—only the position of the camera
relative to the triangle is relevant.
To detect backfacing triangles in 3D, we need the normal of the plane
containing the triangle, and a vector from the eye to the triangle (any
point on the triangle will do—usually we just pick one vertex arbitrarily).
If these two vectors point in basically the same direction (their dot product
is greater than zero), then the triangle is backfacing. A variation on this
theme is to also precompute and store the d value of the plane equation (see
Section 9.5.1). Then the backfacing check can be done with one dot product
and scalar comparison. One quick note about a tempting optimization trick
that doesn't work: you might try to only use the z-component of the normal
of the triangle in camera (or clip) space. Although it might seem like if the
z value is positive, then the triangle faces away from the camera and could
be culled, an example where this isn't true is circled in Figure 10.42.

10.10.6 Rasterization, Shading, and Output

After clipping, the vertices are projected and mapped into the screen co-
ordinates of the output window, according to Equations (10.8)-(10.9). Of
course, these coordinates are floating-point coordinates, which are “contin-
uous” (see Section 1.1). But we typically render pixels, which are discrete.
So how do we know which pixels actually get drawn? Devising an algorithm
to answer this question is surprisingly complicated. If we answer wrong,
then gaps can appear between triangles. Rendering a pixel more than once
can be bad, too, if we are using alpha blending. In other words, we must
make sure that when we render a surface represented as triangles, every
pixel is rendered exactly once. Luckily, the graphics hardware takes care of
this for us and we don't have to sweat the details.
During rasterization, the rendering system may perform scissoring, which
rejects pixels that are outside of the rendering window. This is impossible
if the polygon is clipped to the edge of the screen, but it might be ad-
vantageous for performance reasons to skip that step. The guard band is a
technique that can be used to tune performance trade-offs between clipping
and scissoring (see Section 10.10.4) .