The Last Line of Defense - Advanced Operating Systems and Kernel Applications - page 72

nisms of the operating system may be warranted
(Losocco, Smalley, Mucklebauer, Taylor, Turner,
& Farrell, 1998). Security weaknesses in host
operating systems are therefore a major concern
for the IT practitioner. If unwanted modification
of the host system can be prevented, then the at-
tack may be thwarted despite any weaknesses in
the network which allows the attacker to contact
the host machine.
There has been a distinction drawn in research
between application security and operating system
security. It has become increasingly clear, how-
ever, that such a distinction is academic and that
in practice malicious programs and the individuals
who create them make no such distinction. Mal-
ware such as Code Red exploited weaknesses in
both application and operating system security
(Staniford, Paxson, & Weaver, 2002). What is
required is an end-to-end solution, one that consid-
ers not only the distributed nature of the current
computing environment and the network, but the
close relationship between the application program
and the operating system (Howell & Kotz, 2000;
Saltzer, Reed, & Clark, 1981; Thompson, 1984).
Recent focus on the concept of endpoint security
represents an additional effort to understand and
mediate these risks (Kadrich, 2007).
This chapter will examine specific security
features of the host operating system in a descrip-
tive and exploratory manner. By understanding
the security controls available at the operating
system level and the security weaknesses in those
systems it is possible understand how to better
prevent attacks on these systems.
Operating systems and their underlying secu-
rity mechanisms are clearly a varied landscape
which over time can be quite fluid. This chapter
will focus on two common server operating
systems: Microsoft Windows Server 2003 and
Red Hat Enterprise Linux Server 5. Rather than
refer to specific versions of these operating sys-
tems, this chapter will use the terms Windows
and Linux to refer to Windows Server 2003 and
Red Hat Enterprise Linux Server 5 respectively.

(As this chapter goes to press, the next version
of Windows server operating system, Windows
Server 2008, is in Beta 3; the updates to security
features in this release considered relevant to
this discussion will be identified and evaluated
in this chapter.)

Security and operating System
Security architecture

Early computers operated in closed environments
with experienced and generally trusted personnel.
The introduction of time-sharing with multiple
concurrent processes required the consideration
of how to manage the resources of the computer
relative to the processes using the computer.
Initial computer security concerns had focused
on protecting executing tasks or processes from
each other. Lampson (1974) expanded on that with
a broad definition of protection that involved all
resources under control of the operating system:
memory, CPU registers, files (disk resources) and
password security. Lampson proposed protection
domains to define access rights and objects and
associated object access lists. Under this para-
digm, access to objects is enforced in relation to
the protection domain of the user. The evalua-
tion of operating system security requires a firm
definition of the somewhat nebulous concept
of the functionality and purpose of a computer
operating system. An expansive definition could
complicate qualitative comparisons with other
operating systems which have not been developed
with such a broad definition. For this reason, this
discussion will consider the operating system
the set of software which controls access to the
hardware resources (CPU, disk, peripherals) of
the server and will focus on specific operating
system functionality which is of central impor-
tance to security, in particular, authentication and
authorization (Lampson, 2004). The definitions
used here identify authenticating principals as
those which involve the process of determining
which security principal made a request, and