Aug 20

If you’ve already installed Linux, your partitions are already set up and configured with particular filesystems. You may decide you want to modify this configuration, though. Some changes are tedious to implement. For instance, changing from one filesystem to another requires you to back up, create a new filesystem, and restore your files. One exception to this rule is changing from ext2fs to ext3fs. If you switch filesystems, you may be able to use filesystem-creation options to improve the performance of the new filesystem. Other changes can also be done relatively painlessly. These include defragmenting a disk (that is, repositioning file contents so that they’re not spread out over the entire partition) and resizing partitions to give you space where you need it.

Creating a Filesystem for Optimal Performance

Most filesystems support a variety of options that may impact performance. For instance, large allocation blocks can improve performance by reducing fragmentation and the number of operations needed to retrieve an entire file. Some of these options can be set only at filesystem creation time, but some can be changed after the fact. Not all of these features are available in all filesystems. Across all Linux filesystems, important and popular performance-enhancing (or performance-degrading) options include:

Allocation Block Size  As noted in the earlier section, "Minimizing Space Consumption," small allocation blocks can facilitate more efficient use of disk space, but the cost is a small degradation in disk-access speed. Therefore, to improve performance slightly, you can increase your block size. This option is not easily changed after creating a filesystem. With ext2fs or ext3fs, you can use the -b block-size option to mke2fs; with XFS, the -b size=block-size option to mkfs.xfs does the job. For ext2fs and ext3fs, block-size must be 1024, 2048, or 4096; with XFS, the block size can theoretically be any power-of-two multiple of 512 bytes up to 64KB (65536 bytes), although in practice you can only mount a filesystem with block sizes up to 4KB or 8KB using common CPUs. ReiserFS and Linux’s version of JFS do not yet support adjusting this feature.

Journaling Options  All the journaling filesystems support various journal options. One common option is the location of the journal. By placing the journal on a separate physical disk from the main filesystem, you can improve performance (provided the target disk isn’t too sluggish itself). You can use the -J device=journal-device option in mke2fs or the -j journal-device option in mkreiserfs or mkfs.jfs to set this feature. Ext3fs also supports setting the journal size with the -J size=journal-size option, where journal-size is specified in megabytes and must be between 1,024 and 102,400 filesystem blocks. Specifying a too-small journal may degrade performance, but setting one too large may rob you of too much disk space. If in doubt, let mke2fs decide on the journal size itself.

Reserved Blocks  Ext2fs and ext3fs reserve a number of blocks for use by the superuser (or some other user you specify). The default value of 5 percent reserved space may be overkill on large partitions or on less critical partitions (such as /home). You can gain a bit more space by using the -m reserved-percentage option to mke2fs. Changing this percentage won’t affect actual disk performance, but it may gain you just a bit more available disk space. You can change this option after you create a filesystem by passing the same parameter that mke2fs accepts to the tune2fs program, as in tune2fs -m 1 /dev/hda4 to set the reserved blocks percentage to 1.

Check Interval  Ext2fs and ext3fs force a filesystem check after a specified number of mounts or a specified amount of time between mounts. The idea is to catch errors that might creep onto the filesystem due to random disk write errors or filesystem driver bugs. You can change these intervals by using the -c max-mount-counts and -i interval-between-checks options to tune2fs. For the latter option, you specify an interval in days, weeks, or months by providing a number followed by a d, w, or m, respectively. Altering the check interval won’t modify day-to-day performance, but it will change how frequently the computer performs a full disk check on startup. This disk check can be quite lengthy, even for ext3fs; it doesn’t restrict itself to recent transactions as recorded in the journal, as a forced check after a system crash does.

Directory Hash  ReiserFS uses a sorted directory structure to speed directory lookups, and mkreiserfs provides several options for the hash (a type of lookup algorithm) used for this purpose. You set this option with the -h hash option to mkreiserfs, where hash can be r5, rupasov, or tea. Some hashes may yield improved or degraded performance for specific applications. The Squid Web proxy documentation suggests using the rupasov hash, whereas the qmail documentation recommends r5, for instance. One problem with the r5 and rupasov hashes is that they can greatly slow file creation in directories with very many (a million or so) files. In fact, rupasov is very prone to such problems, and so should be avoided on most systems. The tea hash is much less subject to this problem, but it is also much slower than r5 for directories with more typical numbers of files. In general, you should use the default r5 hash unless you know you’ll be creating many files or the disk will be used by one performance-critical application, in which case checking the application’s documentation or doing a web search for advice may be worthwhile.

Inode Options  XFS enables you to set the inode size at filesystem creation time using the -i size=value option to mkfs.xfs. The minimum and default size is 256 bytes; the maximum is 2,048 bytes. (The inode size can’t exceed half the allocation block size, though.) One impact of the inode size option relates to small file access times; because XFS tries to store small files within the inode whenever possible, specifying a large inode enables storing larger files within the inode. Doing so will speed access to these files. Therefore, if a partition will store many small files (under 2KB), you may want to increase the inode size. Depending on the exact mix of file sizes, the result may save or waste disk space. If few files will be smaller than 2KB, there’s little point to increasing the inode size.

The default filesystem creation options usually yield acceptable performance. Modifying these options can help in some unusual cases, such as filesystems storing huge numbers of files or a computer that’s restarted frequently. I don’t recommend trying random changes to these options unless you intend to run tests to discover what works best for your purposes.

Converting Ext2fs to Ext3fs

One of the advantages of ext3fs over the other journaling filesystems is that it’s easy to turn an existing ext2 filesystem into an ext3 filesystem. You can do this using the tune2fs program and its -j option:

# tune2fs -j /dev/hda4

If the filesystem to which you add a journal is mounted when you make this change, tune2fs creates the journal as a regular file, called .journal, in the filesystem’s root directory. If the filesystem is unmounted when you run this command, the journal file doesn’t appear as a regular file. In either case, the filesystem is now an ext3 filesystem, and it can be used just as if you created it as an ext3 filesystem initially. If necessary, you may be able to access the filesystem as ext2fs (say, using a kernel that has no ext3fs support); however, some older kernels and non-Linux utilities may refuse to access it in this way, or they may provide merely read-only access.

On rare occasion, an ext3 filesystem’s journal may become so corrupted that it interferes with disk recovery operations. In such cases, you can convert the filesystem back into an ext2 filesystem using the debugfs tool:

# debugfs -w /dev/sda4
debugfs 1.32 (09-Nov-2002)
debugfs:  features -needs_recovery -has_journal
Filesystem features: dir_index filetype sparse_super
debugfs:  quit

After performing this operation, you should be able to use fsck.ext2 with its -f option, as described in the upcoming section, "Filesystem Check Options," to recover the filesystem. The newly-deactivated journal will cause fsck.ext2 to report errors even if the filesystem did not previously have them. If you like, you can then add the journal back by using tune2fs, as just described.

Warning

Don’t try to remove the journal from a mounted filesystem.

Defragmenting a Disk

Microsoft filesystems, such as the File Allocation Table (FAT) filesystem and the New Technology File System (NTFS), suffer greatly from disk fragmentation—the tendency of files to be broken up into many noncontiguous segments. Disk fragmentation degrades performance because the OS may need to move the disk head more frequently and over greater distances to read a fragmented file than to read a nonfragmented file.

Fortunately, Linux’s native filesystems are all far more resistant to fragmentation than are Windows filesystems. Therefore, most Linux users don’t bother defragmenting their disks. In fact, defragmentation tools for Linux are hard to come by. One that does exist is called defrag, but this package doesn’t ship with most distributions. Because it is an older tool, it won’t work with most modern ext2fs partitions, much less any of the journaling filesystems.

If you think your system may be suffering from fragmentation problems, you can at least discover how fragmented your ext2 or ext3 filesystems are by performing an fsck on them. You may need to force a check by using the -f parameter. This action will produce, among other things, a report on the fragmentation on the disk:

/dev/hda5: 45/8032 files (2.2% non-contiguous), 4170/32098 blocks

This report indicates that 2.2 percent of the files are noncontiguous (that is, fragmented). Such a small amount of fragmentation isn’t a problem. Unfortunately, the fsck tools for other journaling filesystems don’t return this information, so you have no indicator of fragmentation on these filesystems. If you truly believe that fragmentation has become a problem, you may be able to improve matters by backing up the partition, creating a fresh filesystem, and then restoring the files. This procedure is likely to take far longer than the time saved in disk accesses over the next several months or years, though, so I only recommend doing it if you want to change filesystem types or have some other reason (such as replacing a hard disk) to engage in this activity.

Note

As a general rule, fragmentation becomes a problem only if your disk is almost full. On a nearly full disk, Linux may have trouble locating a large enough block of free space to fit a file without fragmenting it. If you almost fill a disk and then delete files, the remaining files may or may not be fragmented, depending on which ones you deleted. For this reason, keeping your partitions from filling up is best. As a general rule, anything less than 80 to 90 percent full is fine from a fragmentation perspective.

Resizing Filesystems

All too frequently, you discover only after installing Linux that your partitions aren’t the optimum size. For instance, you might have too much room in /usr and not enough room in /home. Traditional fixes for this problem include using symbolic links to store some directories that are nominally on one partition on another partition; and backing up, repartitioning, and restoring data. In many cases, a simpler approach is to use a dynamic partition resizer. Fortunately, partition resizers exist for the most popular Linux filesystems, as well, so you can use these tools to manage your Linux installation.

Warning

Dynamic partition resizers are inherently dangerous. In the event of a power outage, system crash, or bug, they can do serious damage to a partition. You should always back up the data on any partition you resize. Also, you should never run a dynamic partition resizer on a partition that’s currently mounted. If necessary, boot a Linux emergency system to resize your partitions.

Resizing Ext2fs and Ext3fs

Several tools exist to resize ext2 and ext3 filesystems:

resize2fs  This program ships with the e2fsprogs package included with most distributions. The resize2fs program is fairly basic in terms of options. At a minimum, you pass it the device file associated with the partition, as in resize2fs /dev/hda4. This command resizes the filesystem on /dev/hda4 to match the size of the partition. You can also pass the partition size in allocation blocks, as in resize2fs /dev/hda4 256000 to resize a filesystem to 256,000 blocks. The resize2fs program doesn’t resize partitions, just the filesystems they contain. Therefore, you must use resize2fs in conjunction with fdisk to resize a partition and its filesystem. If you want to shrink a filesystem, you should do so first and then use fdisk to shrink the partition to match. If you want to grow a partition, you use fdisk first and then resize2fs. Because getting filesystem and partition sizes to match is tricky, it’s usually best to forgo resize2fs in favor of GNU Parted or PartitionMagic.

GNU Parted  This program provides both filesystem and partition resizing at once, so it’s easier to use than resize2fs. It’s described in more detail shortly, in "Using GNU Parted."

PartitionMagic  This commercial program from PowerQuest (http://www.powerquest.com) supports integrated filesystem and partition resizing operations of FAT, NTFS, ext2fs, ext3fs, and Linux swap partitions. PartitionMagic is easier to use than other ext2fs and ext3fs partition resizers, but it runs only from DOS or Windows. (The package ships with a DOS boot floppy image and a bootable CD-ROM, so it’s still useable on a Linux-only system.)

Resizing ReiserFS

Two tools are available for resizing ReiserFS:

resize_reiserfs  This tool is ReiserFS’s equivalent of the resize2fs program. Like resize2fs, resize_reiserfs resizes the filesystem, but not the partition in which it resides, so you must use this tool in conjunction with fdisk. If you only pass the program the partition identifier, it resizes the filesystem to fit the partition. If you pass an -s option and filesystem size, the program resizes the partition to the requested size, which you can specify in bytes, kilobytes, megabytes, or gigabytes (the last three options require K, M, or G suffixes, respectively). Alternatively, you can specify a change to the partition size by prefixing the size with a minus (-) or plus (+) sign. For instance, resize_reiserfs -s -500M /dev/sda5 reduces the size of the filesystem on /dev/sda5 by 500MB.

GNU Parted  According to its web page, this program supports ReiserFS as well as other filesystems. Unfortunately, as of version 1.6.4, this support is more theoretical than real, because it relies on libraries that aren’t present on most distributions, and that even a fresh build can’t find when everything’s installed according to directions. With luck, though, this support will improve in the future.

The ReiserFS resizing tools are not as mature as are those for resizing ext2 and ext3 filesystems. In fact, resize_reiserfs displays warnings about the software being beta.

Resizing XFS

XFS has long included a partition-resizing tool, xfs_growfs. As the name implies, this program is designed for increasing a filesystem’s size, not decreasing it. Unlike most partition-resizing tools, xfs_growfs is designed to work only on a mounted filesystem. The safest way to use it is to unmount the filesystem, delete the partition using fdisk, create a new partition in its place, mount the filesystem, and then call xfs_growfs:

# xfs_growfs /mount/point

As you might guess, /mount/point is the partition’s mount point. You may also add the -D size option to specify the filesystem size in allocation blocks. Various other options are also available, as described in the xfs_growfs man page.

Although GNU Parted’s web page doesn’t mention XFS support, the source code does include an XFS subdirectory. Parted refuses to work on XFS partitions, but this may change in the future.

Resizing JFS

JFS includes a rather unusual partition-resizing ability: It’s built into the kernel’s JFS driver. You can use this feature to increase, but not to decrease, the size of the filesystem. As with most other partition-resizing tools, you must modify the partition size first by using fdisk to delete the partition and then recreate it with a larger size. After you’ve done this, you should mount the partition as you normally do and then issue the following command:

# mount -o remount,resize /mount/point

This command resizes the filesystem mounted at /mount/point to occupy all the available space in its partition. No other partition-resizing tools are available for JFS, although there is a JFS subdirectory in the GNU Parted source code, suggesting that Parted may support JFS in the future.

Using GNU Parted

Because Parted is the most sophisticated open source partition resizer, it deserves more attention. You can pass it a series of commands directly or use it in an interactive mode. The latter is more likely to be helpful for normal one-time uses. Passing commands to Parted enables you to write scripts to help automate partition resizing. Typically, you launch Parted in interactive mode by typing the program’s name followed by the device on which you want to operate. You can then type commands to resize, create, delete, and otherwise manipulate partitions:

# parted /dev/sda
(parted) print
Disk geometry for /dev/scsi/host0/bus0/target5/lun0/disc: 0.000-96.000 megabytes
Disk label type: msdos
Minor    Start       End     Type      Filesystem  Flags
1          0.023     48.000  primary   ext2
2         48.000     96.000  primary   ext2
(parted) rm 2
(parted) resize 1 0.00 96.00
(parted) quit

This example deletes the second partition on the disk and resizes the first partition to fill all the available space. Unlike most Linux partition-management tools, Parted works in figures of megabytes. This fact can make translating Parted’s partition start and end points to and from the cylinder boundaries upon which fdisk and other tools work tricky. Table 1 summarizes some of the most common and important Parted commands. Although many commands nominally require arguments, in practice they don’t; instead, they prompt for the required information when Parted is run in interactive mode. The part-type code is p for primary partitions, e for extended partitions, and l for logical partitions.

Table 1: Common Parted Commands

Parted Command

Effect

help [command]

Displays information on how to use a command. If the command option is omitted, it displays a summary of all commands.

mkfs partn fstype

Creates a filesystem of fstype on partition number partn.

mkpart part-type [fstype] start end

Creates a partition, giving it the partition type code part-type, with start and end as its start and end points.

mkpartfs part-type fstype start end

Works like mkpart, but also creates a filesystem in the new partition.

move partn start end

Moves the partition to a new location on the disk.

print [partn]

Displays the partition table or, if partn is specified, more detailed information on the partition.

rescue start end

Attempts to recover a partition that was deleted.

resize partn start end

Resizes the specified partition to the specified size.

rm partn

Deletes the specified partition.

select device

Begins editing a new disk device.

 

Resizing Partitions

Most of the filesystem-resizing tools require that you modify the partition using fdisk. (GNU Parted and PartitionMagic are exceptions to this rule.) Precisely how you modify the filesystem’s carrier partition depends on whether you’ll be shrinking or growing the partition. The simplest case is growing a partition. When doing this, you should follow these steps:

  1. Launch fdisk on the disk in question.

  2. Type d in fdisk to delete the partition you want to grow. You’ll be asked for the partition number.

  3. Type n in fdisk to create a new partition in place of the old one. You’ll be asked for the partition number and the start and end cylinders. The start cylinder must be the same as it was originally, and of course the end cylinder should be larger than the original.

  4. Type w in fdisk to write your changes to disk and exit.

  5. Follow the procedure for your filesystem-resizing tool to increase the filesystem size.

Of course, in order to grow a partition, there must be free space on the disk into which to expand the partition. This normally means that you’ll have already deleted or shrunk a partition that follows the one you want to expand. If you want to expand a filesystem into space that’s before it on the disk, your job is much harder. It’s possible to expand the carrier partition as just described, but specifying an earlier starting point, and then use dd to copy a filesystem from later in the new partition to earlier in the partition. This task is tricky, though, because you must compute precisely how far into the newly expanded partition the existing filesystem begins. An error can easily wipe out all your data. Thus, I don’t recommend attempting this task; instead, try creating a new filesystem in the earlier space and mount it at some convenient place in your directory tree. If the empty space is larger than the partition you want to move, you can create a new partition, move the original, verify that the copied partition is intact, delete the original partition, and expand the copied partition and the filesystem it contains.

In order to reduce the size of the filesystem, you must match the size of the partition to the filesystem, which can be a tricky task. Fortunately, there is a procedure that can make this task less error-prone:

  1. Shrink the filesystem, using your filesystem-resizing tool, to a value that’s smaller than you intend. For instance, if you want to shrink a 700MB partition to 500MB, shrink it to 400MB.

  2. Use fdisk to resize the partition to the target size, such as 500MB. This target size should be larger than the filesystem by a wide enough margin to be comfortable.

  3. Use the partition-resizing tool to expand the filesystem into the extra space on the partition, filling it exactly.

As with increasing the size of the filesystem, the start point of the filesystem must remain untouched. When moving space between filesystems, this requirement can create an awkward situation: You can shrink an earlier partition, but expanding the next partition into the freed space is risky.

Tagged with:
Aug 04

A regular part of a database administrator’s job is to do preventative maintenance, as well as to repair things when they go wrong. In spite of the best efforts, data errors can occur, such as in the case of a power failure that interrupts a write. Usually you can correct these fairly painlessly.

There are four main tasks involved in checking and repairing:

  • Optimizing tables

  • Analyzing tables (analyzes and stores the key distribution for MyISAM and BDB tables)

  • Checking tables (checks the tables for errors, and, for MyISAM tables, updates the key statistics)

  • Repairing tables (repairs corrupted MyISAM tables)

Optimizing Tables

Tables that contain BLOB and VARCHAR fields will, over time, become less optimized. Because these field types are variable in length, when records are updated, inserted, or deleted, they will not always take the same amount of space, the records will start to become fragmented, and empty spaces will remain. Just like with a fragmented disk, this situation will slow performance, so to keep MySQL in tiptop shape, you should regularly defragment it. The way to do this is to optimize the table, which can be done in a number of ways. There’s the OPTIMIZE TABLE statement, the mysqlcheck utility (if the server is running), or the myisamchk utility (if the server is not running or there is no interaction with the table).

Optimizing currently works only with MyISAM and partially with BDB tables. With MyISAM tables, optimizing does the following:

  • Defragments tables where rows are split or have been deleted

  • Sorts the indexes if they have not been already

  • Updates the index statistics if they have not been already

With BDB tables, optimizing analyzes the key distribution (the same as ANALYZE TABLE; see the "Analyzing Tables with ANALYZE TABLE" section later in this chapter).

Optimizing Tables with the OPTIMIZE Statement

The OPTIMIZE statement is a SQL statement used when connected to a MySQL database. The syntax is as follows:

OPTIMIZE TABLE tablename

You can also optimize many tables at once, separating each with a comma:

mysql> OPTIMIZE TABLE customer,sales;
+------------------+----------+----------+-----------------------------+
| Table            | Op       | Msg_type | Msg_text                    |
+------------------+----------+----------+-----------------------------+
| firstdb.customer | optimize | status   | Table is already up to date |
| firstdb.sales    | optimize | status   | OK                          |
+------------------+----------+----------+-----------------------------+
2 rows in set (0.02 sec)

The customer table in this instance has already been optimized.

Optimizing Tables with mysqlcheck

mysqlcheck is a command-line utility that can perform numerous checking and repairing tasks besides optimization. A full description of all the mysqlcheck features follows later in the chapter in the section titled "Using mysqlcheck." The server must be running for you to use mysqlcheck. To optimize the customer table from the firstdb database, use the -o mysqlcheck option, as follows:

% mysqlcheck -o firstdb customer  -uroot -pg00r002b
firstdb.customer           Table is already up to date

mysqlcheck allows you to optimize more than one table at a time by listing all the tables after the database name:

% mysqlcheck -o firstdb customer sales  -uroot -pg00r002b
firstdb.customer           Table is already up to date
firstdb.sales              Table is already up to date

You could also optimize the entire database by leaving out any table references, with the following:

% mysqlcheck -o firstdb -uroot -pg00r002b

Optimizing Tables with myisamchk

Finally, you can use the myisamchk command-line utility when the server is down or not interacting with the server. (Flush the tables before running this statement if the server is up with mysqladmin flush-tables. You still need to make sure the server is not interacting with the table, though, or else corruption may result.) This is the oldest way of checking tables. You have to run myisamchk from the exact location of the table or specify the path leading to the table. A full description of all the myisamchk features follows later in this chapter in the section titled "Using myisamchk."

The equivalent of an optimize statement is as follows:

myisamchk --quick --check-only-changed --sort-index --analyze tablename

or as follows:

myisamchk -q -C -S -a tablename

For example:

% myisamchk --quick --check-only-changed --sort-index --analyze customer
- check key delete-chain
- check record delete-chain
- Sorting index for MyISAM-table 'customer'

The -r option repairs the table, but also eliminates wasted space:

% myisamchk -r sales   
- recovering (with sort) MyISAM-table 'sales'
Data records: 8
- Fixing index 1
- Fixing index 2

If you do not specify the path to the table index file, and you’re not in the right directory, you’ll get the following error:

% myisamchk -r customer
myisamchk: error: File 'customer' doesn't exist

Specifying the full path to the .MYI file corrects this:

% myisamchk -r /usr/local/mysql/data/firstdb/customer
- recovering (with keycache) MyISAM-table '/usr/local/mysql/data/firstdb/customer'
Data records: 0

Warning

Tables are locked during the optimization, so don’t run this during peak hours! Also, make sure you have a reasonable amount of free space on the system when running OPTIMIZE TABLE. If you try to run it when your system has almost or already run out of disk space, MySQL may not be able to complete the optimization, leaving your table unusable.

Optimizing is an important part of any administrative routine for databases that contain MyISAM tables and should be performed regularly.

Analyzing Tables

Analyzing tables improves performance by updating the index information for a table so that MySQL can make a better decision on how to join tables. The distribution of the various index elements is stored for later usage. (Analyzing currently only works with MyISAM and BDB tables.)

There are three ways to analyze a table:

  • When connected to MySQL, with the ANALYZE TABLE statement

  • With the mysqlcheck command-line utility

  • With the myisamcheck command-line utility

Regular analysis of tables can help performance and should be a regular part of any maintenance routine.

Analyzing Tables with ANALYZE TABLE

ANALYZE TABLE is a statement used when connected to a database on the server. The syntax is as follows:

ANALYZE TABLE tablename

For example:

mysql> ANALYZE TABLE sales;
+---------------+---------+----------+----------+
| Table         | Op      | Msg_type | Msg_text |
+---------------+---------+----------+----------+
| firstdb.sales | analyze | status   | OK       |
+---------------+---------+----------+----------+
1 row in set (0.00 sec)

The Msg_type (message type) is one of status, error, info, or warning. Here’s what would happen if the index file was missing altogether and you tried to analyze the table:

mysql> ANALYZE TABLE zz;
+------------+---------+----------+----------------------------------+
| Table      | Op      | Msg_type | Msg_text                         |
+------------+---------+----------+----------------------------------+
| firstdb.zz | analyze | error    | Table 'firstdb.zz' doesn't exist |
+------------+---------+----------+----------------------------------+
1 row in set (0.00 sec)

The table will only be analyzed again if it has changed since the last time it was analyzed:

mysql> ANALYZE TABLE sales;
+---------------+---------+----------+-----------------------------+
| Table         | Op      | Msg_type | Msg_text                    |
+---------------+---------+----------+-----------------------------+
| firstdb.sales | analyze | status   | Table is already up to date |
+---------------+---------+----------+-----------------------------+
1 row in set (0.00 sec)
Analyzing Tables with mysqlcheck

The mysqlcheck command-line utility is discussed fully later in this chapter in the section titled "Using mysqlcheck." The server needs to be running for you to use mysqlcheck, and it works only with MyISAM tables. To use it to analyze tables, you use the -a option:

% mysqlcheck -a firstdb sales -uroot -pg00r002b
firstdb.sales                                      OK

You can also analyze more than one table from a database by listing the tables after the database name:

% mysqlcheck -a firstdb sales customer -uroot -pg00r002b
firstdb.sales                 Table is already up to date
firstdb.customer              Table is already up to date

If you tried to analyze a table that does not support analysis (such as an InnoDB table), no harm would be done and the operation would just fail. For example:

% mysqlcheck -a firstdb innotest -uroot -pg00r002b
firstdb.innotest
error    : The handler for the table doesn't support check/repair

You could also analyze all tables in the database with by leaving out any table names:

% mysqlcheck -a firstdb innotest -uroot -pg00r002b

Analyzing Tables with myisamchk

The myisamchk command-line utility is discussed fully later in this chapter in the section titled "Using myisamchk." The server should either not be running, or you must be sure that there is no interaction with the tables with which you’re working. If the –skip-external-locking option is not on, you can safely use myisamchk to analyze tables, even if the server is running. The tables will be locked, affecting access, but there will be no erroneous reports. If –skip-external-locking is used, you’ll need to flush the tables before starting the analysis (with mysqladmin flush-tables) and ensure that there is no access. You may get invalid results if mysqld or anything else accesses the table while myisamchk is running. To analyze tables, use the -a option:

% myisamchk -a /usr/local/mysql/data/firstdb/sales   
Checking MyISAM file: /usr/local/mysql/data/firstdb/sales
Data records:       9   Deleted blocks:       0
- check file-size
- check key delete-chain
- check record delete-chain
- check index reference
- check data record references index: 1
- check data record references index: 2

Checking Tables

Errors can occur when the indexes are not synchronized with the data. System crashes or power failures can all cause situations where the tables have become corrupted. Corruption of the data is fairly rare; in most cases, the corruption is of the index files. These can be hard to spot, though you may notice information being returned slowly or data not being found that should be there. Checking tables should be the first thing you do when you suspect an error. Some of the symptoms of corrupted tables include errors such as the following:

  • Unexpected end of file.

  • Record file is crashed.

  • tablename.frm is locked against change.

  • Can’t find file tablename.MYI (Errcode: ###).

  • Got error ### from table handler. The perror utility gives more information about the error number. Just run perror (which is stored in the same directory as the other binaries such as mysqladmin) and the error number. For example:

    % perror 126
    126 = Index file is crashed / Wrong file format

Some of the other more common errors include:

126 = Index file is crashed / Wrong file format
127 = Record-file is crashed
132 = Old database file
134 = Record was already deleted (or record file crashed)
135 = No more room in record file
136 = No more room in index file
141 = Duplicate unique key or constraint on write or update
144 = Table is crashed and last repair failed
145 = Table was marked as crashed and should be repaired

Once connected to the MySQL server, you can issue a CHECK TABLE command, make use of the mysqlcheck utility (when the server is running), or use the myisamchk utility when the server has been stopped. Checking updates the index statistics and checks for errors.

If any errors are found, the table will need to be repaired (see the "Repairing Tables" section later in this chapter). Serious errors mark the table as corrupt, in which case it can no longer be used until it is repaired.

Tip

Always check tables after a power failure or a system crash. You can usually fix any corruption that has occurred before users notice any problems.

Checking Tables with CHECK TABLES

The syntax for CHECK TABLE is as follows:

CHECK TABLE tablename [option]

For example:

mysql> CHECK TABLE customer;
+------------------+-------+----------+----------+
| Table            | Op    | Msg_type | Msg_text |
+------------------+-------+----------+----------+
| firstdb.customer | check | status   | OK       |
+------------------+-------+----------+----------+
1 row in set (0.01 sec)

CHECK can check only MyISAM and InnoDB tables.

There are five options depending on the level of checking you want to do, as shown in Table 4

Table 4: CHECK TABLE Options

Option

Description

QUICK

This is the quickest check and does not scan the rows to check for wrong links.

FAST

Only checks tables that haven’t been closed properly.

CHANGED

Only checks tables that haven’t been closed properly or have been changed since last check.

MEDIUM

The default option. It scans rows to check that deleted links are correct. It also calculates a key checksum for the rows and verifies this with a calculated checksum for the keys.

EXTENDED

This is the slowest method, but it checks the table for complete consistency by doing a full key lookup for every index associated with each row.

 

The QUICK option is useful for checking tables where you don’t suspect any errors.

If an error or warning is returned, you should try and repair the table.

You can check more than one table at a time by listing the tables one after another,

for example:

mysql> CHECK TABLE sales,customer;
+------------------+-------+----------+----------+
| Table            | Op    | Msg_type | Msg_text |
+------------------+-------+----------+----------+
| firstdb.sales    | check | status   | OK       |
| firstdb.customer | check | status   | OK       |
+------------------+-------+----------+----------+
2 rows in set (0.01 sec)

Checking Tables with mysqlcheck

The mysqlcheck command-line utility can be used when the server is running and works only with MyISAM tables. It is described fully later in this chapter in the section titled "Using mysqlcheck." Table 5 lists the options.

The syntax is as follows:

mysqlcheck [options] databasename tablename[s]

For example:

% mysqlcheck -c firstdb customer -uroot -pg00r002b
firstdb.customer                               OK

Table 5: The mysqlcheck Options That Apply to Table Checking

Option

Description

–auto-repair

Used in conjunction with one of the check options, it will automatically beginto repair corrupted tables after the checks have completed.

-c, –check

Checks tables.

-C, –check-only-changed

Checks tables that have changed since the last check or were not closed properly.

-F, –fast

Checks tables that haven’t been closed properly.

-e, –extended

This is the slowest form for checking, but it will make sure the table is completely consistent. You can also use this option to repair, though itisusually not necessary.

-m, –medium-check

This is much faster than the extended check, and it finds the vast majority oferrors.

-q, –quick

The fastest check, this does not check table rows when checking. When repairing, it only repairs the index tree.

 

You can check more than one table by listing a number of tables after the database name:

% mysqlcheck -c firstdb sales customer -uroot -pg00r002b
firstdb.sales                                      OK
firstdb.customer                                   OK

You can check all tables in the database by just specifying the name of the database.

% mysqlcheck -c firstdb -uroot -pg00r002b

Checking Tables with myisamchk

When the server is shut down or there is no interaction with the tables you’re checking, you can use the myisamchk command-line option (described fully in the "Using myisamchk" section later in this chapter). If the –skip-external-locking option is not on, you can safely use myisamchk to check tables, even if the server is running. The tables will be locked, affecting access, but there will be no erroneous reports. If –skip-external-locking is used, you’ll need to flush the tables before starting the check (with mysqladmin flush-tables) and ensure that there is no access. You may get wrong results (with tables being marked as corrupted even when they are not) if mysqld or anything else accesses the table while myisamchk is running.

The syntax is as follows:

mysiamchk [options] tablename

The equivalent to the CHECK TABLE statement is the medium option:

myisamchk -m table_name 

The default for myisamchk is the ordinary check option (-c). There is also the fast check (-F), which only checks tables that haven’t been closed properly. This is not the same as the lowercase -f option, which is the force option, meaning the check continues even if errors occur. There is also the medium check (-m), slightly slower and more complete. The most extreme option is the -e option (that performs an extended check), which is the most thorough and slowest option. It’s also usually a sign of desperation; use this only when all other options have failed. Increasing the key_buffer_size variable can speed up the extended check (if you have enough memory). See Table 6 for the checking options.

Table 6: myisamchk Checking Options

Option

Description

-c, –check

Ordinary check and the default option.

-e, –extend-check

Slowest and most thorough form of check. If you are using –extended-check and have much memory, you should increase the value of key_buffer_size a lot!

-F, –fast

Fast check, which only checks tables that haven’t been closed properly.

-C, –check-only-changed

Checks only the tables that have been changed since the last check.

-f, –force

This runs the repair option if any errors are found in the table.

-i, –information

Displays statistics about the table that is checked.

-m, –medium-check

Medium check, faster than an extended check, and good enough for most cases.

-U, –update-state

Keeps information about when the table was checked and whether the table has crashed, which is useful for the -C option. Should not be used when the table is being used and the –skip-external-locking option is active.

-T, –read-only

Does not mark the table as checked (useful for running myisamchk when the server is active and the –skip-external-locking option is in use).

 

The following is a sample myisamchk output when errors are found:

% myisamchk largetable.MYI
Checking MyISAM file: Hits.MYI
Data records: 2960032   Deleted blocks:       0
myisamchk: warning: 1 clients is using or hasn't closed the table
properly
- check file-size
myisamchk: warning: Size of datafile is: 469968400 Should be: 469909252
- check key delete-chain
- check record delete-chain
- check index reference
- check data record references index: 1
- check data record references index: 2
- check data record references index: 3
myisamchk: error: Found 2959989 keys of 2960032
- check record links
myisamchk: error: Record-count is not ok; is 2960394 Should be: 2960032
myisamchk: warning: Found    2960394 parts   Should be: 2960032 parts

Repairing Tables

If you have checked the tables and errors have been found, you’ll need to repair them. There are various repair options available, depending on which method you use, but you may not have success. If the disk has failed, or if none of them work, the only option is to restore from your backup. Repairing a table can take up significant resources, both disk and memory:

  • Generally, repairing a table takes up twice as much disk space as the original data file (on the same disk). A quick repair (see the options in the following sections) is an exception because the data file is not modified.

  • Some space for the new index file (on the same disk as the original). The old index is deleted at the start, so this is usually not significant, but it will be if the disk is close to full.

  • With the standard and –sort-recover options, a sort buffer is created. This takes up the following amount of space (largest_key + row_pointer_length) * number_of_rows * 2. You can move some or all of this to memory (and increase the speed of the process) by increasing the size of the mysqld variable sort_buffer_size if you have the available memory. Otherwise, it is created as specified by the TMPDIR environment variable or the -t myisamchk option.

  • Memory usage is determined by the mysqld variables or the options set in the myisamchk command line (see the section titled "Using myisamchk").

If the error is caused by the table running out of space and the table type is InnoDB, you will have to enlarge the InnoDB tablespace. MyISAM tables have a huge theoretical size limit (eight million terabytes), but by default pointers are only allocated for 4GB. If the table reaches this limit, you can extend it by using the MAX_ROWS and AVG_ROW_LENGTH ALTER TABLE parameters. To prepare the table called limited for great things (currently it only has three records), you use the following:

mysql> ALTER TABLE limited MAX_ROWS=999999999999 AVG_ROW_LENGTH=100;
Query OK, 3 rows affected (0.28 sec)
Records: 3  Duplicates: 0  Warnings: 0

This allocates pointers for a much greater number of records. The AVG_ROW_LENGTH is used when BLOB and TEXT fields are present, and it gives MySQL an idea of the average size of a record, which it can then use for optimization purposes.

Repairing Non-MyISAM Table Types

The three methods of repairing discussed in the following sections work only with MyISAM tables. Some of the options have been reported to work occasionally with BDB tables, but they were not designed for this. Currently, the only way to repair corrupted BDB and InnoDB tables is to restore from backup.

Repairing Tables with REPAIR TABLE

You can run the REPAIR TABLE statement when connected to the MySQL server. It currently only works with MyISAM tables. See Table 7 for the options.

The syntax is as follows:

Table 10: Available REPAIR TABLE Options

Option

Description

QUICK

Fastest repair because the data file is not modified. It uses much less disk space as well because the data file is not modified.

EXTENDED

Attempts to recover every possible row from the data file. This option should not be used unless as a last resort because it may produce garbage rows.

USE_FRM

This is the option to use if the .MYI file is missing or has a corrupted header. It will rebuild the indexes from the definitions found in the .frm table definition file.

 

The following is an example of using REPAIR in the case of a missing .MYI file. Let’s delete the .MYI file of an existing table, t4.

% ls -l t4.*
-rw-rw----  1 mysql  mysql    10 Jun 14 02:00 t4.MYD
-rw-rw----  1 mysql  mysql  4096 Jun 14 02:00 t4.MYI
-rw-rw----  1 mysql  mysql  8550 Jun  8 10:46 t4.frm
% rm t4.MYI
% mysql -uguru2b -pg00r002b firstdb

A normal REPAIR does not work:

mysql> REPAIR TABLE t4;
+------------+--------+----------+--------------------------------------+
| Table      | Op     | Msg_type | Msg_text                             |
+------------+--------+----------+--------------------------------------+
| firstdb.t4 | repair | error    | Can't find file: 't4.MYD' (errno: 2) |
+------------+--------+----------+--------------------------------------+
1 row in set (0.47 sec)

The current error message (4.0.3) indicates that the .MYD file cannot be found, when it’s actually the .MYI file that’s missing. The error message is likely to have been clarified by the time you read this. To repair the table in this instance, you need to use the USE_FRM option, which, as the name suggests, uses the .frm definition file to re-create the .MYI index file:

mysql> REPAIR TABLE t4 USE_FRM;
+------------+--------+----------+------------------------------------+
| Table      | Op     | Msg_type | Msg_text                           |
+------------+--------+----------+------------------------------------+
| firstdb.t4 | repair | warning  | Number of rows changed from 0 to 2 |
| firstdb.t4 | repair | status   | OK                                 |
+------------+--------+----------+------------------------------------+
2 rows in set (0.46 sec)

Repairing Tables with mysqlcheck

The mysqlcheck command-line utility is used while the server is still running and works only with MyISAM tables. It is described fully later in this chapter in the "Using mysqlcheck" section. To repair tables, you use the -r option:

% mysqlcheck -r firstdb customer -uroot -pg00r002b
firstdb.customer                                   OK

You can repair multiple tables at the same time by listing the table names after the

database name:

% mysqlcheck -r firstdb customer sales -uroot -pg00r002b
firstdb.customer                                   OK
firstdb.sales                                      OK

If for some reason all tables in a database are corrupt, you can repair them all by just supplying the database name:

% mysqlcheck -r firstdb -uroot -pg00r002b
Repairing Tables with myisamchk

You can use the myisamchk command-line utility (described fully later in this chapter in the "Using myisamchk" section) repair tables (see Table 10.8).

Table 8: Repairing Tables with myisamchk

Option

Description

-D #, –data-file-length=#

Specifies the maximum length of the data file when re-creating it.

-e, –extend-check

Attempts to recover every possible row from the data file. This option should not be used unless as a last resort because it may produce garbage rows.

-f, –force

Overwrites old temporary files (that have an extension of .TMD) instead of aborting if it encounters a preexisting one.

-k #, keys-used=#

Specifies which keys to use, which can make the process faster. Each binary bit stands for one key starting at 0 for the first key.

-r, –recover

Repairs most corruption and should be the first option attempted. You can increase thesort_buffer_size to make the recovery go more quickly if you have the memory. This option will not recover from the rare form of corruption where a unique key is not unique.

-o, –safe-recover

A more thorough, yet slower repair option than -r that should be used only if -r fails. This reads through all rows and rebuilds the indexes based on the rows. It also uses less disk space than -r because a sort buffer is not created. You can increase the size of key_ buffer_size to improve repair speed.

-n, –sort-recover

Forces MySQL to use sorting to resolve the indexes, even if the resulting temporary files are large.

–character-sets-dir=…

The directory containing the character sets.

–set-character-set=name

Specifies a new character set for the index.

-t, –tmpdir=path

Specifies a new path for storing temporary files if you don’t want to use whatever the TMPDIR environment variable specifies.

-q, –quick

Fastest repair because the data file is not modified. Specifying the q twice (-q -q) will modify the data file if there are duplicate keys. Uses much less disk space as well because the data file is not modified.

-u, –unpack

Unpacks a file that has been packed with the myisampack utility.

 

The server should either not be running, or you must be sure there is no interaction with the tables with which you’re working, such as when you start MySQL with the –skip-external-locking option. If the –skip-external-locking option is not on, you can only safely use myisamchk to repair tables if you are sure there will be no simultaneous access. Whether –skip-external-locking is used or not, you’ll need to flush the tables before starting the repair (with mysqladmin flush-tables) and ensure that there is no access. You may get wrong results (with tables being marked as corrupted even when they are not) if mysqld or anything else accesses the table while myisamchk is running.

The syntax is as follows:

myisamchk [options] [tablenames]

You must run myisamchk from the directory containing the .MYI files or supply the path. The following examples show a repair in action, with MySQL deciding whether to use sorting or a keycache:

% myisamchk -r customer
- recovering (with keycache) MyISAM-table 'customer.MYI'
Data records: 0
% myisamchk -r sales   
- recovering (with sort) MyISAM-table 'sales.MYI'
Data records: 9
- Fixing index 1
- Fixing index 2

If you have lots of memory, besides increasing the size of sort_buffer_size and key_buffer_size as described previously, you can also set some other variables to make myisamchk perform more snappily. See the full myisamchk description later in this chapter in the "Using myisamchk" section.

Using mysqlcheck

The mysqlcheck utility is a boon to more recent users of MySQL because, beforehand, much of the repairing and checking functionality could only be used when the server was shut down. Luckily this limitation is a thing of the past with the mysqlcheck utility.

mysqlcheck uses the CHECK, REPAIR, ANALYZE, and OPTIMIZE statements to perform these tasks from the command line, which is useful for automated maintenance of your databases (see Table 9).

The syntax is as follows:

mysqlcheck [options] databasename [tablenames]

or as follows:

mysqlcheck [options] --databases databasename1 [databasename2 databasename 3 ...]

or as follows:


Table 9: Mysqlcheck Options

Option

Description

-A, –all-databases

Checks all available databases.

-1, –all-in-1

Combines queries for tables into one query per database (instead of one per table). Tables are in a comma-separated list.

-a, –analyze

Analyzes the listed tables.

–auto-repair

Automatically repairs tables if they are corrupted (after all tables in the query have been checked).

-#, –debug=…

Outputs a debug log.

–character-sets-dir=…

This specifies the directory where the character sets are.

-c, –check

Checks tables.

-C, –check-only-changed

Checks tables that have changed since the last check or were not closed properly.

–compress

Uses compression in the client/server protocol.

-?, –help

Displays the help message and exits.

-B, –databases

Lists a number of databases to check (all tables in the databases are checked).

–default-character-set=…

Sets the default character set.

-F, –fast

Checks tables that haven’t been closed properly.

-f, –force

Forces the process to continue even if it encounters an error.

-e, –extended

This is the slowest form for checking but will make sure the table is completely consistent. You can also use this option to repair, though it is usually not necessary.

-h, –host=…

Hostname to which to connect.

-m, –medium-check

Much faster than the extended check and finds the vast majority of errors.

-o, –optimize

Optimizes the tables.

-p, –password[=...]

The password with which to connect.

-P, –port=…

The port to use for connecting.

-q, –quick

The fastest check, this does not check table rows when checking. When repairing, it only repairs the index tree.

-r, –repair

Repairs most errors, except unique keys that somehow contain duplicates.

-s, –silent

Displays no output except for error messages.

-S, –socket=…

Specifies the socket file to use when connecting.

–tables

List of tables to check. With the -B option, this will take precedence.

-u, –user=#

Specifies the user to connect as.

-v, –verbose

Prints lots of output about the process.

-V, –version

Displays the version information and exits.

The mysqlcheck utility also has a feature that allows it to be run in different ways without specifying the options. By simply creating a copy of mysqlcheck with one of the following names, it will take that default behavior:

  • mysqlrepair: The default option is -r.

  • mysqlanalyze: The default option is -a.

  • mysqloptimize: The default option is -o.

The default option when it is named mysqlcheck is -c. All of these renamed files can have the full mysqlcheck functionality—it’s just that their default behavior is changed.

Using myisamchk

The myisamchk utility is the older utility, available since the early days of MySQL. It is also used to analyze, check, and repair tables, but care needs to be taken if you want to use it when the server is running. Table 10 describes the general myisamchk options, Table 11 describes the check options, Table 12 describes the repair options, and Table 13 describes other options.

The server should either not be running, or you must be sure there is no interaction with the tables with which you’re working, such as when you start MySQL with the –skip-external-locking option. If the –skip-external-locking option is not on, you can only safely use myisamchk to repair tables if you are sure there will be no simultaneous access. Whether or not –skip-external-locking is used, you’ll need to flush the tables before starting the repair (with mysqladmin flush-tables) and ensure that there is no access.

I suggest you rather use one of the other options if the server is running.

The syntax is as follows:

myisamchk [options] tablename[s]

You must run myisamchk from the directory where the .MYI index files are located unless you specify the path to them; otherwise you’ll get the following error:

% myisamchk -r sales.MYI
myisamchk: error: File 'sales.MYI' doesn't exist

Specifying the path solves the problem:

% myisamchk -r /usr/local/mysql/data/firstdb/sales.MYI
- recovering (with sort) MyISAM-table '/usr/local/mysql/data/firstdb/sales.MYI'
Data records: 9
- Fixing index 1
- Fixing index 2

The table name can be specified with or without the .MYI extension.

% myisamchk -r sales
- recovering (with sort) MyISAM-table 'sales'
Data records: 9
- Fixing index 1
- Fixing index 2
% myisamchk -r sales.MYI
- recovering (with sort) MyISAM-table 'sales.MYI'
Data records: 9
- Fixing index 1
- Fixing index 2

Warning

A common mistake is to try run myisamchk on an .MYD data file. Always use the .MYI index file!

You can use wildcard character to search all tables in a database directory (*.MYI) or even all tables in all databases:

Table 10: General myisamchk Options

Option

Description

-#, –debug=debug_options

Outputs a debug log. A common debug_option string is d:t:o,filename.

-?, –help

Displays a help message and exits.

-O var=option, –set-variable var=option

Sets the value of a variable. The possible variables and their default values for myisamchk can be examined with myisamchk –help.

-s, –silent

Only outputs error messages. A second s can be used to completely silence myisamchk.

-v, –verbose

Displays more information than usual. As with silent, multiple v’s can be used to output more information (-vv or -vvv).

-V, –version

Displays the myisamchk version details and exits.

-w, –wait

If the table is locked, -w will wait for the table to be unlocked rather than exiting with an error. If mysqld was running with the –skip-external-locking option, the table can only be locked by another myisamchk command.

 

By running myisamchk –help, besides the general options, you can see what variables you can change with the -O option and what the current settings are:

% myisamchk --help
..
Possible variables for option --set-variable (-O) are:
key_buffer_size       current value: 520192
myisam_block_size     current value: 1024
read_buffer_size      current value: 262136
write_buffer_size     current value: 262136
sort_buffer_size      current value: 2097144
sort_key_blocks       current value: 16
decode_bits           current value: 9
ft_min_word_len       current value: 4
ft_max_word_len       current value: 254
ft_max_word_len_for_sort  current value: 20

The space allocated by the key_buffer_size is used when doing an extended check or when indexes are inserted one row at a time (using the safe-recover option). sort_buffer_size is used in the default repair, when indexes are sorted in the repair.

To achieve a faster repair, set the sort_buffer_size to about one-quarter of the total available memory. Only one of the two variables is used at a time, so you don’t need to worry about running out of memory by making both values large.

Note

Inside the my.cnf (or my.ini) file, there are separate sections for mysqld and myisamchk. You can quite easily set the sort_buffer_size to a high value for repairing, and keep it lower if your system has other requirements for day-to-day running.


Table 11: myisamchk Check Options

Option

Description

-c, –check

Ordinary check and the default option.

-e, –extend-check

Slowest and most thorough form of check. If you are using –extended-check and don’t have much memory, you should increase the value of key_buffer_size a lot!

-F, –fast

Fast check that only checks tables that haven’t been closed properly.

-C, –check-only-changed

Checks only the tables that have been changed since the last check.

-f, –force

This runs the repair option if any errors are found in the table.

-i, –information

Displays statistics about the table that is checked.

-m, –medium-check

Medium check, faster than an extended check and good enough for most cases.

-U, –update-state

Keeps information about when the table was checked and whether the table has crashed, which is useful for the -C option. Should not be used when the table is being used and the –skip-external-locking option is active.

-T, –read-only

Does not mark the table as checked (useful for running myisamchk when the server is active and the –skip-external-locking option is in use).

Table 12: myisamchk Repair Options

Option

Description

-D #, –data-file-length=#

Specifies the maximum length of the data file when re-creating it.

-e, –extend-check

Attempts to recover every possible row from the data file. This option should not be used unless as a last resort because it may produce garbage rows.

-f, –force

Overwrites old temporary files (that have an extension of .TMD) instead of aborting if it encounters a preexisting one.

-k #, keys-used=#

Specifies the keys to use, which can make the process faster. Each binary bit stands for one key starting at 0 for the first key (for example, 1 is the first index, 10 is the second index).

-r, –recover

Repairs most corruption and should be the first option attempted. You can increase the sort_buffer_size to make the recover go more quickly if you have the memory. This option will not recover from the rare form of corruption where a unique key is not unique.

-o, –safe-recover

A more thorough, slower repair option than -r, which should be used only if -r fails. This reads through all rows and rebuilds the indexes based on the rows. It also uses less disk space than -r because a sort buffer is not created. You can increase the size of key_ buffer_size to improve repair speed.

-n, –sort-recover

Forces MySQL to uses sorting to resolve the indexes, even if the resulting temporary files are large.

–character-sets-dir=…

The directory containing the character sets.

–set-character-set=name

Specifies a new character set for the index.

-t, –tmpdir=path

Specifies a new path for storing temporary files if you don’t want to use the contents of the TMPDIR environment variable.

-q, –quick

Fastest repair as the data file is not modified. Running this option a second time will modify the data file if there are duplicate keys. Uses much less disk space as well because the data file is not modified.

-u, –unpack

Unpacks a file that has been packed with the myisampack utility.

Table 13: Other myisamchk Options

Option

Description

-a, –analyze

Analyzing tables improves performance by updating the index information for atable so that MySQL can make a better decision on how to join tables. The distribution of the various index elements is stored for later usage. This option isthe same as ANALYZE TABLE.

-d, –description

Displays a description of the table.

-A, –set-auto-increment[=value]

Sets the AUTO_INCREMENT counter to the specified value (or increments it by one if no value is supplied).

-S, –sort-index

Sorts the index tree blocks in descending order, which improves the performance of seeks and table scanning by key.

-R, –sort-records=#

Sorts records according to the index specified (index numbers begin from 1; you can use SHOW INDEX to see an ordered list). This can speed up queries that are ordered on this index, as well as ranged selects. It will probably be very slow if you sort a large table for the first time.

Running myisamchk with the -d option produces the following kind of output:

% myisamchk -d customer
MyISAM file:         customer
Record format:       Packed
Character set:       latin1 (8)
Data records:        3  Deleted blocks:                 0
Recordlength:        75
table description:
Key Start Len Index   Type
1      2         4      unique   long
Tagged with:
Jul 06

Linux, Apache, MySQL, and PHP (or Perl) form the basis of the LAMP architecture for Web applications. Many open source packages based on LAMP components are available to solve a variety of problems. As the load on an application increases, the bottlenecks in the underlying infrastructure become more apparent in the form of slow response to user requests. The previous article showed you how to tune the Linux system and covered the basics of LAMP and performance measurement. This article focuses on the Web server components, Apache and PHP.

Tuning Apache

Apache is a highly configurable piece of software. It has a lot of features, but each one comes at a price. Tuning Apache is partially an exercise in proper allocation of resources, and involves stripping down the configuration to only what’s needed.

Configuring the MPM

Apache is modular in that you can add and remove features easily. Multi-Processing Modules (MPMs) provide this modular functionality at the core of Apache — managing the network connections and dispatching the requests. MPMs let you use threads or even move Apache to a different operating system.

Only one MPM can be active at one time, and it must be compiled in statically with --with-mpm=(worker|prefork|event).

The traditional model of one process per request is called prefork. A newer, threaded, model is called worker, which uses multiple processes, each with multiple threads to get better performance with lower overhead. The final, event MPM is an experimental module that keeps separate pools of threads for different tasks. To determine which MPM you’re currently using, execute httpd -l.

Choosing the MPM to use depends on many factors. Setting aside the event MPM until it leaves experimental status, it’s a choice between threads or no threads. On the surface, threading sounds better than forking, if all the underlying modules are thread safe, including all the libraries used by PHP. Prefork is the safer choice; you should do careful testing if you choose worker. The performance gains also depend on the libraries that come with your distribution and your hardware.

Regardless of which MPM you choose, you must configure it appropriately. In general, configuring an MPM involves telling Apache how to control how many workers are running, whether they’re threads or processes. The important configuration options for the prefork MPM are shown in Listing 1.

Listing 1. Configuration of the prefork MPM

StartServers       50
MinSpareServers   15
MaxSpareServers   30
MaxClients       225
MaxRequestsPerChild  4000

In the prefork model, a new process is created per request. Spare processes are kept idle to handle incoming requests, which reduces the start-up latency. The previous configuration starts 50 processes as soon as the Web server comes up and tries to keep between 10 and 20 idle servers running. The hard limit on processes is dictated by MaxClients. Even though a process can handle many consecutive requests, Apache kills off processes after 4,000 connections, which mitigates the risk of memory leaks.

Configuring the threaded MPMs is similar, except that you must determine how many threads and processes are to be used. The Apache documentation explains all the parameters and calculations necessary.

Choosing the values to use involves some trial and error. The most important value is MaxClients. The goal is to allow enough worker processes or threads to run without causing your server to swap excessively. If more requests come in than can be handled, then at least those that made it through get service; the others are blocked.

If MaxClients is too high, then all clients experience poor service because the Web server tries to swap out one process to allow another one to run. Too low a setting means you may deny services unnecessarily. Checking the number of processes running at high loads and the resulting memory footprint of all the Apache processes gives you a good idea of how to set this value. If you go over 256 MaxClients, you must also set ServerLimit to the same number; read the MPM’s documentation carefully for the associated caveats.

Tuning the number of servers to start and keep spare depends on the role of the server. If the server runs only Apache, you can use modest values as shown in Listing 1, because you’re able to make full use of the machine. If the system is shared with a database or other server, then you should limit the number of spare servers being run.

Using options and overrides efficiently

Each request that Apache processes goes through a complicated set of rules that dictates any restrictions or special instructions the Web server must follow. Access to a folder can be restricted by IP address to a certain folder, or a username and password can be configured. These options also include the handling of certain files, such as if a directory listing is provided, how certain filetypes are to be handled, or whether the output should be compressed.

These configurations take the form of containers in httpd.conf such as <Directory> to specify that the configuration to follow refers to a location on disk, or <Location> to indicate that the reference is to a path in the URL. Listing 2 shows a Directory container in action.

Listing 2. A Directory container being applied to the root directory

<Directory />
    AllowOverride None
    Options FollowSymLinks
</Directory>

In Listing 2, the configuration enclosed in the Directory and /Directory tags is applied to the given directory and everything under it — in this case, the root directory. Here, the AllowOverride tag dictates that users aren’t allowed to override any options (more on this later). The FollowSymLinks option is enabled, which lets Apache look past symlinks to serve the request, even if the file is outside the directory containing Web files. This means that if a file in your Web directory is a symlink to /etc/passwd, the Web server happily serves the file if asked. With -FollowSymLinks used instead, this feature is disabled, and the same request causes an error to be returned to the client.

This last scenario is a cause for concern on two fronts. The first is a performance matter. If FollowSymLinks is disabled, then Apache must check each component of the filename (directories and the file itself) to make sure they’re not symbolic links. This incurs extra overhead in the form of disk activity. A companion option called FollowSymLinksIfOwnerMatch follows the symbolic link if the owner of the file is the same as that of the link. This has the same performance hit as disabling following of symlinks. For best performance, use the options in Listing 2.

Security-conscious readers should be alert by now. Security is always a trade-off between functionality and risk. In this case, the functionality is speed, and the risk is allowing unauthorized access to files on the system. One of the mitigations is that LAMP application servers are generally dedicated to a particular function, and users can’t create the potentially dangerous symbolic links. If it’s vital to have symbolic link-checking enabled, you can restrict it to a particular area of the file system, as in Listing 3.

Listing 3. Restricting FollowSymLinks to a user’s directory

<Directory />
   Options FollowSymLinks
</Directory>

<Directory /home/*/public_html>
   Options -FollowSymLinks
</Directory>

In Listing 3, any public_html directory in a user’s home directory has the FollowSymLinks option removed for it and any child directories.

As you’ve seen, options can be configured on a per-directory basis through the main server configuration. Users can override this server configuration themselves (if permitted by the administrator by the AllowOverrides statement) by dropping a file called .htaccess into a directory. This file contains additional server directives that are loaded and followed on each request to the directory where the .htaccess file resides. Despite the earlier discussion about not having users on the system, many LAMP applications use this functionality to control access and for URL rewriting, so it’s wise to understand how it works.

Even though the AllowOverrides statement prevents users from doing anything you don’t want them to, Apache must still look for the .htaccess file to see if there is any work to be done. A parent directory can specify directives that are to be processed by requests from child directories, which means Apache must also search each component of the directory tree leading to the requested file. Understandably, this causes a great deal of disk activity on each request.

The easiest solution is to not allow any overrides, which eliminates the need for Apache to check for .htaccess. Any special configurations are then placed directly in httpd.conf. Listing 4 shows the additions to httpd.conf to enable password checking for a user’s project directory, rather than putting in a .htaccess file and relying on AllowOverrides.

Listing 4. Moving .htaccess configuration into httpd.conf

<Directory /home/user/public_html/project/>
  AuthUserFile /home/user/.htpasswd
  AuthName "uber secret project"
  AuthType basic
  Require valid-user
</Directory>

If the configuration is moved into httpd.conf and AllowOverrides is disabled, disk usage can be reduced. A user’s project may not attract many hits, but consider how powerful this technique is when applied to a busy site.

Sometimes it’s not possible to eliminate use of .htaccess files. For example, in Listing 5, where an option is restricted to a certain part of the file system, overrides can also be scoped.

Listing 5. Scoping .htaccess checking

<Directory />
  AllowOverrides None
</Directory>

<Directory /home/*/public_html>
  AllowOverrides AuthConfig
</Directory>

After you implement Listing 5, Apache still looks for .htaccess files in the parent directories, but it stops in the public_html directory because the rest of the file system has the functionality disabled. For example, if a file that maps to /home/user/public_html/project/notes.html is requested, only the public_html and project directories are searched.

One final note about per-directory configurations is in order. Any document about tuning Apache will tell you to disable DNS lookups through the HostnameLookups off directive because trying to reverse-resolve every IP address connecting to your server is a waste of resources. However, any limitations based on hostname force the Web server to perform a reverse lookup on the client’s IP address and a forward lookup on the result of that to verify the authenticity of the name. Therefore, it’s wise to avoid using access controls based on the client’s hostname and to scope them as described when they’re necessary.

Persistent connections

When a client connects to a Web server, it’s allowed to issue multiple requests over the same TCP connection, which reduces the latency associated with multiple connections. This is useful when a Web page refers to several images: The client can request the page and then all the images over one connection. The downside is that the worker process on the server has to wait for the session to be closed by the client before it can move on to the next request.

Apache lets you configure how persistent connections, called keepalives, are handled. KeepAlive 5 at the global level of httpd.conf allows the server to handle 5 requests on a connection before forcing the connection closed. Setting this number to 0 disables the use of persistent connections. KeepAliveTimeout, also at the global level, determines how long Apache will wait for another request before closing the session.

Handling persistent connections isn’t a one-size-fits-all configuration. Some Web sites fare better with keepalives disabled (KeepAlive 0), and some experience a tremendous benefit by having them on. The only solution is to try both and see for yourself. It’s advisable, though, to use a low timeout such as 2 seconds with KeepAliveTimeout 2 if you enable keepalives. This ensures that any client wishing to make another request has ample time, and that worker processes aren’t idling while waiting for another request that may never come.

Compression

The Web server can compress the output before it’s sent back to the client. This results in a smaller page being sent over the Internet at the expense of CPU cycles on the Web server. For those servers that can afford the CPU overhead, this is an excellent way of making pages download faster — it isn’t unheard of for pages to be a third of their size after compression.

Images are generally already compressed, so compression should be limited to text output. Apache provides compression through mod_deflate. Although mod_deflate can be simple to turn on, it includes many complexities that the manual is eager to explain. This article doesn’t cover the configuration of compression except to provide a link to the appropriate documentation (see the Resources section.)

Tuning PHP

PHP is the engine that runs the application code. You should install only the modules you plan to use and have your Web server configured to use PHP only for script files (usually those ending in .php) and not all static files.

Opcode caching

When a PHP script is requested, PHP reads the script and compiles it into what’s called Zend opcode, a binary representation of the code to be executed. This opcode is then executed by the PHP engine and thrown away. An opcode cache saves this compiled opcode and reuses it the next time the page is called. This saves a considerable amount of time. Several opcode caches are available; I’ve had a great deal of success with eAccelerator.

Installing eAccelerator requires the PHP development libraries on your computer. Because different Linux distributions place files in difference places, it’s best to get the installation instructions directly from the eAccelerator Web site (see the Resources section for a link). It’s also possible that your distribution has already packaged an opcode cache, and you just have to install it.

Regardless of how you get eAccelerator on your system, there are a few configuration options to look at. The configuration file is usually /etc/php.d/eaccelerator.ini. eaccelerator.shm_size defines the size of the shared memory cache, which is where the compiled scripts are stored. The value is in megabytes. Determining the proper size depends on your application. eAccelerator provides a script to show the status of the cache, which includes the memory usage; 64 megabytes is a good start (eaccelerator.shm_size="64"). You may also have to tweak your kernel’s maximum shared memory size if the value you choose isn’t accepted. Add kernel.shmmax=67108864 to /etc/sysctl.conf, and run sysctl -p to make the setting take effect. The value for kernel.shmmax is in bytes.

If the shared memory allocation is exceeded, eAccelerator must purge old scripts from memory. By default, this is disabled; eaccelerator.shm_ttl = "60" specifies that when eAccelerator runs out of shared memory, any script that hasn’t been accessed in 60 seconds should be purged.

Another popular alternative to eAccelerator is the Alternative PHP Cache (APC). The makers of Zend also have a commercial opcode cache that includes an optimizer to further increase efficiency.

php.ini

You configure PHP in php.ini. Four important settings control how much system resources PHP can consume, as listed in Table 1.

Table 1. Resource related settings in php.ini

Setting Description Recommended value
max_execution_time How many CPU-seconds a script can consume 30
max_input_time How long (seconds) a script can wait for input data 60
memory_limit How much memory (bytes) a script can consume before being killed 32M
output_buffering How much data (bytes) to buffer before sending out to the client 4096

These numbers depend mostly on your application. If you accept large files from users, then max_input_time may have to be increased, either in php.ini or by overriding it in code. Similarly, a CPU- or memory-heavy program may need larger settings. The purpose is to mitigate the effect of a runaway program, so disabling these settings globally isn’t recommended. Another note on max_execution_time: This refers to the CPU time of the process, not the absolute time. Thus a program that does lots of I/O and few calculations may run for much longer than max_execution_time. It’s also how max_input_time can be greater than max_execution_time

The amount of logging that PHP can do is configurable. In a production environment, disabling all but the most critical logs saves disk writes. If logs are needed to troubleshoot a problem, you can turn up logging as needed. error_reporting = E_COMPILE_ERROR|E_ERROR|E_CORE_ERROR turns on enough logging to spot problems but eliminates a lot of chatter from scripts.


Summary

This article focused on tuning the Web server, both Apache and PHP. With Apache, the general idea is to eliminate extra checks the Web server must do, such as processing the .htaccess file. You must also tune the Multi-Processing Module you’re using to balance the system resources used with the availability of idle workers for incoming requests. The best thing you can do for PHP is to install an opcode cache. Keeping your eye on a few resource settings also ensures that scripts don’t hog resources and make the system slow for everyone else.

The next and final article in this series will look at tuning the MySQL database. Stay tuned!

Tagged with:
preload preload preload