Previous Page
Next Page

8.7. Logging

It's important to know what is going on on your system. Fedora provides a standardized, network-based logging system and tools to automatically monitor and trim logfiles. Understanding and using these tools effectively will allow you to keep your finger on the pulse of your system with minimal effort.

8.7.1. How Do I Do That?

The syslog facility collects and routes messages in a Fedora system. The file /etc/syslog.conf configures the message routing; the default version of the file looks like this:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

On the left side of each entry is a pattern that consists of selectors. Each selector contains one or more facilities (separated by commas), then a period, and then one or more levels (again, separated by commas).

The facility indicates the origin of the log entry. Possible values are shown in Table 8-3.

Table 8-3. Facility values to indicate the origin of the log entry
authpriv Security, authentication, or authorization systems.
cron Task scheduler (crond and atd).
daemon Server daemons that don't have a category of their own.
ftp File-transfer-protocol daemon.
kern Kernel messages.
local0, local1, local2, local3, local4, local5, local6, and local7 Reserved for custom use on a distribution-by-distribution or site-by-site basis. Fedora uses local7 to log boot messages.
lpr Printing system.
mail Electronic mail.
news Net news (Usenet).
syslog Messages from syslogd itself.
user User-level messages.
uucp Unix-to-Unix copy messages (rarely used).

The level consists of a priority level and can be any of the values listed in Table 8-4, in increasing order of severity.

Table 8-4. Priority-level values, in order of severity
debug Informational software debugging messages.
info General informational messages.
notice Important normal messages that do not indicate an error or problem.
warning Information about an unusual or impending situation.
err Error messages, indicating that something is wrong.
crit Critical conditions indicating imminent danger.
alert Serious, emergency problems.
emerg Emergency situation: the system is in crisis and failing.

Specifying a level means any message of that level or higher (more severe), so the selector kern.crit would match messages from the kernel with a priority of crit, alert, or emerg. To match only crit, an equal sign is added: kern.=crit. An exclamation mark negates a match: kern.!crit matches kernel messages with a priority below crit, while kern.!=crit matches all kernel messages except those with a priority of crit.

An asterisk indicates that the facility or level should be ignored. Therefore, authpriv.* matches messages from the authpriv facility regardless of the priority, and *.info matches messages from any facility which are at the info level or higher. Multiple facilities or priorities can be matched using commas (indicating an OR operation), so mail,local3.* matches any message from the mail or local3 facilities.

Multiple selectors may be included in one entry, separated by semicolons, which indicates an AND operation. The special priority none matches no messages from the specified facility. Therefore *.crit;kern.none matches all messages that are of crit priority or higher, unless they come from the kernel.

On the right side of each entry in /etc/syslog.conf is a destination for the messages. The destination may be:

An absolute pathname

Messages are placed in the specified file. The pathname may also point to a named pipe, providing a method for passing messages to another program, or to a device such as a terminal (such as /dev/tty3) or a printer (/dev/lp0). Adding a hyphen in front of a pathname will prevent syslogd from flushing the buffers to disk after each write, a performance-eating behavior that increases the chance that a message describing the cause of a crash will make it onto the disk.

@ host

Messages are forwarded to syslogd on the remote host.


Messages are written to the terminals of any of these users who are currently logged in.


Messages are written to the terminals of all logged-in users.

The order of the lines in the configuration file does not matter; every line is checked against each incoming message, so messages may be sent to multiple destinations.

The default configuration file routes messages according to Table 8-5; as you can see, /var/log/messages is the prime source of information about the state of the system.

Table 8-5. Message routing as configured in the default syslog configuration file
Type of messageDestination
Everything except mail, authentication, and cron messages, with a priority of info or higher /var/log/messages
Authentication messages (which may contain private information) /var/log/secure
Mail /var/log/maillog
Cron /var/log/cron
All messages of emerg level or higherThe terminals of all logged-in users
UUCP and news messages of crit level or higher /var/log/spooler
Boot messages /var/log/boot.log Interpreting /var/log/messages

The /var/log/messages logfile contains entries similar to this:

May 31 10:40:58 laptop3 dhclient: DHCPREQUEST on eth0 to port 67
May 31 10:40:58 laptop3 dhclient: DHCPACK from
May 31 10:40:58 laptop3 dhclient: bound to -- renewal in 34387
May 31 20:14:05 laptop3 dhclient: DHCPREQUEST on eth0 to port 67
May 31 20:14:05 laptop3 dhclient: DHCPACK from
May 31 20:14:05 laptop3 dhclient: bound to -- renewal in 41631

Each entry consists of a date, time, hostname (laptop3 in this example), program name or other prefix (dhclient), and a text message. Note that the facility and priority are not recorded in the logfile.

Since the /var/log/message file can be very large, it's worthwhile using a tool such as grep to search for specific records. For example, you can view all of the kernel messages with the command:

$ grep kernel /var/log/messages
May 30 04:23:08 bluesky kernel: SELinux: initialized (dev hdd, type iso9660),
     uses genfs_contexts
May 31 20:48:40 bluesky kernel: atkbd.c: Unknown key pressed (translated 
set 2, code 0x85 on isa0060/serio0).
May 31 20:48:40 bluesky kernel: atkbd.c: Use 'setkeycodes e005 <keycode>'
     to make it known.
May 31 21:14:54 bluesky kernel: cdrom: This disc doesn't have any tracks I
     recognize! Creating your own logfile entries

You can generate syslog messages using the logger command-line tool. Simply provide your text as arguments:

$ logger Added host lightning to /etc/hosts

The message recorded in /var/log/messages contains the username as the prefix:

Jun  1 02:32:59 darkday chris: Added host lightning to /etc/hosts

It's convenient to log information about changes you have made on the system in this way, entering them as you work. Your notes will be interleaved with system-generated log messages, making it easy to see the relationship between the changes that you have made and any messages that start or stop appearing in the log as a result.

By default, logger uses the facility user and the priority notice. You can override this using the -p option, and you can override the insertion of the username by supplying an alternate tag with the -t option:

$ logger -p local1.crit -t cooling Stopped water pump

Which would result in this message being logged:

Jun  1 09:54:49 darkday cooling: Stopped water pump

An alias can be used to simplify logging from the command line:

$ alias note='logger -p local4.notice'
$ note Ran yum update

If you are logging a message that contains metacharacters, surround the message with quotation marks.

By adding a custom rule to /etc/syslog.conf, the messages sent to the local1 facility can be placed in their own file (in addition to being logged in /var/log/messages):

local1.*                  /var/log/cooling

The security context of any new logfiles must be set to the same context as /var/log/messages:

# touch /var/log/cooling
# ls -Z /var/log/messages /var/log/cooling
-rw-r--r--  root     root     user_u:object_r:var_log_t        /var/log/cooling
-rw-------  root     root     system_u:object_r:var_log_t      /var/log/messages
# chcon system_u:object_r:var_log_t /var/log/cooling
# chmod 0600 /var/log/cooling      # Optional!
# ls -Z /var/log/messages /var/log/cooling
-rw-------  root     root     system_u:object_r:var_log_t      /var/log/cooling
-rw-------  root     root     system_u:object_r:var_log_t      /var/log/messages Keeping an eye on logs

The -f option to tail provides a convenient way to watch messages that are being appended to a file and is perfect for use with logfiles:

# tail -f /var/log/messages
Jun  1 08:47:14 darkday kernel: hub 1-0:1.0: over-current change on port 1
Jun  1 08:47:14 darkday kernel: hub 1-0:1.0: port 2 disabled by hub (EMI?), re-enabling...
Jun  1 08:47:14 darkday kernel: hub 1-0:1.0: over-current change on port 2
Jun  1 08:47:14 darkday kernel: usb 1-2: USB disconnect, address 4
Jun  1 08:47:14 darkday kernel: usb 1-2: new low speed USB device using uhci_hcd and address 5
Jun  1 08:47:14 darkday kernel: usb 1-2: configuration #1 chosen from 1 choice
Jun  1 08:47:14 darkday kernel: input: Logitech USB-PS/2 Optical Mouse as /class/input/input4
Jun  1 08:47:14 darkday kernel: input: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:1f.2-2
Jun  1 09:54:49 darkday cooling: Water temperature exceeds 70C
Jun  1 09:54:49 darkday cooling: Water temperature exceeds 85C
...(Additional lines are displayed as they are added to the logfile)...

/var/log/messages is normally readable only by root. Although making it readable by other users may reveal a small amount of information about your system (reducing security), it can also reduce the amount of time spent in superuser mode (which, in turn, increases security). To make the messages file accessible to everyone:

# chmod a+r /var/log/messages

This tail command will display the last 10 lines in the file, and then additional lines within a second of the time that they are appended to the file. It can be left running in a terminal window in the corner of the screen while you perform system administration tasks. Configuring remote logging

The syslog service was designed to facilitate remote logging. This is very useful in two circumstances:

  • In the event of a successful system intrusion, an attacker will often edit or delete logfiles to erase any record of his presence. If messages are logged to a remote server, it becomes more difficult to erase the trail because the attacker then needs to successfully attack the machine recording the log in addition to the system originally compromised.

  • In a network, it is convenient to gather logs in one place for centralized analysis. This lets you stay on top of the state of many systems from one location.

To configure a syslog network server, edit that host's /etc/sysconfig/syslog file, which initially looks like this:

# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
# Options to klogd
# -2 prints all kernel oops messages twice: once for klogd to decode, and
#    once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
# set this to a umask value to use for all logfiles, as in umask(1).
# By default, all permissions are removed for "group" and "other".

Change the SYSLOGD_OPTIONS line to include -r (remote logging):


Then restart syslogd:

# service syslog restart
Shutting down kernel logger:                               [  OK  ]
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]

Ensure that your firewall configuration permits connections on UDP port 514.

Next, edit the file /etc/syslog.conf on the machines that will be forwarding log messages to the syslog server, and add this line:

*.*                        @syslogserver

This will forward all messages to the remote host syslogserver (which may be an IP address or hostname). Restart syslogd to activate the changes.

It's important to leave local logging turned on in case the syslog server is unavailable, so don't remove the lines that write to the local logfiles.

The result will be a combined log containing entries from both the syslog server and the host that is forwarding its log messages:

Jun  1 02:52:33 darkday named[13255]: starting BIND 9.3.2 -u named
Jun  1 02:52:33 darkday named[13255]: found 1 CPU, using 1 worker thread
Jun  1 02:52:33 darkday named[13255]: loading configuration from '/etc/named.conf'
Jun  1 02:52:33 darkday named[13255]: listening on IPv4 interface lo,
Jun  1 02:52:33 darkday named[13255]: listening on IPv4 interface eth0,
Jun  1 02:52:33 darkday named[13255]: command channel listening on
Jun  1 02:52:33 darkday named[13255]: zone loaded serial 42
Jun  1 02:52:33 darkday named[13255]: zone loaded serial 1997022700
Jun  1 02:52:33 darkday named[13255]: zone loaded serial 42
Jun  1 02:52:33 darkday named[13255]: zone loaded serial 1997022700
Jun  1 02:52:33 darkday named[13255]: zone localdomain/IN: loaded serial 42
Jun  1 02:52:33 darkday named[13255]: zone localhost/IN: loaded serial 42
Jun  1 02:52:33 darkday named[13255]: running
Jun  1 02:57:22 bluesky chris: VNC service configured, restarting xinetd
Jun  1 02:57:29 bluesky xinetd[15394]: Exiting...
Jun  1 02:57:29 bluesky xinetd[15452]: xinetd Version 2.3.13 started with libwrap loadavg options compiled in.
Jun  1 02:57:29 bluesky xinetd[15452]: Started working: 1 available service

Notice that this log contains entries from darkday (the syslog server) as well as from bluesky (which is forwarding log messages to darkday). Notice also the system administrator's note on bluesky, stating the reason that xinetd was being restarted.

If the syslog server can't convert the IP address on a message into a hostname, the message will be logged with the IP address in the hostname field. Add the corresponding entry to /etc/hosts to make the logfile more readable. Automated log watching

There's not much point in collecting all this information if the logs are never read, but reading logfiles is boring, tedious work. Fortunately, the logwatch package automates this process, sending a daily summary email to alert you to important log entries.

The daily summary is emailed to root on the local machine. Email to the root user should be redirected to a specific user or users by the /etc/aliases file. Edit this file and uncomment the entry for root found at the the end, inserting the name of a user who is responsible for administering the system (or a list of people separated by commas). In this example, all mail for root is redirected to

# Person who should get root's mail

Here is a typical daily logwatch summary:

From:    root <>
Subject: LogWatch for
Date:    Wed, 31 May 2006 04:02:17 -0400

################### LogWatch 7.1 (11/12/05) ####################
        Processing Initiated: Thu Jun  1 02:52:14 2006
        Date Range Processed: yesterday
                              ( 2006-May-31 )
                              Period is day.
      Detail Level of Output: 10
              Type of Output: unformatted
           Logfiles for Host:

--------------------- httpd Begin ------------------------ 
A total of 3 unidentified 'other' records logged
GET /level/16/exec/-///pwd HTTP/1.0 with response code(s) 
   2 404 responses
POST /garethjones/photos/--WEBBOT-SELF-- HTTP/1.0 with response code(s) 
   1 404 responses
GET HTTP/1.1 with response code(s) 
   1 404 responses

---------------------- httpd End ------------------------- 

--------------------- SSHD Begin ------------------------

 Users logging in through sshd:
    chris: 3 times

--------------------- SSHD End -------------------------

--------------------- Disk Space Begin ------------------------

Filesystem                            Size   Used  Avail Use%  Mounted on
/dev/mapper/main-root                 9.5G   2.9G   6.1G  33%  /
/dev/hda1                              99M   9.7M    84M  11%  /boot
/dev/mapper/main-home                 4.9G    24M   4.7G   1%  /home

---------------------- Disk Space End -------------------------

###################### LogWatch End ######################### 

This report will vary according to the services you have installed, but it provides a simple, easy-to-scan summary of log entries that may warrant attention. It also provides a summary of free disk space; if you methodically review these email messages, you won't be caught unaware when your storage needs start to inch upward. Log rotation

Logfiles can grow to be massive. The Fedora logrotate package automatically moves historical log data into history files and keeps a limited number of history files on hand.

logrotate is configured through the master configuration file /etc/logrotate.conf:

# see "man logrotate" for details
# rotate log files weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) logfiles after rotating old ones

# uncomment this if you want your logfiles compressed

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
    create 0664 root utmp
    rotate 1

# system-specific logs may be also be configured here.

The most frequently altered lines are highlighted in bold: logrotate is initially configured to rotate logs every week and to save the last four historical logfiles in addition to the current log. If you have a lot of storage and wish to keep more history, edit the rotate line to increase the number of history files maintained, or change the weekly line to monthly to reduce the frequency of history snapshots (which can make it easier to analyze patterns over a longer period of time without merging data from several files).

The default configuration results in five separate message files being present on the system:

$ ls -l /var/log/messages*
-rw------- 1 root root  86592 Jun  1 02:49 /var/log/messages
-rw------- 1 root root  85053 May 30 02:03 /var/log/messages.1
-rw------- 1 root root 105491 May 26 23:51 /var/log/messages.2
-rw------- 1 root root  74062 May  7 04:12 /var/log/messages.3
-rw------- 1 root root 286194 May  2 13:00 /var/log/messages.4

logrotate also uses per-logfile configuration files in /etc/logrotate.d. These files are installed by various RPM packages that generate logfiles.

8.7.2. How Does It Work?

The main system logging utility is named syslog. It is network-based and uses a server daemon, syslogd, which receives messages from all sorts of system programs through the Unix domain socket /var/log. These messages are matched against the lines in /etc/syslog.conf and written to the selected destinations.

Kernel messages are stored in a buffer that is read by a helper daemon named klogd, either by reading the file /proc/kmesg or by using a kernel system call. klogd then forwards these messages to syslogd for inclusion in the system logs.

A syslog network server listens to UDP port 514 and processes any messages received there through the normal routing decisions.

One significant problem with the syslog implementation is that there is absolutely no authentication performed. Any application can log any message with any facility and priority. Therefore it is relatively easy to spoof log messages or to create a denial-of-service attack by sending huge numbers of logfile entries, eventually filling all available disk space and making it impossible to log further events. (For this reason, it is a good idea to use a separate filesystem for /var/log).

The logwatch and logrotate programs are activated by cron through their entries in /etc/cron.daily.

8.7.3. What About... ...sending log messages to a program?

The standard Fedora syslog program does not support output to a program such as a mailer. However, you can easily write a script that reads a logfile using the tail command and outputs new log entries to a program.

This example emails log messages to a pager or cell phone text service:


tail -0f /var/log/messages|
while read LINE
        echo $LINE|
        mail $DESTINATION

To use this script, place it in the file /usr/local/bin/log-mail and add read and execute permissions:

# chmod u+rx /usr/local/bin/log-mail
# log-mail

You may want to use this script with a lower-volume logfile than /var/log/messages, especially if you pay for each pager message.

To filter messages by content, place a grep command between the tail and while lines in the script.

You can also have log output read to you over the system's speakers:


logger -t log-speak "Starting log reading."
sleep 0.3

tail -1f /var/log/messages|
while read LINE
        # The sed expressions remove the date/time and PIDs
        # from messages to shorten the text.
        echo $LINE|
        sed     -e "s/^.\{17\}[^ ]*//" \
                -e "s/\[.*\]//g"|
        festival --tts
done ...outputting to a named pipe?

A named pipe is a special type of file that can be used to pass messages between two programs. While syslog supports writing to named pipes, the default SELinux security policy prohibits it.

To output to a named pipe, you must first disable SELinux protection for syslogd by setting the syslogd_disable_trans boolean and then create the named pipe with mkfifo:

# setsebool -P syslogd_disable_trans=1
# mkfifo /var/log/messagepipe

Next, create an entry in /etc/syslog.conf, placing a pipe symbol in front of the destination pathname:

*.*                  |/var/log/messagepipe

Restart syslogd. You can then follow the message output with a simple file read:

# service syslog restart
Shutting down kernel logger:                               [  OK  ]
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]
# cat /var/log/messagepipe
...(Messages appear as they are logged)... ...logging messages from printers, routers, and other network devices?

Most network hardware offers the option of logging messages to a syslog server. Simply enter the IP address of your syslog network server into the configuration settings of the device. ...using patterns within the message text to determine message routing?

The syslog-ng package from Fedora Extras can be used in place of the standard syslogd and klogd programs. It uses a different configuration file syntax, and it supports message-text matching and message routing to programs.

The original syslogd and klogd programs are from the package sysklogd.

8.7.4. Where Can I Learn More?

  • The manpages for syslogd, syslog.conf, klogd, logrotate, and logwatch

  • The home page for logwatch:

Previous Page
Next Page