Week 22: Day 058 - Configuring Logging

Hello, it's been quite a while since my last blog post. We had a one week spring break, which was much needed in my case. This week, we're gonna talk about logging and partitions. You ready? Let's get riiiiiight into the blog post.

First, we got a few examples of ways you can write log information. There's Direct Write, rsyslogd, and journald. In terms of journald, it provides an advanced log management system. It collects and store information regarding booting, kernel info, and service. To query/check this, you type "journalctl". When checking the logs for this in /var/log/messages, you'll find that it displays the date and timestamp along with the military time and the host that it came from. It will more importantly show you the service/process name, and the contents of the message. To look at this live do "tail -f <logfile>" and it will show you the live bits of what's going on with the log file (ctrl-c will close this obviously). Here's an exercise show you:

1. Open a root shell.

Obviously.

2. From the root shell, type tail -f /var/log/messages.

This shows you the last ten lines of the the file "messages", which contain just that.

3. Open a second terminal window. In this terminal window, type su - user to open a subshell as user.

This opened another shell under my username.

4. Type su - to open a root shell, but enter the wrong password.

I usually use sudo su. When entering the wrong password nothing will be logged.

5. Notice that nothing appears in /var/log/messages. That is because login-related errors are not written here.

6. From the user shell, type logger hello. You’ll see the message appearing in the /var/log/messages file in real time.

Interesting shortcut.

7. In the tail -f terminal, use Ctrl+C to stop tracing the messages file.

8. Type tail -n 20 /var/log/secure. This shows the last 20 lines in /var/log/secure, which also shows the messages that the su - password errors have generated previously.

This is where it's stored instead of the messages file, it's in the "secure" file.

------------------------------------------------------------------------------------------------------------

Let's move on to rsyslogd. This is similar to journald. It uses facilities, priorities and destinations:

• A facility specifies a category of information that is logged. Rsyslogd uses a fixed list of facilities, which cannot be extended. This is because of backward compatibility with the legacy syslog service.
•  A priority is used to define the severity of the message that needs to be logged. When specifying a priority, by default all messages with that priority and all higher priorities are logged.
• A destination defines where the message should be written to. Typical destinations are files, but rsyslog modules can be used as a destination as well, to allow further processing through an rsyslogd module.

When specifying a destination, it's often a file, and the command must start hyphenated (Ex: -/var/log/maillog). Devices can also be used. The different severity levels are: warn, err/error, crit, alert, emerg/panic. The facilities are different categories of info like cron, dameon, or mail.

In this exercise, you learn how to change rsyslog.conf. You configure the Apache service to log messages through syslog, and you create a rule that logs debug messages to a specific file.

1. By default, the Apache service does not log through rsyslog, but keeps its own logging. You are going to change that. To start, type yum install -y httpd to install the Apache service.

Apache is widely used for Linux web-servers.

2. After installing the Apache service, open its configuration file /etc/http/conf/httpd.conf and add the following line to it:

ErrorLog syslog:local1

3. Type systemctl restart httpd.

4. Now create a line in the rsyslog.conf file that will send all messages that it receives for facility local1 (which is now used by the httpd service) to the file /var/log/httpd-error.log. To do this, include the following line:

5. Tell rsyslogd to reload its configuration, by using systemctl restart httpd.

6. All Apache error messages will now be written to the httpd-error.log file.

7. From the Firefox browser, go to http://localhost/nowhere. Because the page you are trying to access does not exist, this will be logged to the Apache error log.

8. Now let’s create a snap-in file that logs debug messages to a specific file as well. To do this, type echo “*.debug /var/log/messages/messages-debug” > /etc/rsyslogd/debug.conf.

9. Again, restart rsyslogd using systemctl restart rsyslogd.

10. Use the command tail -f /var/log/messages-debug to open a trace on the newly created file.

11. Type logger -p daemon.debug “Daemon Debug Message”. You’ll see the debug message passing by.

12. Use Ctrl+C to close the debug log file.

Moving on to Rotating Log files. To prevent syslog messages from filling up your system completely, the log messages can be rotated. So, if /var/log/messages is rotated on January 17, 2015, the rotated filename will be /var/log/messages-20150115. The default settings for log rotation are kept in the file /etc/logrotate.conf where you can customize the log rotation settings.

--------------------------------------------------------------------------------------------------------------------------

Finally, let's talk about journald. This is an alternative to rysyslog. If you want to see the last messages that have been logged, you can use journalctl -f, which shows the last lines of the messages where new log lines are automatically added. You can also type journalctl and use (uppercase) G to go to the end of the journal.

In this exercise, you learn how to work with different journalctl options.

1. Type journalctl. You’ll see the content of the journal since your server last started, starting at the beginning of the journal. The content is shown in less, so you can use common lesscommands to walk through the file.

2. Type q to quit the pager. Now type journalctl --no-pager. This shows the contents of the journal without using a pager.

3. Type journalctl -f. This opens the live view mode of journalctl, which allows you to see new messages scrolling by in real time. Use Ctrl+C to interrupt.

4. Type journalctl and press the Tab key twice. This shows specific options that can be used for filtering. Type, for instance, journalctl _UID=0.

5. Type journalctl -n 20. The -n 20 option displays the last 20 lines of the journal (just like tail -n 20).

6. Now type journalctl -p err. This command shows errors only.

7. If you want to view journal messages that have been written in a specific time period, you can use the --since and --until commands. Both options take the time parameter in the format YYYY-MM-DD hh:mm:ss. Also, you can use yesterday, today, and tomorrow as parameters. So, type journalctl --since yesterday to show all messages that have been written since yesterday.

8. journalctl allows you to combine different options, as well. So, if you want to show all messages with a priority err that have been written since yesterday, use journalctl --since yesterday -p err.

9. If you need as much detail as possible, use journalctl -o verbose. This shows different options that are used when writing to the journal (see Listing 13.3). All these options can be used to tell the journalctl command which specific information you are looking for. Type, for instance, journalctl _SYSTEMD_UNIT=sshd.service to show more information about the sshd systemd unit.

The command "journalctl -o verbose" is important since it shows all the important information. By default, the journal is stored in the file /run/log/journal. The entire /run directory is used for current process status information only, which means that the journal is cleared when the system reboots. Even when the journal is written to the permanent file in/var/log/journal, that does not mean that the journal is kept forever. Here's how to make the journal permanent:

1. Open a root shell and type mkdir /var/log/journal.

2. Before journald can write the journal to this directory, you have to set ownership. Type chown root:systemd-journal /var/log/journal, followed by chmod 2755 /var/log/journal.

3. Next, you can either reboot your system (restarting the systemd-journald service is not enough) or use the killall -USR1 systemd-journald command.

4. The systemd journal is now persistent across reboots. If you want to see the log messages since last reboot, use journalctl -b.

--------------------------------------------------------------------------------------------------------------------------

Review Questions

1. Which file is used to configure rsyslogd?

/etc/rsyslog.conf

2. Which configuration file contains messages related to authentication?

/var/log/secure

3. If you do not configure anything, how long will it take for log files to be rotated away?

Around 1 month, 5 weeks to be exact

4. Which command enables you to log a message from the command line to the user facility, using the notice priority?

logger -p user.notice "(enter text here)"

5. Which line would you add to write all messages with a priority of info to the file /var/log/messages.info?

"*.=info /var/log/messages.info" in /etc/rsyslog.conf

6. Which configuration file enables you to allow the journal to grow beyond its default size restrictions?

/etc/systemd/journal.d.conf

7. Which command enables you to see new messages in the journal scrolling by in real time?

journalctl -f

8. Which command enables you to see all journald messages that have been written for PID 1 between 9:00 a.m. and 3:00 p.m.?

journalctl _PID=1 --since 9:00:00 --until 15:00:00

9. Which command enables you to see journald messages since last reboot on a system where a persistent journal has been configured?

journalctl -b

10. Which procedure enables you to make the journald journal persistent?

killella -USR1 systemd-journald