Forum

A Guide to systemd ...
 
Notifications
Clear all

A Guide to systemd journal Maintenance [With Examples]


shane
(@shane)
Member Admin
Joined: 2 years ago
Posts: 223
Topic starter  

Arindam

Systemd comes with many built-in features to manage the system logs. In this guide, we explain how you can manage system journals, logs and take action on them such as rotating, archiving, and clear logs. We also explain the manual systems journal clean method and using config file changes.

If your Linux distribution supports systemd, then it collects logs from all processes, applications of the system every second which starts from the boot. All these logging events are managed by journald daemon of systemd. The journald collects all the logs (info, warnings, errors, etc) and stores them as binary data in the disk files. 

As the logs remain in the disk and every second it is collected, it takes up huge disk space; especially for older systems, servers. For example, in one of my test systems which are running for around one year, the log file size is in GBs.

If you manage multiple systems, servers, it is always recommended to properly manage journald logs for efficient operation. Let’s take a look at how you can manage the log files.

The systemd journal Maintenance

Using the journalctl utility of systemd, you can query these logs, perform various operations on them. For example, viewing the log files from different boots, check for last warnings, errors from a specific process or applications. If you are unaware of these, I would suggest you quickly go through this tutorial – “use journalctl to View and Analyze Systemd Logs [With Examples]” before you follow this guide. 

Where are the physical journal log files?

The systemd’s journald daemon collects logs from every boot. That means, it classifies the log files as per the boot. 

The logs are stored as binary in the path /var/log/journal with a folder as machine id.

For example:

image

 

image

 

Also, remember that based on system configuration, runtime journal files are stored at /run/log/journal/. And these are removed in each boot. 

Can I manually delete the log files?

You can, but don’t do it. Instead, follow the below instructions to clear the log files to free up disk space using journalctl utilities.

How much disk space is used by systemd log files?

Open up a terminal and run the below command.

journalctl --disk-usage

This should provide you how much is actually used by the log files in your system.

image

If you have a graphical desktop environment, you can open the file manager and browse to the path /var/log/journal and check the properties.

systemd journal clean process

The effective way of clearing the log files should be done by journald.conf configuration file. Ideally, you should not manually delete the log files even if the journalctl provides utility to do that.

Let’s take a look at how you can delete it manually, then I will explain the configuration changes in journald.conf so that you do not need to manually delete the files from time to time; Instead, the systemd takes care of it automatically based on your configuration. 

Manual delete

First, you have to flush and rotate the log files. Rotating is a way of marking the current active log files as an archive and create a fresh logfile from this moment. The flush switch asks the journal daemon to flush any log data stored in /run/log/journal/ into /var/log/journal/, if persistent storage is enabled.

Then, after flush and rotate, you need to run journalctl with vacuum-size, vacuum-time, and vacuum-files switches to force systemd to clear the logs. 

Example 1:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-time=1s

The above set of commands removes all archived journal log files until the last second. This effectively clears everything. So, careful while running the command. 

Spoiler
journal clean up – example
image

 

Spoiler
After clean up – journal space usage
image

ou can also provide the following suffixes as per your need following the number.

  • s: seconds
  • m: minutes
  • h: hours
  • days
  • months
  • weeks
  • years

Example 2:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-size=400M

This clears all archived journal log files and retains the last 400MB files. Remember this switch applies to only archived log files only, not on active journal files. You can also use suffixes as below.

  • K: KB
  • M: MB
  • G: GB

Example 3:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-files=2

The vacuum-files switch clears all the journal files below the number specified. So, in the above example, only the last 2 journal files are kept and everything else is removed. Again, this only works on the archived files.

You can combine the switches if you want, but I would recommend not to. However, make sure to run with --rotate switch first.

Automatic delete using config files

While the above methods are good and easy to use, but it is recommended that you control the journal log file cleanup process using the journald configuration files which present at /etc/systemd/journald.conf

The systemd provides many parameters for you to effectively manage the log files. By combining these parameters you can effectively limit the disk space used by the journal files. Let’s take a look.

journald.conf parameter Description Example
SystemMaxUse Specifies the maximum disk space that can be used by the journal in persistent storage SystemMaxUse=500M
SystemKeepFree Specifies the amount of space that the journal should leave free when adding journal entries to persistent storage. SystemKeepFree=100M
SystemMaxFileSize Controls how large individual journal files can grow to in persistent storage before being rotated. SystemMaxFileSize=100M
RuntimeMaxUse Specifies the maximum disk space that can be used in volatile storage (within the /run filesystem). RuntimeMaxUse=100M
RuntimeKeepFree Specifies the amount of space to be set aside for other uses when writing data to volatile storage (within the /run filesystem). RuntimeMaxUse=100M
RuntimeMaxFileSize Specifies the amount of space that an individual journal file can take up in volatile storage (within the /run filesystem) before being rotated. RuntimeMaxFileSize=200M

If you add these values in a running system in /etc/systemd/journald.conf file, then you have to restart the journald after updating the file. To restart use the following command. 

sudo systemctl restart systemd-journald

Verification of log files

It is wiser to check the integrity of the log files after you clean up the files. To do that run the below command. The command shows the PASS, FAIL against the journal file.

journalctl --verify
image

Closing Notes

I hope this guide helps you to understand the basics of the systemd journal management process. With these, you can manage the disk space used by the log files in your system or servers by limiting the space, clearing old log files. These are just the guideline commands, you can combine these in multiple ways to achieve your system demands. 


Quote
Topic Tags