Managing log files on Ubuntu servers without root access

So, it’s Friday, you’ve been developing features all week, hotfixing resolutions to bugs and updating your project managers using long words (3 syllables or more) to make your hotfixes sound more awesome than they really are.

You’re winning, have been all week and life is great. 5pm hits, you’re winding down for the week and positioning yourself for a positive start to the next, when, like a lioness pouncing through the long grass of the Serengeti at an antelope, your project manager comes running over.

Your web application is down. As in, not usable. It’s dead, like you, you antelope. You ssh into the server and, frantically, pull out every move in your arsenal to debug what could possibly cause this, only to find that the disk on the server is full. You check memory consumption and find that your log files are consuming ridiculous amounts of memory. Your rpush log is consuming 5gb?!? Your production log 20?!? That ludicrous. Server capacity is a valuable resource and a full disk can cause downtime of an application hosted on said server. Your log files causing your app to drop out of use is not a good situation at all and something that should not cause such a dramatic occurrence. Irrespective of that, this can be managed and moreso, without having root access. How do you do this wizardry I hear you ask? Well, we’re going to use something called Logrotate and configure it through our user (independently of configuration set on the root of the server). Logrotate does exactly what is says on the tin, it rotates logs. It is also designed to compress and even email them. In order to set up independent log management, we must ssh into our server. Once in, we can create a config file for logrotate.

This is typically done in the home directory, but we advise creating a logrotate directory and adding the config file to there.

As a note, throughout all this, we’re going to use vim to set up our config file. mkdir home/deploy/logrotate vim home/deploy/logrotate/logrotate.conf Now that we’ve created the file, we want to set configuration to keep 2 weeks worth of logs and each rotated log to only have log data for that specific day. Firstly we specify the path where our logs are and then we add the necessary remarks to ensure the case defined above is accounted for: /home/deploy/applications/name_of_your_app/shared/log/.log { daily missingok rotate 14 compress delaycompress notifempty } In the path to our logs, the “.log” means that this configuration will be applied to all files with the .log extension within that directory. daily - defines the logs to rotate every day. missingok - moves on to the next log without raising an error message if the log file is missing rotate n - Logs are rotated n number of times before being removed or emailed. compress - Old log files are compressed with gzip. delaycompress - Postpone compression of the previous log file to the next rotation cycle. This only has effect when used in combination with compress. notifempty - ensures that the log is not rotated if it is empty. More configuration parameters can be found in the link provided at the “Recommended Reading” section of this post. Now that we have configured our log rotation, we need to set up a cron job to schedule this script to run every day. Cron is a linux utility that allows us to do this scheduling of the script. Additionally, a cron job is a scheduled task to be run on the server. The following command will open a file allowing for a cron job to be set up: crontab -e Next, we want to set the parameters in which our cron job will occur. We need to specify the timeframe in which this job should take place and the path to the script. 0 1 * * * /usr/sbin/logrotate -s /home/deploy/logrotate/logrotate.status /home/deploy/logrotate/logrotate.conf This creates a cron job to run the log rotate script on a daily basis at 1am. As far as formatting goes, from left to right: 0 - minute 1 - hour * - day (month) * - month * - day (week) The * symbol stands for “any value”. ## Conclusion Ensuring logs are not consuming so much memory on the server that they cause a whole app to drop out is imperative. Additionally, managing your log files will also aid in debugging and chasing potential older bugs, as you’ll be able directly navigate to a particular log file and examine the data within that (so long as they are still in rotation). Keeping your logs maintained and under control is sure to keep the lionesses at bay, so young antelope, live on and stay off the radar, there is much Serengeti to graze. ## Recommended Reading Proof that logrotate is designed to rotate logs: https://manpages.ubuntu.com/manpages/xenial/man8/logrotate.8.html Digitalocean tutorial on setting up Logrotate: https://www.digitalocean.com/community/tutorials/how-to-manage-logfiles-with-logrotate-on-ubuntu-16-04 Logrotate configuration options: https://linux.die.net/man/8/logrotate cronjobs - https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ Generate configuration of your cronjob - https://crontab.guru/