Log rotation on a VPS: why your disk fills up and how to fix it

TL;DR: Web servers, application processes, and various system services continuously write to log files, error logs, access logs, application-specific logs, and without log rotation, a mechanism to periodically archive, compress, and eventually delete older log entries, these files grow indefinitely and can silently consume all available disk space over weeks or months, eventually causing genuine, confusing failures (a database that cannot write, a server that cannot start a new process) that have nothing obviously to do with logging at all. Log rotation is a well-established, standard solution (most commonly implemented via logrotate on Linux systems) that most VPS setups do not have fully and correctly configured for every log source by default, making this a common, avoidable cause of mysterious disk space exhaustion.

Why logs grow unbounded without intervention

Most server processes and applications, by design, append new entries to their log files continuously as events occur, every web request, every error, every application-specific event worth recording, with no built-in mechanism to automatically limit the file's total size or age unless something specifically manages that, since the logging component itself is typically focused purely on writing accurate, complete records rather than also managing its own long-term storage footprint. Left entirely unmanaged, a busy web server's access log, an application's error log, or any actively used log file simply continues growing for as long as the server runs and the corresponding events continue occurring, with no natural ceiling.

The specific failure mode this causes

This becomes a genuine, often confusing operational problem specifically because the connection between "disk space exhausted" and "unrotated log files" is not always obvious to someone troubleshooting a seemingly unrelated failure. A server running out of disk space can produce a range of symptoms that do not obviously point at logging at all: a database failing to write new data (similar in visible symptom to the database connection errors covered in our dedicated guide), a web server failing to serve requests correctly, or a deployment or update process failing partway through because there is no longer sufficient free space to complete a file write operation. Someone encountering any of these symptoms without specifically checking disk usage first might spend considerable time investigating the wrong layer entirely, checking database configuration, checking web server settings, before eventually discovering the actual, underlying root cause was simply an exhausted disk, itself caused by log files that had been quietly growing, unmanaged, for months.

What logrotate actually does

logrotate is the standard, widely used Linux utility specifically designed to solve exactly this problem, typically running automatically as a scheduled daily cron job, checking configured log files against defined rules (rotate when a file reaches a certain size, or on a defined schedule like daily or weekly) and, when a rule's condition is met, renaming the current log file (appending a date or number), optionally compressing it to save space, and starting a fresh, empty log file for the application to continue writing to, while also enforcing a maximum retention count, automatically deleting the oldest rotated, archived log files once a configured number of retained archives is exceeded, ensuring the total space consumed by a given log source's history remains bounded rather than growing indefinitely.

Most Linux distributions include logrotate pre-installed and pre-configured for certain common system log sources by default, but application-specific logs, particularly for web applications, custom scripts, or specific services you have installed yourself, frequently are not automatically covered by this default configuration, requiring you to explicitly add a configuration for them.

Checking your current disk usage and log sizes

Before assuming log rotation is the specific cause of a disk space issue, confirm directly rather than guessing: df -h shows overall disk usage across your server's file systems, quickly confirming whether disk space is genuinely constrained at all. If it is, du -sh /var/log/* (or the equivalent path for wherever your specific application logs are stored, which can vary) shows the size of individual log directories and files, typically making it immediately obvious if one or more specific log files have grown to an unreasonably large size, often clearly the largest items on the entire disk once you look directly, a quick, direct confirmation before proceeding to configure or fix rotation for the specific offending log source.

Setting up logrotate for common log sources

For web server logs (Nginx, Apache access and error logs), most current distributions and web server installations include a reasonable default logrotate configuration already, worth verifying directly (checking /etc/logrotate.d/ for a corresponding configuration file) rather than assuming, since some manual or non-standard installations may not include this default coverage automatically. For application-specific logs, custom application error logs, a Node.js application's own logging output, a PM2-managed process's logs (relevant to our PM2 vs systemd guide, since PM2 itself includes a log rotation module worth specifically enabling if using PM2 for process management), or any other custom log source, you generally need to add your own logrotate configuration file explicitly, since these are not automatically covered by the system's default configuration the way core system logs typically are.

A sample logrotate configuration

A representative example configuration file (commonly placed in /etc/logrotate.d/ with a descriptive filename) for a custom application log:

/var/log/myapp/app.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    create 0640 www-data www-data
}

This configuration rotates the specified log file daily, retains 14 days of rotated archives before deleting the oldest, compresses older archives to save space, and handles edge cases gracefully (not erroring if the log file is temporarily missing, not rotating an empty file unnecessarily). Adjusting the specific retention period and rotation frequency to match your actual needs, how much log history you genuinely need to retain for troubleshooting purposes versus how aggressively you want to limit disk usage, is a reasonable, situation-specific tuning decision rather than a fixed universal setting.

What to do if you're already out of disk space

If you have already reached a genuine, active disk space exhaustion situation, the immediate priority is freeing space quickly to restore normal server function, which typically means manually truncating or deleting the largest unmanaged log files you identified in the checking step above (> /path/to/logfile.log truncates a file to zero length without needing to delete and recreate it, a quick, direct emergency fix), then setting up proper logrotate configuration afterward to prevent the same situation from recurring, since manually clearing the immediate problem without addressing the underlying lack of rotation simply delays the same issue happening again in the future.

FAQs

How do I know if logrotate is already running automatically on my server?

Check /etc/cron.daily/logrotate (a common location for the scheduled job) or check your system's cron configuration directly; on most standard Linux distributions, logrotate is installed and scheduled to run automatically by default, though the specific log sources it actually covers still need to be verified separately, since installation alone does not guarantee every relevant log source has an appropriate configuration in place.

Does compressing rotated logs affect my ability to search through them later if needed?

Compressed log archives can still be searched using tools that handle compressed files directly (like zgrep, the compressed-file equivalent of the standard grep search tool), meaning compression saves meaningful disk space without significantly hindering your ability to review historical log data when genuinely needed for troubleshooting.

How much log history should I actually retain?

This depends on your specific troubleshooting and compliance needs; a common reasonable default is retaining one to a few weeks of daily rotated logs for most application troubleshooting purposes, though certain compliance or audit requirements might necessitate longer retention for specific log types, worth considering against your actual, specific requirements rather than defaulting to either extreme.

Can unrotated logs actually cause a security issue, not just a disk space issue?

Indirectly, since an exhausted disk can cause various services to fail unpredictably, including security-relevant services like logging itself (ironically, a disk-full condition can prevent further logging from being recorded at all, creating a gap in your security visibility at exactly the wrong moment), making proper log rotation a contributor to overall system reliability and observability, not purely a disk-space housekeeping concern.

Does WordPress itself generate logs that need rotation, or is this purely a server-level concern?

WordPress's own debug log (wp-content/debug.log, covered across our various WordPress troubleshooting guides) can also grow unbounded if debug logging is left enabled for an extended period without any cleanup, worth specifically checking and managing separately from server-level logs if you have had debug logging active for a prolonged troubleshooting period.

Is there a graphical or simpler way to manage log rotation without editing configuration files directly?

Some hosting control panels provide a simplified interface for basic log management and rotation settings; for a VPS with direct root access and more custom application logging needs, directly configuring logrotate via its standard configuration files remains the most flexible, standard, well-documented approach, even if it requires a small amount of direct file editing.

How often should I proactively check disk usage rather than waiting for a problem to occur?

Setting up proactive disk usage monitoring and alerting (similar in principle to the uptime monitoring covered in our dedicated guide, many monitoring services also support disk usage threshold alerts) is a more reliable long-term approach than manually checking periodically, ensuring you are notified before disk space becomes critically low rather than discovering the problem only after it has already caused a service disruption.

Conclusion

Unrotated log files are a common, entirely avoidable cause of mysterious VPS disk space exhaustion, growing unbounded in the background until they eventually trigger confusing, seemingly unrelated failures elsewhere on the system. logrotate is the standard, well-established solution, but application-specific and custom log sources frequently require explicit configuration beyond whatever default coverage your system provides for core system logs alone.

Checking your actual current disk usage and log file sizes directly, rather than assuming this is or is not the cause of a given problem, takes only a couple of minutes and either confirms or rules out this specific, common cause quickly before you invest time investigating a more complex, less likely explanation.

HostCloud's VPS plans include disk usage monitoring and alerting as part of our hosting management tools, helping catch unmanaged log growth before it causes a genuine service disruption. See VPS plans at https://hostcloud.in.