Thoughts on Whatnot
Downtime and Docker Logging
November 11, 2016

Note: Several elements of this article are no longer accurate since the site was reworked, but it has been left up for completeness.

For anyone who happened to have checked PS2 Average Login this morning, the little web-app that’s linked at the bottom right of the page, you may have noticed that it went down this morning. Of course, since this blog only has about twenty some-odd readers and they’re all spam bots being proxied through Russia, you probably didn’t notice, but still, I figured I’d explain what happened. Maybe someone will stumble across this at some point and avoid a potential issue.

It’s simple, really. My disk space filled up with logs. For various reasons, PS2 Average Login dumps a ton of logging to stdout. And by a ton, I mean it’s measured in gigabytes, and it was enough to fill up the servers disk space.

Tip: Use Docker’s handy logging options to keep log sizes manageable. Docker defaults to logging to a file stored away inside /var/lib/docker somewhere, and it will quite happily, by default, dump data into it until it fills up your hard drive. This can be configured either when the dockerd daemon is started, or on a per-container basis. In either case, simply use the --log-opt max-size=<size> flag and argument combo to set the maximum size to whatever you want.

If you use Docker Compose, the above can be accomplished by simply adding the following to your docker-compose.yml file:

Version 1:

    max-size: <size>

Version 2:

        max-size: <size>

In either of these cases, these options only work if you are using the json-file logging driver for your container. If you’ve set up logging to syslog, journald, or some other place, the max-size option isn’t available, and you’ll have to configure whatever service you’re outputting to.