Logs and Troubleshooting

journald

journalctl -u pandadev               # one service
journalctl -u pandadev -f            # follow live
journalctl -u pandadev --since "1 hour ago"
journalctl -p err -b                 # errors since this boot
journalctl -k                        # kernel messages (disks, OOM kills)
journalctl --disk-usage
sudo journalctl --vacuum-time=14d    # delete logs older than 14 days

Classic log files

A troubleshooting method

1. Define the symptom precisely. "Site down" or "one page returns 502 since 14:05"?
2. Check what changed. Deploys, updates, config edits, traffic spikes. Most incidents follow a change.
3. Work from the outside in. DNS → network → web server → app → database.
4. Read the logs at the time of the failure.
5. Form one hypothesis, test it, and change one thing at a time.
6. Write down the fix in the runbook.

The usual suspects

df -h                       # disk full?
free -h                     # out of memory?
journalctl -k | grep -i oom # did the kernel kill a process for memory?
systemctl --failed          # crashed services
ss -tulpn | grep 8000       # is the app listening?
curl -I http://127.0.0.1:8000   # does it answer locally?

A 502 Bad Gateway from Nginx means Nginx is fine but the app behind it is not answering: check the app service and its logs next.

Log rotation

Logs grow forever unless rotated. logrotate handles files in /var/log; add one for your app:

/etc/logrotate.d/pandadev:

/srv/pandadev/logs/*.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate
}

Test it with sudo logrotate -d /etc/logrotate.d/pandadev (dry run).

Check yourself

• How do you see only errors since the last boot?
• What does a 502 from Nginx tell you?
• Why look for recent changes first?