Calix CMS - Server Disk Full Errors
Symptoms
The CMS Server reports full disk errors. The LM alarm may look like:
Filesystem Capacity-/home PercentUsed
Datasource:
Filesystem Capacity
Instance:
snmpHRDisk-/home
Cause
CMS may have multiple devices backing up to it as well as automatic internal backups. Daily or even weekly backups of this type are not automatically cleaned out eventually leading to the disk being full. The most likely place to find the files is /home/cmsftp
You can check where the files are using the following process:
- Login to the CMS server as root
- cd /home/
- du -h --max-depth=1
- Confirm that the cmsftp directory has a large disk usage
- [root@cms-server-vm home]# du -h --max-depth=1
96M ./cmsroot
116G ./cmsftp
1.8G ./R15_1_429_linux64
121G
Resolution
Using the du command we can find the size of the various directories in /home/cmsftp. Here we see that smxbackups is the largest directory.
[root@cms-server-vm cmsftp]# du -h --max-depth=1
280M ./C7
4.6G ./C7Backups
198M ./E5Backups
14G ./E7Backups
942M ./software
4.0K ./.cache
0 ./.config
93G ./smxbackups
147M ./ont
116G .
Use find command to delete the old backups
It is generally safe to delete backups older than 30 days. You can use the linux find command to find and delete older files in the backup directory based on file creation time. This command will be destructive and so must only be run against the proper directories (not /home/cmstp directly). The following example shows how to clear out the smxbackups directory shown above
First, before you begin you should snapshot the server in question or have a server admin snapshot the server for you.
Next, try a dry run to make sure you have the command correct:
find /home/cmsftp/smxbackups* -mtime +30 (note that -mtime +30 means older than 30 days)
This will generate a very long list of files. Make sure that the newest file you see is older than 30 days. Backups generally include a datestamp in the filename such as /home/cmsftp/smxbackups/20211129/1638169200_full.tar.gz
Once you are satisfied that the files found are correct we can run the rm command against them as follows. Note that this command is destructive, if you make an error without a snapshot or backup you will not be able to recover the files.
find /home/cmsftp/smxbackups* -mtime +30 -exec rm -f {} \;
Repeat the process on any other backup directories on the server in question such as C7Backups or E7Backups. Do not run this command on software directores such as the C7 directory, ont directory, etc.
Finally, clean up any snapshots of the affected server and verify the disk usage alarms clear.