Understanding inodes

This post covers the basics of inodes – how to identify issues tied to them, and ways to troubleshoot.

What is inode?

An inode (short for “index node”) is a data structure that stores metadata about a file or directory. Each file or directory on the filesystem is associated with a unique inode number which points to information (metadata) such as:

• File type (regular file, directory, symbolic link, device file, etc.)¹.
• Permissions (read, write, execute) for the owner, group, and others.
• Ownership (user and group) of the file.
• File size in bytes.
• Timestamps show when the file was accessed and when the inode itself was last modified².
• The number of hard links to the file³.
• Disk block pointers point to the data blocks on the storage device.

To show the inode number of a file, use the -i flag with ls:

$ ls -i foo
131218 foo

What is inode capacity?

For a filesystem, there is a quota for its inodes. The number of inodes available on a filesystem is determined during the filesystem formatting. This means that if a filesystem runs out of available inodes, we may be unable to create new files or directories, even if there is a free disk space.

Inode exhaustion

Running out of inodes is called: inode exhaustion. This can lead to weird system failures. You might suspect that the system has run out of available inodes when a user, program, or process fails to create a file or directory due to insufficient storage, as the error message indicates.

Generally, it is unlikely to run out of inodes before running out of storage. But this is not impossible, especially when there are a lot of writing small files operations happening on the host over a long period of time.

Reasons for inodes exhaustion

This is a list of factors that can contribute to inode exhaustion:

Lots of small writes (files or directories)

Frequent small writes, such as those caused by excessive writing log files or temp file creation, without log rotation or temp file clean up mechanism can contribute to inode exhaustion. Any write operation creates new inode each time, and on the long run this can lead to a depletion of available inodes.

Software builds and compilation

Application builds that involving frequent compilation can involve writing many temp files and directories. Over time, this process can contribute to inode exhaustion.

Bundled Servers

Mail servers, content management servers (e.g. WordPress), and backup servers can also be reasons for running out of inodes. It’s a good practice to be aware of how these servers deal with I/O operations and how they interact with the file system.

How to identify and fix inode exhaustion?

Identify the issue

• Review error messages, logs, and filesystem reports to determine the root cause of the issue.
• If you encounter an error message indicating insufficient storage capacity, despite being certain that there is available space, it may suggest an inode issue.

Check volume’s capacity using df and du commands. If disk space is good, check the free inodes in the intended volume:

$ df -ih

Output:

Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             242K   345  242K    1% /dev
tmpfs            247K   505  246K    1% /run
/dev/sda1        2.0M   90K  2.0M    5% /
tmpfs            247K     1  247K    1% /dev/shm
tmpfs            247K     2  247K    1% /run/lock
/dev/sda15          0     0     0     - /boot/efi
tmpfs             50K    19   50K    1% /run/user/1000

Delete unnecessary files

• Identify and delete unnecessary files. Check users’ files in /home, and ensure temporary files are cleaned up.
• Check out the /tpm directory; applications usually use this location for scratch/cache data.
• Clear the cache of package managers, like npm, by using their command line options or manually locate and remove their cache directory.
• Use commands like find to locate and delete no longer needed files.
• Clean up old log files.
• If deleting files is not an option, consider compressing unused files.

Prevent it from happening again

• Implement regular automated cleanup tasks.
• Ensure logrotate is enabled and properly configured to rotate old log files.
• If you’re running into this problem often while compiling software in your CI/CD pipeline, you might want to think about using a Docker container to build the app and then stash away the build artifacts using Docker bind mounts or docker exporter. Then delete the build left overs.