RHEL5: Can't create sparse file bigger than 256GB in tmpfs

/var/log/lastlog gets written to when you log in. The size of this file is based off of the largest UID in the system. The larger the maximum UID, the larger this file is. Thankfully it's a sparse file so the size on disk is much smaller than the size ls reports (ls -s reports the size on disk).

On our system we're authenticating against an Active Directory server, and the UIDs users are assigned end up being really, really large. Like, say, UID 900,000,000 for the first AD user, 900,000,001 for the second, etc.

That's strange but should be okay. It results in /var/log/lastlog being huuuuuge, though--once an AD user logs in lastlog shows up as 280GB. Its real size is still small, thankfully.

This works fine when /var/log/lastlog is stored on the hard drive on an ext3 filesystem. It breaks, however, if lastlog is stored in a tmpfs filesystem. Then it appears that the max file size for any file on the tmpfs is 256GB, so the sessreg program errors out trying to write to lastlog.

Where is this 256GB limit coming from, and how can I increase it?

As a simple test for creating large sparse files I've been doing:

dd if=/dev/zero of=sparse-file bs=1 count=1 seek=300GB

I've tried Googling for "tmpfs max file size", "256GB filesystem limit", "linux max file size", things like that. I haven't been able to find much. The only mention of 256GB I can find is that ext3 filesystems with 2KB blocks are limited to 256GB files. But our hard drives are formatted with 4K blocks so that doesn't seem to be it--not to mention this is happening in a tmpfs mounted ON TOP of the hard drive so the ext3 partition shouldn't be a factor.

This is all happening on a 64-bit Red Hat Enterprise Linux 5.4 system. Interestingly, on my personal development machine, which is a 32-bit Fedora Core 6 box, I can create 300GB+ files in tmpfs filesystems no problem. On the RHEL5.4 systems it is no go.

The answer is found in the Linux source, specifically, /usr/src/linux/mm/shmem.c, starting around line 70 on my system (Gentoo 2.6.31-ish):

/*
 * The maximum size of a shmem/tmpfs file is limited by the maximum size of
 * its triple-indirect swap vector - see illustration at shmem_swp_entry().
 *
 * With 4kB page size, maximum file size is just over 2TB on a 32-bit kernel,
 * but one eighth of that on a 64-bit kernel.  With 8kB page size, maximum
 * file size is just over 4TB on a 64-bit kernel, but 16TB on a 32-bit kernel,
 * MAX_LFS_FILESIZE being then more restrictive than swap vector layout.

One-eighth of 2 TB is exactly 256 GB. Larger sizes are possible with a 32-bit kernel, as you discovered with your 32-bit FC6 test system.

It appears that changing the page size may be related to enabling HugeTLB filesystem support in the kernel. However, I don't know enough about the guts of the kernel to say how or why, or what steps you need to take to take advantage of it, or what other implications it might have. To enable it, run make menuconfig, navigate to File systems, and then Pseudo filesystems. The option in question is HugeTLB file system support. The online help for it says:

CONFIG_HUGETLBFS:

hugetlbfs is a filesystem backing for HugeTLB pages, based on
ramfs. For architectures that support it, say Y here and read
<file:Documentation/vm/hugetlbpage.txt> for details.

If unsure, say N.

It might be worth running this by StackOverflow, too. I hope this helps.

One suggestion... could you make an ext3 image in tmpfs and mount that, and put lastlog in there? Something like:

cd /tmp
dd if=/dev/zero of=lastlog.img bs=1024k count=10
losetup /dev/loop1 lastlog.img
mkfs.ext3 /dev/loop1
mount /dev/loop1 /var/lastlog