Discussion between Martin Barker and Wayne Werner

12:16 AM

Q: Why does `chattr +e <file>` fail with no space left on the device when there's plenty?

I have a problem. chattr +e <file> does not work, if the file does not have the +e attr set on it, or if it's removed and re-added. [root@ip-203-0-113-13 log]# df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 3.8G 0 3.8G 0% /dev tmpfs tmpf...

Wayne Werner

@Iain ext4, apparently

Martin Barker

does chmod +e work when chattr is breaking or give you a better error?

Wayne Werner

@MartinBarker chmod: invalid mode: ‘+e’, so... no I don't think so

Martin Barker

@WayneWerner oh sorry I did not mean exactly I meant for the correct group (so Owner, group or other) so can you try chmod o+e <file> replacing <file> with your file name? you will need to set that back i just want to see if you can modify permissions it might provide an error that might help or tell us it could be a busted FileSystem

Wayne Werner

@MartinBarker still doesn't work... e isn't a file permission, it's an attribute. I'm not aware of chmod being able to set file attributes.

Martin Barker

12:16 AM

Just updated to cover that I was thinking if it's erroring as well it might help identify the cause if not it could be a broken FileSystem allocation table / journal .

Have you tried running e2fsck it might show you errors and fix them for you if it is a broken journal

Wayne Werner

Hm. That's an interesting thought

I'm going to dump some strace output in here

Martin Barker

okies

Wayne Werner

execve("/usr/bin/chattr", ["chattr", "+e", "foo"], 0x7ffe758e8f70 /* 21 vars */) = 0
brk(NULL)                               = 0x560b87727000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=24327, ...}) = 0
mmap(NULL, 24327, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f18991fb000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libe2p.so.2", O_RDONLY|O_CLOEXEC) = 3

Martin Barker

According to execve("/usr/bin/chattr", ["chattr", "+e", "foo"], 0x7ffe758e8f70 /* 21 vars */) = 0 it did not error gave a response code of 0 meaning everythings ok and not an error code

Wayne Werner

well, looks like it can't check while the drive is mounted. Which is a problem, as that's where the commands exist, too ;)

I found it particularly interesting that this openat(AT_FDCWD, "/etc/ld.so.cache"... bit, which if you scroll down to the ioctl(3, FS_IOC_SETFLAGS, ...) = -1 ENOSPC

that error seems to be somehow related to /etc/ld.so.cache

I don't know anything about that file, though

Martin Barker

12:28 AM

Well ld.so.cache is ldconfig applications cached Link Library code

yeah it's a FileSystem problem, "The filesystem is returning ENOSPC when attempting to copy metadata." from github.com/trapexit/mergerfs/issues/443

that bug is the same problem just different software

Wayne Werner

other chattrs work fine

that's the weird part

it's just +e that doesn't

Martin Barker

yeah thats what that bug has in it

I don't know a lot about how the ext4 FileSystem works to be able to know much about it I'm afraid but it does appear to be a FileSystem error why that error is occurring i don't know the only thing I can suggest is you try e2fsck and see if it can find an issue in the Journal relating to that file

Just from a programming point of veiw, there are not any locks on the file are there?

Wayne Werner

12:44 AM

nope. Quite literally touch foo; chattr -e foo; chattr +e foo, kaboom

Martin Barker

Really strange FileSystem error then sorry I could not be of more help at least we narrowed it down to a FileSystem error...

Wayne Werner

I'll have to poke around and see if I can create/mount some other volume (or check this somehow? Cause I know I can't run e2fsck while the drive is mounted... or while it's unmounted since all the application is on that drive ^_^)

Martin Barker

If you have access or can get someone to plug in a USB bootable Linux OS you could check it but i don't know past that im not sure what you can do.

Wayne Werner

1:15 AM

not for EC2, heh

last day (15 days later) »

all rooms