knielsen: i can try it on 13.2 and 14.0-CURRENT later

graphics/p5-Image-ExifTool <3

reproduced on 14-current

(current as of a week or so ago)

also reproduced on 13.2-stable as of, um, about Jul 1

Ok, thanks for testing

can you file it as a bug?

Yes, will do. On bugs.freebsd.org?

yes

will do, thanks for the hints and help

_xor: boot/loader.conf is processed before booting the kernel, rc.conf is processed after booting and only when going into multiuser mode

the more interesting question of course is why is it happening

specifically how are we in tmpfs_write without an exclusive lock on the vnode

seems to be agnostic to the state of debug.vn_io_fault_enable

and indeed, tmpfs doesn't seem to declare MNTK_NO_IOPF so vn_io_fault shouldn't be in use anyway

tmpfs doesn't enable MNTK_SHARED_WRITES so there should be an exclusive lock

lock type tmpfs: EXCL by thread 0xfffffe0083f673a0 (pid 814, a.out, tid 100136)

vkarlsen: awesome tool, isn't it? I use it to remove all image/video metadata before uploading anywhere.

aha! mystery solved.

the problem is not in tmpfs_write but rather in the read side

kenrap: Yeah, and it can output to a number of different formats <3

now the interesting question is why doesn't it happen for other filesystems

knielsen: if you haven't filed the bug yet, an important fact is that sysctl debug.vn_io_pgcache_read_enable=0 prevents it

knielsen: the pgcache_read code path does not lock the vnode, which is why the issue occurs

Ok, I'll include that in the report

ooh

and now I can make it happen on real filesystems too

ouch

so it's not just tmpfs

the problem is that the vn_io_fault code path is hiding the bug

if you disable vn_io_fault, and enable vn_io_pgcache_read_enable, then the bug shows up for ufs filesystems

Cool! Just another example of how it's often worth it to get to the bottom of random/sporadic test failures and not just ignore them...

the bug doesn't occur with vn_io_fault because that respects range locking on the data, rather than relying on the vnode lock

(i.e. the write operation registers a lock on the range of offsets it is writing to)

vn_io_pgcache_read_enable is trying to optimize the common case where file data is already in the vm object, but in the process it's sneaking through all the locks

so. workaround is sysctl debug.vn_io_pgcache_read_enable=0

that'll make all tmpfs reads go through the vnode lock and tmpfs_read

anyway, important to stress that even though this shows up mostly in tmpfs, it's a bug in the VFS layer generally

ack

don't worry about putting all the details in your report, I'll comment on it with my findings

next question to figure out is why the foffset_lock mechanism wasn't preventing this

ah, separate open files, obviously

and vn_rangelock is not invoked on this code path

ok, I think my understanding is complete

in fact it's possible that vn_io_pgcache_read_enable is broken beyond easy repair

ah, no, maybe not

it can only be reached if range locks aren't being used, so what it actually needs to do is to lock the vnode... but then again, the sole purpose for its existence seems to be to avoid locking the vnode

RhodiumToad: bugs.freebsd.org/bugzilla/show_bug.cgi?id=272678

Title: 272678 – VFS: Incorrect data in read from concurrent write

ta, will write up my conclusione

how did you discover this knielsen? surely you weren't just writing to the same 367 bytes from two threads?

rtprio: Tests were failing in the MariaDB CI system. It's the binary log (replication log). Writer threads are appending events to the binlog and signalling dump threads that new data is available to send to connected slaves

(I managed to get a ktrace with some specific read/write pattern when the bug triggered, and the test program is just a minimal way to try to reproduce that particular situation as closely as possible)

when a new binlog is created, two writes happen close together from two different threads. That gives the opportunity for one writer threads to wake up the reader so it can run concurrently with the second writer thread

That's why it just so happens that this race/bug is mostly triggered in the testcases at this point of creating a new binlog

ooh, patch from kib, that was quick

looks promising. I'll try it out on my -current vm

looks like it works

I wonder if a simple `tail -f` can trigger it? But may be unlikely to hit the exact timing

probably need multiple writers

tail -f uses a kevent to detect that the file has been extended, so with only one writer there'd usually be no race unless the writer is writing many small blocks in succession quickly

anyway, 10 minutes running the test prog, with kib's patch, has not produced any error

(no reason why it would, since the patch just extends range locking to this case)

that commit message needs MFCing

has anyone tested this with 12?

I have not.

I always forget about 12…

it looks like the bug is not in 12, since the offending read_pgcache code path is not there.

Khelp module "ertt" can't unload until its refcount drops from 1 to 0. <-- huh. haven't seen that before

(just before poweroff)

and of course buildworld decides it has to rebuild everything just because i enabled ccache

to spite you

I have a box on 12.2 and I want to update it but freebsd-update says `This may be because upgrading from this platform (amd64) or release (13.1-RELEASE-p3) is unsupported by freebsd-update. Only platforms with Tier 1 support can be upgraded by freebsd-update. See freebsd.org/platforms/index.html for more info.`

Title: Platforms | The FreeBSD Project

is there anyway to update it it without wiping it?

freebsd-update seems to think it's on 13.1, not 12.2?

what do uname -U and uname -K say?

RhodiumToad, I did -r 13.1 to get it to 13.1

this is uname -a: `FreeBSD freebsd-s-2vcpu-4gb-nyc1-01 12.2-RELEASE-p4 FreeBSD 12.2-RELEASE-p4 GENERIC amd64`

you didn't reboot since updating to 13.1 ?

it won't update to 13.1

what do uname -U and uname -K say?

I get that error when I run 3.1-RELEASE-p3

what do uname -U and uname -K say?

both -U and -K give 1202000

I didn't update to 13.1, I get that message when I run `freebsd-update upgrade -r 13.1-RELEASE-p3`

ok. so 12.2 is well out of support

yes :) It's a little VM I've forgotten about for a while :(

can you update 12.2 to 12.4?

let me see

I think this is looking promising

Thank you

That did work. Thank you

and from 12.4 to 13.1 ?

havn't done that yet

it's starting promising though

Best way update X.Y to X.last, then X+1.last, then X+2.last and etc.

that's why I suggested 12.2 -> 12.4 :-)

Is there a ports equivalent to Debian's 'build-dep'?

what does it do?

Builds all dependencies for a package, but not the package itself

I mainly want it so I can install the deps for weechat, but install my own weechat from the git repos

do you want to build the dependencies, or install them as packages?

Eh, I'd rather build them, but packages would work just as well, I suppose

make depends in the port dir might do it

personally what I'd do is make my own port

interesting

I'll look into make depends, thanks!

I didn't think I had installed anything via ports, so I just went ahead and did the freebsd-upgrade install and now I get `ld-elf.so.1: Shared object "libncurses.so.8" not found, required by "bash"`

whenever I try to ssh in, even if I try to run a command instead of just a shell

can you get in other than via ssh?

not sure? I can see

knielsen: interesting. good fine

*find

RhodiumToad, I have a console shell via the vps provider, but because I'm dumb I cannot find the root password in my password manager

and logging in as my user gives the same libncurses message

can you get in via single-user mode?

let me see

(if it asks for the password even in single user, there's a workaround if you can get the loader prompt)

reasons to keep a root account with a shell from base…

doesn't help if you forget the root password :-)

(i like (t)csh)

yes, I'm in with a root shell

pw youruser -s sh

er

pw usermod youruser -s sh

chsh -s also works, if you can't remember pw syntax

r/o root file system

give me a sec

mount -u /

(ideally, fsck -p / first)

(assuming this is ufs)

That worked, change the shell, now doing a pkg update

I guess I thought pkg upgrade would have happened as part of hte update. It saying to recompile anything I compiled and pkg is all prebuilt I thought

(more trying to not come off as a complete and total moron than complaining)

sshd died now it won't start claminig host keys can't be found... they're there :/

Do they have the correct permissions and owner?

Appears root, 600

what's the exact error

Not host key files found

That's with `service sshd start` and `/usr/sbin.sshd -dD`

did you change the config at all?

Started poudriere, left, VPNed back in, ssh was connwction refused. vpro to the console.

Nope

ls -ld / /etc /etc/ssh /etc/ssh/*_key

poudriere did something.. it won't quit

what OS version?

13.2

It's something not obvious.

poudriere is hunt trying to start in ineractive jail

did you try turning it off and back on?

However, /, etc, and key files are there. and file understands them

I think I'll just restart

I have oob so I can watch it

shurg. restart just fine

anything interesting in /var/log/messages?

RhodiumToad, Thank you, yet again :) I'm updating my application now. The system seems to be working

Just some single 15 for apcupsd.

signal 15 in auth.log for sshd too

That was before the restart

See if poudriere can finish...

ok well the interactive jail still hangs.. but it's killable via poudriere jail -k.. so whatever it was who knows.. now why can poudriere jail -i accept input from stdin like it always could.

echo "cmd; ...." | poudriere testport -ij ... Anyone able to confirm that works in 13.2?