okay so it seems that it would write them out "occasionally," for some meaning of occasionally

in practice, every mounted filesystem has a syncer thread, which schedules writing of every dirty page belonging to that filesystem at appropriate intervals, so that the disk copy of the data doesn't get too stale

this is in addition to the pager's activity of reclaiming pages

right, makes sense.

this is because if you mmap a file and keep writing to it, you usually want those changes written out in the background even if the system is under no pressure to free up memory

you can use MAP_NOSYNC to disable this behavior, btw

the syncer thread also ensures that filesystem metadata is pushed out to disk in cases where it didn't need to be immediately written

but, just to be clear, when doing regular file i/o, there are no more block devices and so the file i/o would not use this unified VM cache and instead it would be send straight to the disk, correct?

no

this is slightly complicated by the fact that although the old buffer pool was eliminated, the buffer _headers_ for it were not, they just now point to VM pages usually

very broadly speaking, what happens if you do a write() call to an open file on a UFS filesystem is something like this:

the write call goes through the file table and the vnode table and arrives at ffs_write, which repeats this sequence:

allocate a buffer header, ask the VM to populate it with existing data if this is an unaligned write, move data from userspace into the buffer,

then tell the buffer management code to possibly schedule a write (immediate write, async write or delayed write depending)

and repeat until no more data is to be moved

but the point is that the buffer headers don't point to some different part of memory, but rather the _same_ memory pages that would be mmapping the file if it is mmaped

so e.g. if you mmap a file that was previously written to, the mmap will just map in the already-existing pages, it won't try and copy anything

the buffer header management code works in coordination with the VM system

it indeed does sound quite "unified." :)

obviously a fair amount of how this works is up to the filesystem code

so for example ZFS does its own thing which is likely to be somewhat different

(which has led to bugs in edge cases)

the main thing to remember is that because it's all unified, there's no expectation that memory will be "free" on a system with any noticable amount of activity

and also, if you look at something like this from top(1):

Mem: 2432K Active, 3593M Inact, 3660K Laundry, 510M Wired, 165M Buf, 263M Free

you shouldn't assume that any of those correspond to either application memory or filesystem cache, almost all of the categories are a mix of both

right, because it reflects the unified VM values

in that particular example, most of the "Inact" category will be file data

but under different kinds of workloads, you might have a large Inact queue containing mostly program memory

the Active / Inact / Laundry distinction is based on whether the data has been recently touched or dirtied, not on what kind of data it is

There are some optimizations specific to files. for example, file read()/write() calls can be detected as being sequential, and optimized accordingly

with freebsd having only raw (unbuffered) disk access and not block (buffered) disk access, how does that relate to the unified VM? does the raw access bypass the unified VM, or do the pages just not stick around in the unified VM as long as the buffer cache did, or are raw accesses actually buffered?

if you write to the /dev/da* (raw) device using write(), those writes are both completely unbuffered and also synchronous

(i.e. the write() call doesn't return until the hardware has said the write is complete - bear in mind the hardware may be lying about that)

does "unbuffered" refer to bypassing the unified VM?

yes

this is why for example when copying an image file to a usb drive using dd, one should use bs=128k or so

(otherwise it will be much slower than necessary)

does this mean that the unified VM is only involved in file i/o when the file is mmapped, but not when you use, say, read/write while the filed is not mapped?

s/filed/file

no, it means that the unified VM is involved in all _filesystem_ I/O, but not to accesses to /dev/*

when you say "filesystem i/o" are you referring to only the directory structure/metadata/etc of the filesystem and not to the data of the files?

no, I mean all access that goes via a filesystem, including the file data

because if file i/o is always unbuffered due to lack of block devices, and "unbuffered" refers to bypassing the unified VM, then I don't see how file i/o would go through the unified VM

i.e. writing to /home/andrew/myfile is filesystem access, but writing to /dev/da0 is not

I literally just explained to you how writing to a file goes through the unified VM

okay, I think I see what you're saying

there are only two common cases where the lack of block devices makes a difference. One is when you copy data to/from raw devices using dd, for example to make disk images or boot media

the other is when porting FUSE filesystems from linux

okay so it sounds like the old buffer cache cached data after it was written to disk, but the new unified VM cache caches data after it's written to the file system

so it basically shifted where it was in the processing pipeline (and got unified with the page cache, of course)

so file i/o is still cached, just not at the same point

well the old-style cache didn't cache only after writing, it was a writeback cache

right

thanks for the information, it's been very helpful

also bear in mind that the "new" unified VM has been around for, I think, at least as long as I've been using FreeBSD

which is over 26 years now

where can I find doc on how to configure ipv6 DHCP ?

got a kind of urgent one if anyone is around...

?

BTX loader ("1.00") BTX Version 1.02

Any way to make it NOT probe a bad disk and wait?

oof, old-style

sec

Like I know if I get to the next stage (beastie menu) there's lots to do, but I'm stumped on this.

which disk is dead?

I don't even need the bad drive to function, I know it's bad.

It's part of an old zfs pool that's all HDDs.

It's reading out all the drives OK (2 boot SSDs, 2 HDDs on one SATA controller - 4 SSDs on the other), but the part where the spinner should be spinning, it's spinning like 1 notch every 15 minutes.

what messages have come up before it hangs?

ie: i.imgur.com/rl3BlNT.png

That's it...

The BIOS alerts me that one of the drives is marked bad by SMART (which I know).

can you disable the drive in the bios? that should be enough

BTX is a dos-extender, i.e. it's still actually using the bios for doing i/o

(you might then have to use the loader prompt to stop the kernel from accessing it)

Is there a prompt for BTX? I'm really fuzzy on it these days.

Also dug through the BIOS and was not seeing a way to disable a drive - I could set hot swap on/off, set drive type to SSD/HD and that's it.

pretty sure there's no way to get to a useful prompt before loader is up

how many drives are there and which one is bad?

(I see there are 8 in the list, is that all of them?)

8 drives and I'm not sure which one maps to which drive letter

ok, so the fact that it prints the list means that it probed all the drives

I'm torn between waiting it out and going back into the BIOS

This is the first boot into 12.4 - just did a freebsd-update from (cough) 12.2.

so it's hung at some point after bd_init, probably at the point where it tastes all the disks it found

And of course, the host is remote, so I can't physically pull the problematic drive.

was it booting ok under 12.2, or had it been up for a long time?

/* XXX this gets called way way too often, investigate */

gotta love comments like that

Up for about 4 months - last boot was a zfs scrub having issues with this same drive and it paniced a few times.

fortunately that's not the problem

ok, so the drive was known-bad from way back

It came up OK post-panic, then after I moved everything off the bad pool, I manually rebooted w/o issue.

hm

so it didn't get as far as printing the memory size

which is literally the line after devinit(); so that's where it hung?

so the next probe after that is vdisk_dev, which is probably not doing anything,

hold on, just rebooting again to get into bios and dig

but then the next one is zfs_dev

It was still at the same spot as that imgur png above (if you saw that)

so it looks like it's the point at which it tries to figure out the available zpools that's the problem, not unexpected

I'm not seeing any way to disable individual disks before that point.

Kind of a not great design to have the thing that's necessary to boot have no config...

well it does have a config, it's just that it wants to know what devices it has before doing a prompt

and without UEFI, it's a bit constrained on functionality

the host has UEFI, but never set it up on the FreeBSD side (not that I know how)

i.imgur.com/dV1etKc.jpg - sata setup it's the first HGST

no option anywhere to disable that data port entirely?

nope - I've got spinup (which is actually spinup delay enable/disable) and hot swap yes/no

Oh, the kicker? Boot drives are UFS.

Sadly my ssd pool goes across the two built-in controllers, so if I disable my SATA controller with the HDs on it, I then have another pool not working.

I'm digging through the code, but I'm not seeing any good solution yet

I hate The Cloud, but I do like "hardware is someone else's headache".

so zfs_probe probes every partition on every physical disk, and I don't see any way to exclude anything

(and it doesn't print anything until they're all done)

I wish I could know whether it's just probing very slowly or if it's just locked itself up.

The cursor under the spinner blinks and the host does reboot with ctrl-alt-del.

there are no keyboard commands here right? Like you can't cancel out or anything.

Well, I'm going to let it try booting, eat dinner (yes 1 a.m.), pull backups for two things that aren't redundant, and slap them elsewhere and then see about remote hands.

I don't see any keyboard interaction.

the zfs code isn't even calling twiddle() anywhere i can see, which explains why it isn't spinning

ugh

Would I be in this mess with UEFI?

I believe that with UEFI you'd have more chances to disable the drive before getting into the loader. but it would depend on the UEFI implementation

Is the BTX loader still used in UEFI boot or no?

not the BTX version, no

it's the same loader, but using UEFI services to scan and access the hardware rather than the legacy BIOS

the UEFI environment potentially allows executing other programs before booting, but I haven't experimented with it much

(other than under bhyve, which has a very minimal UEFI)

Hmmm, interesting.

I really appreciate you digging through all this stuff, I'm so rusty. I started back at FreeBSD 2.1.6 or so.

me too :-)

it doesn't even have boot options, does it (bhyve uefi)

but I've been using fbsd as my main desktop and development environment ever since

ixmpp: last I looked it had no support for persisting any variables between boots

ixmpp: also, I think it lacks a shell

very minimal indeed

it can load and run bootx64.efi and that's about all

RhodiumToad: i.imgur.com/ZpvBlpv.jpg it has printed "ZFS: can't find vdev details" 3x, about once/hour.

and it just booted it seems!

every time i try building with poudriere my pc freezes lol. i tried to limit the memory in poudriere.conf to a low number..does it not care if the host runs out of memory?

i.e should i be shutting all my programs and making sure the host doesn't use any memory while the jail builds stuff?

hehe froze again

trev: that sucks

wait.

just poudriere, or ports in general?

trev: can you throw your poudriere.conf in a pastebin, and tell us a bit about your system (number of threads according to sysctl -n hw.ncpu, amount of memory, et cetera)?

Also, what sort of things are you building? Does the graph include llvm/gdb, and things like Firefox or Chromium?

s/gdb/gcc/

By default poudriere will use one job per builder jail, but if you've overwritten that in order to get compilers or browsers to build faster, it's not impossible that you're ending up with only half a GB of memory per thread (which isn't a lot, when doing multi-threaded compiling). If on top of that you're putting all your artifacts (ie. distfiles, workdir, object files, et cetera) in tmpfs, that's even

more memory contention.

meena: seems work on regular ports builds

i am trying to build kitty terminal

but honestly trying to build it makes me want to not use it cause it is a bloated mess

i reduced tmpfs to 1GB

and gave 8GB poudriere under MAX_MEMORY

debdrup: let me pastebin for you

debdrup: termbin.com/qxo3

ignore the neofetch on the bottom. i though that would work to show system specs but it's just a mess

just do `head -n30 /var/run/dmesg.boot`

MAX_MEMORY and TMPFS_LIMIT are both per-jail, and with 4 threads that means you need 36GB memory.

-_-

So yes, you're having severe memory contention. ;)

Poudriere is meant to run on build servers with hundreds of GB of memory.

good to know...i just wanted to keep my system clean and then wipe the jail after the build :\

let's try again with lower numbers

if i leave then you know i crashed :')

Does your system have swap? You might be able to get away with halving MAX_MEMORY.

Swap on an NVMe SSD, I mean.

no not on nvme

just regular sata ssd

welp

instant freeze again

some people aren't just meant to be a beastie

I see that the CPU isn't exactly new; is it possible that you need to reapply the thermal grease, because the CPU is overheating?

Typically FreeBSD won't freeze when it's running out of memory, it'll just start killing processes that take up the most amount of memory.

ey.... please... a bit of respect.

cgit.freebsd.org/src/tree/sys/vm/vm_pageout.c#n1900 this is the code that handles OOM, and it's pretty good at it.

Title: vm_pageout.c « vm « sys - src - FreeBSD source tree

could be the case, but i am able to build other software

ugh

unless freebsd can't control my fans

so regarding powderkeg, I should use a sacrificial nvme instead of tmpfs?

Well, I've no real answers for you, other than that you're still probably contending on memory.

If FreeBSD can't control your fans, the CPU has its own fan control that takes o ver.

Ellenor: I've no idea how you came to that conclusion.

See my comment about poudriere being built with the assumption of a LOT of memory available.

ah

Swapping to an NVMe SSD _can_ make it possible to go a little over - ie. trev has 16GB system memory and, I assume, 16GB swap, and 4x(4+1)=20GB as the maximum memory that the builders+tmpfs can take up.

Hm.

It's not gonna as fast as it could be, but FreeBSD shouldn't be freezing from it.

aye.

trev: you can blacklist certain packages from putting wrkdirs in tmpfs

I do this for rust because my poudriere VM has 16 GB RAM

sure it slows the process down but at least there is no oomkilling going on

Oooooh, this looks pretty cool. Going to have to check this out...

Title: GitHub - ast-grep/ast-grep: ⚡A fast and easy tool for code searching, linting, rewriting at large scale. Written in Rust

"You can install it from npm or cargo!"

why on earth would i want to install anything from npm, if i don't absolutely have to?

devops.

Remilia: i turned tmpfs off

also, is there just a public repo of ports that i can just use to get bins?

cause i don't have a good enough build machine i guess

pkg install

meena: we went over this before...

it's not in the pkg repo

i have very short memory

am i missing something though?

unless I'm actually involved, it gets swapped out very quickly

you were trying to build kitty terminal?

yeah

Not sure how anything JS specifically constitutes DevOps

DevOps is just a means to an end, and is relatively tool agnostic

You could make an entire devops flow with zfs and Make if you wanted to

"devops flow"

s/zfs/zsh/g

nah, I want the other one. zfs. we store data in zfs props

i had a setup with GNU make, that provisioned VMs long before discovering Puppet

I have strong opinions about Puppet, but they're probably several years out of date

trev: poudriere supports fetching binary packages for dependencies using the -B flag, if memory serves.

That way you only use poudriere to build the package that you're looking for.

I like the idea of storing releases in ZFS snapshots or similar though, and doing a blue/green release gradually over a fleet.

Feels like a summer project

portsfallout.com/fallout?port=x11%2Fkitty%24 seemms like there's fallout for kitty though, so I'm not sure that'll work.

Title: Fallout list - FreeBSD pkg-fallout

(Make is awesome for that kinda thing because it's idempotent, or can be, if you write it correctly)

debdrup: and the error is not in kitty

trev: bugs.freebsd.org/bugzilla/show_bug.cgi?id=270871

Title: 270871 – x11/kitty fails to build with invalid UTF-8 in comment using poudriere on stable-13.2 amd64

i know this bug, but i think it's patched already

Remilia: nonetheless, this is exactly what the bugzilla is for

my ports have the latest patch

maybe the ports in my jail don't...

bugs.freebsd.org/bugzilla/show_bug.cgi?id=270703 this is the update, which mentions nothing of the bug as far as I can see

Title: 270703 – x11/kitty: Update to 0.27.1

ah yeah, correct

So this appears to have been an XY problem all along.

oh well. i've spammed the chat with this crap enouhg

i will just use st

Ligatures is a font thing, nothing to do with the terminal emulator.

heh..try it then

it was a controversial topic on the alacritty repo

trev: Irrespective of that, it's always a good idea to precisely state what you're doing, and to iterate it if the conversation seems to drift off.

XY problems aren't ever going to go away, but that doesn't mean we can't try to reduce them.

I wasn't privy to the prior conversation (or don't remmeber it because of chemobrain), so you trying to build it and running into memory contention issues from less-than-optimal configuration seemed like the problem whereas what you were actually struggling with is a bug that's already been reported.

debdrup: true. i should have looked harder for current issues with kitty. i am new to freebsd and a bit overwhelmed, while trying to rush to get things set up in my free-time. need to read more

trev: that's fine, we all start somewhere. I have a few hints though, since you're new.

If something in ports/packages is broken, going to freshports.org/x11/kitty (using kitty as an example) will present you with a bunch of icons next to the version and category (under the name and short description)

Title: FreshPorts -- x11/kitty: Cross-platform, fast, featureful, GPU-based terminal emulator

these are quick links to find dependencies, find bug reports, and find ports fallout (ie. if something is broken, you can locate the logs)

Hovering over the icons should tell you exactly what they do

That might help you ask better questions :)

If you ever need to file a bug report, docs.freebsd.org/en/articles/problem-reports is invaluable.

Title: Writing FreeBSD Problem Reports | FreeBSD Documentation Portal

thanks debdrup!

Oh, in case anyone was wondering, in talking about scrypt in crypto(9) to replace PKCS#5, Colin brought up the excellent point that in the loader environment, there isn't a whole lot of memory available (especially not on BIOS+MBR systems, but it's not exactly great in UEFI+GPT+ESP land either).

meka: your blog appears to have 404'd.

i wonder if you can build vnet transport on top of man.freebsd.org/vale(4) for interconnecting local jails

Title: vale(4)

meena: I've wished someone would do that for.. at least 10 years.

how hard can that be??

the TODO list starts with tidy my daughter's room, and goes sharply down hill from there

meena: such TODO lists are best kept separate -- things I enjoy (BSD, food, reading), another for things I have to do quickly but do not enjoy (cleaning daughter's room, lawn work) -- good luck

coreystephanphd: hi dr. corey. big fan here. every time i read/watch your work i wonder how this guy does all that he does!

sooo... for sanity checking purposes, is there any reasonable upper bound for what anyone might set kern.maxfilesperproc to?

gotta love code where the comments totally disagree with the actual logic

"At most it can be 1/6 the number of physical pages." if (maxfiles > (physpages / 4)) maxfiles = physpages / 4;

hint, 4 and 6 are not the same number :-)

hm, are there systems with >=8TB of RAM around in the wild?

because with 8TB, maxfiles = imax(MAXFILES, physpages / 8); isn't going to do the right thing

hi folks! Can anyone have a look at this? I think I'm getting stupid. Or huge bug in "ls" –> dpaste.org/gavd6

Title: dpaste/gavd6 (Plain Text)

I suppose ls refers to which filesystem item to stat when talking about "reference". stat can be used to find out the actual referenced file.

use the -l option as well as -L

if you're confused by the name not changing, that's because the name is always what you specified on the command line (or read from readdir)

"ls shall write the name of the link itself and not the file referenced by the link." posix description for -H

(part of, if you are asking why it shows the link and not the file it points to)

same for -L

(if you think about it, the purpose of `ls` is to list what the names are _in that directory_, so it would make no sense to list the target name)

(yes, in this case the target is also in the same directory, but that's a special case - in general, symlinks can point anywhere)

trying to remember which awful #unix channel i used to hang out in which had an FAQ that started with: I just discovered a huge bug in rm! — no, you didn't.

hrm… bhyve can work with vale

trev That means a lot -- thanks :)

RhodiumToad: in the wild is a relative term. Yes, vendors do claim they can sell you a system, but if you ask for a price you can't afford it.

yeah, I'm just looking at code in the kernel that goes to UB when memory exceeds some threshold, and wondering if anyone ever hit it

I've seen one system with 2TB memory.

so far, the worst assumption I've found is that physpages / 4 fits into a signed 32-bit int, which would break at 4TB

er, 8TB

I think mjg is likely to run into it first, with his attempts to make things scale.

no wait, 2TB

eurgh, can't calculate today

8TB in 4k pages makes physpages not fit in signed 32 bits, which is ok because physpages is stored as a long

16TB would make physpages / 2 overflow a signed int

so 32TB is the point at which the worst assumption I've seen so far would break

RhodiumToad: I guess that's just one more sign that `ls` is a CLI tool, not one for scripts.

michelem: why so?

(admittedly in scripts it's often the wrong tool)

what is confusing there is that all arguments (-L -H -P) have different symlink-specific semantics but produce the same output. Their description is ambiguous enough to allow thinking otherwise, and that's compounded by the fact that `ls` does list extra-dir information with `-l` . `-l` is not a usable parameter for scripting though. Indeed, `stat` is the tool for that. low-level, but never disappoints.

they only produce the same output in cases where you're not actually asking for any information beyond the filename which you already know

thanks for looking into it and for your friendly and competent answer.

though when the link is to a directory you'll see more significant differences

Hmphm. Tried searching dmesgd for 2199023255552 to find the system I thought I remembered, but either that's wrong or I remember wrong.

RhodiumToad, "Actually, finding a disagreement between the comment and the code tends to mean both are wrong" - Steve McConnell.

yeah, I know :-)

But more often code and comments and consistent, but both wrong

That could be. But the adage I quoted saves you from having to decide which is correct when they differ.

(I'm no longer a software dev. Haven't been for several years.)

I know this isn't a FreeBSD question specifcally, but I'd figure ill throw it outthere to the general public: If i self-sign a certificate for a website, is it normal to get a padlock in the searchbar but still have it say Connection Not Secure?

Yes

But depends on the browser really

Firefox in this case

the browser will throw up the insecure warning page if it can't trace the certificate to something it trusts

obviously a self-signed cert won't pass that test unless you also tell the browser to trust it

these days, of course, you can just use an LE cert instead

royalyork: google letsencrypt. you may get trusted cert for free... , other solution would be create own ca and install ca certificate on browser storage , then just sign your certs with own ca

letsencrypt is the best way to go by far if you expect anyone who isn't you to visit the website

I'll look into that, thanks