Ed M on pf: hachyderm.io/@emaste⊙ms/111132944611315615 ; hachyderm.io/@emaste⊙ms/111133934066815915 ; hachyderm.io/@emaste⊙ms/111138399698480459

Title: Ed Maste: "The FUD about #FreeBSD's pf port is rather tediou…" - Mastodon

That 3rd link is awesome.

_xor: Is the syntax the same? If they are, we could use the same lexer and parser even if the backend is handled differently.

TommyC: i think there's more than just the lexer/parser

Hmm, is there a signal that triggers the process to exit with a success code?

Front-end would probably be the least of the worries, I would think.

I imagine it would be the implementation-specifics of the processing/matching engine, with my comment earlier basically alluding to that thread-safe in FreeBSD and can be scheduled on multiple cores or whatnot.

I did read a comment saying that packet processing doesn't benefit as much from multi-threading since the nature of processing/matching typically involves a window of just a few packets before all operations would be blocked (e.g. re-assembling fragments might be one example).

_xor, Most signals are due to an exceptional condition and therefore the process should exit with a non-zero exit code.

man 3 signal

Then look at the action and description of the action.

However a program can be written to do other things. And actually incorrectly handling signals is a too common bug in many programs.

Why do you want to kill a program and have it exit zero?

Packaging mishap in acme.sh? Extracting acme.sh-3.0.6_1: 100% ## [: !-f: unexpected operator

Interesting

Would that be in some post-install script?

yes parv , looks like it. happened on two different jails

moviuro, Do you have the file name?

nope

Title: freebsd-ports/security/acme.sh/pkg-install at main · freebsd/freebsd-ports · GitHub

Line 6: if [ !-f /var/log/acme.sh.log ]

^-- SC1035 (error): You are missing a required space here.

moviuro: wanna submit a bug / patch?

meena: should I do that on freebsd bugs or on github?

(and yes, I can do that)

Also, I'm not sure that script is pkg -c / pkg -r safe

imagine if man.freebsd.org/pkg-lua-script(5) had examples of how to convert bad shell scripts to good lua scripts

Title: pkg-lua-script(5)

meena, I was searching for a file in "acme.sh*.pkg" but could not find one. Oh, that is stuffed right in +MANIFEST

meena: if you want to voice your concerns: bugs.freebsd.org/bugzilla/show_bug.cgi?id=274348

Title: 274348 – security/acme.sh pkg-install bug in 3.0.6_1

moviuro: thanks

meena: doesn't pkg(8) pass some env variables like $DEST_ROOT or similar?

I don't know

i'd have to read the code, or create a package that puts env output into a log file

PKG_ROOTDIR

${PKG_ROOTDIR}${PKG_PREFIX}

instead of of /usr/local. and i think /usr/bin/install can stay the way it is, since it's not a binary that needs to be of a certain ABI for this operation to succeed

o/

RhodiumToad, you there?

?

damn, that was quick

do you have few minutes to speak a bit more about make and makefiles?

sure

awesome

So I have read yet few more articles and man page once again.

First question - makefiles in usr/share/mk are generic makefiles that we can/should include in our own projects or maybe these should be used only when building FreeBSD from the source?

bsd make includes /usr/share/mk/sys.mk automatically (iirc) and obviously therefore anything it includes.

I still didn't figure out how to force make to move my *.o files to `build` directory, that's how I have found bsd.obj.mk

but the rest is really best left alone unless you're building freebsd itself

when it comes to having a separate build dir, that's getting into territory where bsd make differs from other makes, and so you need to start thinking about portability.

hmmm

okay, one step at a time

I went through the bsd.obj.mk file and found comments how we can set custom build directory

personally, I use exclusively GNU make unless I'm working on freebsd (src or ports)

if I create `obj` in my source tree it will be used automatically

but there should be a way to set custom one with env MAKEOBJDIR=<stuff> but it doesn't work for me

maybe I should switch to GNU make, there is no really a good reason why I'm forcing myself to use it

except the fact that it's here by default and I'm trying to learn/focus on what's available out of the box

I can stick with `obj` directory for now, not a big deal though - was just curious why it doesn't work

since my main motivation is to keep learning things

eh. you can't build more than like 3 ports before something pulls in gmake as a build dep

I see, good to know

there are bunch of materials about gnu make, saw that wildcards feature which makes such things pretty simple

maybe I exaggerate slightly but gmake is very commonly used by ports

one thing that gmake isn't very good at is numeric comparison, that's about the only feature bmake has that gmake doesn't

however, bmake's syntax for most complex stuff looks like linenoise, whereas gmake has actual names for functions

I see, will stick with learning bmake basics and then move to gnu make then

do you mind asking few more questions?

go ahead

last time you mentioned keeping separate files for list of source files only in the subdirectories, do you have an example of that somewhere so I can see?

er

probably not, unfortunately

let's say I have `src` directory with `foo.c` and `bar.c`

inside

should I create there `Makefile` with sth like `SRC:= foo.c bar.c` and then .include that in my main Makefile?

I wouldn't call it Makefile in such a case

okay, so it can be any file

since Makefile is the default name used by make, its presence in a directory suggests you can run make in that directory to do something useful

.include "file" will simply attach content of that file in place of the .include line? like #include in C?

yes

does the order matter? do I need to .include "src/notaMakefile" before I will use that variable inside?

not sure if make parses Makefile once or more

the order sometimes matters and sometimes doesn't. Remember that when you assign a variable with = rather than := (and you should almost always use = unless you know why not to), the value isn't expanded until referenced

the makefile is parsed once

lemme try in the background, one sec

looks like it doesn't matter where I include this file

now I have another problem though

after including that file in my Makefile, my ${SRCS} variable (defined in a separate file in a subdir) stores "foo.c bar.c" but should have "src/foo.c src/bar.c"

I have used some fancy replacement to define objs based on srcs, OBJS= ${SRCS:%.c=%.o}

are you planning to extend this to multiple subdirs?

probably sth similar can be done with adding dir in front but I have no idea what kind of syntax that is to even start searching for some docs

at this stage I planned to have only "src", "inc" and "tests"

stuff like adding the dir in front is another thing that varies between different makes

;_;

looks like I will have to read about GNU Make sooner than I initially planned

two last questions and I need to go with a dog, sorry

1. do you have different makefiles for building binary and library or maybe you keep everything in one Makefile and simply add new targets?

2. what kind of syntax this is "OBJS= ${SRCS:%.c=%.o}"? where I can read more about it to solve other issues when I need it?

I think $(SRCS:%=src/%) might work in both bmake and gmake but I've not tested it recently

for bmake see "Variable modifiers" in the manpage, for gmake see "Substitution references" in the info or html doc

bmake has a huge array of variable modifiers that can make variable refs look like linenoise, whereas gmake relies on named functions instead

e.g. $(SRCS:M*.c) in bmake vs. $(filter %.c,$(SRCS)) in gmake

note that while both ${foo} and $(foo) work in both bmake and gmake, outside of the BSD world the $(foo) form is universally used

I see

thanks a lot once again

will spend some time on these resources before bothering you again :)

have a good one, gotta go with the doggo

o/

RhodiumToad: how hard would it be to add those GNU functions to bmake?

dunno

currently in bmake, $(foo bar) is a legitimate reference to a variable named "foo bar"

so adding gmake-style functions would break that

how do you declare such a variable?

indirectly

foo = bar baz $(foo) = quux

"... This allows almost arbitrary variable names, however names containing dollar, braces, parenthesis, or whitespace are really best avoided!" -- so sayeth the manpage

is there any way to print all default variables that make uses behind the scenes? there are simply too many options in the debug mode (-d) :P

GNU make has these in man but I see none for bmake

(going through -d A right now)

rwp: I was pretty sure exiting 0 on a signal wasn't something that was expecting to happen, but I just wanted to make sure.

Also, true and false being inverted in shell scripts is...annoying sometimes.

shell scripting is unnecessarily hard

I made a mostly working port of ee in zig: git.sr.ht/~fancycade/eez

Title: ~fancycade/eez - Zig port of ee - sourcehut git

fancycade: cool

if I'm just bending strings I prefer Tcl

i did a make installworld installkernel and i can't rm the DESTDIR, even with sudo

i get operation not permitted errors

some of the files will have flags set. probably schg, maybe some others. you can see this with ls -lo

chflags noschg on the files should fix that up for you

ls -lo in lib/ says schg on them

schg is set on the most critical files, like libc and rtld

s in schg mean secure or system?

system

there's a uchg flag which is the user equivalent

wow cool

the owner of a file can set/clear uchg, but only superuser can set schg, and not even the superuser can clear schg if the securelevel has been raised

is it good practice to always wipe the build dir before the build step?

for building the source tree? no, not usually

no sorry not build dir, the destdir of installworld installkernel

isley ty that worked. it's kinda tedious going into each dir with schg flag files and setting noschg on each. to do it recursively i type sudo chflags noschg -R dir?

oh, the destdir of installworld/kernel should usually be cleaned out first, yes

chflags -R noschg dir

that worked great, tyvm

meena: Yes, it is.

meena: I've been using github.com/go-task/task for a good while now and it's definitely been a net positive

Title: GitHub - go-task/task: A task runner / simpler Make alternative written in Go

There's also just, which I used for a bit, and it's good too, but I preferred yaml (yeah, yeah, I know) over make file syntax.

_xor: yaml is not an adequate language

never mind, someone did metacpan.org/pod/YAMLScript

Title: YAMLScript - metacpan.org

RhodiumToad how do you script a freebsd installworld installkernel? i have to chflags -R the DESTDIR then rm -rf it before clean the env but that has to be run as sudo. so do you put sudo ... lines into the script, or leave them out and remember to run the whole script with sudo like sudo myscript?

ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell there's probably worse things, but this is a nice summary

Title: The yaml document from hell

polyex: I generally check for uid=0 and if that doesn't work then I test sudo and see if it fails.

meena: I know too much about YAMLs shortcomings.

I just haven't had to deal with them too much...yet.

I don't want to have to rely on sudo when scripting stuff.

...and I'm definitely not going to setuid.

_xor, Regarding signals: A typical programing bug is to trap a signal and then exit incorrectly. Either using exit code 0 or not exiting with a signal.

Therefore if you are dealing with some locally constructed program from a friend I think it actually highly probably that signal handling is buggy with it.

Hey guys... just had one of my machines stop responding to network... logged in via IPMI only to see a "ping: sendto: No buffer space available" ... Where do I start looking for the issue?

Yup, saw that in the man page and the table of the expected behavior per-signal.

_xor, The ultimate work on signal handling is this: cons.org/cracauer/sigint.html

Title: Proper handling of SIGINT/SIGQUIT

Onepamopa: netstat -m

Onepamopa, Memory use in top?

RhodiumToad I'll add a few imgur screens here because ... ipmi

Onepamopa: most likely something has used up all the mbufs, which means either there aren't enough mbufs or mbuf clusters, or something is leaking networking memory

vmstat -z | grep mbuf might also have relevant info

prnt.sc/27QtRtcsgA6d  prnt.sc/GYt08ywpm_9c

Title: Screenshot by Lightshot

Do you already have trend monitoring set up such that you could look to see if something has happened before the event? Something like Munin or other?

Title: Screenshot by Lightshot

And of course if not then I suggest setting something up so that when this happens again there might be clues stored in the trend monitoring.

influx + grafana working collecting data about network, firewall states, etc - no spikes - everything normal

that's the first thing I've checked..

hm. no requests for mbufs denied or delayed

what OS version?

Hmm... Review the /var/log/messages log file and see if there is any clues logged there?

memory allocated to network doesn't seem too excessive.

13.2-STABLE 13-n255439-0bccb454ae96

does ping still return the error?

yeah...

this is actually the 4th time it's happened

roughly takes around 20 to 30 days ....

now it's 46 days uptime ..

hm, I don't see that commit tag, do you have some local modifications in play?

I've checked 4 times now and I don't see what could be causing this ...

yes, but nothing related to memory

just a small PF patch to allow me to keep the "added time" for IPs in the table when resetting counters

what's the branch point between your modifications and upstream?

how do I check

Hmm... "swapinfo -m"??

used: 0

Isn't that? freebsd-version -kru

13.2-STABLE 3 times

rwp: this is networking memory it's complaining about, which is all wired, so swapinfo isn't relevant

I am not very practiced reading the network memory diagnostic data RhodiumToad has suggested by comparing those to my systems those all look very similar.

rwp I already compared with another machine currently working... values are ~same ...

Onepamopa: git merge-base between your branch and origin/stable/13 ?

It was the 30-50 days of uptime causing problems. I have had systems where things leak memory and grow but everything seems okay until there just isn't enough resources. Looking for things that are squeezing the system. But none are found so...

try 9bccb454ae96

0 was a typo

aha.

ok, so this is -stable as of early March.

So.... ifconfig ix0 down && ifconfig ix0 up .... ping is finally working

what type of NIC is in use?

but .... what was the causae...

intel 10gig one

ixgbe driver? the one in the tree?

the default one

RhodiumToad, How do you feel about "netstat -na | grep -e CLOSE -e FIN | wc -l" for looking for problems?

Onepamopa: I ever met this "no buffer space avaiable". It seems that your default route changed if you have 2+ netcards.

I feel it's not relevant here

I have had systems be attacked by botnets which kept network memory overwhelmed holding many resources in close wait and fin wait states.

Onepamopa, Is this system on a LAN where it is relatively safe? Or on the Internet WAN where it must fend off Internet hostilities?

kyonsalt It's a dual port NIC but the 2nd port has no link and I've set it "down" manually on boot.

there's no indication here that network memory in total was overwhelmed

it's in OVH

notice the number of mbufs and clusters in use was not excessive and there were no counts for mbuf requests delayed or denied.

and no intr queue drops

that bouncing the interface fixes it suggests that the problem was it failing to enqueue packets to the hardware

the driver itself doesn't seem to mention ENOBUFS, but it's an iflib driver and iflib can return that in some cases.

you're suggesting a driver issue or hardware failure ?

there's some changes to ixgbe and iflib since March, but nothing that seems obviously relevant

these days distinguishing between "driver bug" and "hardware bug" can be rather ... vague

I am suspicious it might be a high network load caused by external agents. A botnet for example.

I've not used this hardware myself, so I don't know whether it's worth trying the port drivers

Bringing the interface down will flush all internal resources, and then up will reestablish connectivity. And the network disruption _might_ cause abusive attackers to disconnect and move to the next IP address.

Since the Internet packet lifetime is 2 minutes I have in the past downed interfaces for 2 minutes to force all network connectivity to fail off as a temporary brute force hack to nudge attacking botnets to move on to another IP address. It's a hack though I admit.

rwp let me show you ... prnt.sc/dyAOM_Yui0yW

Title: Screenshot by Lightshot

does that look like a ddos to you? because it doesn't to me...

this is the past 3 hours, 1 second resolution

so.. let's get back to figuring what's causing this...

Very good that you have trend data! Does not look like an attack to me from that data. It looks like everything is okay. Right up until it isn't. And then it's wedged.

exactly

At least with it back on the net you can log in using ssh instead of needing ipmi and be a little more comfortable.

Since you haven't found a specific problem I would fish further out and look through all of the logs of anything this system is doing. If it is Nginx then I would look through those logs looking for clues. And so on for whatever it does.

I've already ssh'd in and even placed some traffic back on it

rwp I've already done that and haven't found anything to indicate even remotely of an issue

which is why, even though I'm super tired and sleepy - I went here to get some help in figuring this..

because, as I said.. it's the 4th maybe even 5th time this is happening, and each time I'm pulling the latest src, compiling kernel and world, installing, doing a fresh reboot and placing traffic back

Tricky problem! I always hate it when there are few clues to what might be the problem.

and 30-40 days after that - whoa - it happens again

It happens to this one specific system? Or it happens to any of your systems?

it's happened on multiple systems

all with this same nic?

most often on this one because it's most loaded

yes

all ryzen 9s with 10gig nic

in different datacenters

I would have a hard time thinking it is a hardware failure if it is happening on multiple different hardware systems. But could still be a driver issue.

could be but ... who can tell for sure ?

I certainly can't ..

After search the wisdom of the web for this problem there are an amazing number of hits on people reporting the same problem and they mostly say ifconfig down and then up resets them back to working and then the trail ends. I assume it is a temporary reset and not a permanent fix though.

so... if any of the devs lurk around anonymously... make yourselves known :)

JFTR but are these systems running a pf or ipfw firewall? Just gathering data...

rwp: 19:18 <Onepamopa> just a small PF patch to allow me to keep the "added time" for IPs in the table when resetting counters

Thanks meena! I had not seen that in the scrollback.

it only registered peripherally and tingled my spidey sense, because I believe to remember seeing a bug or thread about memory leaks in pf

but i can't look it up now, really

Onepamopa, Since you have already taken the interface down/up reseting things and it is late time in your timezone and you say you are exhausted if it were me I would be inclined to sleep now and then hit it fresh with a clear head tomorrow. Since probably that reset restarts the 30-40 day clock I would expect things will continue through the night into morning okay. Since you have explored everything you/we can think of at this time.

meena the patch is basically a single if () to not re-set the time of the IP when table counters get flushed

If you think that can cause a memleak ... it would've happened a lot sooner I think.. not after 30-40 days

I don't think it's a memory leak as such

Onepamopa: i didn't mean your patch is causing that

oh..

I'm really out of ideas.. the only thing I can do is pull the latest, rebuild all and hope it won't happen again

but considering I've been doing that 4-5 times now ...

have you tried the port drivers?

no

what do I try ?

compile ixgbe driver from ports instead of using the one in the kernel ?

take a look at net/intel-ix-kmod in ports

check the descriptions for whether your card is supported

I haven't used this hardware so I have no personal experience with it

hm, kldstat shows its using if_igb

what does it show exactly?

 5    1 0xffffffff8215c000    74f30 if_igb.ko

link at 1g media: Ethernet autoselect (1000baseT <full-duplex>)

what does pciconf -lv show for the device?

Ethernet Controller X550

the whole output pleaser

s/r$//

ix0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x1849 subdevice=0x1563

    vendor     = 'Intel Corporation'

    device     = 'Ethernet Controller X550'

    class      = network

    subclass   = ethernet

do you have anything explicitly loading an if_igb? e.g. in loader.conf or kld_list in rc.conf

yes

@ loader

it's from an older server, i'll remove it

that's the em driver, which shouldn't be applicable to an x550 card

Id Refs Address                Size Name

 1   65 0xffffffff80200000  1f42b38 kernel

 2    1 0xffffffff82143000    11ab8 ipmi.ko

 3    3 0xffffffff82155000     3cb0 smbus.ko

 4    1 0xffffffff82159000     2870 accf_data.ko

 5    1 0xffffffff8215c000    74f30 if_igb.ko

 6    1 0xffffffff821d2000    716f8 pf.ko

 7    1 0xffffffff82244000     ab70 opensolaris.ko

 8    1 0xffffffff8224f000     2e88 accf_http.ko

 9    1 0xffffffff826e5000     3378 acpi_wmi.ko

10    1 0xffffffff826e9000     3218 intpm.ko

11    1 0xffffffff826ed000     3380 if_urndis.ko

12    1 0xffffffff826f1000     3178 uether.ko

13    1 0xffffffff826f5000     3340 uhid.ko

14    1 0xffffffff826f9000     4350 ums.ko

15    1 0xffffffff826fe000     3380 usbhid.ko

16    1 0xffffffff82702000     32b0 hidbus.ko

17    1 0xffffffff82706000     54b8 if_gre.ko

paste site!

no if_ix there?

Title: dpaste/Btpk (Plain Text)

nop...

does kldstat -v show if_ix loaded as part of kernel?

Title: dpaste/cpE0 (Plain Text)

ok, pci/ix is it

are you using GENERIC or a custom kernel config?

generic

device ix is in the GENERIC config, so that's ok

I don't _think_ loading igb will have been doing any harm, it just shouldn't be attaching to anything

certainly not necessary though.

I've already removed it from loader

so, you suggest I build the ix driver from ports and manually load it on boot?

I don't recommend that you do it or don't do it, it's just one option you might try

is that driver newer than the one in the kernel?

I don't know, you could check that. All I know is that it exists

hm, the kernel ix driver is char ixgbe_driver_version[] = "4.0.1-k"; the port is 3.3.10 ?

is there such a thing in a shell script as getting a tmp dir generated that i can use then delete after script is done?

polyex, mktemp(1)?

wtf it's perfect!!!

Hello, all.

in a shell script being run by sudo i don't suppose there's any way to get the $HOME of the user calling sudo myscript?

heya

when i make a scripted bsdinstall out of an extracted iso, i put my custom.txz distribution in usr/freebsd-dist. but trying to do the same thing from src, when i build and installworld installkernel somewhere, there's no freebsd-dist dir. so where do i put my custom.txz?

polyex: sudo by default exports SUDO_xxx environment variables to executed program, for example SUDO_USER which you can use to determine the home directory of the caller

like some command like mktemp that i can call with $SUDO_USER to retrieve its $HOME?

getent passwd "$SUDO_USER" |cut -d ':' -f 6

weird but cool ty

how do i do a conditional check in bash? if ["$(id -u)" -ne 0]; then... doesn't run when i execute it as non-superuser. how can i make it work right?

wow i just needed spaces between the [] !

The [ square bracket is a built-in alias for the "test" operator. if test"$(id -u)" -ne 0 would not work either and also needs a space.

oh wow

Try "man [" for example. The "[" is just a command but happens to be builtin for performance reasons. But it must be treated as if it were an external for all purposes of quoting and passing arguments.

Meanwhile... | & ; ( ) < > are metacharacters and special to the shell individually and don't need spaces. They separate words. If used plainly they must be quoted.

Meanwhile... ! { } are shell keywords. Used for negation and lists. Being a keyword they need spaces. So { echo foo ;} is a list. But {} is not { nor } but {} and does not need quoting. This confuses people routinely when using find's -exec echo {} + sequence and I often see people quoting it like "{}" "+" which is unnecessary extra superfluous and redundant quoting.

i kinda don't like symbols being so different

am i alone thinking that's mentally taxing?

It all evolved that way back in the 1970's and changing it after all of these years would break a lot of stuff. Best not to stir the pot.