Alrighty, everything is now update to 15.1-STABLE

nice

im still rc3 going to reinstall later just because

well not rc

Well that wasn't dramatic. pkgbasify was quick and just worked.

yep

rtprio: Welcome to the world of scientific research. I've been using nixos since 2016 on various systems, including nixos unstable on my work laptop. Though I'm conservative when it comes to running production servers and still prefer a more proven system there

There's a lot of interest in systems like nix in the functional programming research community, too

I did it. My VM is on 15.1-RELEASE.

scientific research‽

MelanieUrsidino: edolstra.github.io/pubs/phd-thesis.pdf

o_O

i've never seen a p0 release

is that from the bug that was sent out in the mailing list a couple days ago?

hc: have you looked at the software bill of material (SBOM)? The concept of being able to "track" software and its dependencies have always been a "fun" avenue as inconsistencies are that constant variable to balance.. back to reading it a little more

voy4g3r2: Haven't looked into it beyond briefly trying out syft, but it seems to have limited support for .cabal files

I like the idea of sbom. It is an attempt to make an important aspect of software security (transitive dependency versions) more visible

just upgraded a vm from 15.0 to 15.1 using freebsd-update and it went smoothly, thanks for making this process so easy

Q. I'm debugging something, and I need to be sure --- does `fetch` do keep-alive / reuse connection if I call it multiple times on the same site in a short timespan?

Or is each run of `fetch` a new connection, always?

You mean the command fetch as in /usr/bin/fetch? The shell should fork,exec on each invocation, how can it preserve state and reuse a connection?

hc: Correct. And you never know, there could (in principle) be a `fetchd` to preserve it over short timespans, or the process forks a background process that stays around for a minute or so with some per-site socket.

Just doing a sanity check because I'm dealing with a heisenbug of networking ......

(TL;DR most fetches work, occasionally I get a "connection reset by peer" ...... to an internal [to bhyve/VM] network)

(my reverse proxy is even weirder, it's pretty consistently every 2nd request that fails)

DarkUranium: true. I don't know fetch enough to be able to say with certainty, though I'd be really surprised

Same, but at this point, I felt like I needed the sanity check.

The issue's driving me crazy >_<

Yeah, I know you need a sanity check; you need absolute certainty and I can't help you with that

Fair, thanks anyway ^^

Okay, this is weeeeeeeeeird.

The jail has a 3.8-12% packet loss (too small sample size to tell for sure) to a *container in VM on the same machine*.

The host (of the jail) has 0.0%.

Maybe too early to start asking this, but intel or Realtek nic?

scoobybejesus: It doesn't go via the NIC, it's jail -> host -> bhyve VM -> container

But I can check if it matters.

DarkUranium: Have you run fetch with -v yet? It should give you some clues as to whether it uses connection keep alice

I just ran it with two URLs and it looks like even when specifying two URLs in one go to the same host it will establish two connections in sequence

(-vv might be more helpful even)

Apparently, I was being DoS'd yesterday. But that was yesterday.

Okay, I *think* I managed to get a bit further.

`ping` form the jail has a packet loss. `ping` from host does not.

(seems to be kinda random, 0-12%)

traceroute is different, from the jail, it goes via 10.88.0.1, a bridge

DarkUranium: and your virtual network path issue?

sig`: Hm? Er, which path issue?

DarkUranium: your host will be good because the jail has loss because only the jail path crosses epair and bridge

vnet jails use their own isolated net stack and epair/bridge

just learned this too

Sure, but I'd expect 0% loss since it's not an *actual* cable. But maybe.

Either way, the packet loss is no longer showing up *at all*, *ever* (as of a re-test a minute ago) ...... but nginx is still getting reset connections.

hc: FWIW, not only does it *not* use Keep-Alive, it actually sends `Connection: close` explicitly.

So it's safe to say there's no reuse, at least.

hmm, tcp connections being reset?

Yeah, to summarize (to put it all in 1 place again):

My setup is `jail (nginx reverse proxy) -> bhyve VM -> Linux -> container`, in terms of what connects to what.

All but 1 container have this problem. 1 works, in the same Linux system. No idea why.

(out of 5 or so)

Every 2nd request (or so, so far it seems like it's *actually* every 2nd request, but that's hard to confirm), nginx gets me a "502 Bad Gateway" in the browser, and logs it as: kevent() reported about an closed connection (54: Connection reset by peer) while reading response header from upstream,

Upstream IP is correct in the log, so it's not some weird round-robining going on in nginx.

probably not nginx....

nginx shows the coorect upstream ip?

Exactly. I'm thinking either keepalive (between nginx/proxy & backend, not user & proxy) or a firewall misconfiguration or an IP conflict.

any old upstream connections?

No, because I also tried restarting nginx and such.

In fact, *reloading* (not even restarting) nginx's config on the proxy seems to pretty reliably make the next request fail.

DarkUranium: ok

(but not 100% reliably, nothing's 100% here, sadly ... which is why I'm struggling with diagnosing it)

you have 1 jail working well out of the 5?

1 Linux (podman) container, but yes.

The only difference, AFAICT, is that the one that works well uses a non-wildcard certificate. But I don't see how that could *possibly* affect the connection in this way.

compare the working one app runtime with the broke ones?

hm, can you share more of your networking setup (ifconfig and routing tables)?

have you tried disabling upstream keepalive to see if that fixes it

Good point, I've not done it at upstream yet.

... so, disabling keepalive makes it not work at all. Which makes sense as to why reloading would break things (presumably, it drops keepalive connections due to [potential] config changes)

ah yeah

Every 2nd request is how 504 Gateway Time-out. So it goes 502 -> 504 -> 502 -> 504, instead of 502 -> ok -> 502 -> ok

HI

DarkUranium: a few thoughts 1.) is your network topology setup 2.) what type of firewall setup? 3.) what network card

for example.. host firewall and then have a firewall that is on the individual vnet jails itself.. unwinding what you can on the network and what is the number of interfaces and bridges that you are using..

voy4g3r2: It's an Intel i210, but note that none of the traffic passes through it.

None that's relevant to the problem at hand, anyway. The issue is between the reverse proxy and the contain behind it.

I want to ask a question.. could I but every part of my system in separate partition like usr, var, tmp.. and get a fresh 14 version on usb

then upgrade through it

s/contain/container/

or will go into problem for?

voy4g3r2: it's pf on FreeBSD end, iptables on Linux/VM.

Podman in Linux as well.

So, turns out it *does* reach the backend after all. It was misconfigured and hiding error_log.

> [info] 345#345: *783 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too (104: Connection reset by peer) while connecting to upstream

(this is backend's own frontend)

In other words: proxy thinks the backend closed the connection prematurely, and the backend thinks the proxy did it.

does anyone know how to completely remove a previously `fetch`ed release with AppJail? i used `appjail fetch destroy -v $v default` and it's not longer in the list, but there are still three datasets mounted with $v in the name

Is the tool expecting you to destroy the datasets manually?

it removes other dataset(s) correlating to $v. so my options are to just remove the other residue myself, create a ticket/minimal example or investigate the source. my hope was, that i am not the only one using AppJail around here