"zrepl status" always errors out for me, on two hosts

while on the latest bloody bits but is fine on the previous ones

That's strange, but at least it's a narrow window to look at!

Since it's happening on OI too, it must be some change in gate, and I think richlowe narrowed it down to between june 9th and 30th

Well, not "must", but likely..

just booted back to my previous be as it was getting annoying

I am not seeing anything strange in vim. Are you seeing this on the console or is it an ssh session or something?

ssh session

vim was a mixed bag

zrepl status triggered it 100% of the time for me

(And I am debugging a sync issue so it was a problem)

That looks fine for me too. iterm2 and mac terminal, various sizes

weird

Where is your primary console?

serial

Mine is on serial, where zrepl/vim have always been funcky for me

client side I tried iterm2 terminal.app and putty

So it must be something in combination with a configuration I have I guess

But what

Well, you and yuripv (who saw it on OI too)

Once I have fixed the zrepl issue, I'll update again, assuming there will be another bloody build soonish with the openssh stuff fixed.

I'll only do one then so it's easier to compare A and B

The updated openssh is already in bloody.

But yes, somebody who is seeing this problem will have to bisect packages and try and find where it was introduced

Sadly neither of mine are easy to reboot, it takes a long time

gist.github.com/sjorge/defa35d58989ba40bf2c0dcf1680f432 does this help?

That's the panic zprel throws from last night

I still had a terminal window open

Doesn't seem to be any changes between now and the working be that would come even close to terminal handling I think

5.11-151047.0:20230609T142524Z

Is the exact ts from the working build

At least I found the zrepl not syncing problem now I can actually run zrepl status, my certs expired

I guess it's time to move that to wireguard + tcp transport instead as I had the same issue lat year too

I'll switch back to the new be on one of hte nodes after

re-updating one of my hosts now

Done and I can trigger it again with zrepl

Let me do an truss on both I guess

OMGFG

when I run it though truss it works

without truss it fails -_-

Aha it's not truss

I guess like yuripv I can't trigger it all the time I just got unlucky ;(

I did get a truss output now from when it fails

It seems to panic shortly after a TCSETS (L1003)

Based on the lines above it started to draw it's box layout and then calls that ioctl and it poofs out of existance

That would also explain why andyf could not reproduce it as it's indeed 'random' I must have just gotten unlucky last night to repeatedly hit it

sjorge: im doing zrepl over tailscale with the tls certs. I think I set the to expire in a year so I will likely be puzzled in a years time when things break haha. I should add a cron job to alert me when they are close to expiring

Does zrepl support using certs that you provide? I wonder if you could use tailscale-provided certs...

I think "tailscale serve" might even auto-renew them for you...

papertigers I was like I can either rotate them or... push it over wireguard

I wanted to setup a tunnel between the tiny backup box at my mom and home anyway

Since it's encrypted over the internet via wg I just went to tcp transport

No point in double encryption tbh

Seems to work well enough, with nahamu work on getting wireguard-go into omnios-extra

Is the wg connection using my bits, or is the tunnel handled by a non-illumos kernel?

See above :p

asked and answered

nice!

I wrote my own wg-quick manifest... then found the one from the omnios package

So i dropped mine

If your old one had any improvements over mine, please let me know.

It did not

Only downside is that the interfaces are named tunX tbh

yeah, that's a shortcoming of the tun driver.

but at least you can name your config file however you want.

Yeah I noticed that

I had it called tun0.conf first but that got messy so now I'm back to wgX.conf naming

And let wg-quick handle it

the tun driver works just well enough that no one has enough drive to write a better one.

Having it fold into ipadm some day would rock, but I guess that means we need a none wireguard-go implementation first

Exactly :D

Eh its a bit quircky but it works well enough -> setup and stop caring about the quirckiness

uh, my tailscale port does use ipadm. does my wg-quick script not??

It does fro the ip stuff

but something like ipadm create-wg -l wg0 ... would be nice too

Well I guess you need a dladm create-wg + ipadm create-addr technically

Ah, yeah, we'd need to port wireguard into the kernel. Someone smart enough might be able to port the FreeBSD kernel port over...

We can probably get the wg dev to look over a port before integration if one ever get smade

How the freebsd one went was a mess, but he did end up reviewing and fixing a ton of bugs in the first attempt, ideally it would just be reviewing but the port for freebsd has a whole history about it -_-

It was the hallway talk at the EuroBSDcon I went to before covid

Yeah, I think the current version is fine, but yeah, the first version was a mess.

we still don't know what's wrong with the netbsd version :)

nbjoerg: is it based on the FreeBSD one or somethign else?

written from scratch

("no, you are not supposed to do that, you are not smart enough!")

Wrong as in doesn't work or wrong because it's not 'approved'

rmustacc: both

according to him

He does seem very opinionated on the crypto stuff used

But that not necessarily bad (assuming he knows what he's doing and not a total ass about it)

the amusing part is if his choices ever prove to be wrong, the protocol is designed such that you can't practically do any sort of rolling upgrade

there's really no way to specify a 'v2' set of mechanisms... basically upgrade everything everywhere at once or stand up entirely parallel infrastructure and move things over

yeah, that's my pet peeve with the design

i also find the whole 'you must assign every client a static IP' thing annoying

that and the lack of a configuration protocol

like that's fine for tiny deployments

i don't see how that really works well in large deployments

it doesn't, that's why some vpn providing shrinkwrapped wireguard

kind of like the various commercial vpn tools on windows

...so compared e.g. to openvpn, it can actually be a step back

Yeah I do hate the needs static assignment

I like my pipes encrypted and setup, addressing should not be part of the pipe laying so to speak

But the performance is a lot better for me than openvpn (which I also use for mobile devices -> home connections)

isn't openvpn basically vpn over http[s] ?

Yes

Well TLS

I run mine on port 443 for 'random wifi that is not mine' compat reasons

jbk: I think that's tailscales value add. Management layer on top of wg that seems to work really well

i had (back at joyent) started on the _very_ beginning bits for kernel support

starting with chacha20-poly1305 support

though the openbsd code i was using to try to add to our existing chacha20 support was headache inducing

because of the rampant type punning happening everywhere

which by extension made it rely on the implicit C struct layout and padding for correctness, which made me uncomfortable

at least from my recollection

[illumos-gate] 15793 When preferred_dc is set, DNS outage can cause smb/server and idmap startup to timeout -- Matt Barden <mbarden⊙rc>

in retrospect, it probably would have been easier to not even try to reuse the existing chacha20 code and just have a completely separate chacha20-poly1305 implementation

(I remember being kinda of shocked with the quality of the code given openbsd's vaunted reputation for security)

but the thought was to add the needed mechanisms to the crypto api, then could implement the protocol in the kernel using those

[illumos-gate] 15794 Want file name in oplock break timeout messages -- Gordon Ross <gwr⊙rc>

and to maybe extend pf_key

since IIRC at least openbsd (and maybe freebsd) basically requires the kernel to hold all the info for every client in kernel mem

which (again) seems bad for larger deployments

and might allow for more flexibility on how you manage client keys

andyf: I didn't narrow it down, I thought I had, but it isn't happening to me in general.

I thought I narrowed it based on me being fine on the 9th and yuri broken on the 30th

but I'm also fine in july

or I'm mis-organizing the repro steps.

i had a 100% reproducible rate yesterday but like 1/25 ish now

i hope it's not timing based

i did capture it with truss

for me it's like `sudo vim ...` is broken, and then everything else starts to behave weirdly

did the openssh update and reboot, it's now close to a 100% again for zrepl status :/

moin, assuming I want to understand what goes wrong in illumos when I get: NOTICE: NVRM: GPU 0000:01:00.0: Failed to copy vbios to system memory. - how would I start debugging it?

ej tsoome!

Agnar: So I can't find the string vbios in illumos.

The nvrm makes me think this maybe is nvidia related. Is that the case?

rmustacc: yes, sorry I forgot to mention the important part. It's the /kernel/drv/amd64/nvidia driver version 470 that ships with OI. It should work, the card IS supported but it seems we have some issues in OI with it

rmustacc: the driver from freebsd works, so does Linux

OK. I'm not really sure how to suggest debugging that. I guess try to see what APIs it's calling around when it emits that message.

rmustacc: the question is, can I somehow get more informations about the driver with kmdb? ::modinfo seems not to provide more informations

rmustacc: I also think of trying Ghidra on the module...

It would be very helpful if we had some illumos folks at nvidia :/

There are some folks who used to work on illumos there.

Anyways, I would use kmdb to step through stuff until you find where the message is coming from or similar.

I don't have great suggestions here.

I don't think the people we know at nvidia are at the part of nvidia that would be interesting to you.

Well, actually, if you try to load that module again after boot can you get that message to appear again?

If so, then you can start doing a bit of DTrace. Though is there even a symbol table?

nvidia used to provide "source" within the package they themselves provided, to the non-juicy bits.

back when it was an SVR4 package, anyway.

it's unlikely to tell you anything about NVRM, but might get you a little somewhere.

richlowe: I guess starting Xorg will trigger that again. good idea.

richlowe: I'll play with that tomorrow, thanks so far!

If you can get it to reproduce the message then you can use DTrace to get a stack trace for what it's worth.

rmustacc: yes! I'll get my dtrace book tomorrow and try to get a stack trace and maybe I get an idea why the copy fails :)

I mean, if you can get that message to reproduce reliably after boot then you could do something like fbt::cmn_err:entry{ stack(); }

there are even fbt probes in the nvidia module

I was under the impression that fbt can automatically find function boundaries to trace at runtime rather than needing those probes to be configured at build time.

That's correct. fbt is just looking at the symbol table basically.

sdt is build time static probes.

ah!

well, it's late, I'll try that tomorrow. thank you all so far