[illumos-gate] 15424 loader: support SOURCEDEBUG -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 15429 loader: error in libsa/Makefile.inc -- Toomas Soome <tsoome⊙mc>

hm... testlib.h:29:10: fatal error: mevent.h: No such file or directory

getting build error in usr/src/cmd/bhyve/test/tests/mevent, anyone else seen it?

ahm, apparently my own issue.... :)

tsoome, I don't get this SOL and KVM over ip with Intel AMT. Why not having connected monitor or dummy monitor, even relates to OS using both SOL and console post boot. Both are working while in loader, but not afterwards.

erm?

you lost me there:)

is illumos lacking drivers for intel AMT devices. So can't use then for local console nor continues to use serial console. Loades says not recognising any serial port, but displays that message.. on serial over ip ..

ok, lets write it down step by step:)

you have system missing the console?

yup, if monitor is there during the boot, all is nice, VT terminals are up, console working. If monitor is not there during boot, low res display of loader messages and never getting console and VT services are not working (it's triton headnode)

that is, with serial on lan, you get loader console, but not OS console?

is it UEFI or bios mode boot?

Hmm, got to check it i settings.. if UEFI or BIOS boot..

prtconf -v | grep efi-version

if no output, then its bios

name='efi-version' type=string items=1 so it's uefi

You know what, I think I should not bother you about graphical console, because it's not real-blown KVm anyway in Intel AMT, it requires dummy monitor adapter or monitor adapter connected to work, and that's it.

But I were thinking, if there is Serail over IP working pre-boot.. why wouln't serial over Ip be working..

meaning post-boot. like it does not recognize serial ports, ok, they ar enot there, but what serial over IP is, if it is not serial port.. ?

ok, with UEFI, the things can be more complicated. if you can boot it to loader, I'd like to see output from commands: show console, framebuffer get, efi-show -g global -v ConOut

and if you could post the output from prtconf -v

if you have older setup, it is possible your framebuffer is ignored if it is not announcing to be VGA compatible device (13075 No console messages after early boot on non-VGA graphics)

non-VGA graphics? such as CGA? :)

with UEFI you do not have to provide VGA comparible gfx

Woodstock: UART? :)

compatible*

it even does not have to provide memory mapped framebuffer.

[illumos-gate] 15439 Console noise: NOTICE: hma_vmx_init: CPU does not support VMX -- Gordon Ross <gordon.w.ross⊙gc>

i'd love if x86 servers did serial console without all of the gyrations and inconsistencies that exist today

being able to copy/paste the content is sooooooooo useful when there's a problem

Yep! See my recent experience with OCR-ing VMware screen output for fenix illumos#15431 e.g.

BUG 15431: Corner-case: pool import vs. pool taskq can cause panic (New)

↳ illumos.org/issues/15431 | code.illumos.org/c/illumos-gate/+/2673

(I mean, it's cool MacOS can do OCR on the screencap but the '0' to either '8' or 'B' is discombobulating...)

I spent many years trying to drag system vendors into having usable BMCs. Guess my current employment shows how well that turned out.

we actually do have tvs_screen_buf in tem, it does have textual data with color attributes.

rmustacc: If you have a moment, would you share your thoughts on whether we should be using xsaves or xsavec given that xsaves has a modified optimization that xsavec does not have? openzfs/zfs #14557#issuecomment-1452323431

why wouldn't you want to run a java applet to get access to your console? /s

ryao: I would focus on mitigating the issue. Don't use xsaves on affected AMD platforms.

I can maybe look at the discussion, but it is unlikely to be soon.

rmustacc: That is fine. Thanks.

If anyone has tips for me on how to fix WireGuard/wireguard-tools #17#issuecomment-1456664190 please let me know.

ipadm getting upset about e.g. "ipadm create-addr -t -T static -a local=10.254.254.1/32,remote=10.254.254.1/32 tun0/tun00" vs "ipadm create-addr -t -T static -a local=10.254.254.1,remote=10.254.254.1 tun0/tun00"

(note the "/32")

nahamu: I thought /32 was an invalid address most of the time because it is its own broadcast address.

i thought there was a bug filed for that.. maybe it was a smartos bug

ryao: you're probably right.

I see. Linux permits it here...

thought maybe it's a man page bug

I was more curious whether I should just have my code strip off the number of mask bits, or whatever.

I could also specifically only strip "/32" and perhaps "/31" but otherwise leave it alone??

This goes into a highly specific part of networking on which I know very little, so I will stop talking now.

nahamu: i remember hitting that with ikev2.. i thought there was an issue somewhere

IIRC, the remote should always just be an IP

so the local might be fine, but the remote needs to be stripped?

I'll do some testing.

nahamu: You could try asking in #networking about what should be done here.

nahamu: yeah IIRC, i think you're expressing 'send local (subnet) to this remote IP' so the remote is always a single IP

it's been a while though, so i might be remembering the details wrong

that should do it.

cmd ipadm create-addr -t -T static -a "local=$1,remote=${1%/*}" "$REAL_INTERFACE/$2"

that should fix it.

Thanks jbk!!

ryao: point-to-point links don't really have broadcast addresses.

setting both ends of the tunnel to the same IP is going to confuse things somewhere.

sommerfeld: Thanks for the correction. I usually do not use PtP links over VPN connections, so I am fuzzy on the details of what is right here.

sommerfeld: so far it's worked well enough for all my wireguard and tailscale stuff.

But I'm always open to ideas for improvements.

The remote ip of a PTP is really just a token/nonce that goes in the next hop ip address. some ip stacks just let you specify outbound interface when adding a route (and some require it). for a true point-to-point link, that's all you need to identify where the packet should go.

For a non-point-to-point link -- when the link needs to know which one of several nodes you're forwarding to -- you also need the next hop address (which the link will convert to a link-specific layer-2 address).

What does the illumos stack do?

it follows the (alas) less-than-ideal BSD model - routes must have a next-hop ip address and the kernel intuits the outbound link (interface) from that unless you specify the outbound interface with -ifp

should I be changing how I'm setting up my wireguard tun device addresses?

I couldn't answer that without knowing more about your config..

sommerfeld: the config is a tun link (from the 3rd party driver), driven by software for wireguard, wrapped in tailscale.

there's not _really_ an address at the other end of the tunnel exactly.

In this case tailscale wasn't at play, but all the rest was.

I don't remember what the third party tun does with the other address, or what the system knows after that.

I do know the 3rd party tun is bad, and I wanted to rewrite it, but then got very heftily involved in something else.

sorry nahum.

heh

s'all good.

I'm making another push towards getting the wireguard-go bits upstream.

copec was kind enough to exercise my wg-quick code and find a bug.

I stopped using your wireguard bits (directly) because your tailscale bits are so convenient.

same.

I should give tailscale a try

the package is there in both SmartOS pkgsrc and omnios-extra.

I don't think I'd have put so much effort into wireguard-go if it weren't for tailscale. I'd have worked with papertigers on his port of the rust-based boringtun.

I'd like to use the rust based one. I'll play with it more

I ending up taking a different approach -- settled on an alpine VM under bhyve running straight wireguard

yeah, running the real client is a good thing though I think.

maybe someone smart and ideally not a criminal will give us in-kernel wireguard someday

do a real tun driver too though, please, smart person :)

Do you think there are less copies with a VM than with a TUN based?

doing a tun driver would be fun, but I would definetly need guidance. Not something I could probably cook up on my own

copec: copies in what sense? Copying the packet data around?

yeah

My hunch says there's a good chance. The linux kernel wg would probably write directly to the vitio queue which is viona which I believe tries it's best to avoid copying data.

are you looking for perf?

virtio*

Yeah, I have a Linux VM now that matches the throughput I can push/pull from this smartos based box at ~300Mbit, but the wireguard-go seems to max out around 50Mbit

^With default iperf3

copec: how are you testing? I haven't seen performance quite that bad...

haha, asked and answered.

Would iptun(4D) just need enhancements? Or would a native tun(4D) need something else entirely?

Good question. I don't really know.

a question I don't know the answer to, but I believe they're entirely different.

I think iptun just lets you have IP-in-IP

not letting userland entirely take the wheel like tun does.

danmcd: you ship the 3rd party tun in smartos, you can look.

it's not great, but it usually works.

copec: I can push closer to 500Mbit.

(through tailscale)

It's a STREAMS module. /me would almost wanna read the protocol spec and start over.

yeah

danmcd: _that's_ why I've been encouraging people to start over :)

and/or write an in-kernel wireguard

papertigers, boringtun is building fine, I don't know what I was doing last night, thanks for the followup though

which I know offends your IPsec sensibilities

so much I even bothered to spell it IPsec :)

Thank you.

danmcd: I believe we need a gldv3 device that would "tunnel" the traffic through to userland similar to how the STREAMS tun drive works today

Something like that. I don't even know how the packets flow, etc. etc.

I think the tun driver largerly works the same across OSes in that regard. It's a way to let a userland process inspect/transmit packets

I remember thinking it'd be a new mac link type, yes.

it bugs me that the streams one we have today is not zone aware. So you get tun# based on who requested a device first

I like Solaris/illumos IPsec is the only implementation that has worked well that I have used. It's always been sort of fragile on everything else with too many pieces. I like the WG model

*I think the...

"WG model" Sorry if I'm fuzzy, WG == ?!?

WireGuard

Ahh.

danmcd: sorry to have distracted you here too. Are you good for the asm push?

ie, can I?

Yes! I prob. wont *merge* it until late tonight or tomorrow, but that's MY problem, not yours, and I *do* want this in for this week's SmartOS. Thank you!!!

(cue Salt 'N Pepa...)

well, don't thank me until we find out how I botched it and broke everything somehow :)

confidence is always a mistake :)

[illumos-gate] 15374 stop using aw(1onbld) and feed asm through the compiler frontend -- Richard Lowe <richlowe⊙rn>

(honestly my major worry is whether that's screwed Oxide)

They may need a local putback buddy like I do (for the LX brand stuff).

their tree contains an extra platform or 2, and things like that, there might be some merging git's weird rename logic doesn't like too

Just don't do it on your mac. :)

it'll be fine on a mac with `git pull`

shit, I knew I missed something in the heads-up

that patch/git are foolish about case preservation.

so git will work, but anything going via a textual patch will be, at best, a pain in the ass.

sommerfeld: are you satisfied with the change to code.illumos.org/c/illumos-gate/+/2665 ?

→ CODE REVIEW 2665: 15271 document viona tunable to force copy of tx buffers (NEW) | illumos.org/issues/15271

sjorge: oh, sorry, I meant to get back to that. the question was more around more context about *why* the host wouldn't return the tx buffer promptly (long tx queues on the real device? handed to local TCP for a local socket that was being left unread?)

I think that's one for pmooney, probably.

I think I remember sjorge and pmooney having talked it through, at least.

NICs generally consume the packet promptly, but local netstacks are under no such pressure

like you point out, a packet could end up in a tcp/socket buffer for a long time

so cases where the traffic terminates on the host rather than going on the network is where this has been seen?

yeah, traffic which ends up in a local netstack

are there similar issues for vm-to-vm traffic?

generally no

assuming they're both bhyve, we'll either have space in the guest RX ring to place the packet, or we'll drop it immediately

sommerfeld: i was never able to accurately recreate it locally, but when I was looking at dumps, it appears somehow we end up chaining potentially megabytes of data via b_cont at the sockfs layer when going from bhyve->native netstack

nahamu, I'm not sure if this is a wireguard-go specific thing, but with a Linux host acting a hub, all the wireguard peers can reach each other through the hub by simply having an AllowedIPs being a subnet that contains all of them

if the destination guest isn't running with viona, then normal socket buffers may be in place, so there are less guarantees

illumos.org/issues/13463 has what i was able to see at the time

→ BUG 13463: bhyve locks up when viona ring buffers on loan to a local zone are not returned by sockfs (New)

it's rather unfortunate, because allocating a buffer and copying the packet data definitely does have a cost

hence why we default to not doing the copy

nahamu, my smartos zone is working from routing packets between the tun and the lan, but not between the peers

Indeed.

(we do copy the headers for TOCTOU reasons)

I was able to reproduce this 100% reliably

I would be in an OmniOS VM where I would build a disk image (2G file) and I would "rsync -Paz disk.img gz:disk.img" from inside the VM

and then every time be sad that I'd forgotten I can't do that as the VM becomes uncontactable for a while

it seemed like somehow we were bypassing the hi water marks that should have produced some back pressure, but I was never able to dig deeper before I left joyent

The problem in this case is that rsync wants to read more data than we are able to send and so it just waits

256 ring entries by 1500 is like 350KiB of data

is this the change where we document a tunable? I was nervous about that too

but I couldn't tell if it was useful as a real property on something.

I mean, this is 100% broken today in a well-understood way

So I think we should do literally anything to unbreak it and then figure out how to make it less bad after that

right, I don't want to get stuck in the situation we have in the past where well-known and documented tunables live forever, and get passed around without being understood

but I don't know what else is appropriate, so I left it alone, until just now.

if you look at the analysis, there's around 350k of data in excess of the high water mark (sopp_rxhiwat) queued (so_rcv_queued)

all while the userland process is waiting in port_get

Setting the tuneable definitely fixes the problem, with likely an acceptable performance impact

nahamu, nevermind, I figured it out, it was nat'ing the packets between peers

providing a mechanism to un-borrow those buffers from inside sockfs is going to be a challenge

We don't know why the tuneable exists though - except that it is on for a particular type of network card

ideally you'd treat the loaned packet as a hot potato and copy it if it was going to leave the fast path.

sommerfeld: yes

but that is, as pmooney notes, complex

Hard to know when the fast path ends

yes, many ways off the fast path.

like it seems pretty legit for those buffers to be loaned for a few ms here and there

just not ... way more than that

like, it may end up in a sockfs buffer where it is _immediately_ picked up by a recv syscall

which implies then that you have to be able to _come back_ to it later and unloan them

but doing that unloaning requires a new paradigm for stuff accessing those mblks

which is what I would like it to do, but I acknowledge how expensive that will be to arrange haha (in terms of work hours)

andyf: I had just said to jclulow that I'm perhaps more in favour of setting it by default right now than documenting it, but I guess what you said changes that too

to know that they are well and truly not in use at the time you do the unloan

leaving it configurable by /etc/system and friends, but defaulting to on (copy all packets) seems like a reasonable compromise

It seems like it could be relatively easy to also decide not to copy in specific circumstances we know to be safe

something adaptive might work well enough (loan if you have headroom, but start copying if you've loaned out more than X% of your TX descriptors?)

Yeah that could help too

but like we must be able to tell somewhere along the way if a buffer is going to sockfs for userland, or if it's going to another viona, say

I know we can't claw them back, but we could decide copy policy _on the way in_

copec: that's a relief.

But it really feels like we should just set the tuneable by default (which will slow things down, but they will work) and then work incrementally on economically motivated performance improvements later

jclulow: yes, agreed; defaults should be safe.

I didn't mean to dip out immediately after posting but i got called on the oncall number.

but yeah, ideally it would just work without a tunable and for a lot of people it does.

i just seem to be extremely good at hitting it (multiple vms -> nfs server in a zone on one host)

and i'm personally not seeing any noticable impact on performance, but it's not like i did a lot of benchmarks or somethings

just happy the vms dont get stuck 4+ times a day