09:27:55 [illumos-gate] 15424 loader: support SOURCEDEBUG -- Toomas Soome 09:34:04 [illumos-gate] 15429 loader: error in libsa/Makefile.inc -- Toomas Soome 11:37:40 hm... testlib.h:29:10: fatal error: mevent.h: No such file or directory 11:39:24 getting build error in usr/src/cmd/bhyve/test/tests/mevent, anyone else seen it? 11:48:31 ahm, apparently my own issue.... :) 12:00:27 tsoome, I don't get this SOL and KVM over ip with Intel AMT. Why not having connected monitor or dummy monitor, even relates to OS using both SOL and console post boot. Both are working while in loader, but not afterwards. 12:01:32 erm? 12:02:03 you lost me there:) 12:03:09 is illumos lacking drivers for intel AMT devices. So can't use then for local console nor continues to use serial console. Loades says not recognising any serial port, but displays that message.. on serial over ip .. 12:04:42 ok, lets write it down step by step:) 12:04:56 you have system missing the console? 12:30:54 yup, if monitor is there during the boot, all is nice, VT terminals are up, console working. If monitor is not there during boot, low res display of loader messages and never getting console and VT services are not working (it's triton headnode) 12:32:04 that is, with serial on lan, you get loader console, but not OS console? 12:32:31 is it UEFI or bios mode boot? 12:34:20 Hmm, got to check it i settings.. if UEFI or BIOS boot.. 12:34:44 prtconf -v | grep efi-version 12:35:03 if no output, then its bios 13:11:50 name='efi-version' type=string items=1 so it's uefi 13:12:48 You know what, I think I should not bother you about graphical console, because it's not real-blown KVm anyway in Intel AMT, it requires dummy monitor adapter or monitor adapter connected to work, and that's it. 13:13:33 But I were thinking, if there is Serail over IP working pre-boot.. why wouln't serial over Ip be working.. 13:15:37 meaning post-boot. like it does not recognize serial ports, ok, they ar enot there, but what serial over IP is, if it is not serial port.. ? 13:19:03 ok, with UEFI, the things can be more complicated. if you can boot it to loader, I'd like to see output from commands: show console, framebuffer get, efi-show -g global -v ConOut 13:19:35 and if you could post the output from prtconf -v 13:20:57 if you have older setup, it is possible your framebuffer is ignored if it is not announcing to be VGA compatible device (13075 No console messages after early boot on non-VGA graphics) 13:25:55 non-VGA graphics? such as CGA? :) 13:26:16 with UEFI you do not have to provide VGA comparible gfx 13:26:22 Woodstock: UART? :) 13:26:32 compatible* 13:27:12 it even does not have to provide memory mapped framebuffer. 14:04:08 [illumos-gate] 15439 Console noise: NOTICE: hma_vmx_init: CPU does not support VMX -- Gordon Ross 15:07:47 * sjorge wishes every system had a usable UART 16:05:03 i'd love if x86 servers did serial console without all of the gyrations and inconsistencies that exist today 16:05:22 being able to copy/paste the content is sooooooooo useful when there's a problem 16:45:20 Yep! See my recent experience with OCR-ing VMware screen output for fenix illumos#15431 e.g. 16:45:21 BUG 15431: Corner-case: pool import vs. pool taskq can cause panic (New) 16:45:22 ↳ https://www.illumos.org/issues/15431 | https://code.illumos.org/c/illumos-gate/+/2673 16:45:57 (I mean, it's cool MacOS can do OCR on the screencap but the '0' to either '8' or 'B' is discombobulating...) 17:17:14 I spent many years trying to drag system vendors into having usable BMCs. Guess my current employment shows how well that turned out. 17:18:31 we actually do have tvs_screen_buf in tem, it does have textual data with color attributes. 17:40:57 rmustacc: If you have a moment, would you share your thoughts on whether we should be using xsaves or xsavec given that xsaves has a modified optimization that xsavec does not have? https://github.com/openzfs/zfs/issues/14557#issuecomment-1452323431 17:44:10 why wouldn't you want to run a java applet to get access to your console? /s 17:46:25 ryao: I would focus on mitigating the issue. Don't use xsaves on affected AMD platforms. 17:47:26 I can maybe look at the discussion, but it is unlikely to be soon. 17:50:05 rmustacc: That is fine. Thanks. 18:04:41 If anyone has tips for me on how to fix https://github.com/WireGuard/wireguard-tools/pull/17#issuecomment-1456664190 please let me know. 18:05:18 ipadm getting upset about e.g. "ipadm create-addr -t -T static -a local=10.254.254.1/32,remote=10.254.254.1/32 tun0/tun00" vs "ipadm create-addr -t -T static -a local=10.254.254.1,remote=10.254.254.1 tun0/tun00" 18:05:29 (note the "/32") 18:05:33 nahamu: I thought /32 was an invalid address most of the time because it is its own broadcast address. 18:05:45 i thought there was a bug filed for that.. maybe it was a smartos bug 18:05:46 ryao: you're probably right. 18:05:56 I see. Linux permits it here... 18:06:02 thought maybe it's a man page bug 18:06:36 I was more curious whether I should just have my code strip off the number of mask bits, or whatever. 18:07:31 I could also specifically only strip "/32" and perhaps "/31" but otherwise leave it alone?? 18:08:24 This goes into a highly specific part of networking on which I know very little, so I will stop talking now. 18:13:41 nahamu: i remember hitting that with ikev2.. i thought there was an issue somewhere 18:13:52 IIRC, the remote should always just be an IP 18:14:10 so the local might be fine, but the remote needs to be stripped? 18:14:20 I'll do some testing. 18:14:53 nahamu: You could try asking in #networking about what should be done here. 18:16:39 nahamu: yeah IIRC, i think you're expressing 'send local (subnet) to this remote IP' so the remote is always a single IP 18:16:49 it's been a while though, so i might be remembering the details wrong 18:19:04 that should do it. 18:19:23 cmd ipadm create-addr -t -T static -a "local=$1,remote=${1%/*}" "$REAL_INTERFACE/$2" 18:19:26 that should fix it. 18:19:29 Thanks jbk!! 18:50:19 ryao: point-to-point links don't really have broadcast addresses. 18:50:51 setting both ends of the tunnel to the same IP is going to confuse things somewhere. 19:06:23 sommerfeld: Thanks for the correction. I usually do not use PtP links over VPN connections, so I am fuzzy on the details of what is right here. 19:06:59 sommerfeld: so far it's worked well enough for all my wireguard and tailscale stuff. 19:07:24 But I'm always open to ideas for improvements. 19:10:09 The remote ip of a PTP is really just a token/nonce that goes in the next hop ip address. some ip stacks just let you specify outbound interface when adding a route (and some require it). for a true point-to-point link, that's all you need to identify where the packet should go. 19:12:55 For a non-point-to-point link -- when the link needs to know which one of several nodes you're forwarding to -- you also need the next hop address (which the link will convert to a link-specific layer-2 address). 19:18:05 What does the illumos stack do? 19:24:58 it follows the (alas) less-than-ideal BSD model - routes must have a next-hop ip address and the kernel intuits the outbound link (interface) from that unless you specify the outbound interface with -ifp 19:27:57 should I be changing how I'm setting up my wireguard tun device addresses? 19:28:57 I couldn't answer that without knowing more about your config.. 19:52:24 sommerfeld: the config is a tun link (from the 3rd party driver), driven by software for wireguard, wrapped in tailscale. 19:52:44 there's not _really_ an address at the other end of the tunnel exactly. 19:53:19 In this case tailscale wasn't at play, but all the rest was. 19:53:22 I don't remember what the third party tun does with the other address, or what the system knows after that. 19:53:41 I do know the 3rd party tun is bad, and I wanted to rewrite it, but then got very heftily involved in something else. 19:53:44 sorry nahum. 19:53:47 heh 19:53:57 s'all good. 19:54:34 I'm making another push towards getting the wireguard-go bits upstream. 19:55:04 copec was kind enough to exercise my wg-quick code and find a bug. 19:55:46 I stopped using your wireguard bits (directly) because your tailscale bits are so convenient. 19:55:58 same. 20:02:31 I should give tailscale a try 20:03:03 the package is there in both SmartOS pkgsrc and omnios-extra. 20:04:06 I don't think I'd have put so much effort into wireguard-go if it weren't for tailscale. I'd have worked with papertigers on his port of the rust-based boringtun. 20:04:57 I'd like to use the rust based one. I'll play with it more 20:16:31 I ending up taking a different approach -- settled on an alpine VM under bhyve running straight wireguard 20:17:21 yeah, running the real client is a good thing though I think. 20:17:38 maybe someone smart and ideally not a criminal will give us in-kernel wireguard someday 20:18:03 do a real tun driver too though, please, smart person :) 20:18:45 Do you think there are less copies with a VM than with a TUN based? 20:18:46 doing a tun driver would be fun, but I would definetly need guidance. Not something I could probably cook up on my own 20:19:22 copec: copies in what sense? Copying the packet data around? 20:19:29 yeah 20:20:30 My hunch says there's a good chance. The linux kernel wg would probably write directly to the vitio queue which is viona which I believe tries it's best to avoid copying data. 20:20:51 are you looking for perf? 20:21:11 virtio* 20:23:56 Yeah, I have a Linux VM now that matches the throughput I can push/pull from this smartos based box at ~300Mbit, but the wireguard-go seems to max out around 50Mbit 20:28:40 ^With default iperf3 20:28:41 copec: how are you testing? I haven't seen performance quite that bad... 20:28:48 haha, asked and answered. 20:31:47 Would iptun(4D) just need enhancements? Or would a native tun(4D) need something else entirely? 20:33:16 Good question. I don't really know. 20:33:18 a question I don't know the answer to, but I believe they're entirely different. 20:33:38 I think iptun just lets you have IP-in-IP 20:33:50 not letting userland entirely take the wheel like tun does. 20:34:24 danmcd: you ship the 3rd party tun in smartos, you can look. 20:34:35 it's not great, but it usually works. 20:34:37 copec: I can push closer to 500Mbit. 20:34:43 (through tailscale) 20:37:39 It's a STREAMS module. /me would almost wanna read the protocol spec and start over. 20:38:25 yeah 20:38:37 danmcd: _that's_ why I've been encouraging people to start over :) 20:38:44 and/or write an in-kernel wireguard 20:39:01 papertigers, boringtun is building fine, I don't know what I was doing last night, thanks for the followup though 20:39:02 which I know offends your IPsec sensibilities 20:39:06 so much I even bothered to spell it IPsec :) 20:39:13 Thank you. 20:40:57 danmcd: I believe we need a gldv3 device that would "tunnel" the traffic through to userland similar to how the STREAMS tun drive works today 20:41:22 Something like that. I don't even know how the packets flow, etc. etc. 20:41:25 I think the tun driver largerly works the same across OSes in that regard. It's a way to let a userland process inspect/transmit packets 20:43:47 I remember thinking it'd be a new mac link type, yes. 20:46:05 it bugs me that the streams one we have today is not zone aware. So you get tun# based on who requested a device first 20:47:02 I like Solaris/illumos IPsec is the only implementation that has worked well that I have used. It's always been sort of fragile on everything else with too many pieces. I like the WG model 20:47:44 *I think the... 20:50:01 "WG model" Sorry if I'm fuzzy, WG == ?!? 20:50:09 WireGuard 20:50:48 Ahh. 20:52:04 danmcd: sorry to have distracted you here too. Are you good for the asm push? 20:52:09 ie, can I? 20:52:37 Yes! I prob. wont *merge* it until late tonight or tomorrow, but that's MY problem, not yours, and I *do* want this in for this week's SmartOS. Thank you!!! 20:52:45 (cue Salt 'N Pepa...) 20:52:59 well, don't thank me until we find out how I botched it and broke everything somehow :) 20:53:17 confidence is always a mistake :) 20:56:49 [illumos-gate] 15374 stop using aw(1onbld) and feed asm through the compiler frontend -- Richard Lowe 20:58:48 (honestly my major worry is whether that's screwed Oxide) 21:02:01 They may need a local putback buddy like I do (for the LX brand stuff). 21:02:22 their tree contains an extra platform or 2, and things like that, there might be some merging git's weird rename logic doesn't like too 21:05:20 Just don't do it on your mac. :) 21:05:58 it'll be fine on a mac with `git pull` 21:06:02 shit, I knew I missed something in the heads-up 21:06:15 that patch/git are foolish about case preservation. 21:06:47 so git will work, but anything going via a textual patch will be, at best, a pain in the ass. 21:15:09 sommerfeld: are you satisfied with the change to https://code.illumos.org/c/illumos-gate/+/2665 ? 21:15:10 → CODE REVIEW 2665: 15271 document viona tunable to force copy of tx buffers (NEW) | https://www.illumos.org/issues/15271 21:17:28 sjorge: oh, sorry, I meant to get back to that. the question was more around more context about *why* the host wouldn't return the tx buffer promptly (long tx queues on the real device? handed to local TCP for a local socket that was being left unread?) 21:20:20 I think that's one for pmooney, probably. 21:20:31 I think I remember sjorge and pmooney having talked it through, at least. 21:21:29 NICs generally consume the packet promptly, but local netstacks are under no such pressure 21:22:00 like you point out, a packet could end up in a tcp/socket buffer for a long time 21:23:01 so cases where the traffic terminates on the host rather than going on the network is where this has been seen? 21:23:21 yeah, traffic which ends up in a local netstack 21:23:25 are there similar issues for vm-to-vm traffic? 21:23:34 generally no 21:23:51 assuming they're both bhyve, we'll either have space in the guest RX ring to place the packet, or we'll drop it immediately 21:24:02 sommerfeld: i was never able to accurately recreate it locally, but when I was looking at dumps, it appears somehow we end up chaining potentially megabytes of data via b_cont at the sockfs layer when going from bhyve->native netstack 21:24:08 nahamu, I'm not sure if this is a wireguard-go specific thing, but with a Linux host acting a hub, all the wireguard peers can reach each other through the hub by simply having an AllowedIPs being a subnet that contains all of them 21:24:12 if the destination guest isn't running with viona, then normal socket buffers may be in place, so there are less guarantees 21:25:08 https://www.illumos.org/issues/13463 has what i was able to see at the time 21:25:09 → BUG 13463: bhyve locks up when viona ring buffers on loan to a local zone are not returned by sockfs (New) 21:25:19 it's rather unfortunate, because allocating a buffer and copying the packet data definitely does have a cost 21:25:34 hence why we default to not doing the copy 21:25:45 nahamu, my smartos zone is working from routing packets between the tun and the lan, but not between the peers 21:25:45 Indeed. 21:25:51 (we do copy the headers for TOCTOU reasons) 21:26:15 I was able to reproduce this 100% reliably 21:26:51 I would be in an OmniOS VM where I would build a disk image (2G file) and I would "rsync -Paz disk.img gz:disk.img" from inside the VM 21:27:10 and then every time be sad that I'd forgotten I can't do that as the VM becomes uncontactable for a while 21:28:19 it seemed like somehow we were bypassing the hi water marks that should have produced some back pressure, but I was never able to dig deeper before I left joyent 21:28:49 The problem in this case is that rsync wants to read more data than we are able to send and so it just waits 21:29:03 256 ring entries by 1500 is like 350KiB of data 21:29:28 is this the change where we document a tunable? I was nervous about that too 21:29:45 but I couldn't tell if it was useful as a real property on something. 21:30:02 I mean, this is 100% broken today in a well-understood way 21:30:15 So I think we should do literally anything to unbreak it and then figure out how to make it less bad after that 21:31:38 right, I don't want to get stuck in the situation we have in the past where well-known and documented tunables live forever, and get passed around without being understood 21:31:48 but I don't know what else is appropriate, so I left it alone, until just now. 21:32:04 if you look at the analysis, there's around 350k of data in excess of the high water mark (sopp_rxhiwat) queued (so_rcv_queued) 21:32:24 all while the userland process is waiting in port_get 21:32:50 Setting the tuneable definitely fixes the problem, with likely an acceptable performance impact 21:32:54 nahamu, nevermind, I figured it out, it was nat'ing the packets between peers 21:32:59 providing a mechanism to un-borrow those buffers from inside sockfs is going to be a challenge 21:33:04 We don't know why the tuneable exists though - except that it is on for a particular type of network card 21:33:20 ideally you'd treat the loaned packet as a hot potato and copy it if it was going to leave the fast path. 21:33:25 sommerfeld: yes 21:33:31 but that is, as pmooney notes, complex 21:33:39 Hard to know when the fast path ends 21:33:43 yes, many ways off the fast path. 21:34:00 like it seems pretty legit for those buffers to be loaned for a few ms here and there 21:34:05 just not ... way more than that 21:34:09 like, it may end up in a sockfs buffer where it is _immediately_ picked up by a recv syscall 21:34:16 which implies then that you have to be able to _come back_ to it later and unloan them 21:34:37 but doing that unloaning requires a new paradigm for stuff accessing those mblks 21:34:41 which is what I would like it to do, but I acknowledge how expensive that will be to arrange haha (in terms of work hours) 21:34:46 andyf: I had just said to jclulow that I'm perhaps more in favour of setting it by default right now than documenting it, but I guess what you said changes that too 21:34:50 to know that they are well and truly not in use at the time you do the unloan 21:36:02 leaving it configurable by /etc/system and friends, but defaulting to on (copy all packets) seems like a reasonable compromise 21:37:37 It seems like it could be relatively easy to also decide not to copy in specific circumstances we know to be safe 21:37:39 something adaptive might work well enough (loan if you have headroom, but start copying if you've loaned out more than X% of your TX descriptors?) 21:38:29 Yeah that could help too 21:39:05 but like we must be able to tell somewhere along the way if a buffer is going to sockfs for userland, or if it's going to another viona, say 21:39:14 I know we can't claw them back, but we could decide copy policy _on the way in_ 21:39:17 copec: that's a relief. 21:39:39 But it really feels like we should just set the tuneable by default (which will slow things down, but they will work) and then work incrementally on economically motivated performance improvements later 21:40:24 jclulow: yes, agreed; defaults should be safe. 22:50:58 I didn't mean to dip out immediately after posting but i got called on the oncall number. 22:51:27 but yeah, ideally it would just work without a tunable and for a lot of people it does. 22:52:12 i just seem to be extremely good at hitting it (multiple vms -> nfs server in a zone on one host) 22:55:18 and i'm personally not seeing any noticable impact on performance, but it's not like i did a lot of benchmarks or somethings 22:55:44 just happy the vms dont get stuck 4+ times a day