btw, the OpenZFS 2.2 released with many new feature and improvement, if the ZFS in illumos kernel will get same feature and improvement in recently?

jbk: Thanks for the help, changed line 50 in usr/src/cmd/cmd-crypto/decrypt/Makefile from '$(ROOTDECLINK):' to '$(ROOTDECLINK): $(PROG)' , then compile and get the error as this : pastebin.com/m1ME4Uhh is there still some issue with some Makefile ?

i'm not sure..

it's strange since i've been building that exact same commit just fine

though i need to rebuild some stuff anyway... and i have to run for a bit, so I'll clean and do a full build and see what happens

well don't _need_ to, but now that I'm able to talk to my tpm2 module, I want to see if I can get it working as a kernel RNG provider

jbk: Thank you, I’ll wait for the fix, thanks for the great help

hmm.. i just did a build which included that commit and didn't have any issues

jbk: the build with commit: e5c93d6afd576eba5ed58a0f188357e3cc604b61 , the ‘Makefile’ has changed on this commit

eee, install is missing all target there, decrypt is not getting built

that is, after make clobber, make install does not build anything.

Something tops sftp transfer speed from LX zone that is behind other gateway zone to 100Mbit and I don't know what causes it.

tozhu: looks like a fix for your issue is up for review now

i had the wrong line

jbk: Thank you, what’s the fix line?

I’m going to verify the fix in my env

on line 48 -- add 'all' to the list of dependencies

okay, I have changed it manually, and is going to verify it now

Hi, I've just seen the "OS-8496 Deprecate docker registry access". I'm using docker images on standalone smartos - this goes away now?

jbk: I have verified, the fix works well

nahamu: did you get anywhere with further testing of the wireguard-go rebase?

@jbk & @tozhu (pardon latency) ==> That fix needs to be in -gate as well.

See illumos#16057 ( jinni )

I'll be RTI-approving it today if it flies by.

yeah.. andyf fixed it and has it up for review

just this time slot every day is pretty much always booked

and often runs over

also, i really wish the materials on kcf had made it out...

jperkin: not yet, no.

But I don't think that the one I did is the one to ship. I want to rebase from a tagged version rather than arbitrary commits on master.

Does anyone have any of the Intel CPU models mentioned here?

I don't, but would like to find someone who does.

Is there a way to get smartos to cough up the id numbers instead of the human name for the installed processors?

danmcd: We have Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz

for testing from vendor. It's in semi-production now, but I can probably check with boss how long he wants to test it and squeeze some time before we return it

if you want to run smartos on it. otherwise it runs debian now and I have root to get some data

Perfect, that's Ice Lake: intel.com/content/www/us/en/product…-cache-2-10-ghz/specifications.html

How do you want your SmartOS? .iso? .usb? Platform image for Triton?

(I was hoping "today or tomorrow" since we cut SmartOS tomorrow night.)

pjustice: `psrinfo -vp` will emit something after the "GenuineIntel" at the bottom that will help.

It's supermicro, I can boot ISO, I'll check with the boss in the morning, but my guestimate is week or two ETA

Oh damn... that's gonna be too long. (I may put the updated ucode into SmartOS anyway if I can't get it into -gate in the next 24 hours).

other box we're testing has 'Intel(R) Xeon(R) E-2378 CPU @ 2.60GHz'

Ok, I have at least one 606A6 Ice Lake Xeon Scalable Silver 3309Y

but probably same timeframe

pjustice: I'm building SmartOS with this as I type.

i forget, do we have the CPUid and platform ID values somewhere exasily accessible?

i might have one of those CPUs

`psrinfo -vp` look for the string after "GenuineIntel" on the bottom

danmcd: I also have Intel(R) Xeon(R) Bronze 3106 CPU @ 1.70GHz that's not in production

(ark.intel.com/content/www/us/en/ark…6-processor-11m-cache-1-70-ghz.html)

well it's a Xeon -D but it says '50663'

Nope.

For those who speak intel code names we need "Ice Lake" or later.

I have Xeon Ds of that flavor, jbk

x86 (GenuineIntel 50654 family 6 model 85 step 4 clock 2200 MHz)

Intel(r) Xeon(r) D-2123IT CPU @ 2.20GHz

Looks like all of the relevant processors are 6xxxx or higher.

I have two of the 606A6 family machines.

pjustice: what flavor of SmartOS can you try? Or do you want to use ucodeadm and apply them yourself?

(ISO, USB, or PI ?)

PI

Lemme push it to kebe.com

Does the ucodeadm require reboot?

You're packaging 0d0003B9?

You don't need to reboot with ucodeadm, IIRC.

(I haven't done it myself ever so I have no experience upon which to draw.)

PI is here: kebe.com/~danmcd/webrevs/16058/platform-20231113T204650Z.tgz

MD5 == b4a5ebc3da29a35187b2d88898bad14e

If you swap PI you will DEFINITELY need to reboot.

aye

pjustice: also before-and-after outputs of theses:

psrinfo -vp

ucodeadm -v

Would be nice to avoid a reboot if possible. Does ucodeadm get you enough info?

I think it might. Lemme check history. What are you running right now PI-wise, pjustice?

joyent_20230504T000449Z

Hmmm. I was hoping it'd be recent enough to include illumos#15846 in it ( jinni )

Still...

Well, lemme declare an emergency reboot, then we can just be sure. :)

Thanks and sorry.

This is damned near what 20221116 will be (which in addition to being a SmartOS release, will ALSO be a Triton release, so it'll go on the `release` channel.)

danmcd: is that the Xeon gold family?

"Ice Lake"?

If it's the right *generation*. Yes, Ice Lake (not all Xeon Golds are Ice Lake, older ones are Skylake)

wait I have "Sky Lake" does that work for what you need? I'm on the last "release" channel PI for triton on SmartOS

Skylake doesn't.

Skylake isn't affected.

pjustice: I hope you took "pre-reboot" output of `psrinfo -vp ; ucodeadm -v`

I have noticed recently a strange behavior when provisioning a new bhyve VM on E5- series (haswell/broadwell) and on "Sky Lake" where the VM just hangs on start until I do a `vmadm reboot UUID -F` then it boots up like it should.

Fedora38 is the guest.

vmadm console is just black

Does this guest use a fabric network? There's still a race between varpd coming fully up and VM boot.

You'll see something in /var/adm/messages on the GZ about it.

It does, on Intel 40GB NICs

It's device-independent.

what should I grep out?

It's about zones beating out varpd going to fully initialized.

I see different errors which could be what you're referring to

Something like this:

Interesting I hadn't seen that one but it would make sense

larry root: [ID 702911 daemon.error] zone fa7130e6-9907-4851-8980-72d2f4ed9588 failed to create overlay device sdc_overlay4385813 with command 'dladm create-overlay -e vxlan -s svp -p svp/host=portolan.kebecloud.work.kebe.com -p svp/underlay_ip=192.168.69.14 -p vxlan/listen_ip=192.168.69.14 -p mtu=8500 -v 4385813 sdc_overlay4385813

So "failed to create overlay device"

I dont have anything like that

2023-11-14T17:34:05.683912+00:00 40-a6-b7-22-54-20 ip: [ID 722105 kern.warning] WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp@0:udp: error 13

Okay so much for that theory.

2023-11-14T17:34:05.687992+00:00 40-a6-b7-22-54-20 mac: [ID 736570 kern.info] NOTICE: vnic1057 unregistered

2023-11-14T17:34:05.689636+00:00 40-a6-b7-22-54-20 mac: [ID 736570 kern.info] NOTICE: vnic1056 unregistered

(That's a shutdown warning, noisy but not helpful.)

2023-11-14T17:34:18.422486+00:00 40-a6-b7-22-54-20 genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@0 (zcons0) online

2023-11-14T17:34:19.570779+00:00 40-a6-b7-22-54-20 mac: [ID 469746 kern.info] NOTICE: vnic1058 registered

2023-11-14T17:34:19.570829+00:00 40-a6-b7-22-54-20 mac: [ID 435574 kern.info] NOTICE: vnic1058 link up, 40000 Mbps, unknown duplex

2023-11-14T17:34:19.675923+00:00 40-a6-b7-22-54-20 mac: [ID 469746 kern.info] NOTICE: vnic1059 registered

2023-11-14T17:34:19.730053+00:00 40-a6-b7-22-54-20 genunix: [ID 408114 kern.info] /p

Thats the gist of them

Unknown duplex looks interesting

ah

Guess I'll figure out how to get a dump next time it happens. I already sdc-oneachnode rebooted all of the offenders.

I provisioned 20 guests at the same time and 2/3rds of them did this

on 20 different compute nodes with no workload at the time.

OH... this is post-provision. Yeah, probably losing races of a DIFFERENT flavor.

Might be more of a #triton problem.

Yeah its definitely on triton. I provisioned them from node-triton in a for loop

frustrating because after provisioning I ran an anisble script to set them up. I find it everytime because the ansible_host cannot SSH into the IP

anyway, sorry to jack your thread

oh another question for you danmcd

Do you know remember the openbgpd issue that another illumoser had opening bgp messages to other neighbors? I dont remember exactly what the issue was but you had mentioned it was an illumos bug

I dont expect you to remember with that shitty description either lol

But if you do I was just wondering if it had been fixed yet

I have no memory of that right now, sorry.

Cool, no expectations

I think we were talking about that when both of us were still at joyetn

probably a bit late for jperkin to know pkgsrc status. I'm just tired of running quagga.

I did successfully port rustybgp a couple of years ago, along with gobgp client binaries. May look back into that too.

danmcd: before/after yagi.h-net.org/2023.4-IPU-psrinfo.txt

You forgot `ucodeadm -v`

If you have another same-CPU server you can run it on there for "before" and on your rebooted-PI one as "after"

Sorry for not seeing that sooner pjustice

Sorry, failed at reading.

File at the above link is updated.

But I seem to have the same ucode on both machines? (Only rebooted one.)

"before" ucodeadm comes from the one I didn't reboot.

No you don't. ...3b9(new) != ...389(old). This is good.

This also tracks with Intel data (and that MAN your old PI is old... you're missing an update):

| ICX-SP | Dx/M1 | 06-6a-06/87 | 0d0003a5 | 0d0003b9 | Xeon Scalable

Gen3

THANK YOU @pjustice !!! I owe you a reasonably-priced beverage of your choice.

Did I mention blind? :)

danmcd: boss says the tests should be finished by end of next week. then the boxes are available.

old PI was May.

I thought.

Hang on...

You missed this from August:

15834 Update Intel microcode to 20230808

that explains it.

jvl: Not as urgent now. But please boot the latest smartos you can if you do.

Yeah, ok. That trafcks.

-f

danmcd: will do and get back to you.

pjustice: you're now mentioned (as is your URL) on the bug report for illumos#16058 ( fenix )

Infamy!

Is your workload shared by adversarial parties (e.g. a public cloud)? IF SO you really really ought to update to this week's fresh PI.

As the REX prefix flaw here can lead to privilege escalation.

Kinda glad it affects Ice Lake and later (which we@MNX do not have as of yet).

I've got 3 racks of ice-lake coming ~30th november. Glad this was fixed first lol.

We don't have directly adversarial workloads, but otoh, we run PHP, so...

HAH! (php...)

What NICs are you using with those 3-racks-worth?

Us? Whatever was on board.

Oh.... I really would like to see the spec sheet on it.

I'm not sure, with no 100GB Intel support I tried to get Chelsio in place. But our mfg arrangement was difficult. We may be stuck with those.

Hopefully we got the Chelsio's. They were supposed to procure them. But if not then I guess linux compute nodes it is

100GB? Chelsio is your best choice. I'm trying to get Mellanox 100G working (it works, but not very fast right now).

I love Chelsio. VP of sales was working very diligently with me but his last day was last friday. They're literally an uber drive from our integration facilities in San Jose but mfg refused to procure in the US. Had to be done in Taiwan

And by "I" I actually mean "Alex Wilson, Robert Mustacchi, and I". I just happen to have more motivation (if less clue).

and also had to be with their preferred distributor. It was a nightmare. So I either need to find 120 Intel 40GB NIC's used on Ebay and shelve these until 100GB is ready...or use linux :(

lol

I worked through the linux computenode setup issues. But I feel like it still has a ways to go. I dont even really want to use it either. But if push comes to shove. I can't have $1.5m of hardware sitting idle.

OCP was rolling out 1.6TB switching fabric this year.

danmcd: I have a question on how piadm installs the PI on all drives? last week, I stepped on a rake ... I had mirrored pool with 6 devices (3 mirrors) and needed to remove a device pair. All went well, but on reboot, the bootloader went directly into panic. I booted from an usb stick, removed all existing PIs and installed latest on those 4 remaining drives, but it was ... unexpected.

I think port speeds are 800GB aggs

jvl: You can't boot a pool with multiple vdevs. (3 mirrors == 3 vdevs)

That you were able to at all is, honestly, kinda miraculous, or a bug.

danmcd: intel dpdk is the hold back on drivers right?

eh? :) why am I always the one who doesn't know that something can't be done and does it? :)

Nope. There is an ice(4D) driver that exists in FreeBSD and Linux; it looks similar to i40e in some ways. THere may even be a prototype out there somehwere.

Just wondering if illumos community writes 1 if it would "theoretically" work on the up coming 400,800 etc

ah. I knew BSD/Linux had it already. Is the intention to port BSD version?

Probably depends on who's doing the work. When I started it, I skipped the common code.

But it became a non-thing for me.

I would not hold my breath that the same NIC logic would hold for a 200/400/800G part.

danmcd: pastebin.com/DkyMx0Z8 this boots fine

There is a case for not using the common code. (I think of 13230 in i40e and my blood pressure rises.)

jvl: seriously how did "piadm bootable" work?

The I/O engine is the same in i40e and ice, so finding a way to reuse some of the same logic would probably help a bunch.

Damn ... so this is a "big deal" then?

barfield: it's nontrivial. I'd rather spend my cycles on mlxcx(4D) all the same, thank you.

hehe

Or finding someone to regression test the 13230 changes. (It's a big test space, unfortunately.)

Means I need to fly back to San Jose and kiss some ass with Chelsio and company

I may have someone but would need to check his bandwidth

jvl: Serious question, did you install standalone smartos on this with 3-mirrors? Or did you start as a USB or ISO booter and then uttered "piadm bootable -e zones" ?

If you needed 100 GbE today and don't want to do driver work really, Chelsio's T6 or a Mellanox CX-5 are probably where I'd start.

Because "zpool create -B <your layout>" will fail. "zpool create <your layout>" will succeed, and maybe "piadm bootable -e" needs to be more careful

danmcd: the devices were 4x 480G SSD + 2x 240G SSD. I'm not 100% sure, but I think I've installed to 4x 480G in mirrored pool with two vdevs and then attached another mirror

Also jvl --> `piadm bootable -r <pool>` will do its best to update all the devices (intended for single-vdev mirrored or raidz)... maybe you needed to do that before yanking drives?

rmustacc: I told them T6 was perfect but they really want me on T7's which come out Q1. T7's have a very interesting new offloading engine included.

danmcd: now I was just removing the 2x 240G SSDs. I was tyring bootable -d and -e from the stick, but that didn't help. I just removed all the PIs and grabbed the latest one and then it booted.

I've been using T5's on my BSD IPS appliances for probably 8 years at this point. Those are great NICs

I'm aware of what the T7 brings, don't worry. And waiting for samples. But Q1 isn't today, which i why I haven't mentioned it.

*which is

lol, I trust me I get it. Additional workload not needed at this time.

danmcd: it was really new installation to boot from the SSDs from day one. not upgraded to boot from local devices later

Worse than the 3 new racks we have coming we're rearranging all of our DC ops. 90 project to move into all equinix facilities

90 day*

Which is proving to be nontrivial with 415v 3 phase power requirements

jvl: not sure what to say right now save I may have some questions for you later.

danmcd: no worries, I'll re-test the same thing then I have those servers available end of next week. I usually lurk here in tmux, so if something crosses your mind, I'm here.

Thanks.

likewise

danmcd: Dan, fwiw, I've just installed latest smartos from iso into virtualbox VM with 4 32G drives. Installer suggested raidz1, do I manually did zpool create zones mirror d0 d1 mirror d2 d3 and went through. removed the virtual iso and booted just fine.

AHA!

Thank you!

By default installer tries to create with `-B` and will fail on your two mirrors scenarion, then it will jump back to just without -B and succeed.

what does it mean? that only first drives are bootable or that it just tries something else and makes all of them bootable?

I guess you're booting BIOS too, not EFI. You cannot boot a pool with EFI unless you created with with 'zpool create -B'.

yes, this is legacy boot

"piadm bootable -r <pool>" will attempt to update MBR (and ESP on EFI-bootalbe pools) for every drive in the pool.

refresh, got it. thanks!

raidz1 should be bootable too? last time I tried I wasn't all that lucky ...

Raidz1 is officially bootable (you can even do raidz1 with `zpool create -B`).

`zpool create -B` is the gold standard for bootable pools.

I was trying on old hp dl360g7 with 8 drives and it was trying to get blocks in 64K increments and after 30 minutes *I* gave up ...

You don't have to boot zones either if you've, say, a spare drive, you can dedicate a boot pool.

My triton head node on in-house "Kebecloud" does this.

Unsure of your TZ, but mine is US/Eastern and it's dinnertime.

before you did you magic with piadm, I was perfectly fine with remotely upgrading USB stick and rebooting, without KVM attached ... for like 6-7 years :))

Europe/Prague. Have a good one and enjoy your meal. Thanks for all! :)

Thank you!