jperkin: changed our tls-cert today and got a cert error while doing pkgin ug (we an rsyned clone of pkgsrc to speet up the updates :-). Might be because the intermediate from GlobalSign changed on 29.01.2024 and it is not yet in the pkgsrc trust store? Will it be included in the next updates?

The link to GlobalSign Intermediates: support.globalsign.com/ca-certifica…/alphassl-intermediate-certificates

yeh almost certainly, we're at 20231115.

jperkin: Can you say when the next trunk updates will land?

no, but I'll take a look at it tomorrow

in the meantime you should be able to skip the validation with SSL_NO_VERIFY_PEER=1 in your environment

jperkin: Thx a lot for your help!

hello all, I remember that The max vCPU is limited for 32 for Bhyve VM on SmartOS, if I want to create a Bhyve VM with 96 cores on SmartOS, I have to changed the SmartOS source code and re-compile, so my question is where defined the limitation? (I have a machine with 128 core/256 thread (AMD 9754), I need to change it ), a big thanks

You just need to change VM_MAXCPU in usr/src/uts/intel/sys/vmm.h

Has anyone here tried the igc(4D) build on their Intel I225/I226 systems?

(ahh fixed)

andyf: Thank you

tozhu - it's limited because several structures in the kernel are sized based on that, so there is extra memory overhead for every VM. Eventually I imagine that things will be sized dynamically.

With 32 vCPUS, sizeof (struct vm) = 0x49c8

With 64, sizeof (struct vm) = 0x8eb0

Thank you very much, I checked, current is set to 64 by default

I think 64 cores for a VM is okay for most case

thanks again

Is there any tricks to running a PXE / DHCP server in a bhyve virtual machine ? I can see it offer DHCP but my machine isn't accepting and booting

you need allow_dhcp_spoofing

I did that

You also have allow_ip_spoofing?

[DHCP] Offering to boot ac:1f:6b:9e:df:47

yeah, I set all of them allow

in adminui*

all includes mac spoofing?

restricted traffic?

So just to be clear, your dhcp *server* is the one in bhyve?

And when you tcpdump/snoop you can see the dhcp request come in and dhcp offer go out?

yea, it might be something on my server side cause I am seeing DHCP coming in

and the offer go out so it should be working*

Ok, next step, I would snoop from the gz on the physical interface to see if you see it exiting.

If the kernel is dropping it due to some anti-spoof protection then you won't see the packet leave.

Just wanted to make sure there wasnt anything with BHYVE that limits it

It might also be good to use like scapy or something from a running host to send dhcp request packets and see what kind of response you get.

Well, I can't say for certain that there's not. pmooney would be the person who would be able to say for sure. But as far as I know there's nothing inherent in bhyve or viona that will prevent it from being a dhcp server.

Side question: Any reason you're not running the DHCP server in an os zone?

ahhh im dumb github.com/danderson/netboot/blob/main/pixiecore/README.booting.md

Pixiecore proxies the DHCP offers

when I ran this in my KVM setup for testing, i didn't realize libvirt was actually doing the dhcp and not netboot*

Looks like I'm sending out an address but the machine isn't getting passed that

What I'm learning right now is

EFI PXE Clients on network cards is really awesome.

i sense.... sarcasm

:)

jbk: you know the pain.

firmware is terrible :)

maybe oxide can sell a openfirmware 100gb NIC

or vendors can have an option to flash to some open firmware

what a dream

If my server would come with NICs that didn't hate me./

i suppose (if i was really bored and had the time, i might consider it as a lab experiment), I suppose given the programming manual for a card, you could probably write your own, though how to safely flash it and such might be an interesting exercise on its own

i mean, at the end of the day, there's basically a bare bones NIC driver and then the PXE code that uses it... though there might also be some other bits to initialize the hardware that a NIC doesn't need to concern itself with

Mellanox connectX5 even booting off the ipxe.iso im getting "No configuration methods succeeded" really fun

like trying to troubleshoot apache :) my experience has been at least 80% of the time, it never actually tells you why something doesn't work and you just have to fumble around in the dark until it magically does

"denied"

why?

"denied"

*turns logs up to 11

why?

"denied"

pretty much

like there's someone that's running a custom php app that was written for an older version of php

the problem they're running into now though is that the version of openssl is too old to suppor tls1.3

but the version of openssl that support that is too new for the version of apache that's running

and upgrading apache (installed to a separate dir) it refuses to actually run the php code

with no indication of why at all

which is frustrating

(the app is strictly internal, and not reachable from the internet)

servers should come with a way to turn off the 1tb of dram

like give me 8gb so it doesn't take 45 minutes to boot

People awake

Smithx10: Probably some startup check?

Major issue I have is to boot with DAS/HBA, it takes forever

or training DIMMs

It's way faster for me to unplug the SAS cable, boot the machine with internal zones only. Then connect the SAS and do import tank after the system is up and running

at least with EFI it's easier to control the boot order so you don't have 'let's wait 30s for this PXE interface' (repeat for each NIC in the system), now let me try each HBA

granted an old Sun E10K laughs at your puny 45min boot time :P

SuperMicro X10 and X11 mostly

if you enabled full diagnostics (granted that was generally only done as a post-install burn-in test), it could take > 24 hours to boot

48 hours did not suffice in this setup

Not sure what hangs; good thing I found out I could simply detach external SAS cable to NetApp 4246 in order to have the system come up.

nahamu: pretty sure I might be hitting ipxe/ipxe #1091

applied your commit to main and and doing build now to embed into pixiecore

Wish me luck "_" :P

@TyrfingM1olnir how many disks in the DAS?

in the the? :)

hello, i was wondering if it's possible to run a firewall in a zone and have it act as a gateway for a mixture of zones and bhyve vm's? or should i run the packet filter from the hypervisor itself?

if this is a pretty common use-case could someone point me to some recent docs or reference implementations? thanks!

s/hypervisor/global zone/

You can do both.

I use a native zone as a router & NAT (it could be a FW but I approach that problem differently). It's on OmniOS, so you may need some extra SmartOS magic (forwarding requires some lnk-layer protections to be disabled..)

You can also, if you want a different/more-familiar-admin, use a BHYVE VM (again with some SmartOS link-protections disabled) to run some FW appliance.

ok thanks danmcd! i think i'll need to spend some time experimenting with etherstubs then

i'm looking to replace a freebsd hypervisor where i use pf and nginx for controlling access to private instances. i'm hoping i can do this more cleanly with smartOS. at least that's what i'll find out :)

nahamu: yep.... that was it "_"