triton gurus, looking for assistance on a compute node, which upon setup, continously claims to fail, all though it is setup, and the job has no errors, says it is setup successfully and is updating cnapi, although cnapi never gets updated. I cannot seem to find any logs pertaining to server setup failure. It even drops the /var/svc/setup_complete file upon completion.

I'm at a loss

I'm not sure why it didn't update my nick

Those are john__'s are from barfield

The only thing that I do see occurring is that `ur` is running fine on the node, but on the headnode `sdc-server list` shows status as "unknown" after a few minutes. I checked disk space. No issues. I restart ur `svcadm restart ur` it pops back up as running in the sdc-server list output but after about a minute it says unknown state again.

It also only appears to install the amon-agent and show the agent package version. cn-agent, net-agent, basically *-agent* doesn't exist on the CN.

This is a CN in a different rack with rack-aware networking deployed as a site-note.

side*

bahamat: the ntp issue yesterday ended up being a restriction in /etc/inet/ntp.conf to only the original admin network subnet. The RAN subnets were not in the config file. I updated the config and it works now. I am assuming I need an ntp.conf in /usbkey/config.inc in order to make this permenant. I cannot remember if this is how we did it at joyent though for some reason

danmcd: Do you know if the lastest intel 100GB NIC's are working correctly in SmartOS as of recently? Do they happen to have the same memory issues as i40e in libdl?

another weird thing I am noticing on this separate compute_node issue is that: sdc-oneachnode -n <UUID> 'CMDS' work just fine. It isn't giving the old "missing compute node" error like you would expect from a CN with an "unknown" status.

sorry to be a pest the last couple of days lol

I don't remember exactly how we handled that for SPC, but I know there was an allowance for it.

yeah I was thinking the same thing. Maybe a cnapi setting somewhere. I'll dig into the source

weird looks like agent setup may be failing because of a lack of usbkey

We handled it in chef :-(

Aha, I found the issue. cn_tools.tar.gz is "maybe" missing from the assets zone

that's definitely not a normal situation.

its definitely there

[root@SCS_5MPHXQ1 (dfw003) ~]# /usr/bin/curl -sSf 192.168.32.21/assets/extra/joysetup/cn_tools.tar.gz

curl: (22) The requested URL returned error: 404 Not Found

But getting this

that* I mean. Looks like a DNS issue. like the "assets" server name is required in the URL

yeah super weird. can ping sapi, napi, etc, but not assets.

barfield: We don't support ice (their 100G 800-series part).

danmcd: shit. So only chelsio?

what libdl memory issue?

there's an issue with the driver+jumbo frames+kernel memory fragmentation

There was an issue where dladm would use up a bunch of contigious memory blocks on vnic creation and cause contention with bhyve VM's

At least that was my understanding

though hopefully I can get a fix for that upstreamed (we've been testing it, and have started rolling it out to customers)

Yeah I think that is it. I may not have described it perfectly

ahh.. that's actually the driver

dladm just triggers it :)

haha, that makes sense

I've hit the issue a lot

vnics create additional rings, and each entry wants a mtu-sized chunk of physically contiguous memory

yeah

but its because our customer purchased nodes that really didn't have enough RAM to begin with. Like 128GB in the node and wanted to assign 112GB to the BHYVE VM with a 40GB network in the loop

our fix is we cap the size of the buffers used on the RX ring to 4k (PAGESIZE)

and let the HW split larger packets

I've seen it on freebsd with netmap interfaces too

which should help

of course if the system is tight on memory in general, it's not going to help

I think it was the 202306 PI that got us past the GZ crash hump here

but at least it shouldn't have to work extra hard for jumbo frames

What is the deal with the intel 100GB NICs? This concerns me because our supplier sourced 100 of them for some racks we're building out even though I asked multiple times for chelsio nics

Or would mellanox be a better choice?

well i think we have drivers for chelsio or mellanox, but not intel

Oh so its a vendor issue not a community issue. My ignorance just lead me to believe that "100GB intel couldn't be that different from 40GB" lol.

i haven't looked at the drivers to know what the difference is

Yeah me either

Was just checking to see the status is all before I get a bunch of compute_nodes that we can't use.

it's possible that maybe the existing driver just needs a small amount of update, it oculd be different enough to do a whole new driver, dunno

Sorry if my humor doens't come across as humerous in the chat :)

oh it's ok

i have a rather dark sense of humor, though i try to limit it to friends that expect it :)

LOL same here, had to create a "dark humor only" channel on slack for some select co-workers

one thing to keep in mind with chelsio is that there is still no support for mac groups, so no hardware classification for VNICs, not sure how much that matters

the intel ice driver would be significantly different from i40e IIRC

like, nothing shared

I'm not sure if mac groups affects us or not. Just running triton compute node workload in the traditional sense

annoyingly, i don't think intel has published any programming guide for it either

yea, if your zones aren't heavy on network I/O, then it shouldn't matter as much

so it's all going off of existing drivers

Another random question: Is anyone going to opencompute world in October?

jbk: Robert started a driver for it back in the Joyent days, very very bare bones, lots left to do, I imagine he had a PG at that time

one discussion we had was eschewing the common code this time around and just going our own way

barfield: Mellanox (now nVidia) should get some improvements into mlxcx(4D) SOMETIME this year or early next.

update from my earlier compute node setup issue...looks like the admin network in the other rack which is: 10.24.20.0/22 was not allowed to access imgapi. I think that was the issue. But will know for certain after this node finishes booting. I found logs in /opt/smartcd/agents/log/install.log which pointed me to this

*/opt/smartdc/agents/log --- excuse the typo

barfield: Yeah, the default fwrules for the admin account is only to allow the admin network prefix to tag smartdc_role. When setting up RAN you need to manually add the new subnet(s).

I think there is still more. It still failed.

WTf

lol

Unfortunately RAN is a quasi-supported feature. The parts that work do work, but how to set it up isn't fully documented.

Its okay I'm doing things like I always do anyway, the hard-way

Its how I learn best anyway

I don't even have a RAN setup to try to test any of this.

When I run the joysetup.sh script locally it always just ends at the last line "exit 0". How do it run it locally like sdc-server would remotely? So that I can watch -x output and try to find something failing

We (MNX) don't use it (yet), and we don't have any paying customers (hint, hint) that use it so filling in the gaps isn't the best use of our time, when considering other things we know we'd like to get done.

The output goes to a log file. The best way is to follow that.

I'll write some docs when it is done

Yes, but it isn't leading me anywhere. I think that I am missing some environment variables set by sdc

I'll source the configs maybe thats it...+ /lib/sdc or whatever the path is

yep that did it

yeah failed again. seen_states are zpool_created filesystems_setup and imgadm_setup...must be failing on tools

barfield: I just created OS-8484.

Nice