Is there a common misconfiguration that would cause instances launched on a compute node with a gigabit external interface to be stuck at about 10Mbps speed? Everything looks fine on my routers and switches, and the CN isn't showing any basic networking issues (packet errors, etc.) I've tried both bhyve (debian) and joyent (base-64-lts) instance types and they all will only transfer at under 10Mbps.

On the CNs themselves, I can transfer at gigabit speeds with no trouble (using scp to move big files around).

Sounds almost like an mtu mimsmatch or something, but that's mostly guessing at things.

That's the only thing I thing I can think of initially.

Okay, thanks. It's all 1500 at the moment - I started to go down the road of bumping it all to 9000 on everything, but that seems to be troublesome (have to change the nic_tag, which doesn't want to be changed, and then the nic, etc.)

If you look at the joyent instance type does anything stand out in say nestat -s?

...starting one up now

I don't see anything immediately suspicious: gist.github.com/justindthomas/cdd480eda7b9757f059663baac3e3aaa

I assume that's while you have something like iperf going?

That was after installing speedtest. I just actually ran the speedtest and updated the gist.

There's more I can investigate on my switches and routers. I'll start there. Maybe something with the VLAN is wonky.

I also have a couple of 10Gb bnx interfaces I could pull into service on these machines and see if that changes anything. Right now I'm just using the built-in igb interfaces.

I would trust the igb more than the bnx/e.

Okay, g2k

jdt: And just to be sure of the basics, `dladm show-linkprop -p speed -z global` shows the link speed you expect?

ha, no it doesn't:

LINK PROPERTY PERM VALUE DEFAULT POSSIBLE

bnxe0 speed r- 0 0 --

bnxe1 speed r- 0 0 --

igb0 speed r- 1000 1000 --

igb1 speed r- 10 10 --

Why would that be, given a pretty normal installation?

Dunno. I'd start with getting the output of dladm show-linkprop igb1, which should have some of the phy information, IIRC.

On the HN (identical hardware) the speeds look fine.

But that'll be a bit large that you probably don't want to drop it in here directly.

So we are definitely advertising more stuff. I guess this doesn't have what the link partner reported, that's annoying.

We have that in the driver.

The switch is reporting a 10Mb link too. Maybe it's just a bad cable.

Yeah, the headnode shows 1000Gbps on igb1 on the same switch. I'll start with the physical layer. Thanks for looking.

Sorry, 1000Mbps

Yep, bad cable. Swapped it out and back to 1Gbps. It's always the basics.

Glad that's what it was. As annoying as it is, an easier problem to solve!

Does it make sense to enable SR-IOV in the BIOS of Dell servers? Does SmartOS use it / plan to use it with some cards?

no, and no

thank you

It shouldn't hurt to enable it in the BIOS.

ssh ellie

hunter2