nomad: One thing to check is: does it go faster (in aggregate) if you run multiple iperf streams in parallel

It's only letting me run one stream at a time, regardless of trying multiple -c on one host or on different hosts.

I'm guessing the server only allows one stream.

and trying three streams from fs2 (going to hvfs2, fs1, and fs3) concurrently gives an aggregate of about the same bad number I get with just one stream.

nomad, to run parallel you pass -P to the client

hadl : Thank you very much for your help with PPPoE.

I wrote a howto on unitedbsd:

ik5pvx, thanks. Still getting the same low numbers.

[ ID] Interval Transfer Bandwidth

[SUM] 0.00-10.00 sec 2.78 GBytes 2.39 Gbits/sec sender

[SUM] 0.00-10.00 sec 2.78 GBytes 2.39 Gbits/sec receiver

util.c:155:14: error: 'F_OFD_SETLK' undeclared (first use in this function); did you mean 'F_SETLK'?

just to confirm... if I install a new (dual) NIC (x540) in an OmniOS box, touch /reconfigure, and reboot, that device should show up in prtconf -v and dladm, right?

Did I miss a step somewhere or do I have an actual hardware problem?

you don't need to touch /reconfigure if you add a device

just adding it should make it show up in prtconf

Woodstock, that's what I thought but when it didn't show up after first boot I gave /reconfigure a try.

The card shows link when cabled to another device so it has power.

prtconf -d should show device names (from the pci id database, perhaps related to your product)

prtconf -dD shows that and the driver which attached (if any).

you may have a software problem (available drivers don't support the device you added).

prtconf -d shows 4 540 entries, which is the same as we had before I added this card.

nomad: if the device isn't seen by prtconf there's probably something else wrong. every device in the system should show up there, even if there's no driver available.

As I feared.

We had a problem years ago with a fileserver that wasn't seeing half the bus because it only had one (of two) CPUs installed but this host has both CPUs so that *shouldn't* be the problem.

when in doubt, power off, pull & reseat card.

sommerfeld, that's what I was just about to say :)

then do a tree walk in bios to look for work/don't work options.

I guess I'm heading back to the server room for another round of opening up the huge host. <sigh>

If that was a reference to an EFI thing I don't think this host has that. I'll have to check but I'm pretty sure it's old-school BIOS.

Though I will check to see if the BIOS sees the card/offers any enable/disable options for it.

just meant "walk the tree of menus looking for anything that might possibly cause the device to be ignored"

As I feared, it was in fact a hardware problem.

I swapped the NIC with the HBA and the NIC appeared and the HBA disappeared.

I've managed to find a temporary solution so on with network testing.

well lookie here.

[ ID] Interval Transfer Bandwidth

[SUM] 0.00-10.00 sec 10.8 GBytes 9.26 Gbits/sec sender

[SUM] 0.00-10.00 sec 10.8 GBytes 9.26 Gbits/sec receiver

that's an iperf3 -P 3 run on a single dedicated link between two OmniOS hosts.

When I don't use -P 3 the numbers go down to match the numbers I was seeing yesterday.

no, actually, they don't. They go down to about double what we were seeing yesterday.

there is no consistency between yesterday's results and today's. :/

Is it cooler in the DC today? haha

jclulow, you laugh but ... well, no, the answer is no. But I get where you're coming from.

performance is challenging because of how many variables there are

I definitely had a network weirdness problem here that went away when I reseated some calbes

*cables

Which is very unsatisfying

indeed. Which is why I dropped in a dedicated card to match another spare I had in the other test server.

I ran cables between them and that was going to be my 'dedicated, controlled' environment.

except now I'm seeing better throughput from hosts and networks I didn't touch.

[ ID] Interval Transfer Bandwidth

[ 5] 0.00-10.00 sec 3.92 GBytes 3.37 Gbits/sec sender

[ 5] 0.00-10.00 sec 3.92 GBytes 3.37 Gbits/sec receiver

vs

[ ID] Interval Transfer Bandwidth

[ 5] 0.00-10.00 sec 8.75 GBytes 7.51 Gbits/sec sender

[ 5] 0.00-10.00 sec 8.75 GBytes 7.51 Gbits/sec receiver

mind you, *that* particular test was between a production host and the host I rebooted to install the new NIC in so...

hmm.. I think I might have found a clue. pastebin.com/Rm19SDWC

difference between i40e and ixgbe based aggregates.