hello all, what’s this tcp parameter means? tcp_rst_sent_rate_enabled , when should it be set to 0 ? I’m running PostgreSQL rdbms, should it be set to 0? or use default value 1 ?

it is in an isolated network

It's a protection against packet amplification (it rate-limits RST packets). WHat problem are you trying to actually solve?

I *must* be doing something wrong. I'm trying to set a MTU of 9000 (for jumboframes) on an ixgbe device and I'm told it isn't valid. When I check, I see this:

ipadm show-ifprop -p mtu ixgbe4

IFNAME PROPERTY PROTO PERM CURRENT PERSISTENT DEFAULT POSSIBLE

ixgbe4 mtu ipv4 rw 1500 -- 1500 68-1500

ixgbe4 mtu ipv6 rw 1500 -- 1500 1280-1500

Is 1500 really the max MTU?

(looking at the POSSIBLE column.)

yes, but no

do

illmos separates out the data link layer bits from the IP bits

nomad: dladm(8)

danke

so you need to see what the MTU is on the underlying link

annoyingly though, to change it, you'll need to tear down your IP interface(s)

change it, then recreate them with ipadm / ifconfig

that's ... ungood.

The good news is that when you change it with dladm(8) it persists.

LINK PROPERTY PERM VALUE DEFAULT POSSIBLE

ixgbe4 mtu rw 1500 1500 1500-15500

yeah, i've floated around some sort of 'apply at next 'start' option to dladm as a possible way to deal with it, but haven't really done much more than what I just said

so, I have to ipadm delete-if then dladm set-linkprop then ipadm create-if to reset it?

well, I can test to see if setting jumbo frames impacts iperf3 test results but I'm going to be stuck if it actually turns out to potentially matter.

I guess I'll worry about that after the tests. Thankfully I have two test hosts I can do this on without impacting prod.

the reason for the limitation is basically once you start 'using' the link, (e.g. ipadm create-if, not the only way, but by far most common) pretty much every driver (for performance reasons) creates a pool of MTU-sized buffers for TX and RX (that are pretty much ready to go)

so changing the MTU would mean having to reallocate all of those, which would be rather complex to do if you're also actively using the buffers to pass traffic

(i'm simplifying a bit, but but that's basic gist)

hrm.. i think this has been asked before (and the answer is 'no'), but anyone put any thought into what NVMe over fabric would look like?

well, the good news it doesn't seem to actually matter to the speed tests I'm doing. The bad news is, it's still slower than it should be. pastebin.com/TJh7WgNz

danmcd: Thank you very much, I just read the parameters, my application is postgresql, and I hope network could run with best performance for postgresql

RST wont' help you.

There are PG-savvy folks on the developer list who might be able to help.

okay, thank you very much

@nomad ==> I wonder if you're single-CPU-stream bound? Did you try running multiple iperf server processes on multiple ports, followed by clients connecting to those ports?

iperf isn't MT, but your apps are.

For a 10G, two should do nicely.

IF they add up to 10, you're CPU (and yes, possibly driver and/or TCP stack) bound.

danmcd, -P 2 has much better totals.

Okay.

I forgot about that because iperf3 on a different host sees the full 10G (or close enough).

[SUM] 0.00-30.00 sec 33.0 GBytes 9.44 Gbits/sec sender

[SUM] 0.00-30.00 sec 33.0 GBytes 9.44 Gbits/sec receiver

well... in one direction it does.

Responder is still single-threaded IIRC.

the same hosts going in the other direction are still way low.

[SUM] 0.00-30.00 sec 17.6 GBytes 5.03 Gbits/sec sender

[SUM] 0.00-30.00 sec 17.6 GBytes 5.03 Gbits/sec receiver

iperf3 is good for only one thing: Single-stream tests.

what would you use for testing?

Two iperf -s procs with different ports, and then run two iperf clients, one each to each server.

You can use pwait(1) as a starting line:

sleep 3600 &

( pwait `pgrep sleep` ; iperf -c <... one server port>) &

( pwait `pgrep sleep` ; iperf -c <...other server port>) &

pkill sleep

BOTH go off to the races.

Someone should write an iperf server variant that spawns threads upon accept() per connection.

I'll poke at that shortly.

I'm getting a combined total of 8.94Gb/s in one direction and 9.57Gb/s in the other.

so now to tear the interfaces down and reset them to the 1500MTU again.

so, yeah, MTU not making a difference in iperf3 timing. As would generally be expected.

which leaves me confused why it made such a huge difference on the FBSD tests I did over the weekend.

[illumos-gate] 16603 acl_totext(3SEC) can truncate users and groups -- Gordon Ross <gwr⊙rc>

[illumos-gate] 16623 Want tests for libsec acl text conversions -- Gordon Ross <gwr⊙rc>

[illumos-gate] 16591 nvme_field_validate swallows more specific error messages -- Andy Fiddaman <illumos⊙fn>

[illumos-gate] 16592 Cannot update NVMe firmware on Micron 7300 -- Andy Fiddaman <illumos⊙fn>

[illumos-gate] 16593 nvme panic when committing partially loaded firmware -- Andy Fiddaman <illumos⊙fn>

[illumos-gate] 16596 nvmeadm: some firmware activation controller errors are not -- Andy Fiddaman <illumos⊙fn>