Sorry for me doing this in first place.

I forgive you all. :-P

if you have a packet that's segmented into 2+ mblk_ts (i.e. mblk_t->b_cont != NULL), how does that relate to mac_hcksum_set()? do you set that on the first or last mblk_t, or do you have to go through and set that on each mblk_t

In general most metadata is expected to be on the head and it's required that if someone removes the head information it would get duplicated to the tail.

As an example, if you have initial information from a NIC that's doing inner/outer checksums due to encap and the encapped packet is in the tail, when you removed the link and passed it on, then you'd need to update the checksums.

ok

Conversely, a driver should only ever look at the head.

That help jbk?

yes.. exactly what I needed to know

thanks

I think the other useful thing to keep in mind is that the stack will basically take the fast path only when it as all the L2-L4 headers in one mblk and will otherwise pull it up at some point.

ok.. i'm just looking at being able to RX a packet (likely jumbo frame) using >1 segment but not header splitting -- just to avoid allocating > PAGESIZE chunks of DMA memory

Even if it was header splitting it'd be fine.

(may look at the header splitting feature later, so that's good to know just to be sure)

Basically you just don't want to split L3/L4 headers if you can avoid it.

ok

If you're splitting at the end of the headers / data it may mamke sense.

But to be honest, then if you start using encapsulation were you want that split to be really changes.

It also may not buy you anyting unless the stack actually frees the leading block sooner.

hi, I was wondering where can I find some ressources for starting with IllumOS ? I'm looking for some resources about the basic stuff like how to check the listening ports or how to manage the services

in other stacks, with the assistance of the NIC in picking the split point, header split + page flipping has enabled zero-copy RX. But a lot of other things have to align for that to work.

We had zero-copy RX at one point, but IIRC the historical context of losing it was, "sun4u machines did copyin/copyout faster than the overhead of ZC setup". That was a long time ago.

Assuming dma binding right now we shouldn't have copies going on at least until sockfs.

That is when we copy to the user address space.

At least, in the happy case.

The historical ZC was pre-sockfs, FWIW.

(or damn near after it...)

[illumos-gate] 15349 dtrace should accept comma after last item in enum (fix tst.enum.d) -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 15169 incorrect argument declaration for pthread_rwlock_destroy -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15246 scsi_hba_attach_setup(9F): Typo dev_bus_ops -> devo_bus_ops -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15055 mc_tx(9E) has incorrect function name -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15361 ld doesn't fill out PT_DYNAMIC sufficiently for binutils 2.40 -- Richard Lowe <richlowe⊙rn>

[illumos-gate] 15260 ld crashes if group members are stripped by -s -- Richard Lowe <richlowe⊙rn>

Chat-ron: service management is mainly through svcs, svcadm, and svccfg commands. netstat -a will show port usage (see the man page for other useful flags)

sommerfeld >> thanks for the hints ; I've seen there is no -l flag for netstat, so I assume there is no equivalent to a netstat -l ?

No, but it'd probably be fairly straightforward to add.

Chat-ron: netstat -a -f inet gets you close enough. Other than filtering out sockets there is seldome much noise

adapt -f inet to -f inet6 for ipv6

yep I've seen this flags

but I thought that, as in FreeBSD there is sockstat for this (in addition of netstat), maybe that IllumOS there would be another tool too for this

Yes, connstat is similar to sockstat.

uuuu connstat is nice. Adding to toolbelt

snmp over /dev/arp ftw (if it's the tool I'm thinking of)

connstat andyf?

if you're talking about connstat then no. but SNMP over /dev/arp? now you got me curious

Oh you mean using the mib interface and not raw snmp.

Yes, I should have been more precise. It's still a very odd thing the first time you look at it, or was for me.

Where can I look at it?

So it basically uses SNMP MiB's to read things from the system?

thx connstat is nice !

toasterson - that's how I think of it - github.com/illumos/illumos-gate/blo…ster/usr/src/uts/common/inet/mib2.h has some more information.

netstat uses it too

got i hate iscsi

err god

jbk: He would agree with you :)

sort of like DNS of HTTP, not one stopped to ask if we should do this :)

You mean DNS over HTTP, right? (And yeah, both DoH and iSCSI are immorally equivalent IMHO.)

DoH is just horrible, while there is a perfectly good DoT

err yeah

i just spent 4+ hours w/ a customer preceeded by another 2 hours of conference calls

so i'm brain fried now

Like either iSCSI or DoH alone wouldn't do that to you already.

that was just the icing on the cake

after everything else

'oh now iSCSI'

Quick question about the upstream overlay. Where are the plugins located that need to be added on the commandline?

you mean for the inner IP->outer IP mapping bits? (there's also the possibility of different encapsulation plugins in the kernel)

it says missing encapsulation plugin when I simply type dladm create-overlay -v 1 ofl/net0

So I guess I need to add more flags

The overlay(5) man page has the details

ah, keep forgetting there are more pages than just man1

If you're still stuck, I can dig up some examples

I'd appreciate it if anyone could take a look at code.illumos.org/c/illumos-gate/+/2647 - two-line fix (plus some tests) ported from OpenZFS

→ CODE REVIEW 2647: 3816 "zfs allow" mis-displays filesystems with "allow -c" but not "allow -s" set (NEW) | illumos.org/issues/3816

andyf: thanks that helps. With that I can start testing

It works really well in my experience, you just might need to drop the MTU on anything created on top of an overlay

So make the default MTU 1500 on vnics on overlays?

toasterson : no jumbo frames?

neuroserve: All vnics are Jumbo frames IIRC

So unless I set the real nics to Jumbo Frames I'll have to drop the MTU

But I guess even then I have to drop the MTU on an overlay a bit to make space for the header

I usually set them to 1400

Etherstubs default to 9000, so vnics on etherstubs use jumbo

for some testing, it would be useful to be able to get the kernel into a state where physical memory is highly fragmented (i.e. few if any contiguous chunks of ram free)... I'm wondering if artifically lowering the amount of RAM via /etc/system might allow the system to get into that state easier (by just generating lots of activity)

or might there be an easier way?

honestly, we're so awful about memory fragmentation, I would imagine you don't need to try too hard

believe it or not it's actually a problem

at least on systems with lots of ram

I don't know of any tweaks that make it happen faster, though.

basically for several drivers, if they do DMA allocations > PAGESIZE for packets (mostly rx since tx you can usually use multiple cookies) that can cause umm... undesirable behavior

like dladm create-vnic can block in the kernel for 10+ minutes