tsoome_: oh, so we and lld do what it expects, but binutils doesn't now, and won't?

despite both of us having implemented it based on what gnu used to do

[illumos-gate] 16317 SHA2Update() is wrong for 512 MiB or bigger blocks -- Bill Sommerfeld <sommerfeld⊙ho>

It was fixed by illumos 13970 (fenix), so 'startstop.patch' is no longer required for building illumos.

BUG 13970: loader: BIOS loader ld script needs to use KEEP statement with linker sets (Closed)

↳ illumos.org/issues/13970

I think it's been long enough now that we can drop that patch in omnios - I don't think we're building any gate versions from 2021 any more.

hi all, i have panic at nvme with latest illumos updates

1 panic[cpu1]/thread=fffffe000aaebc20: programming error: invalid NS/format in cmd fffffe065806ee00

it is panic under vmware 6.7 with nvme controller

With VMware's virtualized controller or passthrough?

So, uh, that's an identify namespace command with nsid 1 set. So at first glance seems pretty legal? What's the state of the namespace?

rmustacc: added NVMe virtual controller to vm

no drives attached to this controller yet, one drive attached to SATA virtual hba

@andyf: Thanks for the info I removed the patch from my commit again.

rmustacc: no panic if i have one nvme attached to controller.

i have panic if i have just NVMe virtual controller without attached drives

I guess we'll need to look at the virtual controller state then to understand why asking for this is crashing. Is it actually advertizing a namespace being present and active?

it wouldn't surprise me if VMware is perhaps playing loose with the NVMe standards

IIRC, it advertises trim support (whatever the NVMe term is.. every protocol uses a different name), but then returns an error if you actually issue the request

... though I wonder now if maybe instead on VMware we should do write zeros

that's what you have to do with virtual scsi disls

err disks

(which reminds me, I need to upstream that here at some point)

basically on vmware virtual disks, it uses WRITE SAME to write out 0s instead of UNMAP

which will actually shrink a thin provisioned image

because vmware doesn't support unmap for virtual disks

while zfs can compress an all-zero block to a null blockpointer..

this is in lieu of issuing a SCSI UNMAP command

so below zfs

(for VMware disks)

so you can zpool trim a pool on vmware and shrink the vmdk size

Beats having to shut down the VM and use the hole punch thing from the ESX CLI

Or maybe both are necessary?

i mean, internally, when i issued the ioctl to a virtual disk, i could see the size of the vmdk shrink without any other intervention

so I don't think so

it doesn't help there's actually two different ways to unmap in the scsi standard

(because of course why not)

there's the actual UNMAP op

but you can also set a bit in a WRITE SAME op to UNMAP

Things may have improved since I last used VMWare in earnest, which was mostly 5.5

i just need to pull out a few bits -- mostly because I don't think they'd be wanted upstream (for disks that support absolutely nothing of unmap/etc., it resorts to WRITE SAME w/ a block of zeros to basically erase those areas of a disk... which turned out to be useful for other stuff for us, but probably not something that'd be useful more generally

andyf: they haven't

their SCSI support is still pretty shit IMO

and for some reason, i suspect that's unlikely to change any time soon :)

just call it a hunch

:)

jbk: vaguely related in terms of thin provisioning -- I've wondered for some how hard it would be to get swap to use UNMAP or equivalent.

ISTR some OSes (can't remember if its Linux or FreeBSD) attempt to trim the swap device when it's added

though I guess we'd probably want to make it an additional flag to swap -- since people could still be using the swap device as a dump device as well

in terms of while the system is running, that I don't know -- i mean it just needs to call ldi_ioctl(), but haven't looked at teh swap code too closely to know how problematic that might be in practice

rmustacc: i have no ideas and it was working before latest refactoring of nvme driver

[illumos-gate] 16020 Borrowed abds ignore metadata flag during allocation -- Jason King <jason.brian.king⊙gc>

good evening. asuming you are a nvidia graphics driver for solaris, which kernel function would you use to reserve memory to copy your vbios into? I dtraced on :cmn_err:entry and printed the stack(), however the deepest call I see is from :nvidia:nv_printf:entry - and this is pretty sure not the call that produces the error...

If I were such adriver, I might call one of the functions in github.com/illumos/illumos-gate/blo…sr/src/uts/common/sys/gfx_private.h since they were specifically created for me and me alone to use

but I don't know what the actual driver does

alanc: thanks for the pointer, I have so no clue about driver programming I just poke around and hope to touch a nerve :)

and I was just reminded by the comment in the file that while it was originally created for the nvidia driver it was later extended to use by the drm graphics drivers as well

ah! I'll browse through that in a minute

jbk: one of the things I've noticed is that the swap block allocator is a clock algorithm, which means that, over time, your originally-sparsely-provisioned swap vdev gets fully populated.

trim on swapoff/shutdown would also be good for transient vm's (no need to retain swap contents while the thing is off).

sommerfeld, that used to be classically well known that Solaris swap was the only *nix whose swap worked well

sommerfeld, you should check out Solaris Internals 2nd edition, it really hammers home how much it is a highly engineered <s>Boeing<s>Airbus, compared to Linux (which I still love Linux)

Every class of hypervisor still has to implement all the core kernel components to share the machine. As good as Xen is now, it never made sense to me to discard all the hard earned knowledge

alanc: after reading throught that I would suspect gfxp_alloc_kernel_space() being called, would you agree?

copec: I'm not sure what your point is. TRIM basically didn't exist in the timeframe you're talking about. Illumos should evolve to use new features of both physical and virtual hardware.

sorry, I tangent-ed off of the solar swap

I agree, although modern SSDs internal allocation seem to manage just fine with or without TRIM

but it takes a size_t as an argument, and I should be no problem if you ask for x +size_t, you get it. maybe something in that function

I have many SATA SSDs in SAS enclosures that don't use TRIM and I compare their performance to the same disks in enclosures with SATA, and the performance difference after some time is indistinguishable

There are all "enterprise" level disks though

what's more, illumos by default swaps on a zvol rather than a dedicated slice of physical disk; a "trim" (via the ldi_ioctl mentioned above) would free blocks to the pool for reuse even if there was no TRIM going to the physical hardware.

Yeah, TRIM in that regard makes total sense. The swap zvol shouldn't be sparse though, and at least for me that zvol is a drop in every zpool I have, so it doesn't really make the difference in terms of a zpool getting towards full and slowing allocations

ok, it's not fbt:gfx_private:gfxp_alloc_kernel_space

sommerfeld: it looks like maybe if we added an ldi handle to struct swapent (and allocated a flag to incidate if trim should be done), we could maybe issue the ioctl in swap_phys_free()

ok, different approach. paste.ec/paste/eIj27fAE#4cWppNTJaVyAcqLy6W0wB3MkGJSc6dnyCsyL4WuuVjl shows a call to nvidia`_nv001009rm, however dtrace -l doesn't find that one

jbk: yeah, but can you do I/O from where swap_phys_free is called without creating bigger problems?

I tought fbt are generic probes?

IIRC the fbt provider will decline to create a probe for a function if the prologue or epilogue are not what it's expecting.

sommerfeld: what does that mean?

Agnar: the function prologue is the set of instructions that first execute on entry to a function that do things like set up the stack frame, save registers, etc.; epilogue undoes that on the way out.

fbt has some assumptions about the structure of the prologue and if it's not shaped the way it expects it doesn't create fbt probes for that function.

ah,understood

so for "private" functions in closed source code, this could happen and we have no clue of course.

yes, there are compiler options that affect how the prologue is generated and if you're going for performance above all other considerations you probably will step outside the line of what dtrace can cope with.

which makes sense for a graphics driver

heck, I need to sleep...thanks so far for your help, alanc and sommerfeld

if you disassemble that function, we should be able to tell whether dtrace would like it

if I remember, the most important part is that it begins by pushing a frame

if I remember correctly, the nvidia drivers are cross-compiled using gcc on Linux, and thus likely without the stack-frame patches in the illlumos gcc

sommerfeld: good question, unfortunately comments don't really answer that.. an alternative would be more complicated

e.g. you'd probably need to track both 'needs trim' and 'free/ready for reuse' so you're not holding the swapinfo_lock while trimming (since that'd block allocation/freeing)

(side note, it looks like the swap code is using it's own homegrown bitmap implementation to track free/in use... wonder if it'd be nicer to convert it to use sys/bitmap.h

copec: It's actually fine for the swap zvol to be sparse, FWIW.

You can make a reservation for it, obviously, to make it thick

But it is not required. And pre-zeroing it does not improve performance; if anything it makes it worse because of COW.

I think that's what sommerfeld mentioned, they can be sparse but algorithmically stop being sparse as fast as possible

jclulow, I mean you'll also have other problems at that point, but what happens if the zpool fills while the sparse zvol can't allocate?

You can't page more things out

It is not ideal, but it is not necessarily immediately fatal

you'd probably want a reservation, at least.

I mean I think you want _some_ reservation. But the tension created by never-overcommit + fork() leads to some hard bargains

I can't believe it's the future, and fork v. vfork still matters

memory big, software bigger

I mean really we should posix_spawn(3C) wherever we can -- and probably look at moving it into the kernel

Is anyone here using Tribblix? would love some pointers with setting up wifi (wificonfig)

yeah, if no one else gets to it first, it's something I'd like to look at (though seems like there's a lot of pieces to get right).. IIUC Solaris made it a separate syscall at some point

pfr: I don't see ptribble here right now. Is "wificonfig" a Tribblix tool?

Oh, no, that's part of the base OS

I can't imagine it is Tribblix specific

jclulow: not sure tbh.

I have read the man page but I cannot figure out how or where to add my password

'wificonfig connect' doesn't have a password flag

pfr: I fear you may be in for some disappointment

wificonfig is the base OS, but I think old

pfr: I would actually look at the dladm page

I think you want dladm *-wifi

yeah

I didn't find wpa_supplicant in any of the tribblix overlays

and we need to work out why wificonfig is (still?) there?

richlowe: Yes I am unclear.

pfr: The details are managed for you by "dladm *-wifi" I believe

richlowe: yes, I also looked at dladm, but was also unclear how to connect. The tool looks pretty comprehensive for complex networks. I just need to connect to a private home network

You would use "dladm connect-wifi"

and I'll presumably be prompted for the SSID and password?

Sorry, I'm at work procrastinating so not on my illumos machine rn

I think to stick a password in there you want to create a secobj, so "dladm create-secobj"

e.g., see "Example 3 Creating a WiFi Key" and then "Example 4 Connecting to a Specified Encrypted WiFi Link" I guess

in dladm(8)

thanks. I'll look into that

igork: Yeah, I get it. Just the more you can provide me, the more I can try to help. I don't understand how that's currently an illegal request without more context about the state of the controller.

It does look like "dladm create-secobj -c wpa somenameforkey" would prompt for the password yeah

[illumos-gate] 16324 Want Micron 7300, 7400, 7450 log support -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 16325 nvmeadm: want ability to write log page to raw file -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 16326 Update NVMe error status codes for 2.x -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 16327 wdc nvme assertion clearing support -- Robert Mustacchi <rm⊙fo>