looks like a reboot between runs is advisable to clear stuff that got mounted by tests and not unmounted.

Yes, and I've also had occasions where the dump device is left set to a volume on the testpool, preventing it being destroyed.

Is that specific to the ZFS tets that you're seeing that?

yes.

you can also clean up /var/tmp between runs without having to reboot

I saw at least one left-over mount point.

It really seems like the kind of test suite that you don't want to run on a machine with things you care about

i.e., it should probably run in a throw-away VM

haha no..

which reminds me.. at some point i need to finish setting that up

i managed to get pkg(5) working in a smartos zone and i was going to create a dedicated test vm separate from the one used to build/dev

it's silly, but i don't want to give up my current omni vm because by some cosmic conincidence, the UUID ended up with the first 5 digits of pi (hence the name) :)

jclulow: indeed, but these issues should be tracked down and fixed because they cause damage to other tests in the same suite!

oh, to be clear, I am also a broken record on the subject of needing someone to make a complete catalogue of flakes in the test suite

I just think, even if it's 100% reliable, that I would still only expect to run it in a throw-away location

Unlike, say, the libc test suite or whatever

yeah, josh has been... vocal about how this needs fixing

the challenge is that it requires such a throwaway setup to run, and many of us don't have one

or have one unique enough it's hard to tell if the problem is the setup, or the tests

jbk: it _is_ silly, but whenever I get a "good" commit hash, I'm sad to lose it

if zfs tests are 100% success, then it must be safe to run; failing tests are obviously potentially harmful.

you need to make that more obviously deadpan

in my test vm, I have list of 9 failing tests, but it may not be 100% true.

yeah, i have my list -- though I thought I filed tickets for all of them (to help others) even if I wasn't able to root cause all of them

for example, in my vm, I have FAIL l2arc tests, while andyf had them PASS.

yeah, some are unreliable

danmcd runs the ZFS tests as part of the smartos release process

so is a good person to talk to

yep, that too.

A think that rmustacc did with nvme, which is wonderful, is split his test suite into "destructive" and "not"

that would help the zfs tests in so many ways (even if there are not that many non-destructive)

^^^

I have a VMware Fusion VM specifically tailored for taking the abuse of such a run. It has 6 virtual SATA drives just for ZFS test, a virtual NVMe drive that appears to pass all but one corner-case test I'm chalking up to VMware madness...

... and it's called "new-from-scratch-SmartOS" so its usage is right there on the label.

Some are very unreliable.

I was using iscsi luns to get trim tests running in vmware fusion vm.

for trim tests, this one is only one failing: Test: /opt/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_verify_trimmed (run as root) [00:02] [FAIL]

I have a test setup in a bhyve vm with multiple host-zvols imported as additional disks.

yeah, I have an illumos-image-builder setup that did it (though not with me)

it's a bit hinky to use, but jclulow's image builder set up to build a fresh disk out of your proto area is a _wonderful_ test setup.

and I'm queuing up a couple test fixes for some of the tests that are failing for me to go along with the vdev_indirect fix I'm testing.

thats nice.

I feel confident they will be appreciated as much as, if not somehow more than, the fix

yeah, with all due respect to the work that'I don't think I'd ever put myself in a position where I'd want to use zpool remove on a system I cared about.

err, that escaped early.

with all due respect to the folks who did the work on it, I don't think I'd ever put myself in a position where I'd want to use zpool remove on a system I cared about.

Yeah, in 20240418 there were 24 skips all ZFS trim related.

It's good that the capability exists.

49 failures (not counting fenix OS-8542 ):

OS-8542: SmartOS bash's built-in echo breaks illumos#16437 (Open)

↳ smartos.org/bugview/OS-8542

4 are BHYVE ones that wont' work due to nested-virt stuff

1 mapfile due to the bizarre way we seem to build libgcc

but it's not needed often enough that it is exercised enough to be dependable compared to alternatives like zfs send|zfs recv

2 aforementioned NVMe test failures

2 ksensor failures (also likely VMware-caused)

and 40 ZFS failures.

ksensor should work despite vmware.

But likely illumos.org/issues/15873 which I saw had been filed the other day.

→ BUG 15873: ksensor tests should accept the ksensor_test driver already being installed (New)

That is the ksensor tests don't rely on any actual hardware thing.

Oh, and it's one ksensor failu7re not 2 The other non-zfs failure is svr4 packaging, which we don't ship.

40 still? huh, could you mail me a link?

Hang on...

@rmustacc /opt/os-tests/tests/ksensor/ksensor_basic.32

I was kind of hoping that since we both run tests on fusion vm, the resulting list should be similar:D

```

@rmustacc

ksensor_basic.32: TEST FAILED: failed to open /dev/sensors/test/test.volt.0.1: No such file or directory

tsoome: list forthcoming...

danmcd: the ksensor failures are a race condition

I'd talked to robert about this on arm, the problem is that the pre-test hook loads the test module, but doesn't (can't?) wait until they're ready

so if you're unlucky, the actual tests then run before the test module is initialized

Oh yes!!! That's right, I have to do that DIFFERENTLY becuayse it assumes it's there.

(Perhaps I should put a delay in?)

I'd have thought about this harder, but I thought it was unique to emulated arm, I'm sorry.

I even filed a bug for it I think...

danmcd: if there's something that you can poll for, a timed polling loop would be good. but a fixed sleep is going to be either too long (in which case it wastes test time) or too short (in which case it's flaky..)

and if there isn't something you can poll for for driver init, maybe there's a missing interface...

the devices existing, would be the obvious thing.

fenix illumos#14886

BUG 14886: vmm_drv_test needs to be less IPS-dependent (New)

↳ illumos.org/issues/14886

but I'm not sure if that comprimises what the tests are testing

(And it was an inconsistency...)

ANYWAY... I owe tsoome a list of tests.

danmcd: that's vmm _not_ doing what ksensor does (14886)

what you want in 14886 is what ksensor does, what you want in ksensor is something to poll for readyness of the test driver _because_ it does what you want

(I hope that made sense)

I had forgotten the bit on ARM and a race there. I can try to look at fixing it up. Sorry about that. It is unfortunate, but there is probably a cheap way to guarantee it's ready with devfsadm.

me too, really, it didn't occur to me anyone else could lose that race

But I expect a devfsadm call should block on that much, but hard to say.

is there any reason our various kernel printf functions don't support '%#x'?

Probably just a lack of traditional support for the alternate form / need.

rzezeski: I have been asked before in code review not to use it, because people didn't immediately know what it meant

perhaps related

(and to be fair, outside of %x, I don't know what it means for any other format)

Okay, well FWIW it would help to have it because stuff like the cxgbe common code uses it all over the place. And it means instead of getting useful data in the log you just get the character 'x'.

Yeah, to be clear, I'm very in favour of it existing and working and being used

We may find it confusing, but for better or worse we got other's code in our gate that like to use it.

richlowe: How hard is this to support (I know nothing about printf stuff, I thought it was part of the compiler)? I could just add it as part of this T7 wad.

rzezeski: %#x goes back at least as far as C89 (I have the ansi spec in dead tree form).

rzezeski: it's implemented in C as part of the library in userland and in the kernel for kernel.

rzezeski - if the kernel printf functions you're using are tagged with __KPRINTFLIKE, you will also need to teach gcc that it's ok

rzezeski it would be nice to have:) we use it in few places in loader ;)

rzezeski: take a look at usr/src/uts/common/os/printf.c

and unfortunately then wait for everyone to upgrade their gcc

sommerfeld: k, well maybe I'll see if I can get it working

rzezeski: it should be easy for simple cases like that

but I've been wrong in _ndprnt-like places before

sommerfeld: though i don't think that's used in userland, libc has it's own (slightly convoluted) _doprnt

or whatever it's called

it does, userland supports %#x obviously, because standards.

the kernel has a different non-standard printf

which is also where andyf's comment about possibly having to teach GCC that # is supported on %x comes from (on the other hand, there's a better than 50% chance that it thinks it's supported already...)

Well, yes, but I hoped the same about %h and %z (or something in that area)

I mean, if this has been around since C89, wouldn't everyone already know about it :)

I think they were supported by the kernel, but gcc didn't believe so

Well, I'm placing the bet because ryan says that his code is already using the format and it's just not working.

if gcc were grumpy, it would presumably be grumpy _now_ :)

Yea, this is code shared across FreeBSD/Linux/us

so I'm guessing they have it

no, the kprint stuff is illumos-specific

but it'd still be firing on your format strings already, if it were upset

sorry I can't words good

rzezeski: actually you want to fix vsnprintf in usr/src/common/util/string.c

Haha, yea, all I mean is, this code is compiled/running on other platforms. And I figure it works there since Chelsio choose to useit.

richlowe - good point. I don't see it in github.com/illumos/gcc/blob/il-10_4_0/gcc/config/sol2-c.c#L43-L75 but it might be somehow silently allowed

digging around in ancient history: looks like 7th edition unix doesn't document %#x while 4.2BSD documents it.

so it's probably established enough to be safe to use, i mean we're not postgres :)

we have absolute control of our compilers jbk, we can use all _kinds_ of things we want to, when they work.

the downsides are annoying, and I see why folks complain, but when you get the upside it sure is nice :)

and for the kernel/standlone printf which doesn't do floating point types, the # modifier only has to deal with #x and #o

computers are so much fun

it's a laugh a minute

../../common/inet/ip/keysock.c:2457:26: error: unknown conversion type character '#' in format [-Werror=format=]

cmn_err(CE_NOTE, "Test %#x\n", keystack);

It looks like an easy patch to gcc though, if we end up adding support.

andyf: help me out here because I'm dumb, the cxgbe code is already using it and compiles just fine, it just results in bogus output. What am I not understanding?

Which printf-like functions is it using with that?

and is the error (-Wformat) possible turned off?

*possibly

snprintf and then passing that to vcmn_err

I see nothing about turning off the format warning

cxgb_printf()

that's what it calls

Perhaps the compiler can't follow the chain?

Unless cxgb_printfs is declared as printf-like, the compiler won't follow the chain.

KPRINTFLIKE in this case

If you were to mark cxgb_printf as __KPRINTFLIKE(3) or whatever, you'd probably get the error

(for it to fail)

but yeah, we should add support to the illumos gcc, even if you're safe right now

yeah, KPRINTFLIKE for kernel printf rules.

because then we can all use it

richlowe: yea, I mean I can go and update all of the cxgbe common code, but then the next person to update this driver might not realize we don't support # and reintroduce this issue. Or we could add the KPRINTFLIKE so it's impossible to miss. But it does add additional noise to the delta between us and upstream. It just seems like we should allow #.

Oh, definitely, we should allow it, and update our gcc branches.

yeah, I'm apparently being super unclear, I absolutely thoroughly want you to allow it (though it's not really up to me, obviously, it's up to you). We're just saying where the rest of the bear trap is hiding

unfortunately, the best person we have at translating what I'm trying to say has already said no for today :)

richlowe: I was also being unclear in that I was agreeing with you. haha. Consider this 3 votes for adding # support.

I may be too obsessive about this archaeology because I found and was just looking at 4.1BSD's doprnt.s, which has # support that isn't mentioned in the manpage.

(and, yes, it's in VAX assembler...)

the unix history repo does that to a person

and there is a doprnt.s.old that doesn't have it. so that looks very much like at least one version of %# was born in or around 4.1bsd

rzezeski: be glad that our printf core is in C, not assembler.

after my first build for debug and non-debug I could start doing incremental builds in nightly?

Or just use bldenv.

if you did debug _and_ non-debug, and don't have multi_proto set, incrementals are weird

in general if you're developing and what incrementals and convenience, only build one way or the other, until your RTI build

I'd just use bldenv after a nightly. incrementals might work ok if the nightly is either debug or non-debug but not both at the same time in the same tree

s/what/want/

oh ok, thanks for the pointers. I'll read about bldenv and the whole workflow again, I usually was doing full rebuilds (omnios bi) but is taking around 3 hrs.

`bldenv <the same env file you use with nightly>` will spawn a shell you can work in where 'make install' will work to put things into the proto area

much more convenient, I find, is `bldenv <env> <command>`, so like `bldenv ../foo 'make -wej 28 install'`

(the documentation is probably clearer and better at explaining)

richlowe thanks!, I'll spend more time in the docs

neirac: illumos.org/books/dev/workflow.html#incremental-building in case it helps.

rmustacc thanks, I was looking at the illumos.org/docs/developers/build. I 'll check the workflow next to have the whole picture.