Is there a way to get bhyve to tell me why my VM won't start? The only thing in the logs is "Feb 10 23:59:54: bhyve exited with status 4" which is incredibly un-helpful.

daemon: I do appreciate how listening on all makes it easier to move stuff from host to host.

bhechinger: what prints that message?

bhechinger: using vm-bhyve?

Yes, vm-bhyve. It's printed in the logfile

bhechinger: check the "debug" option. enabling it will get you a bhyve.log with bhyve's error messages.

device emulation initialization error: No such file or directory

Well I now know more than I did.

That's only slightly less un-useful, but we're getting somewhere now. :-D

might be related to PCI configs

I'm not doing anything funky like passthrough.

This VM did used to work then I went mucking about with the networking on the host. I can't imaging rebuilding the VM would make a difference but maybe that'll make a difference?

bhechinger: google suggests that's possibly a missing interface (ie: tap2 when you want tap1 or similar)

dvl: just played with this a bit on a 12.4 VM w/ipv6 and this gets me a static v6 IP using a static gateway and an alias that works:

ifconfig_vmx0="DHCP"

ifconfig_vmx0_ipv6="inet6 2600:4040:a9da:xxxx::beef/64"

ifconfig_vmx0_alias0="inet6 2600:4040:a9da:xxxx::dead/64"

ipv6_defaultrouter="2600:4040:a9da:xxxx:1a03:73ff:fee1:7b35"

both IPs reachable from outside

spork_css: I can't see where that would be though. I'm used named bridges and everything lines up.

spork_css: Thanks. I'll look at that. To be clear, IPv6 works. Not all IPv6 aliases get created.

is vmm.ko loaded?

(for bhechinger)

spork_css: I notice /64 vs prefixlen 64.

Yeah, I'm lazy.

spork_css: yeah, it is

Title: Snippet | IRCCloud

dvl: I tried with one alias, no problems, rebooting with 4 now...

spork_css: Yes I get one alias only

I do wonder why my aliases shouldn't have the ipv6 equivalent of a /32 mask

Thanks for the help everyone. I'm going to call it a night. I'll bug you again another day if I haven't made any progress myself.

dvl: success with 4 aliases

spork_css: good.

dvl: definitely incrementing the alias number? :)

hrm, wasn't there an announcement that 13.1 was dropping sendmail by default, some new MTA?

i can't fin dit

does your console.log with rc debug enabled show anything interesting?

spork_css: Yes, good question, yes.

dvl: I know this shouldn't matter, but if you start at "alias0" instead of "alias2" does that change anything?

spork_css: not all mine are shown, that's a grep of inet6

reversing the numbers on the alis

changing the numbers doesn't change the results.

Demosthenex: i think it's only in main for the moment? definitely not in 13.1

dvl: anything at all about v6 in the console.log? Do you have rc_debug="YES" set?

rc_debug="YES yes

yuripv: the git commit was 2022, but it looks like maybe 14.

np

pizza just arrived. I may be done for a while. ;)

dvl: good luck!

I'm thinking my testing is not correct. 'service netif restart' does not seem to be restarting igb0 off from scratch.

dvl: for reference my testing is in a VM that takes like 20 seconds to reboot, so I've been rebooting. I don't trust the "netif" script myself, doesn't really seem to get you in the same state you'd be in after a reboot.

I think I was mistaken. I think it's doing the right thing. Still not getting aliases to work yet though.

I posted forums.freebsd.org/threads/trouble-with-ipv6-aliases.88008

Title: trouble with IPv6 aliases | The FreeBSD Forums

dvl, Shouldn't that be ifconfig_igb0_ipv6_alias0="inet6 2610:1c0:2000:11::100 prefixlen 64" instead?

I am not sure but I had thought that the IPv6 stack was handled in parallel with IPv4 but that meant that alias declarations are in parallel too.

rwp: I'll try it, but see grep alias /etc/defaults/rc.conf

well, well, well...

rwp: I think you win the internet.

You are right that the sample inet6 line in that file is the other way.

I win the internet? But I don't think I can handle the internet! That might be the type of gift which is just too much. :-)

Also as I read through "man rc.conf" and look at the section "ifconfig_⟨interface⟩_ipv6" that matches the sample and your original.

yeah, "ifconfig_igb0_ipv6_alias0" contradicts the rc.conf manpage

"Aliases should be set by ifconfig_⟨interface⟩_alias⟨n⟩ with “inet6” keyword. For example:

ifconfig_em0_alias0="inet6 2001:db8:2::1 prefixlen 64"

It still is not clear to me if ifconfig_igb0_ipv6_alias0 worked or not though. Regardless of the documentation saying it should be different.

IPv6 after 20 years should be in a mature and solid place but unfortunately is still often not quite there yet. :-(

ipv6 has always worked for me, whenever i've had an ISP that provides it

I just use FreeBSD on a regular desktop computer though, nothing special.

Anyone here have any success with net/unfs3?

What is the purpose (asks me lazy ass)?

would there be any benefit in writing a chess engine as a kernel module?

alip: not really, but it could it be a fun exercise

What is confusing about the ipv6 alias - why was ifconfig_vmx0_alias0 working for spork_css but not for me, yet ifconfig_igb0_ipv6_alias0 worked for me?

rwp: took HTTP/1.1 until HTTP/2 for broad enough adoption of not HTTP/1.0

Reading man rc.conf, and finding that etc/start_if.<interface> files can be used for configuration

and further testing is not conclusive here. I'm still not getting the expected aliases.

Next step: stripping down rc.conf

Linode doesn't have ready FreeBSD nodes, any other cloud services that work well with FreeBSD?

I liked DigitalOcean

I should use it again.

I ran a mailing list with FreeBSD and DigitalOcean a few years ago

worked well

was easy to get a PTR record

(unlike amazon lightsail)

I thought DigitalOcean had recently stopped offering/supporting FreeBSD

are there ready freebsd nodes?

oh...

that sucks if true, parv

ah, it is true...

guess I'll have to use Vultr next time

not only no default freebsd, they block 25 port by default and even if not, they have blacklisted whole ip ranges

promising?

I think they block 25 too

Dropped FreeBSD to "empower our customers by providing them with simple, reliable cloud infrastructure" forums.freebsd.org/threads/vps-friends-of-freebsd.85048 💩🙄🤣

Title: VPS friends of FreeBSD? | The FreeBSD Forums

but vultr was okay when I used it

but many providers have easy setup with custom image

but for example on digital ocean custom image does not support ip6

how interesting and bizarre. ipv6 is so important

only if you are able to spin first like their distro and then change it to freebsd

i did openbsd there with ipv6 without issue, but first spin up debian, then dd from console

but if I upload and boot custom image, ipv6 will be disabled for that droplet

but out of that and that they are assholes to support practically just shit now ... vps works good there

without mail obviously wich is another shit if one pay for vps

yeah, I want to run my own mail server again, this time for personal email addresses

tired of Google locking me out of my gmail accounts every month

so depends on country, big providers block 25 or their ips are blacklisted

u in europe?

washington state, USA

oh, no idea there, but maybe hetzner?

shall check it out

on vultr you pay monthly? Jeez, I thought amazon price structure is non-transparent, but yeah lol

I guess you'll see on the invoice, lol

ye? they change it to pay upfront?

I think DO stopped providing FreeBSD nodes, but you can still create them. I'm not sure how though.

yes digital ocean provides nothing now, just centos which even does not exist more, rocky alma both like centos and ubuntu and debian

so practically just centos and debian

but custom images works

I think people host email on hetzner

I usually never heard of any complain about hetzner

I stopped running my own MX server and leave that to others.

borbela: ah, they do charge hourly and there's a monthly cap

I say that because port 25 was mentioned above. I used to be quite keen about running my own mail servers and I understand the eppeal. However, I thought the ongoing work involved was better spent elsewhere when I can pay someone else $50/year to do it for me. I loved running Postfix and Dovecot, great tools.

This host is getting lots of 'shutdown -r now' practice in the past 12 hours. :)

If you do it as "shutdown -r now ; exit" you can save your tcsh history ;)

true minimalist

I am grateful for a solid ipmi interface (it's a Dell R720 with iDRAC7, hosted by NYI).

xtile: I don't get it... a joke? Sorry, it's 5AM here and I woke up early at 3AM

How does IPMI of SuperMicro compare?

it's not a joke: i guess the joke part is whether you'd want to save your history on that machine or not, which, i assume, is, not really

parv: Not so well. java vs html5 on the Dell. I have two supermicros and I'd prefer not to have to run java.

dvl, Thanks. What is vintage of your SuperMicro machines?

parv: Now that you ask, one of my projects is to retire one of the supermicros by combing its function and a Dell R720 into a Dell R730.

dvl, I have X1[0-2] (at work)

parv: dan.langille.org/2022/12/31/slocum-8 and dan.langille.org/2022/12/31/knew-8

Title: slocum – Dan Langille's Other Diary

dvl, Sweet.

parv: sorry, I see now you'd have to click a few times perhaps to find the answer

parv: knew is a X10SRA-F-O

parv: and a X9DRE-TF+

dvl, Yeah saw that slocum has X9*

dvl, Do you put labels on your disks when they are more than a handful in a machine?

dvl, I take it despite the issues with upgrade of firmware/software of iDRAC, upgrade was worth it?

parv: I add the serial numbers as labels. For example, I did this very recently: gist.github.com/dlangille/f70e2e8d3…00065a54ea#file-zpool-create-data01

Title: These are the file systems I need to combine if I want to merge two servers into one · GitHub

e.g. sudo gpart add -t freebsd-zfs -a 4K -l Y7P0A02MTEVE da1

dvl, Sorry I meant printed labels.

parv: Yes, there are difficulties sometimes in getting Dell firmware updated. Often, I think, because it's older and getting it newer. However, help is there with updateyodell.net

Title: Update Yo Dell, foo! | An FTP server with life cycle repos for G11-G14 Dell PowerEdge Servers

parv: Yes, i have put labels on the drives, but I have not done that recently.

parv: There's the iDRAC firmware, the BIOS firmware, the HBA firmware, etc. And sometimes you have to go in steps, update to 1, then 2.2, then 3.1 etc. And sometimes it takes multiple iterations to get them all updated.

OK, good progress this morning. I stripped out /etc/rc.conf and started putting pieces back in. Now I'm at the point where services come up as expected and ifconfig is doing what I need.

dvl, I am skittish when it comes to updating firmware for fear of bricking, or possibility of need to invest ever more time to resolve whatever issues would come. So unless I really, really, truly have to, I don't

dvl, That's great

parv: yes. I am similar. I usually update the firmware when I first get the host. At work, we update firmware each time we do a bigger update (e.g. FreeBSD 12.3 to 12.4)

parv: but we have service contracts etc. ironically, we prefer to buy Dell instead of our own brand because customers get preferred support not internal stuff. :)

this, despite we get hardware at 10% retail.

dvl, Ah, I see. Your outfit is much more adventurous|bold than ours.

How would one go about specifying IP address settings (static IP address, mask or DHCP) based on host name instead of hard coding the strings with "ifconfig_<driver>"?

I am thinking: found a host name in /etc/rc.conf; now find the file with IP address data & stuff /etc/rc.conf accordingly.

parv: I don't know that it can be done based on hostname. But that would be interesting.

Anyway, later ...

this is now my life today. Add more entries back into /etc/rc.conf, shutdown -r now, inspect, save /etc/rc.conf, repeat.

Surely there's a way to reload rc.conf without restarting your machine, dvl?

mexen: Yes, I've tried this, but it's not quite the same as rebooting: service netif restat && service routing resart

I wonder what's missing

rwp: I must take back the internet. It seems the problem was not solved by that alternative rc.conf format for ipv6 aliases.

I wonder what parv was talking about.

Supermicro IPMI hasn't been Java for literally.....5? 7? years

for the entire X1[0-2] line

OUt of intellectual curiosity...

why should zroot/var/tmp be its own dataset on the default ZFS install?

like, what problem is there, that is solved, by having zroot/var/tmp as its own dataset, instead of just a subdirectory under zroot/var

These are the things that capture my attention on a Saturday morning.

DrKK`: hier(7).

good idea

man 7 hier

Despite many people assuming it to be true, /var/tmp and /tmp are not the same directory.

whoops would help to do that at the prompt

well,

it's not about /tmp and /var/tmp being different

my question is about why it needs to be a separate dataset

from /var

At a guess it'd be because zroot/var has different properties from zroot/var/tmp

I will guess that the reasoning is that then /var/tmp can have different properties from the other datasets in /var

yes, but,

Jinx! :-) Race condition.

why would someone need that

and if they did need it,

they could make their own dataset easily and mount it

so the question is, why should it be the default situation upon install

DrKK`: for the same reasons zroot/var/log, zroot/var/crash, and other datasets have their own datasets.

Do you know why ZFS on root has the set of datasets it has, and not just one (or a few) dataset?

Picking everything up and replacing a non-dataset with a dataset is easy enough but still not trivial. Changing properties on an existing dataset is trivial.

There is one very important reasons that it is the way it is.

For example setuid is off on /var/tmp by default installation but on for /var and I have no idea why it is on there.

oh that's interesting, isn't it.

That's not it, though.

Boot environments are the reason; the idea behind them is that you can move between boot environments, and all of the system related changes that might happen during an upgrade can be reverted by going back to a prior boot environment, whereas all the user-created data is left alone.

not to be difficult debdrup,

Boot Environments are one of the best features that is implemented by default upon install.

you'd still have boot environment isolation

if you had the /var/tmp

as a subdirectory under the /var dataset

which is why it's perplexing.

at least, unless I misunderstood what I was looking at this morning

well whatever, I'll take another look. Thanks for the answers.

Careful examination of the directories that've got their own dataset will show you that if they were included as part of the zroot/var dataset, they'd shift around if boot environments are switched.

That'd mean user-created data is changed.

ah

I will guess that some decade ago when zfs started to become popular in FreeBSD that the person implementing it had learned a few things based upon experiences with Solaris zfs and set things up as a best guess. Then some minor changes have likely occurred in the layout since then. But it works. It's not broken. Stability is important. So changes happen very slowly.

Boot environments come from Solaris/Illumos.

Title: Makefile - OpenGrok history log for /illumos-gate/usr/src/cmd/beadm/Makefile

a ha

zroot/var mounted=no

got it

:)

both zroot/usr and zroot/var are not mounted but act as a parent for other datasets to inherit from.

That way /usr/local is included in the zroot/ROOT/default Boot Environment.

Take a look at "zpool history" or "zpool history | grep -v -e zfs-auto-snap" if auto snaps are enabled.

The first dozen or so lines will be the installer's commands to set up the zfs pool initially.

thanks

I'll note that for zroot/usr and zroot/var that the mount point is set explicitly because otherwise it would mount under /zroot/usr and /zroot/var and children of them too.

By explicitly setting the mount point for them then later children inherit that and mount by default in the desired location.

The item I don't understand and am exploring is zroot itself. It canmounts by default. Why have /zroot mounted?

I set the canmount property to off for zroot as an experiment not long ago (a month?) and am waiting to see what problems I might encounter.

So far so good. (shrug) But I wonder why it was set that way as a default.

After unmounting /zroot I also removed the empty directory too as associated cleanup.

Regarding Boot Environments, I have this bookmarked as required reading: forums.freebsd.org/threads/howto-freebsd-zfs-madness.31662

Title: HOWTO: FreeBSD ZFS Madness | The FreeBSD Forums

hmm. "net/tigervnc" is makred for for removal at the end of March

... Is that because there are now "net/tigervnc-{server,viewer}"?

That'd be a decent reason for the removity.

Sorry, I read terribly mistakenly. Port to be removed is "net/tightvnc"

Nevermind

DrKK`: iirc, it had to do with setting noexec on it or something

DrKK`: yes, /var and anything without its own dataset, fall into the boot environment, so are swapped when you switch BEs (like /var/db/pkg), but /var/tmp is global between them all

etc

I shuddered after seeing new port called "git-branchless"

parv: 🎶 what is it good for 🎶

(absolutely nothing)

(to repeat) "vfs.zfs.arc_free_target" takes number of pages. So how/where do I find how many bytes are in a page? I am on amd64 system

meena, ;-J

parv: There's hw.pagesize, but there's also hw.pagesizes

not sure whether multiples are used concurrently or not

mason, Thanks much. (hw.pagesize is 4KB here)

Oh I am sorry, s/KB/KiB/

Which is to say, "what we all mean by KB" :P

8-)

I thought "vfs.zfs.arc_{min,max}" would have been retired in favor of "vfs.zfs.arc.{min,max}"; see both on -Current

Computers are always binary powers of two. SI units are nice for dealing with physics and the real world. But not for computers.

Powers of ten are themselves arbitrary based on the average count of fingers on a human.

The World is arbitrary, capricious, terrible

I have often wished for six fingers as standard and then we would have base 12 or base 60 which at least would have a larger number of integer divisors.

:]

Whoops, forgot to re-run xmodmap.

Which see: bpa.st/GJO2K

Title: View paste GJO2K

But then I think... Tentacles... We could get to base 16 pretty easy with tentacles!

Is that due to your new keyboard regime? Or something that predates it?

It long predates it, but I have been swapping keyboards, and the mapping resets when I do.

Hello; I'm trying to install FreeBSD 13.1 on a newly-refurbished Dell Latitude 7390. I flashed the `-memstick.img` to a USB stick and can boot into the installer just fine, but it doesn't recognize the hard drive (`geom disk list` only shows the USB stick). I see nothing obvious in `dmesg`. I'm not sure what to do from here.

what harddrive is there?

also try escaping to shell from installer and checking dmesg

fran: It's conceivable you want to change how the fixed disk is presented in your BIOS/UEFI/whatever.

`PNY USB 2.0 FD`, the installer disk, is showing up as `da0`. And yes, I escaped to the shell to run `geom` and `dmesg`. Nothing immediately jumps out at me, although I'm not 100% sure what I'm looking for.

Try: sysctl -n kern.disks

Try: camcontrol devlist

or: ls -l /dev/ada*

for the barbaric, lazy way

sounds so '00, everyone uses nvd*/nda* nowadays :D

rwp, those two commands return, respectively, "da0" and an entry for "<PNY USB 2.0 FD PMAP>"

No /dev/ada* devices are showing up.

yuripv: Check your BIOS to see if it has options for how it presents your disk.

fran: *

yuripv: Good point.

Hmm... I don't know but... Is this an NVMe system? And perhaps the live boot does not have a driver loaded for it?

well, let's first get an answer about the drive installed

BIOS ought to show that as well.

I'm not super clear about the hardware specs; I bought this particular model because it looked promising in the "Laptops" list on the FreeBSD Wiki, with a link to a narrative of someone installing FreeBSD 12.1-STABLE with seemingly no problems.

Rebooting into the BIOS now to see what I can find.

("ships without hdd installed" :D)

yuripv: The first desktop system I was ever issued that had an NVMe, I thought they'd shipped it without storage until I realized.

The BIOS system information tab has two entries, "M.2 SATA = {none}", and "M.2 PCIe SSD-0 = (serial number of some sort)".

Under System Configuration -> Drives, there's checkboxes (both checked) for "SATA-2" and "M.2 PCIe SSD-0".

yuripv was right about my ada0 then

nvme/nvd(/nda) are in generic so it should be detected

There's also a SATA Operation tab with three radio buttons, "Disabled", "AHCI", and "RAID On" (this last one is checked).

That's all I can see in terms of hard drive options in the BIOS.

set that to ahci, but you don't have sata drive anyway

now that we know you have nvme drive, try `grep ^nvme /var/run/dmesg.boot`

"nvme0 at channel 8 on ahci0"

Sweet!

so now disk is there?

The disk doesn't show up in `geom disk list`.

grep ^nvd /var/run/dmesg.boot

Nothing for `^nvd`.

nvmecontrol devlist

Two lines, one beginning "nvme0: IDENTIFY (06)" and one beginning "nvme0: ABORTED - BY REQUEST".

ugh

was there anything else besides "nvme0 at channel 8 on ahci0"?

Nothing else.

may be try setting the sata operation to disabled?

i'm not sure why nvme0 is being on ahci0

The only other line in the dmesg output containing "nvme" at all was "ahci0: Detected 1 nvme remapped devices" a few lines earlier.

aha

probably some compat stuff for some ancient OS

Hmm, this looks related. lore.kernel.org/linux-pci/20190620061038.GA20564⊙ld/T

Title: [PATCH v2 0/5] Support Intel AHCI remapped NVMe devices

It was. I went back to the BIOS and switch that one setting from "RAID On" to "AHCI", and now the hard drive is showing up as /dev/nvd0.

you mean you switched it to "disabled"?

(i thought you did the "raid" -> "ahci" previously)

I read that as fran had not yet rebooted with AHCI change

I forgot to make that change.

In any event, it's working now. Thank you for the help.

Have fun!

fran: i'd try setting it to disabled anyway, "remapping" could affect performance

I'll try that at some point and see if it still works. Thanks.

Ok, poking around and fixing some other things and I think I know the cause of the vm boot error. I had deployed an old template that used network0_switch but as I'm no longer using vm-switch I changed that to network0_device. That's when the booting issues start.

Title: Snippet | IRCCloud

So, I mean, it's there?

Title: Snippet | IRCCloud

oh, do I need the full path to the device?

DrKK`: yeah, I'm really liking the current Supermicro situation with IPMI - not only is it "free", but it's been a real pleasure to use since the move away from Java

We did splurge for the $30 key to allow remote BIOS updates, and it was pretty cool to update the BIOS on 4 machines from my couch a few weeks ago.

bhechinger: is this vm-bhyve ?

rtprio: yes, vm-bhyve

Oh, wait, network interfaces don't get dev entries. Heh.

no, they don't, they're tun entries in `ifconfig -l`

in sysctl dev., but not in /dev

so my sadly incorrect theory is wrong. :-D

ok, so what can't bhyve find when I use networkX_device?

can you share the sierra.conf

Title: Snippet | IRCCloud

both internal and external switches exist?

bridges, but yes

well, 'switch' in vm-bhyve is the bridge that vm-bhyve creates

I'm not using vm-switch, I'm managing the bridges external to vm-bhyve

Title: Snippet | IRCCloud

i realized that after i typed it

It happens. :-D

Ok, time to take the kids to see Avatar, be back quite a bit later. Thanks for the help so far!

Awoooooo~

gosh, i hope they're seeing Avatar, the last airbender…

After every few minutes screen flickers on 14/xorg (i915kms)

In name of increasing battery life of Framework laptop, tried disabling power on "external ports", thinking that would affect only the 3 pass-through Thunderbolt ports. Realized after booting that that affected the WiFi card too as the inteface kept toggling between being up & down

(4th Thunderbolt port is used to supply AC power)

fun

I'm trying to figure out what's going on with dhcp6c (specifically, the size of prefix my ISP will give me) but I can't find the logs. I don't see anything about dhcp6c in /var/log/messages*. Where should I be looking for these logs or how should I be enabling them? The dhcp6c* man pages dont mention logging.

Well, the man page for dhcp6c(8) mentions syslog(8) but I don't seem to have that man page either?

for syslog?

I do not have the syslog(8) man page as referenced in dhcp6c(8).

i would suggest just running dhcp6c in the foreground until it's worked out

but that must be a linux-ism as there's no syslog(8)

I was hoping to not run dhcp6c in the forground (because I think I'd like to have the logs regardless after this investigation) but it'll at least let me figure this bit out.

it should show up in daemon.log if i had to guess

Actually, daemon.log looks to be right. I hadn't checked the rolled logs. Now my issue is that it only seems to be logging errors. The only entries in there are errors when my (wonderful nine fives) ISP went down. Is there a standard way in rc.conf to add command line parameters to a service?

(In order to pass the -d debugging flag)

check the rc file, in /usr/local/etc/rc.d for options

they're generally documented at th top of the file

It wasn't documented there but I took a guess that the rc system does some automagic around *_flags variables and that looks like it did what I expected.

👍

dvl: got any debug output to share from 'rc_debug="YES"' in rc.conf? I saw lots of interesting stuff in there when I was tinkering.

ooohh