i have a bhyve zone running fbsd15-current; on tcpdump I see lots of (looks like almost all pkts) cksum 0x7e1f (incorrect -> 0x5ca8) types of pkts. heard this might be a vtnet driver problem, not sure if I should be creating the smartos bhyve with any special options to improve behaviour.

think i found some bugs on the freebsd bugzilla, sorry for the noise

I don't know if it is my fault but installing SmartOS as a Virtualbox guest on Linux host, with virtio network adapters, it starts installing, but hangs at 0.2% while Downloading pkgsrc bootstrap..

I used fixed IP for admin interface, seems like that installation passes without pkgsrc, but 'piadm avail' hangs , even the ping works.. so something about networking..

And I wonder why in graphical console, in vim, Delete key is backspace and backspace is delete ..

jperkin: Latest fail2ban in trunk is brocken with python 3.12. Error message ist: "ERROR No module named 'asynchat'".

Got it fixed with pkgin in py312-setuptools py312-pip; pip-3.12 install pyasyncore pyasynchat;

May you can add the three dependencies setuptools, pyasyncore and pyasynchat to the latest fail2ban build?

Only adding setuptools was not enough, see the comment in this pr: void-linux/void-packages 613d8c3

Actually, while installing , installing does do download pkgsrc bootstrap, but it is very slow.. I still hope it is somehow my local networking is the problem, but don't see how.

and funally curl gives up

jfqd1: can you create a ticket and I'll take a look on Monday? thanks

nikolam: I've not heard of any other reports of such slow networking, good few MB/s from here in the UK

you can also try pkgsrc.mnx.io as a cloudflare-backed alternative

I do have a strange issue after rebooting a lx zone, errors are:

WARNING: i40e1: failed to allocate 32768 bytes of DMA for I/O buffers

WARNING: i40e1: failed to allocate DMA resources for RX descriptor ring

WARNING: i40e1: Failed to allocate ring memory

Never had this before. Any ideas how to fix it? No restart, no new vm creation is working on this machine.

Do I need to rebbot the hypervisior?

The machine hat plenty of free RAM…

jfqd1, what distro is in LX zone, just as a sidenote, what image name , SmartOS version , check disk space

Also zfs list | grep UUID will tell if maybe LX zone disk is full / need to give it more space via zfs set ?

Thx for your ideas, ut the vm as the hypervisor has plenty of free disk space. vm is ubuntu 22.04. never had this issue before.

Unfortunately it's looking for continguous memory for DMA.

So free memory isn't the same as contiguous free memory.

rmustacc: what is "contiguous free memory"? Sorry, never heard of.

All good, let me try to explain.

So DRAM is laid out as a couple regions of physical address space.

Ignore for the moment the question of how that maps to a given DIMM itself.

In general programs and the kernel use a concept called virtual memory. That is through page tables, the program's address space can get mapped to any region of physical memory.

For some weird reason, upon fresh install of SmartOS 20240905 , in /usbkey/config there is admin_gateway= set to the same address as admin ip= as is compute_node_ntp_hosts , i switched it to right gateway IP (that I also provided during install and compute_node_ntp_hosts=dhcp and now imgadm works..

The implication of that above is that when a program asks say for 1 GiB of memory it doesn't need to find 1 GiB of DRAM all in a row that's unused.

It just needs to find enough pages that it can program into the tables that map things around.

On x86 the smallest page size is 4 KiB.

What this means is that physical memory often becomes fragmented, pieces of it are used up and if you look at the regions of free physical memory, there aren't many regions that are necessairily contiguously free.

The DMA allocation there is where the wrinkle comes in.

While programs can use virtual memory, devices by default cannot unless something that's called an IOMMU is in place, which isn't available right now on SmartOS (or illumos).

In this case the driver is trying to find 32 KiB, or 8 contiguous pages of physical memory and cannot.

rmustacc: Ah ok got it. So it will try and try and eventually get thr needed ram, right?

I'm not certain.

Depends why it's fragmented.

rmustacc: After 10 minutes the nics were created and the network was usable. Strage issue, never hat this before with Opensolaris, Openindiana, OmniOS or SmartOS before.

Yeah, what's happening here is realistically specific to the i40e driver.

It has a large number of rings and groups, but is trying to conserve some of the DMA allocations to when you create the vnic against it, but then the system has been up for so long that getting the DMA memory is difficult.

ok, so I should consider a reboot of the hypervisor next.

Certainly memory will be more contiguous at boot and probably the same set of VMs will come up faster; howerver, if it's been up for a long time there's no promise this won't happen again, unfortunately.

At least, if I have the right understanding of this.

rmustacc: Thx a lot for taking time to explain t to me!

Alas, it's probably my fault, so least I can do...

And I have seen that when I set SmartOS HVM guest Network hardware to virtual Intel LAN hardware,on Virtualbox Linux host, it is all well. When I set it to virtio-net it is not working for SmartOS guest .(Ping is going through, but commands like imgadm avail , piadm avail hang forever)

a perhaps naive question, is there no way to just mark a specific section and set aside for contiguous use that is only used as a last resort when everything else is allocated, or is that just plugging the wrong hole?

For DMA jperkin?

yeh

Maybe. But for i40e it's a fair bit of memory.

As it's not just the 32K ring, but all the packet rx/tx buffers.

And that memory itself may get chewed up by other DMA allocations, but dunno.

Changing the packet memory to not need contig allocs so it's all page size would probably go a long way (especially with jumbo frames).

For the rings, I'd have to check if the device allows non-contig DMA rings so you could use cookies.

Otherwise longer term, especially for this relatively static alloc the iommu would be better maybe.

virtio-net on SmartOS gues on VBox, seems to work better, with promiscuous mode enabled on VBox's HVM NIC.

jperkin: if you knew the maximum number of guests a given machine would ever be provisioned, one approach would be to define a way to preallocate some number of rings at boot time, before memory gets fragmented.