freebsd13 enable ipv6, could get ipv6 address, can't ping6 ipv6.com(no route to host), any idea?

kyonsalt: ipv6.com has no ipv6 adress in the dns

try "ping6 www.kame.net"

:),same response:"no route to host", "netstat -nr" shows ipv6 gateway and could ping that ipv6 gateway address. Confused...

what does "traceroute6 -n www.kame.net" say?

$ traceroute6 -n www.kame.net

traceroute6: Warning: mango.itojun.org has multiple addresses; using 2001:2f0:0:8800:226:2dff:fe0b:4311

connect: No route to host

 

any fw active?

also what says route -n6 get 2001:2f0:0:8800:226:2dff:fe0b:4311

$ route -n6 get 2001:2f0:0:8800:226:2dff:fe0b:4311

route: route has not been found

can you paste the output of netstat -6nr

no firewall

too long, but there have 2 ipv6 global address with "link#4 U/UHS em3"

use a paste service like bsd.to

Title: dpaste

I have a server with one SSD and 40x hdds configured with zfs in 20 mirrored vdevs

when I do rsync from/to the SSD, I can hit at most 400MB/sec

the hdds are on sas3 backplane rated @ 12gbps

shouldn't I be seeing close to 1GB/sec or more ?

rsync with what options?

ah it's a local copy, so just rsync --info=progress2 src dst

to destination files that don't already exist?

it's a single 38Gb file

which doesn't exist yes

25,682,083,840 66% 416.80MB/s 0:00:30

ok, and if you run gstat -p while it's running, what do the busy percentages look like?

ah, hang on a second

SSD limit is 400MB/sec lol

not sure why I was expecting more

well there you go

however, I also have a zfs ssd pool made up of 5 x 2 mirrored vdevs

and there it's also 400MB/sec to/from the 20 x hdd vdevs

ok, and gstat -p shows what?

bursts of 11% busy on all drives, but it's not constant

Title: dpaste/Qwic (Plain Text)

Title: View paste MFFWS

this is better

the drives don't seem to be all that busy

I tried disabling the sync on the target zfs pool, didn't make any difference

now do top -CSH -s10 -c2 (it'll sleep 10 secs and redisplay and exit, paste the final output)

top: invalid option -- c

-d2 sorry

interestingly enough I'm see more activity on the ssd pool than on hdd pool

Title: View paste B7I54

smaller pool will generate proportionally more activity for a given load

this is the top output: bpa.st/NCWEE

Title: View paste NCWEE

er

true, but I was expecting 5 x 400Mb = 2GB/sec output for this ssd pool

that's a lot of cpus, can you do it on a larger screen?

or maybe add -I to the flags

no, that won't help

want to see system threads in the output, but the per-cpu idle threads are getting in the way

Why could "idle" threads not be filtered out?

What controller is driving these devices? Might it be the bandwidth limit?

ah, -z is the flag needed

It's a standard LSI 9308-8i + 9308-8e HBAs

the 8i drives the internal SSD pool, the 8e drives the external HDD expansion enclosure

Never mind

last1: top -CSHz -s10 -d2

it's all supposed to be 12gbps

Should be good then. It's good that you are looking into this performance side.

Title: View paste C5HVS

rysnc busy raking the disks

haha

the cpus are CPU: Intel(R) Xeon(R) CPU E5-4667 v4 @ 2.20GHz , 18 cores each, 36 for both, I'm guessing with HT it shows as 72 total cores

I am not a fan of HT and have benchmarked tasks where things are faster without HT than with it. But not file system tasks. So, don't know here.

rsync is using enough cpu that it might be the limiting factor, depending on the exact pattern of system calls it's doing

not in terms of total cpu usage obvious;y, since the box is >96% idle, but in terms of single thread usage

interesting

so if I start two in paralell it might show total transfer of 800MB/sec but it will be bound on different cpus

and then I can tell if the limit is IO or cpu

might be interesting to try a few completely independent rsyncs in parallel to see what the aggregate i/o is like

cool, let me try that

kevans: also I was especially peeved since this is the second time for lua 5.4 alone, though the first time I was keeping up enough to catch the problem in a release candidate and argue them out of actually releasing an ABI break

this time I was just too late :-(

Title: View paste JOXQS

yep, it seems the CPU is the limiting factor

two in parallel I hit 800MB/sec

well, either CPU or rsync single thread performance

last1, Another thing to try is --while-file option (& --sockopts)

rsync isn't necessarily going to be the fastest tool for this, since it's written for a much more general use-case

s/--while-file/--whole-file/

parv: when the destination file doesn't already exist that should make no difference

since it'll do whole file mode then anyway

Right

so what's the fastest way to copy a file then ?

weirdly enough, from another server I can hit 500MB/sec using scp

with: -c aes256-gcm⊙oc

no I can't

Replace all the disks? I read about "Infiniband"

ha, weird stuff, check this out

(replacing the disks in terms of moving a set of disks from one enclosure to another)

from backup server, I start a copy from remote, I get 300MB/sec using scp

from the remote server I start a copy to backup server, also using scp , I get 500MB/sec

why would it matter where I initiate the transfer frmo

*from

So rsync is good for *incremental* transfer. And also for plain copying. But for plain copying just plain cp is also okay.

it may affect the distribution of encryption vs. read/write workload between threads

I might try a test using "pv" (a utility I like with nice progress indication) and then try a test without rsync.

I'm not sure how scp divides things up on the initiation end, but on the destination end I think it'll end up using separate processes for decrypt and write

RhodiumToad: that may indeed be the case, the source server has a more powerful cpu

last1, When you would chance, also try a "zfs send | mbuffer | zfs recv"

last1: again, if you investigate using top -CSHz you may be able to tell whether certain specific threads are a possible bottleneck

.oO( Heh that works even without "have" :-)

I guess what's needed is something like torrents

split the file automatically in x chunks and transfer those in parallel

RhodiumToad: oh yeah, totally get it... and roberto's responses have left a lot to be desired

very "no one cares" vibes, when it's the promise they made. you can't just de-weight some APIs like that because it's convenient

bad wording, some parts of the API

silly question, when copying, even with cp a file from a different pool, why does it show 99% cpu usage ?

isn't copying mostly IO ? what's the cpu doing ?

often the answer to that is "figuring out what to write where"

at least if the device has no DMA, then processor needs to handle transfer of each byte in I/O

as the raw i/o becomes faster, the software overheads become more of an issue

well, cp is about 3x faster than rsync

completes in 27 seconds

so roughly 1GB/sec I think

yeah that makes sense

I think in recent versions, cp uses copy_file_range

which may bypass some of the CPU cost of doing the obvious read/write loops

on ZFS, it is also dependent on fragmentation

yeah, and iirc rsync always forks and execs an rsync receiver to copy to, even for a local copy; can't really optimize it in that way

er, forks

would the zfs compression show up under the cp cpu usage ?

I don't think so, I'd expect to see that in a zfs thread

it is more disk iops

but my zfs knowledge is weak

check it with gstat

nerozero: last1's case involves a lot of disks and a high-speed controller

see disk "busy" status and ops/s

yeah, we already determine the issue is the cpu

RhodiumToad, sorry, just joined dont see previous conversation

maybe a faster cpu can do faster copes than 1GB/sec

*copies

ok

what is the CPU specs then ?

Intel(R) Xeon(R) CPU E5-4667 v4 @ 2.20GHz

yeah

thing is, 36 cores is a lot of aggregate CPU power, but you're still limited in how much speed you can deliver to a single thread

so I'm hitting a pretty basic limit here, I can't imagine even with more powerful CPUs hitting more than 2-3GB/sec

probably not when you're bottlenecked by 1 thread, no

THere is 18 cores, the rest is hyperthreading which performance is iffy

but still it is alot

2 packages, I think?

it's 2 cpus, with HT it's 72 'cores'

and yeah, HT won't be helping at all in this context

hard to tell whether it's just not helping or whether it's hurting at all... I would guess it probably isn't hurting, but that'll depend on other factors

still thiking this is an issue of disk iops, in case of zfs raid, had same issue after having a lot of volume type zfs entries

we ran two rsyncs in parallel, got twice the speed

it's really a single thread speed issue

nerozero: we got gstat -p output before, the disks are a long way from being saturated on ipos

*iops

so I wonder, how do big data companies transfer giant files that are like 100-200Tb

if it's sequential it will take forever

by switching hard drives :D

heh, true

alas, it's been a good many years since I worked with anything considered "big"

like mirror pool - take half, u got a second copy :D

;->

how do you know which to take though ?

it is mirror, dont care :D

this time I'm taking photographs of each disk's serial # before putting it into its final bay for production

Mirror!

yeah, but the order they give you after isn't related to their position

though then I often had to transfer significant data volumes between San Jose and Amsterdam, and part of how that was done involved avoiding having single huge files and instead using larger numbers of moderate size files

Title: View paste NFCGC

^^ now that is impressive

cheap mans backup server. 280Tb total capacity

at this many drives the RAID controller itself can cause a performance issues, bus speed limitations

My experience limited by 10 drives max so... I'm a bad advisor here :(

the raid controller is fine, it can do 12gbps

it's an hba actually

ah wait, another click, 12gbps = 1.5GB/sec which is pretty close to cp's speed

possible

plus some overhead

guess I have to look @ 24Gbps sas

Imho moving data by replacing a part of the drives by bringing them offline is still a solution ...

this is an ongoing backup, I can't shuffle physical disks 10 times per day

but I'm happy with my answers

3 way mirror - not loosing redundancy

rsync is cpu limited, cp is io limited

thank you everyone for your inputs & help

good luck !!!

In regular RAID 5, I know that drives can be hot swapped in RAID 5, which means a failed HDD can be removed and replaced without downtime. Is this fact valid for also ZFS/raidz1?

as RAIDZ1 = RAID 5

Is hot swapping possible with ZFS and raidz1? If so, would simply replacing the failed disk with the new one, be enough? Or some set of commands is needed?

21.2.3

Title: Chapter 21. The Z File System (ZFS) | FreeBSD Documentation Portal

tercaL: ^

hotswapping needs zfsd(8), that you enable the autoreplace zpool property, and that the disks have an identifiable enclosure path

depending on what you mean by hotswapping, i gues

RAIDZ is roughly RAID5

debdrup: In cases like, getting a drive rerror, shutting the server down, replacing the disk and re-starting the OS.

if you mean the actual electrical hotswapping, that needs a backplane with capasitors to handle the inrush current

*a drive error

tercaL: it's not just a question of software, though

Hmmm

electrically, there's a risk of arcing due to inrush when hotswapping

that's why you need a capasitor

as far as software goes, you can use zpool offline and zpool online to remove and re-add a disk

that's what zfsd can do automatically, using enclosure identifiers

as an example, in my SAS enclosure, if i pull a disk, and insert a new disk (usually bigger, but at least the same number of LBAs of the same size), my zpool will automatically begin the resilver process

this isn't really a good idea if you're booting from the pool though, because boot pools require partitioning

but yes, you can probably do what you want, it's just a question of how

if you can schedule downtime, that's absolutely the best idea, unless you're working with a backplane that's made for drive hotswapping

but zfs itself doesn't really care, so long as there's sufficient blocks, which is a question of array layout

auto resilver also will not work when you use geli encryption

true

but yeah, I have replaced 6 out of my 8 disks in the last year, without taking down my nas

using zfsd with autoreplace and autoexpand enabled does feel like magic, though

yeah, I should move to zfs encryption instead of geli on my home nas

i wouldn't.

Title: OpenZFS open encryption bugs (public RO) - Google Sheets

ok, maybe I should stick with geli

thanks :)

debdrup: Very strong and super cool info, thanks. Final question, when I created a pool with: zpool create tank raidz1 da0 da1 da2 da3 da4 da5, the pool was created but the console had errors output like; GEOM da0: the primary gpt table is corrupt or invalid. And it goes the same messages for all da1, da2.. disks. Should I worry about this one?

and it continues: "using the secondary instead -- recovery strongly advised"

This helped: forums.freebsd.org/threads/corrupt-gpt-table.77585/post-483348

Title: Solved - Corrupt GPT Table | The FreeBSD Forums

tercaL, Since you have six drives I recommend raidz2 (effectively raid6) instead of raidz1 (raid5) as it reacts better in the presence of fault conditions.

Speaking of 13.1-RELEASE to 13.2-RELEASE, would you sooner update a base jail so that its respective thin jails update "automatically," or would you rather bootstrap a fresh base jail and edit the `fstab`s of the thin jails to mount the filesystem from the fresh base jail?

With constrained resources on a VPS, the latter is easier for me. It has worked thus far

I don't use base jails because storage space isn't the premium it used to be.

Base jails and thin jails, rather.

True. I appreciate the response. It's much to better to hear that than "the two are not equivalent and you are asking for trouble."

There's another option, too.

VPSes tend to be overprovisioned (YMMV, i suppose). My issue is compute. I forgot to reboot prior to upgrading last night, and it took forever. Rebooting tends to get cores with much more headroom initially

Another option?

If you set it up so that third-party software configuration is handled via unionfs(5), instead of updating each jail sequentially, you can install a new jail, zfs snapshot then zfs clone that snapshot into a new dataset that's used for yet another jail where you install third-party software into, then mount the configuration files into the jail.

That way, you're only spending diskspace for the third-party software, as zfs snapshot+clone will cause all the unmodified data to simply exist as referenced data.

It needs ZFS to work, and is fairly labour-intensive over simply running freebsd-update, but saves a couple of GB (or however much diskspace the base system takes up).

I suspect some/most of it could also be done via poudriere-image(8) using the zfs snapshot+boot environment feature, but I haven't looked into that yet to make sure.

It also requires careful control over the jails, so maybe it's more effort than is worth it.

It's mostly just something I've been thinking of playing around with, if I ever get the time/energy.

As far as tinkering goes, I like the idea. Skinning a cat in a new way helps to either cement prior knowledge or realize shortcomings. But, yeah, I think I lack the time (and need) for this type of setup.

Not sure I'm comfortable with that phrasing.

There's more than one way to knit a sweater...

Hi, if I allowlisted a dynamic IP via a dns name that's auto-updated will pf still allow the new IP in?

like does it resolve my dns often?

Nameserver lookups should never be part of firewall configuration.

That risks a catch-22.

ok, what would be the best way to whitelist a dynamic IP that has a dns entry?

(I'm new to freebsd)

o/

how can I find CPU and / or motherboard temperature ?

the only thing so far I have found is: sysctl hw.acpi.thermal.tz0.temperature

ha ... kldload coretemp, then sysctl -a | grep -ie temp

esselfe: ip range with cidr notation

do a whois on your ip to find out what range it's in, then figure out the vlsm

ok, thanks

how can I install this? man.freebsd.org/cgi/man.cgi?query=a…path=FreeBSD+13.2-RELEASE+and+Ports

Title: ag(1)

the "ag" command is not available on the system ...

freshports doesn't seems to have listed it too

found it under textproc/the_silver_searcher

Title: FreshPorts -- textproc/the_silver_searcher: Code-searching tool similar to ack but faster

dvl :)

thanks

^ ignore the branch, I blindly copy/pasted after searching for Silver Searcher

i'd be cursious how ag performs compared to ripgrep

otis need it for vim

they are all compatible.

Hej! Might anyone have ideas for this boot issue on a Ryzen 7 Pro-based ThinkPad? bugs.freebsd.org/bugzilla/show_bug.cgi?id=270707

Title: 270707 – Installer media doesn't boot on Thinkpad T14s Gen 3 (Ryzen 7 Pro 6850U)

Hi VimDiesel! I've missed you!

<3 <3

heh

VimDiesel, you tried with ACPI disabled?

CrtxReavr: you mean michaeldexter

Um. . .. yeah.

whoops

I remember a painful period in the transition from PS/2 to USB devices. . . had to side-load a kernel with fdc(4) support removed.

ugh, ok the root issue of scli not working wasn't my jail or freebsd, it's that signal-cli is a year out of date and the ssl certs expired. signal is such a pita.

michaeldexter: try: set debug.acpi.disabled="acad cmbat" from the loader prompt

Trying that now... Thanks.

yuripv: That now stops earlier at psm0: model Generic PS/2 mouse, device ID 0

what if only cmbat in that line?

yuripv: That now stops at the next message, acpi_acad0...

yuripv: I tried including only "acad" but the same result.

yuripv: One type of of the space bar gives: AcpiOsExecute: task queue not started

usually when boot hangs it's not the fault of whatever did the last message, but rather the thing _after_ that, which of course you never see

hrmmm

do we have a fixed boot oder?

no

well

maybe try disabling all the acpi modules, or just enabling them one at a time?

Trying that...

there's also a debug.acpi.avoid option to tell it not to try and parse parts of the acpi data, but figuring that out would probably require getting a dump of the acpi data from another OS

RhodiumToad: set hint.acpi.0.disabled="1" gives a panic: APIC: Could not find any APICs.

Has anyone had luck with getting a FreeBSD working on proxmox? With all the defaults, I can only acheieve a maximum screen resolution of 1280x1024. Wondering if anyone has had success with getting 1920x1080

*** getting a FreeBSD DESKTOP working on proxmox

michaeldexter: debug.acpi.disabled="all"

RhodiumToad thank you we got some progress! :D

RhodiumToad: Indeed! Now I get mountroot> with no disks, but I investigate further. Thank you both!

maybe try disabling "acad button cmbat cpu ec lid mwait quirks thermal timer video" and if you get anywhere with that, try selectively removing those until it breaks

Will do!

RhodiumToad: That stops at whatever is after psm0

But thank you!

michaeldexter: there are also debug output options in acpi(4)

could try that with ACPI_ALL_DRIVERS and ACPI_LV_ALL_EXCEPTIONS

I'll try that yuripv

yuripv: No luck with ACPI_ALL_DRIVERS and ACPI_LV_ALL_EXCEPTIONS and them individually, plus a few more from the manual page, but huge thanks!

michaeldexter: it does not print any additional information?

looks like ACPI_DEBUG is needed in kernel config for that..

or there's also debug.acpi.enable_debug_objects (try setting this to 1 with previous debug settings?)

yuripv: No, that did not provide any more debug information. I'll try debug.acpi.enable_debug_object

Same behavior.

michaeldexter: add sysresource to the list I suggested

michaeldexter: if that doesn't help, try isa

the problem is that you likely need bus, children, pci and pci_link to get at important hardware

probably need isa too for that matter, but try it

Will do.

Just to confirm RhodiumToad: At the loader prompt: set debug.acpi.disabled="acad button cmbat cpu ec lid mwait quirks thermal timer video sysresource isa"

Same behavior

I guess I can build opions ACPI_DEBUG into a kernel.