06:57:40 <razamatan> i have a bash alias that mounts and encrypted (loopback) file.  this worked fine until i upgraded my vm instance from base-64-lts 21.4.1 to 23.4.0.  now, it doesn't bother waiting for me to enter the encryption password and just fails to attach the loopback.  the alias is this: alias myMount "mount \$(lofiadm -a /enc/file/path -c aes-256-cbc) /mnt/path"
07:06:51 <razamatan> if i avoid nesting the commands, the alias works as expected and waits for password input on the lofiadm command: lalias workingMount "lofiadm -c aes-256-cbc -a /enc/file/path /dev/lofi/2 && mount /dev/lofi/2 /mnt/path"
07:07:13 <razamatan> unfortunately, the working way doesn't play nice if something else is using /dev/lofi/2
08:58:51 <jperkin> lofiadm comes from the platform rather than the image so it's hard to see how that would affect things here
08:59:11 <jperkin> I'd have a look at any surrounding infrastructure that may have changed at the same time
09:04:38 <razamatan> looking at lofiadm source in the illumos, nothing seems to have changed except for the Makefile in the last few years: https://github.com/illumos/illumos-gate/tree/master/usr/src/cmd/lofiadm
09:05:09 <razamatan> i verified that bash works fine w/ nested tty read blocking with other commands...
09:19:36 <razamatan> ok.. i just spun up the old vm instance and it exhibited the correct behavior.. weird
09:27:48 <razamatan> bash, mount and lofiadm are all the same binaries provided by the platform which is constant between the two vm instances
09:32:59 <razamatan> would termcap differences cause this?
09:40:42 <razamatan> ok.. looking at the termcap info using infocmp, there is a difference in the rin value... the old value (that causes the original nested lofiadm to work) is just `rin@` whereas the new vm value is `rin=\E[%p1%dT` which doesn't work.  off to learn about termcaps....
15:04:03 <rmustacc> razamatan: So I wouldn't dig into termcap first for this.
15:05:05 <rmustacc> There are a bunch of changes in the kernel lofi driver. So can you share the two platforms you changed between?
15:20:12 <danmcd> Also... is your bash coming from platform (/bin/bash) or pkgsrc (/opt/local/bin/bash)?
15:20:18 <danmcd> I ask because of the failure mode...
15:20:20 <danmcd> "now, it doesn't bother waiting for me to enter the encryption password and just fails to attach the loopback."
15:20:43 <danmcd> This seems like a can't-reach-the-tty problem.
15:21:59 <danmcd> Platform bash won't change unless you change the platform, which I can't tell if you did that or not, given you also said you updated the zone's image from "base-64-lts 21.4.1 to 23.4.0."
15:22:35 <danmcd> Knowing if you changed platforms as part of the pkgsrc base is helpful, so if you did, also let us know the platform versions.
15:25:04 <danmcd> If you use pkgsrc bash, it COULD be something there related to $(SHELL-COMMAND) processing that changed
20:14:19 <razamatan> danmcd: from platfrom?  it's /usr/bin/bash in both cases
20:15:01 <danmcd> Yeah, so that's platform bash alright.  Did you change platforms when you updated your zone's pkgsrc base?
20:15:04 <razamatan> danmcd: i am actually running a single platform image (joyent_20240125T000404Z) with the old and new vm that can repro this problem
20:15:18 <danmcd> Ahhh.   Interesting.
20:15:22 <razamatan> very
20:15:35 <razamatan> set -x bash output is the same flow
20:15:57 <danmcd> So the old pkgsrc image on this same PI  works, but the new pkgsrc image does not?!?
20:16:05 <razamatan> correct
20:16:42 <danmcd> These running concurrently on the same physical machine?
20:16:51 <razamatan> binaries wise, they're all the same, afaiu.. to me, it seems there's something in the vm images that's causing the tty read to fail in the new vm
20:16:54 <razamatan> yes
20:16:57 <danmcd> Damn....
20:19:20 <razamatan> rmustacc: yeah.. the termcap diff was a red herring
20:19:53 <danmcd> So tell me, what does "pldd `pgrep lofiadm`" say (run it in the same zone) while your lofiadm is asking for  a password on the one that works?
20:19:56 <danmcd> ALSO:
20:20:15 <danmcd> what does "vmadm get $VM | grep fs_allowed" say for both zones?
20:21:08 <razamatan> danmcd: fs_allowed is the same between them:   "fs_allowed": "ufs,pcfs,tmpfs",
20:21:16 <danmcd> Had to check.
20:21:33 <danmcd> pldd?  Does it have anything in /opt being linked in?  I doubt it, but I have to ask.
20:22:30 <razamatan> non-working new zone pldd: 38341:  lofiadm -c aes-256-cbc -a /zshares/archive/.private /dev/lofi/10
20:22:32 <razamatan> /usr/sbin/lofiadm
20:22:34 <razamatan> /lib/libc.so.1
20:22:36 <razamatan> /usr/lib/libpkcs11.so.1
20:22:38 <razamatan> /lib/libcryptoutil.so.1
20:22:40 <razamatan> /usr/lib/security/pkcs11_softtoken.so.1
20:22:42 <razamatan> /usr/lib/libsoftcrypto.so.1
20:22:44 <razamatan> /lib/libavl.so.1
20:22:46 <razamatan> /lib/libmd.so.1
20:22:48 <razamatan> /lib/libgen.so.1
20:24:43 <razamatan> working old zone pldd is the same output for the paths.  only difference is the pid (39409 instead of 38341)
20:25:44 <razamatan> the uuid of the two base images is: 85d0f826-0131-11ed-973d-2bfeef68011c and 8adac45a-aca7-11ee-b53e-00151714048c
20:41:21 <danmcd> Wait... both ask for the password?  I thoguht you said the new one doesn't?
20:41:44 <razamatan> i had to use the workaround approach (non-nested &&) to capture the pgrep lofiadm
20:41:51 <danmcd> Oh.
20:41:52 <razamatan> otherwise, it dies too fast
20:42:26 <danmcd> Yeah... the workaround works... WHY it works is the question.  CLearly it's not some /opt library lofiadm is bringing in for some bizarre reason.
20:42:57 <danmcd> Is your privilege set the same in both zones?
20:43:00 <razamatan> yes
20:44:34 <danmcd> So literaly `ppriv $$` on invoking shells on both zones show the same thing.
20:44:40 <danmcd> Damn.
20:45:16 <razamatan> danmcd: outside of the bash pid being different, the diffs show nothing else
20:45:20 <danmcd> Damn.
20:46:46 <danmcd> So if you utter "lofiadm -c aes-256-cbc -a /enc/file/path" all by itself on a CLI you fail on the 23.4 zone?
20:46:59 <danmcd> (Trying to minimize the reproduction...)
20:47:24 <razamatan> no.. it only fails when i nest it inside of a mount: mount $(lofiadm -c ... -a ...) /mnt/path
20:47:33 <danmcd> Ahhhh so
20:47:41 <danmcd> echo $(lofiadm -c....)
20:47:50 <danmcd> succeeds on the 21.4, fails on 23.5 ?
20:47:54 <danmcd> 23.4 I mean ?
20:48:09 <razamatan> doing it by itself or doing the workaround of serializing the lofiadm and mount works fine
20:48:12 <razamatan> i'll do the echos
20:49:12 <razamatan> yeah.. echo repros
20:51:00 <razamatan> actually.. this may be due to sudo
20:51:05 <danmcd> WHA?
20:51:07 <danmcd> You're using sudo?
20:51:14 <danmcd> hold on...
20:51:23 <danmcd> I saw no mention of sudo.
20:51:25 <razamatan> yes.. i omitted it b/c i thought it couldn't possibley repro 
20:51:28 <razamatan> this is my mistake
20:51:41 <razamatan> 1000x apologies
20:51:55 <danmcd> Oh no worries.  I'm glad I could help narrow it down.
20:51:57 <razamatan> so the main delta in terms of binaries is sudo which does live in the vm image
20:52:04 <razamatan> really, thank you
20:52:09 <danmcd> YW, glad to help.
20:52:27 <danmcd> Gonna go install a USB-key now downstairs.  I have to bring this NUC back online after a weekend of failed experiments.
20:52:37 <razamatan> i'll have to figure out why sudo is so bad... visudo in the new vm image doesn't seem to handle the term correctly...
21:04:26 <razamatan> gah.. the fix is to disable use_pty in sudo
21:06:38 <razamatan> the just enabled it by default in the last few months.  sorry for the noise folks, and thanks again for the help danmcd.  also, thanks in general for a great os!
21:43:08 <danmcd> YW, and don't forget props to jperkin and rmustacc as well.
22:01:58 <razamatan> yep