Hello sir's & sirette's!

I got a Q regarding SmartOS

Hence being a type-1 hypervisor, can you set it up to protect flashable mcu's/soc's from writing w/ tools like 'flashrom' etc.? does it do that out of the box, or how do you configure that? Looked thru documentation & can't find this either there or on websearches. I know Secureboot disallows it,though it can be overridden.

Also got another question regarding the LTS version (or any version), have any undergone code-auditation?

And third question i got,say i install a Linux-dist on top of Smartos where i want the virtualized O/S getting networking data "first" say thru a stateful,DPI firewall, can that be achieved? (All questions is to be understood from AMD-architecture)

And a smaller Q, can i by configuration restrict access to ioctl, ionice cmd's & syscalls or do i need like syzkaller etc. for that? What are recommended here?

zt3phan: The website has an example of a NAT zone which is a smartOS image. This can be used to handle outward-facing networking for all VMs/zones. I have done the exact same thing, but, with openBSD in a bhyve VM, instead of illumos.

duncan: thx. is it in the documentation or external site?

thx mate

sorry my screen died, did anyone happen to answer any above q's except duncan (much appreciated!)?

(while i was discon'ed ofc)

danmcd, are there any updates regarding the support for Broadcom 3816? ...I've seen the thread on -dev but haven't noticed any reference to stable support in the changelogs... am I mistaken?

There are none as of yet.

The patches are available, but I think it needs real testing.

Someone from nexenta dropped the patches a while ago on illumos-dev. I made some small mods to it, and it probably needs to be filed, put in gerrit, etc.

(Someone might've filed an illumos bug already?)

geez these timeouts..finally reg'ed to nickserv

:( I need to purchase a Dell server as it's the only option for the HBA. I need to replace a problematic Supermicro server.

zt3phan: as far as things like flashing and hardware interface, those operations can be done from the administrative plane only (i.e., the global zone). Tenant instances are not permitted to interface with real hardware.

oops, too late.

xmerlin: Is this standalone smartos? Or a Triton node (and Compute or Head node if Triton?)?

ty bahamat, much appreciated

danmcd, standalone smartos

I'll spin a 20240111 version with the 38xx in it. If you notice, there are Oct 2023 PIs in kebe.com/~danmcd/webrevs/mpt_sas-38xx

I'll update with 20240111 PIs.

danmcd, Thank you, when I receive the server, I will test the image

Looks like I'd built PI, ISO, and USB; I'll do the same here. Expect something in <= 3 hours. (And expect the timestamp to be 20240118, but it's just 20240111 + 38xx support.)

When do you think that'll arrive xmerlin? I ask because I'm tempted to land this support into SmartOS now, but I don't have 38xx HW to test it with.

(IOW, your test results would be vastly appreciated.)

danmcd, I'll let you know as soon as I get a confirmed delivery date

This is a Dell... is it the HBA-only 38xx? Or is the MegaRaid, which might be supported by `lmrc` ?

(Actually, use the one I spin; lmrc is in -gate, but 38xx for mpt_sas isn't; you win either way.)

danmcd, hba only HBA355i

-> dl.dell.com/topicspdf/hba355_ug_en-us.pdf page 9

first column

What you recommend for hardening the SmartOS?

@xmerlin --> uploaded. Look for platform-20240118T163954Z*

I checked out devsec's ansible roles, seems good. Any other tip's?

zt3phan: Hardening against what exactly? I saw you mention FW flashes. Also, what's you're threat environment? Adversarial users on a cloud environment? Public-Internet exposure?

(And I'll note that at some spaces I won't have enough knowledge to answer your questions.)

i check the server-logs in between not to miss anything. Public inet, phones gettin' hacked etc. the latest year.

One thing I never could work out was why rpcbind runs in the global zone ;)

danmcd, thank you

Still wondering :(

i checked out galaxy.ansible.com/ui/repo/publishe…hardening/content/role/os_hardening

It seems to be an Illumos thing because it's enabled by default on OI and OmniOS as well

dancmd: yes the common stuff this hackergroup do is flash i2c/SPI mcu's (or what else they can flash), and put microkernels w/ FreeRTOS variant with included malware-kit spreading like wildfire. I transparent-proxied and found their dumpsites etc, really competent .ch based group.

So i'm looking to defend in all possible ways.

*sigh* timeouts,timeouts..

I think bahamat stated it well; non-global zones (NGZs) should be safe against hardware-interface attacks as NGZs can't access hardware without explicit hard-to-do configuration.

sounds great.

zt3phan: I think you missed what I said earlier, so i'll repost it:

zt3phan: as far as things like flashing and hardware interface, those operations can be done from the administrative plane only (i.e., the global zone). Tenant instances are not permitted to interface with real hardware.

If your attacker has access to the global zone (GZ), it's either bad configuration or an NGZ-escape. NGZ-escapes are high-priority bugs.

Yes ^^^

zt3phan: SmartOS doesn't generally need additional hardening, because it's hardened by default.

okay, so i should avoid devsec os-hardening?

In fact, when Joyent was purchased by Samsung, they sent in their SecOps folks to provide recommended hardening guidelines, and we already met or exceeded every measure.

i c :D

has there been a code-audit at any version by a third-party?

I'm happy to discuss SmartOS's security and threat model, and we're always open to constructively examining potential improvements.

No. But it's 100% open source and all code is available for audit.

and another thing, even though i plan to run on AMD-SEV im wondering if SmartOS encrypts itself to memory?

We even have an open Jenkins instance where builds can be observed by 3rd parties.

No, it doesn't encrypt itself to memory. Doing so is effectively impossible. At the end of the day, there must be unencrypted runtime data in memory in order to perform compute on it.

allright, yes i saw that, thou like Zephyr-RTOS 2.7.x LTS are code-audited officially.

But we do support encrypted zpools. It's intended for use in the context of Triton, but it can still be done on SmartOS standalone if you add some additional components to it.

just small wondering of mine.

E.g., the decryption key source is intended to be kbmapi that runs on the headnode. For standalone you need to provide your own key source.

Any operating system that claims to be encrypted in memory is either playing word games or is outright lying.

So, in regards on the kernel, does it use a shadowstack or?

No, we use a different mechanism called stack smashing protection.

i see.

It's much more efficient than shadowstack.

They're diffrent mechanisms actually.

The shadowstack is usually part of CFI protections in the CPU.

That's pretty orthogonal to stack smashing protection tbh.

Yeah, but I mean that both are intended to mitigate buffer overflows.

We don't use shadowstack to prevent buffer overflow, we use SSP, which is entirely different, but is still buffer overflow protection.

Yeah, but we do want to have both some day.

(in illumos at least)

Mhm yeah..do you adopt any own "grsec"-stuff, but of your own? I know they really invented much of the stuff used today, it's standardized & the other non-free, so what im asking is if you wrote any own of it, besides rbac?

Well, not being Linux we can't use grsec. And we do have several security features. It's best to discuss individual threats and mitigates rather than lump everything together by saying grsec-like.

What is considered grsec-like will vary greatly from person to person.

PaX?

kernel lockdown?

I think so.

kernel lockdown no, but tenant instances don't have access to anything that would need locking down.

ah yeah, i suppose. gotta ask since i never used bhyve - what are the pros/cons vs kvm?

BHYVE has a lot more modern support and active development.

bhyve is an all original vmm created by FreeBSD.

When comparing FreeBSD vs Linux KVM, there may be various points made by each side.

For illumos/SmartOS bhyve is a better choice for us.

^^^

in security context, what would you say?

KVM is GPL, which there are differences of opinion on how relevant that is.

At first glance, no differences, but I will say that since the KVM is an older fork and BHYVE is an ongoing care-and-maintenance project, you're less likely to find old bugs in BHYVE.

And when I say "KVM" I mean "the KVM used in SmartOS and other illumos distros."

We have a better working relationship with FreeBSD, so changes from bhyve made by us or them flow pretty freely, whereas we had a lot of difficulty upstreaming KVM changes.

KVM in Linux is under development too, but none of that was pulled into the illumos kvm.

And @bahamat nails the other reason.

On SmartOS, both KVM and Bhyve run inside a zone with stripped down permission.

Yes, the "double-hulled" virtualization.

So even if there is say, a vm breakout vulnerability in kvm or bhyve, the place you broke into is actually a smaller, more secure prison.

Even if BHYVE or KVM is escaped, you then have to escape a non-global zone if you want to do real damage.

kvm/bhyve brand zones are restricted from doing things like forking, reading files, listening on the network, sending packets, etc.

To truly break out of an HVM on SmartOS you first need to escape the vmm, then defeat the capabilities restrictions in the zone brand, then have a zone escape.

i understand.

Also, a guest running in HVM is free to implement their own security protocols, which could include installing grsec, adding another entire layer of mitigations that need to be defeated.

It would probably be pretty easy for operators to use our image creation process and add on grsec for every image, so that your own tenants always have it installed by default.

If i setup a zone w/o any specified rules, do the root user on guest vm still able to do ioctl for instance?

bahamut, ofc i understand that.

So, technically yes, but that doesn't allow them to do just anything.

There are a lot of things that use ioctl to perform various permitted tasks.

But root in a non-global zone can't use ioctl to do anything malicious to the global zone.

right right, but by advanced evasion techniques, do you know any case a breach have been done thouroughly?

There have been zone breakout issues in the past. There are currently no known methods for escaping a zone.

okay.

gotta check cve's i guess.

If such an issue were discovered or reported, it would receive the highest priority for fixing it.

sounds good to me, notice you update very frequent.

Yeah, we have biweekly releases, and if there are security critical issues we will publish updates out of cycle as needed.

Our advisories are published here: security.tritondatacenter.com

have you ever thought of the possibility to run S-Os on like an SoC or from disk?

And we send announcements to smartos-discuss@ or sdc-discuss@ as necesary.

thank you,bahamut

Running on an SoC would depend on the architecture of the device.

If there were an x86_64 SoC system and we had driver support for everything, then it should work no problem.

We don't currently run on ARM or RISC-V.

like FPGA for AMD?

There's some work going on to run on ARM, but even once that's incorporated into illumos there is some upstack stuff we'd need to do to make sure that SmartOS works well in that context.

that requires ofc a microkernel and diff install-method

Well illumos isn't a microkernel.

And there isn't really an install method. You just boot the platform image.

but that could be done via kconfig'ing and only have the platform-specific's,right?

"install"

Where that image is stored can vary. It can even be stored on locally attached disks, but that's not really the same as "installing" the OS like you would with Linux, *BSD, or even OmniOS.

something like that

FYI, we're having our office hours on Discord, so my replies may not be as prompt until that's over.

if you look at the approach Zephyr-rtos takes, they offer support to ESP etc, dunno if you looked at their project.

(?)

If you're interested, the discord is discord.gg/rgkeBVRw

I haven't particularly looked at zephyr, though I'm aware of the project.

bahamut, that's good to know. i can't get into discord on the device im on.

No worries. I felt it might be rude to mention that I was participating in a public forum elsewehere, but not provide the link :-)

they use platform-specfic builds to the kernel upon "installation", thou it is'nt really 'easy' to use.

Oh, no I don't think we'd use something like that.

Our infrastructure makes the running system auditable because it can be cross referenced against known approved images.

that's where SmartOS actually got my attention, plus the image-type system looks great.

The boot image hash is verified, and it includes a manifest of all objects and their hashes shipped in the platform image.

and i need some more knowledge to illumos also, all bsd stuff always been rigid

We don't have explicit tooling for "please verify the running system", but a knowledgable operator can perform those checks manually.

can you use TPM directly from the SmartOS?

tpm2.0*

didnt see it in pkgs

no

there currently isn't a driver

or the tpm driver currently only supports the older tpm1.2 devices

really?

How do i deal w/ that, from guest-vm?

you'd never really want to pass through a HW tpm though

for secure boots,yes? or why not?

because the state of the hw tpm isn't going to be correct for secure boot

upstream bhyve added some hooks for supporting virtual tpms, not sure if they've made it back yet

(then would need to do a bit of wiring up)

if you sign the illumos-kernel?

would'nt that be wanted?

or what you mean state incorrect?

still.. the problem is secure boot is kinda like a blockchain.. order matters

each bit hashes the next thing

so each step compares the values to known values to determine if things are 'ok'

i''m not 100% on the tpm, but verifying the kernel & then firmware..? or do you mean the boot-process of a usb?

Verified boot systems typically have a root of trust, TPM, which signs the rest. Passing through the TPM to a VM wouldn't guarantee anything about the VM, and might have implications for the host

E.g. on Chromebooks, the Google TPM (firmware of which is OSS BTW) signs vboot stuff (it's a coreboot thing), the coreboot payload (depthcharge bootloader) then uses things like dm-verity to verify filesystem from which the system is to be booted

The other main use of a TPM is to store secrets, you probably don't want these to be shared between the host and VMs

the other wrinkle I suppose is that hardware TPMs usually live behind a PCI-LPC bridge (more or less an ISA bus) which bhyve doesn't have pass-thru support unless I suppose you did the entire bus

which would probably grab several other things on the system

so the booted vm still is completely isolated by the illumos kernel, then protection of that should only be of interest, because the vm's could be compromised and thus needing to be scrapped, main importance the illumos-kernel is intact? but i suppose your entirely right as i don't have your expertise..

zt3phan: So for the context of instances, an instance is a zfs clone of an image, which itself is a zfs dataset. These come with all of the protections for data integrity built into ZFS. Image hashes are verified before importing, and can be cross referenced with images available publicly.

So if your question is how can you know that the image you're running is the image you think you're running, there are already strong protections and cross references against publicly verifiable data that can be used to verify this.

As far as ensuring the platform image you're running is the image you think you're running, I already discussed that above.

no, i mainly wanted to use TPM as it includes other protections, again where writelocks to specific mcu's lie..

Neither of these use hardware enforcement, so if you have a hard requirement on TPM that's something that will be a concern.

yes i see.

But in lieu of hardware enforcement, we do already have some strong verification mechanisms in place.

right.

Anyway, its mainly the communication (driver) with the chip that's the main issue atm, do you know if any work is being done on that?

is that correct btw?

If someone wanted to replace all the firmware on your computer, lack of driver is not going to be a problem

But you realise how difficult and targeted that would be, right?

>.<

yea..

anyway, my last Q i got, is the system you currently use built on pre-built, specific images of guest-OS'es or can i use any OS as long as i dump it to an image and sign it? if the former i'd like a url to see the images available.

We provide images of popular guests, but you can also create your own instances from scratch, and even perform the install from some media.

there's a mostly working tpm2 driver under development, just not finished yet

bahamut, thanks yet again & thanks to jbk, duncan & dancmd for your time and answers! i'll be idling here some and go thru documentation thoroughly before i get into this project, which im pretty confident i will. & pleased to meet ya'll, ttyl! :)

still needs TAB support and testing w/ TPMs that use CRB for it's interface, and more testing in general

i c, jbk. thx. looking forward to it. if you got any url you think i should go for besides the official site & github, please don't hesitate to throw it :)

unfortunately, i don't have an easy way to distribute an image w/ the driver

if you're adventureous to build your own smartos image, github.com/jasonbking/illumos-gate/tree/tpm2-squashed should have it (though it's based on a slightly older smartos release, so you'd want to rebase it)

though it's still a work in progress, so treat accordingly.. it may not work on your system, or may cause a kernel panic (hopefully not, but so far just been tested on a few systems)

oh hrm.. looking at that branch, i haven't squashed down the most recent stuff...

there updated

just had a look at jasonking's git & also noticed hardenedbsd (which is based on illumos) got tpm2 working, but i dunno if it's of any use to you, as there is prolly more implications i suspect(?)

im not that adventureous as of nowm but in near future i could make an attempt.

now*

hardenedbsd is a FreeBSD derivative

!

this is why i need to get more into the bsd-world. xD

well it's called hardenedBSD for a reason

netbsd,openbsd,freebsd..it's a bit confusing, which actually makes a linux-lowlife like me think they're kinda-the-same ;p

read on the illumos-gate NVMe topo-node is'nt complete either in terms of tracking temp,health etc...anyone heard there are any other troubles w/ NVMe disks & running S-OS guest on 'em?

ah yes, there was another thing i was thinking about. when the vm_guest is running i suppose there is a syscall-fuzzing, how does that approach look? do i need to do the fuzz on my own?

how tedious w/ timeouts...gotta get a real client, but i'll read the logs on the website in topic if anyone wanna answer the above at any point, thanks in advance <3

zt3phan : log.omnios.org

Are there any recommendations for BIOS settings in new generation servers? I typically disable all energy savings and set everything to maximum performance. However, what do you suggest for settings like 'MADT Core Enumeration: linear or round robin,' 'NUMA Nodes Per Socket: 0/1/2/4,' 'Enhanced REP MOVSB/STOSB: enable or disable,' 'Fast Short REP MOVSB: enable or disable,' and 'REP-MOV/STOS Streaming: enable or disable'?

ISTR (though could be confusing things) the MOV* settings might be related to hw vulnerabilities (side channels I think)

so it probably depends on what you plan to run on the system.. if it's multi-tenant or untrusted stuff

might not want it enabled