12:23:49 Does anyone know which log file shows cloud-init errors during provisioning? The JSONified cloud config in the customer_metadata gets ignored and I can not find any error related to this issue. 12:44:11 TheTim0Nat0r : what kind of instance is it? 12:44:42 neuroserve: lx and kvm 12:48:10 there could be logs in /zones//logs - at least the console log should have some cloud-init output 13:49:51 Does agetty have something to do with it? Thats the only thing I could find: "agetty: cannot connect on UNIX port" 13:56:40 Never mind, found this in the guest : https://i.gyazo.com/48fc296f3cd3766afb964ba2a2fd53d4.png 14:06:51 you should be able to update the metadata of an instance with sdc-updatemachinemetadata (https://docs.tritondatacenter.com/public-cloud/tags-metadata/metadata#use-cases-for-metadata) 14:21:59 Been updating metadata multiple times, the string has changed but not the behaviour unfortunately 16:21:27 danmcd rmustacc: any idea of what to do with the sestopo output to find the slot / light it up? 16:24:27 does the slot show up in fmtopo -V ? 16:26:00 jbk https://us-east-storage.solutions.iqvia.com/bruce_dev/public/topo-v 16:27:35 is that cut off? it seems to stop at the first bay 16:27:54 hmmm 16:27:56 let me reupload 16:29:09 Same url, 6264 lines 16:29:57 yeah looks like it stops there.. i wonder if there's something it's not liking about the enclosure 16:31:01 my understanding is that the t10 standards for ses can be a bit.. open to interpretation by vendors 16:33:13 lol, so what do I tell the DC guy who has to replace the drive? 16:34:56 does sestopo output anything? 16:38:17 https://us-east-storage.solutions.iqvia.com/bruce_dev/public/sestop-ses0 https://us-east-storage.solutions.iqvia.com/bruce_dev/public/sestop-ses1 16:53:45 try 'TOPOSESDEBUG=1 fmtopo -V' -- that should output debug info which might help explain why it's not enumerating the contents despite sestopo showing they exist 16:54:12 though that output goes to stderr 16:54:46 so you might need fmtopo -V > topo.txt 2>debug.txt (or 2>&1...) to capture it 16:56:58 "/usr/lib/fm/fmd/fmtopo: failed to get properties for bay=0: (null)" 16:57:48 "/usr/lib/fm/fmd/fmtopo: failed to get properties for FAN6=0: (null)" 17:03:11 that is strange... the '(null)' appears to be coming from topo_strerror(), but unless there's something smartos-specific going on there, it should always return a string, and never NULL 17:05:39 Nothing AFAICT we change there: 17:05:41 kebe(~/ws/ij-cr)[0]% diff {~/nfslinks/nowhere/ws/illumos-gate,.}/usr/src/lib/fm/topo/libtopo/common/mkerror.sh 17:05:41 kebe(~/ws/ij-cr)[0]% 17:05:56 just to see.. if you do 'UMEM_DEBUG=guards TOPOSESDEBUG=1 fmtopo -V' 17:06:06 does it crash? 17:06:53 danmcd: looking at topo_strerror() -- it seems like it should always return a string 17:06:55 no 17:07:00 unless i'm missing something 17:13:01 and looking at that error message from fmtopo, there appears to only be one spot where that happens 17:52:02 Smithx10: I've not had any chance to review or dig, sorry. 18:06:44 one thing that's (maybe) interesting is there's an ARRAY_DEVICE element that seems like it's actually a 'container' for the actual drive bays.. i haven't dug too deeply, but the impression I get is that there's an assumption that ARRAY_DEVICE is a slot/bay 18:06:57 and it's also enumerated first 19:29:18 hmm 19:30:26 I guess to make it easier I could take this drive out of the pool 19:30:42 and maybe make the pool do work? 19:30:57 and the non blinky drive could be the other spare.... or it.... 19:52:35 that might work.. 19:53:03 you could also try doing a dd if=/dev/dsk/ of=/dev/null to make the other spare light up