Any idea what could be causing this?

18:59:48.897 [warning] |00000000000|kz_amqp_connection:388(<0.208.0>) failed to connect to 'amqp://guest:guest⊙1143:5672' will retry: timeout

18:59:48.897 [warning] |00000000000|kz_amqp_connection:388(<0.206.0>) failed to connect to 'amqp://guest:guest⊙1132:5672' will retry: timeout

I've check the amqp host and the server is running and says it has multiple connections from the host reporting this error.

the amqp host says "user 'guest' authenticated and granted access to vhost '/'. That last log entry is from 12 minutes ago but, the kazoo hosts show the same error message every few seconds.

Ysean: maybe try tcpdump to see what's being sent/received during the connection attempt?

seems odd that rabbit would have connections but timeout on this one

right

mc_ : fwiw, it was a SDN issue. I had a VM cluster member go down over the weekend and it was housing most of the Kazoo hosts in this grid. It just so happened that the SDN config didn't migration when I manually recovered the VMs.