Having installed OmniOS on my SuperMicro X9DR7-LN4F I configured a link aggregation to my Cisco SG300-28 switch, but found the network connection to be very erratic. It turns out that OmniOS by default uses a L4 LACP hash policy (piping by mac-address, IP-address and TCP/UDP port) whereas Cisco SG300 to my knowledge (i.e. the SG300 administration interface) only supports L3 LACP hash policy (mac-address, and IP-address). In other words, what traffic goes down what pipe was not in sync between OmniOS and the switch.

The fix is simple, specify the hash policy when creating the link aggregation (aggr0):

1

dladm create-aggr -P L3 -d igb0 -d igb1 -d igb2 -d igb3 aggr0

or update an existing link aggregation (aggr0) using:

1

dladm modify-aggr -P L3 aggr0

That stabilized network connections.