We noticed that if you create a mixed bond (intel, mellanox, broadcom)the order of the bond initialization of the bond itself, matters. This might be a firmware/SFP+/driver issue.
Here is a working config from /etc/network/interfaces:
auto bond1
iface bond1 inet manual
bond-slaves enp3s0f0np0 eno1
bond-miimon 100
bond-mode active-backup
mtu 9216
And here is a non working stanza:
auto bond1
iface bond1 inet manual
bond-slaves eno1 enp3s0f0np0
bond-miimon 100
bond-mode active-backup
mtu 9216
Note: Look at the bond-slaves order. After changing the order, all links are up and fail-over correctly.