problem
With a persistent network offering, CloudStack builds the network's bridge on every host in the zone. But when the network is deleted it only cleans the bridge up on hosts that actually ran a VM on it. On every other host, the bridge and its VXLAN interface stay behind forever.
Investigation
There is a difference between the filters when creating a persistent network, and removing a persistent network.
Creation:
DefaultHostListener.setupPersistentNetwork() creates the bridges on hostConnect/hostEnabled for all networks from getAllPersistentNetworksFromZone() - no isolation-method filter.
Removal:
NetworkOrchestrator.cleanupPersistentnNetworkResources() is gated by networkMeetsPersistenceCriteria(), which requires the broadcast URI scheme to be Vlan, so for vxlan:// networks CleanupPersistentNetworkResourceCommand is never sent. (hostAboutToBeRemoved() sends the same cleanup with no scheme check either - the VLAN-only gate exists only on the network-delete path.)
Hosts that ran a VM on the network are cleaned up via the normal VM-lifecycle teardown, so the leak only affects uninvolved hosts.
versions
4.22.1.1, KVM, advanced zone with VXLAN isolation, persistent network offerings.
The steps to reproduce the bug
- Advanced zone, KVM, VXLAN isolation, 3+ hosts.
- Create a VPC with two tiers on a persistent offering.
- Start VMs so they land on only some hosts. (Restarting
cloudstack-agent on an uninvolved host also creates the bridges there via hostConnect.)
- Delete the tiers, then the VPC.
ls -d /sys/class/net/brvx-* on each host.
Expected behaviour:
The bridge and VXLAN interface are removed from every host in the zone.
Actual behaviour:
The VNI is still present on hosts that never ran a VM on the network; CleanupPersistentNetworkResourceCommand never appears in management-server.log during the delete.
What to do about it?
Accept Vxlan alongside Vlan in networkMeetsPersistenceCriteria(), or drop the scheme check to match the setup path.
problem
With a persistent network offering, CloudStack builds the network's bridge on every host in the zone. But when the network is deleted it only cleans the bridge up on hosts that actually ran a VM on it. On every other host, the bridge and its VXLAN interface stay behind forever.
Investigation
There is a difference between the filters when creating a persistent network, and removing a persistent network.
Creation:
DefaultHostListener.setupPersistentNetwork()creates the bridges on hostConnect/hostEnabled for all networks fromgetAllPersistentNetworksFromZone()- no isolation-method filter.Removal:
NetworkOrchestrator.cleanupPersistentnNetworkResources()is gated bynetworkMeetsPersistenceCriteria(), which requires the broadcast URI scheme to beVlan, so forvxlan://networksCleanupPersistentNetworkResourceCommandis never sent. (hostAboutToBeRemoved()sends the same cleanup with no scheme check either - the VLAN-only gate exists only on the network-delete path.)Hosts that ran a VM on the network are cleaned up via the normal VM-lifecycle teardown, so the leak only affects uninvolved hosts.
versions
4.22.1.1, KVM, advanced zone with VXLAN isolation, persistent network offerings.
The steps to reproduce the bug
cloudstack-agenton an uninvolved host also creates the bridges there via hostConnect.)ls -d /sys/class/net/brvx-*on each host.Expected behaviour:
The bridge and VXLAN interface are removed from every host in the zone.
Actual behaviour:
The VNI is still present on hosts that never ran a VM on the network;
CleanupPersistentNetworkResourceCommandnever appears inmanagement-server.logduring the delete.What to do about it?
Accept
VxlanalongsideVlaninnetworkMeetsPersistenceCriteria(), or drop the scheme check to match the setup path.