Skip to content

VXLAN persistent networks create bridges on hosts that never ran a VM on them, but don't clean them up #13966

Description

@MitchDrage

problem

With a persistent network offering, CloudStack builds the network's bridge on every host in the zone. But when the network is deleted it only cleans the bridge up on hosts that actually ran a VM on it. On every other host, the bridge and its VXLAN interface stay behind forever.

Investigation
There is a difference between the filters when creating a persistent network, and removing a persistent network.
Creation:
DefaultHostListener.setupPersistentNetwork() creates the bridges on hostConnect/hostEnabled for all networks from getAllPersistentNetworksFromZone() - no isolation-method filter.
Removal:
NetworkOrchestrator.cleanupPersistentnNetworkResources() is gated by networkMeetsPersistenceCriteria(), which requires the broadcast URI scheme to be Vlan, so for vxlan:// networks CleanupPersistentNetworkResourceCommand is never sent. (hostAboutToBeRemoved() sends the same cleanup with no scheme check either - the VLAN-only gate exists only on the network-delete path.)

Hosts that ran a VM on the network are cleaned up via the normal VM-lifecycle teardown, so the leak only affects uninvolved hosts.

versions

4.22.1.1, KVM, advanced zone with VXLAN isolation, persistent network offerings.

The steps to reproduce the bug

  1. Advanced zone, KVM, VXLAN isolation, 3+ hosts.
  2. Create a VPC with two tiers on a persistent offering.
  3. Start VMs so they land on only some hosts. (Restarting cloudstack-agent on an uninvolved host also creates the bridges there via hostConnect.)
  4. Delete the tiers, then the VPC.
  5. ls -d /sys/class/net/brvx-* on each host.

Expected behaviour:
The bridge and VXLAN interface are removed from every host in the zone.

Actual behaviour:
The VNI is still present on hosts that never ran a VM on the network; CleanupPersistentNetworkResourceCommand never appears in management-server.log during the delete.

What to do about it?

Accept Vxlan alongside Vlan in networkMeetsPersistenceCriteria(), or drop the scheme check to match the setup path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions