DNS record orphan mitigation

[maleck13]

What
In some circumstances, it is possible that a dns record is left behind by a controller:

• Bad clean up during namespace deletion (IE I delete the namespace with the controller in it but leave DNSRecords around)
• Manual removal of finalisers

We want to enable users to identify records that have become orphans so that action can be taken to clean them up.

How

Identification:

Each controller managing a DNSRecord should be able to see the owner id of each other controller working on the record set for that rootDomain. Adding a section to the DNSRecord status such as:

  domainOwners: 
        - id1
        - id2
        - id3

• Add all owners to DNSRecord Status Add all domain owners to dnsrecord #209
• Add metrics and alert template to identify orphan records
• Document how to remove orphan records

and exporting this via a metric something like: dnspolicy_status_root_domain_owners{name="mypolicy",namespace="mynamespace",root_domain_owner="id3", root_domain=a.b.com} 1 and dnsrecord_status_root_domain_owner{name="myrecord1", namespace="mynamespace", root_domain_owner="id1", root_domain="a.b.com"} 1 it should be possible to trigger an alert if the number of DNSRecords is less than the number of owners . We should work with @david-martin @R-Lawton to work through what metrics would make sense to achieve the goal the above is just an idea.

dnspolicy_status_root_domain_owners{name="mypolicy",namespace="mynamespace",root_domain_owner="id1", root_domain="a.b.com"} 1
dnspolicy_status_root_domain_owners{name="mypolicy",namespace="mynamespace",root_domain_owner="id2", root_domain="a.b.com"} 1
dnspolicy_status_root_domain_owners{name="mypolicy",namespace="mynamespace",root_domain_owner="id3", root_domain="a.b.com"} 1

and a record can only have 1 owner, it might look like this:

dnsrecord_status_root_domain_owner{name="myrecord1",root_domain_owner="id1", root_domain="a.b.com"} 1
dnsrecord_status_root_domain_owner{name="myrecord2",root_domain_owner="id2", root_domain="a.b.com"} 1
dnsrecord_status_root_domain_owner{name="myrecord3",root_domain_owner="id3", root_domain="a.b.com"} 1

Then a join query might look like this:

count by(name) (dnspolicy_status_root_domain_owners * on(root_domain_owner) group_left dnsrecord_status_root_domain_owner)

That would give you a value of 3 (as there's 3 dnsrecords that it can join on).
If you want to alert if that number is wrong, you can compare it to the count of root_domain_owners on the policy:

count by(name) (dnspolicy_status_root_domain_owners * on(root_domain_owner) group_left dnsrecord_status_root_domain_owner)
!=
count(dnspolicy_status_root_domain_owners)

Clean Up:

Talking with @mikenairn we could craft a method that would result in creating a new DNSRecord with the missing owner id to be reconciled by an existing controller and then delete this record to force the controller to remove the records related to that owner id.

[maleck13]

Related: Kuadrant/kuadrant-operator#649 We may want some form of visibility in the dashboard around this

[maleck13]

#209

[maleck13]

https://github.com/Kuadrant/gateway-api-state-metrics/pull/46/files

[maleck13]

Kuadrant/kuadrant-operator#862

[maleck13]

Kuadrant/kuadrant-operator#862