[Task Manager] Throttle TM health logging

[chrisronline]

Relates to #101751

Some possible ideas:

• Gate entire logging changes behind a config switch (off by default)
• Log a message telling users they can switch config on when the status changes (from green -> warn/error)
• While in a warn/error status, perhaps log at some throttled interval (every 1h?)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[tsullivan]

Chiming in to say I find this log message could be have more useful context information:
[warning][plugins][taskManager] Detected delay task start of 69.115s (which exceeds configured value of 60s)

Can you include the "task type" in the message? For example, I suspect my example log message applies to a report:execute task.

[chrisronline]

@tsullivan That's a good idea! I added it to the PR in 4321ec8

[spong]

Not sure if this is the right place to mention, but also noticed these new TM health logging and saw a misspelling in one of the Latest Monitored Stats fields.

capacity_requirments vs capacity_requirements (source)

Looks like this is new for 7.14 as of #100475, so may still be able to fix before release if we want.

[chrisronline]

@spong Thanks, I'll log a ticket for it!