Allow users to select the cloud metadata providers #13812
Conversation
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| package add_cloud_metadata |
There was a problem hiding this comment.
don't use an underscore in package name
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| package add_cloud_metadata |
There was a problem hiding this comment.
don't use an underscore in package name
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| package add_cloud_metadata |
There was a problem hiding this comment.
don't use an underscore in package name
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| package add_cloud_metadata |
There was a problem hiding this comment.
don't use an underscore in package name
urso
added
:Processors
libbeat
in progress
|
Jenkins is under maintenance right now. Let's rely on Travis for now. |
|
|
||
| func (c *config) Validate() error { | ||
| // XXX: remove this check. A bug in go-ucfg prevents the correct validation | ||
| // on providerList |
There was a problem hiding this comment.
Is there a bug we can reference so that we know when it's ok to remove the check.
There was a problem hiding this comment.
Not yet. The bug is quite bad I'd say, so I will adress this tomorrow.
There was a problem hiding this comment.
go-ucfg issue: elastic/go-ucfg#133
We introduce a new setting 'providers' to the add_cloud_metadata processor. By now all the implementation for metadata providers requires developers to mark a provider as 'local'. The alibaba and tencent providers are not marked as local by now. If the 'providers' setting is not used, then no all providers marked as 'local' are applied. This is a breaking change, because alibaba and tencent providers will not be enabled anymore by default. If the providers setting is used, only the selected providers will be used.
urso
force-pushed
the
cloud_metadata_selector
branch
from
cd8d811
to
45d9ce2
Compare
|
The libbeat testsuite fails in the elasticsearch output. But required unit tests succeed. There are no system tests in libbeat that test add_cloud_metadata. |
| visited := map[string]bool{} | ||
|
|
||
| for name, ff := range providers { | ||
| if visited[ff.Name] { |
There was a problem hiding this comment.
Is "visited" to account for the redundant short names like "alibaba" and "ecs"? A clarifying comment would be nice
|
CI is as green as it currently gets. All related tests passed. |
* Allow users to select the cloud metadata providers We introduce a new setting 'providers' to the add_cloud_metadata processor. By now all the implementation for metadata providers requires developers to mark a provider as 'local'. The alibaba and tencent providers are not marked as local by now. If the 'providers' setting is not used, then no all providers marked as 'local' are applied. This is a breaking change, because alibaba and tencent providers will not be enabled anymore by default. If the providers setting is used, only the selected providers will be used. (cherry picked from commit dc99773)
* Allow users to select the cloud metadata providers (#13812) * Allow users to select the cloud metadata providers We introduce a new setting 'providers' to the add_cloud_metadata processor. By now all the implementation for metadata providers requires developers to mark a provider as 'local'. The alibaba and tencent providers are not marked as local by now. If the 'providers' setting is not used, then no all providers marked as 'local' are applied. This is a breaking change, because alibaba and tencent providers will not be enabled anymore by default. If the providers setting is used, only the selected providers will be used. (cherry picked from commit dc99773)
|
Thanks so much @urso! 🚀 |
|
Something has gone wrong the list of providers in the docs, I think the list item symbol is https://www.elastic.co/guide/en/beats/filebeat/master/add-cloud-metadata.html: |
elastic#13815) * Allow users to select the cloud metadata providers (elastic#13812) * Allow users to select the cloud metadata providers We introduce a new setting 'providers' to the add_cloud_metadata processor. By now all the implementation for metadata providers requires developers to mark a provider as 'local'. The alibaba and tencent providers are not marked as local by now. If the 'providers' setting is not used, then no all providers marked as 'local' are applied. This is a breaking change, because alibaba and tencent providers will not be enabled anymore by default. If the providers setting is used, only the selected providers will be used. (cherry picked from commit fa6d344)
Resolves: #11145
We introduce a new setting 'providers' to the add_cloud_metadata
processor.
By now all the implementation for metadata providers requires developers
to mark a provider as 'local'. The alibaba and tencent providers are not
marked as local by now.
If the 'providers' setting is not used, then no all providers marked as
'local' are applied. This is a breaking change, because alibaba and
tencent providers will not be enabled anymore by default.
Although it's a breaking change, I prefered to disable non-local providers by default. (I epxect the fix to go into 7.4.0).
If the providers setting is used, only the selected providers will be
used.
The change supports alises for different providers. But the setup will only initialized a provider twice if the alias is used.