Report host name as FQDN on all platforms

[pickypg]

Apparently Go does not provide the FQDN on all systems (particularly Linux, it seems) because it checks /bin/hostname rather than /bin/hostname -f.

Here's the Go test showing that they accept the shortname: https://golang.org/src/os/os_test.go#L1341

It would be great if we could fill in the actual FQDN rather than an unexpected subset, but it does mean providing an appropriate mechanism for every supported system (or falling back to the subpar behavior that comes out of the box).

As a workaround, #715 will allow environment variables to fill this in, but it still requires something else in the environment to be used.

/cc @andrewkroh

[dMopp]

+1

[Lollaskates]

+1, running topbeat on my linux hosts returns FQDN, but running topbeat on windows hosts (Server 2012R2) only returns the hostname. Would like to have it normalized.

[JakeDEvans]

my linux systems (cent 7.2) report fqdn, but my windows report the shortname.

I would prefer the shortname (which is how my rsyslog data comes in). can we update this request to allow you to specificy the command used to aquire the hostname?

windows: %COMPUTERNAME%.%USERDNSDOMAIN% or %COMPUTERNAME%
linux: hostname -s or hostname -f

[trevorndodds]

Please make this an option, I for one prefer the shortname as it keeps the field length shorter. Some windows domains have really long FQDNs.

[richlv]

it might be preferable to allow running a custom command to determine hostname. on some systems even hostname -f returns short name, and user might run a custom command to determine fqdn.

[dMopp]

A good idea, too but hostname -f shows the short version only if the system is wrongly setup (or didnt have a domain)

[jvkassi]

How can i force this ? We need fqdn

[popstas]

@mib4fun, we replaced host field with fields_under_root, fqdn fills with ansible:

filebeat.prospectors:
- input_type: log
    fields_under_root: true
    fields:
      host: "fqdn"

[popstas]

Also you can define beat.name: just set name on top level of filebeat.yml

[jvkassi]

@popstas ! Its' working !

[andrewkroh]

Duplicates:

• Feature request - Beats to report FQDN #6741
• Packetbeat host.name not fqdn like winlogbeat #16507

[DPattee]

Found this report while trying to figure out how to do the opposite...

I only want the hostname part available in a field, not the entire fqdn. I.e. what hostname -s returns.

Having two fields, one with just the hostname and one with the fqdn, would be a fine solution for me.

Reasons:

1. Clutter. All machines reporting in to a single elastic cluster belong to the same domain the way I built things out. Having the extra domain information just clutters up dashboards / viz labels.
2. Mac OS X bug that occasionally reports the .local version of the hostname instead of the correct . version (https://discuss.elastic.co/t/mac-hostname-showing-as-localhost/48878)

[willemdh]

This is really an issue. The inconsistencies in host.name usage result in all kinds of unwanted results. As this field is used as the primary key for many stuff, this should get cleared out asap, as every query, dashboard, saved search, watch, alert or ml job will have to be revisited and updated once there is finally a consensus on using shortname or fqdn for all host.name values in all beats.
As some organizations might index events from different domains where some hosts might have the same name, imho there is no choice but to move to lowercase fqdn. (for example dc01.domain1.net and dc01.domain2.net)
To make stuff convenient for users who don't need / want fqdn, an additional field, such as host.shortname could be added?

I can understand that it takes some time to update beats host.name output and processors etc, but at least we should asap evolve to a consensus where this is going long-term, so we can make the necessary preparations and if time permits even add the domain ourselves untill Elastic provides it.

@MorrieAtElastic @andrewkroh @webmat @MikePaquette Please think about a long term plan for host.name and communicate about this.

[webmat]

Agreed, there's a few things we'd like to shore up with regards to host identifiers. I'll make sure fqdn vs shortname is part of the discussion.

[StefanSa]

hi there,
please fix it quickly, that is otherwise totally confused with all action.
An example of many here, because of the inconsistent handling of the host.name.
In the Metric App / Inventory i see most hosts twice "with" and "without FQDN".
Currently, with Metricbeat, Filebeat and Packetbeat, the FQDN is not sent as host.name but only the hostname (without FQDN).
However, with one exception, Winlogbeat sends the FQDN as host.name.

[willemdh]

@webmat Any news about this?

[webmat]

On the ECS side we've just started a more holistic discussion around all the host identifiers, it's RFC 0006. Still very early stage, though. Should start moving in the coming weeks.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dMopp]

Still an issue :(

[mschiegl]

+1

[rylnd]

@joshdover thanks for the ping. Since this proposal looks to be about a forward-looking convention rather than a forced migration, I don't envision significant issues on our end. As others have mentioned, we primarily use host.name as an assumed primary key, so in general both old and new datasets should continue to function as expected; any concerns are going to be centered around attempting to correlate across these datasets.

1. Host/Network pages may show "duplicate" entries for old/new host.name values (but this is already the case today)
2. Exceptions will continue to function on old data, but new exceptions may need to be added/modified for new data
3. Prebuilt rules may need to updated if they make assumptions about host.name (cc @brokensound77 )

We'll probably want to communicate this change to Security users so that they can respond accordingly, but this will always be specific to the user's data and their consumption of it.

[norrietaylor]

@ferullo, I ensure you are in the loop about @joshdover's proposal. @mitodrummer has started a PR here to move this along.

Any concerns?

[ferullo]

No, I have no concerns. Endpoint can align with Agent/Beats on this, we'll just need to know the exact release to make the change in.

The only part of the Security App that I can think of that could be affected by this is Endpoint or Detection Engine rules and I really really doubt any Elastic rules are based on host.name. But @brokensound77 can you confirm? Sorry for not having looped you in on this sooner.

[AndersonQ]

folks, a question about the domain. Depending on the system configuration the domain might be missing, thus the FQDN would be just the hostname.

On Linux hostname -f uses .lan if no domain name is defined. I'm reproducing that on go-sysinfo, the lib we use to get the host infos.

Any objections on that?
@ferullo is it fine for Endpoint?

[willemdh]

@AndersonQ What about Windows, will it take the workgroup as domain? I don't see a problem with using .lan for Linux servers not in a domain.

[AndersonQ]

As Josh recommended on 8.x it'll be optional, thus a config is required, therefore I propose:

• Elastic Agent (configuring the host provider)
  providers.host.use_fqdn: true|false

• Beats standalone (configuring the add host metadata processor)
  processors.add_host_metadata.use_fqdn: true|false

[joshdover]

Thanks for sharing, @AndersonQ. A couple questions:

• We don't currently support configuring any providers.* keys in agent policies from Fleet today. Do you anticipate any challenges on the agent side in adding this to the agent policy? (I'm not familiar with how/where agent applies the agent policy from Fleet)
• When providers.host.use_fqdn: true is set, will Agent automatically configure processors.add_host_metadata.use_fqdn on config passed to Beats? Or will Beats be required to change the default for the add_host_metadata processor based on the agent config? Something else?

[AndersonQ]

We don't currently support configuring any providers.* keys in agent policies from Fleet today. Do you anticipate any challenges on the agent side in adding this to the agent policy? (I'm not familiar with how/where agent applies the agent policy from Fleet)

I don't foresee any problem, as long as we add it to the "policy parser", it should be fine. But I'll confirm with the team later today.

When providers.host.use_fqdn: true is set, will Agent automatically configure processors.add_host_metadata.use_fqdn on config passed to Beats? Or will Beats be required to change the default for the add_host_metadata processor based on the agent config? Something else?

I'd say the agent would configure the processors.add_host_metadata for the beats as well. But it could be the agent would do it only for the monitoring beats. I'd say it's more a product decision, either way shouldn't be too difficult to implement

[joshdover]

I'd say the agent would configure the processors.add_host_metadata for the beats as well

I think the issue is that Agent v2 removed all knowledge and assumptions about the underlying streams configuration. AFAIK we don't manipulate it anymore.

However, I think we want to have consistent behavior across all integrations (@nimarezainia please confirm). Once we have the shipper appending these fields, this is more straightforward, but until then we need another solution.

Please do discuss with the team today (I won't be there).

[jlind23]

@AndersonQ Please keep us posted in order to take a decision today.

[AndersonQ]

After discussing with the team, we agreed on having a feature flag config agent.features.fqdn.enabled: true|false. It'll be added to the control protocol and the applications the agent runs should be able to understand them.

The generic version is:

agent:
  features:
    feature_name:
      enabled:
      config1:
      config2:
      ...
      configN:

That way the agent will be able to enable and disable features and the features can have complex configs beyond just enable/disabled

[jlind23]

Thanks @AndersonQ for the update. Feel free to link Beats and Agent PR to this issue in order to consider this as complete.

[joshdover]

@AndersonQ Great thank you. Let's make sure this field is explicitly documented in the control protocol, along with the expected behavior of the application. We should then share that elastic-agent-client PR with the Endpoint developers as soon it's available for them to start integrating with.

[joshdover]

One question: does this mean that we are only planning to support this feature in Agent or do we still plan to add a Beats configuration for this?

[ferullo]

I have no objections to this approach to adding a default domain when needed. Also this config looks good to me, please just make sure it's passed to Endpoint over the v2 comms.

@softengchick is going to take over tracking this issue from the Endpoint side.

[DPattee]

folks, a question about the domain. Depending on the system configuration the domain might be missing, thus the FQDN would be just the hostname.

On Linux hostname -f uses .lan if no domain name is defined. I'm reproducing that on go-sysinfo, the lib we use to get the host infos.

Any objections on that? @ferullo is it fine for Endpoint?

Anderson can you run your test on a Mac ? I’m pretty sure the default expected domain would be .local

[AndersonQ]

folks, a question about the domain. Depending on the system configuration the domain might be missing, thus the FQDN would be just the hostname.
On Linux hostname -f uses .lan if no domain name is defined. I'm reproducing that on go-sysinfo, the lib we use to get the host infos.
Any objections on that? @ferullo is it fine for Endpoint?

Anderson can you run your test on a Mac ? I’m pretty sure the default expected domain would be .local

done. We'll mimic as much as possible the OS behaviour

[AndersonQ]

@AndersonQ What about Windows, will it take the workgroup as domain? I don't see a problem with using .lan for Linux servers not in a domain.

No, the complete behaviour is described on elastic/go-sysinfo#144. For windows, as we can request the FQDN directly to the system, it just returns what we get.