Add a custom User-Agent to queries

[freak12techno]

Problem: when cosmos.directory is doing queries towards the node, it does the request with the User-Agent of the original query:

<IP> - - [03/Jun/2024:12:59:53 +0200] "GET /aaaa HTTP/1.1" 501 68 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:126.0) Gecko/20100101 Firefox/126.0"

It's difficult to distinguish queries that go from cosmos.directory proxy and the clients directly.

I think it makes sense to use a custom User-Agent for all queries from cosmos.directory (as it's the cosmos.directory backend that does querying?) while somehow preserving the original User-Agent and IP.

Not sure what's the best way to do so, so I wonder what's the team's opinion on that.

[tombeynon]

@freak12techno I've thought about this a few times - completely agree it would be valuable info to have. I'm not sure if re-writing the user agent is the cleanest way to do it, it would be nice to keep that as-is. I was considering using a header, there doesn't seem to be a standard for this sort of thing but I was considering x-forwarded-by. x-forwarded-for is used to identify the original request's IP address when a request is proxied, so x-forwarded-by could make sense for the inverse.

Another option could be x-requested-with but it could be set by the request so we would have to overwrite it (same issue as user-agent).

Any thoughts here @freak12techno ?

[freak12techno]

@tombeynon I think altering the original User-Agent would rather do more harm than good; my idea was to add a new header like x-original-user-agent or something (I am sure there might be a better naming but not sure which one) and set it to original User-Agent, while replacing the actual User-Agent header with something that identifies the real source (as technically it's cosmos.directory backend doing queries from its IP, it just forwards everything to a node, right?)

Also, not sure about this, but does it use x-forwarded-for? If not, we can (and should, I think) also add it (as in, adding original request IP as x-forwarded-for).