Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node 18.18.2 : unable to override the Host header in fetch #50305

Closed
fieteboerner opened this issue Oct 20, 2023 · 13 comments
Closed

Node 18.18.2 : unable to override the Host header in fetch #50305

fieteboerner opened this issue Oct 20, 2023 · 13 comments
Labels
fetch Issues and PRs related to the Fetch API

Comments

@fieteboerner
Copy link

Version

v18.18.2

Platform

Linux hostname 5.15.131-1-MANJARO #1 SMP PREEMPT Thu Sep 7 11:05:18 UTC 2023 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

fetch("https://postman-echo.com/get", {
    headers: {
        Test: 1234,
        Host: "foo.postman-echo.com",
    },
})
    .then((r) => {
        return r.text();
    })
    .then((text) => console.log(text))
    .catch((e) => {
        console.log(e);
    });

run this code

The test works successfullty with v18.18.0 & v18.18.0

How often does it reproduce? Is there a required condition?

ive run the tests numerous times
the only condition I know is to run with v18.18.2

What is the expected behavior? Why is that the expected behavior?

  • should send a request topostman-echo.com but should override the Host to foo.postman-echo.com Header
  • the reason: this is how a http client should behave
  • see vor example: curl -H 'Host: foobar.com' https://postman-echo.com/get

What do you see instead?

  • the host header has not been overidden by the headers config
  • the initial set host in the request url is taken

Additional information

  • this was also tested in several docker images
    • node:18-alpine (contains bug)
    • node:18.18.2-alpine (contains bug)
    • node:18.18.2-alpine (no bug)
@panva
Copy link
Member

panva commented Oct 20, 2023

Host is a forbidden header in the Fetch Standard. The Friday October 13 2023 Security Releases disallowed its (and other forbidden headers') use. This also aligned the Host header behaviour with other Fetch implementations such as Deno, Bun, Workerd, and others.

@panva panva added the fetch Issues and PRs related to the Fetch API label Oct 20, 2023
@paulrutter
Copy link

paulrutter commented Oct 20, 2023

@panva @mcollina as i also mentioned in nodejs/undici#2318 (comment), there are there are valid usecases for setting the host header on a request in Node.js environment.
For example, if you have a multi-tenant application with specific hostnames, you want to able to supply the IP address + host header, not just the hostname in the url.

We are running into this issue when upgrading from Node 18.17.1 to Node 20.8.1, as the host header doesn't seem to be passed along to the application anymore (thus, breaking the integration).

Is this really the desired behavior?
It's not a backwards compatible change, and i think the RFC should not be followed when not in a browser environment.
I also don't read in the RFC that the host should not be overridable, just that the agent should generate one (if not available via the custom headers).

Is there a workaround to still allow setting the host header via fetch? Or are we back to using third-party HTTP clients for this?
Java has a workaround by supplying a property: -Djdk.httpclient.allowRestrictedHeaders=host.

@mcollina
Copy link
Member

Or are we back to using third-party HTTP clients for this?

TL;DR, you'll have a much better production experience anyway. fetch() is significantly slower than undici.request() and almost on par with http.request(). Your use case seem specific enough that a custom client for Node.js is recommended.

@paulrutter
Copy link

paulrutter commented Oct 20, 2023

Thanks for your response @mcollina, but this is not really what i would have expected.
Isn't fetch supposed to be the HTTP client to use? I was under the impression that fetch would be the preferred HTTP client to use (after years of using request, got, axios and node-fetch). It would be a shame to now go back to a userland library again.

I also think that using a host header is not that specific that it warrants a custom HTTP client; lots of applications have custom hostnames. Is something like the java solution a path forward? Java has a workaround by supplying a property: -Djdk.httpclient.allowRestrictedHeaders=host.
Or Deno's solution to the same issue:
denoland/deno#17968 (comment).

Another question; isn't undici what drives the fetch API under the hood? How can it then be slower than using undici directly?

@paulrutter
Copy link

Decided to use undici.request instead of fetch. Then we can still leverage the same HTTP client as fetch uses under the hood, but get the freedom we need.

See nodejs/undici#2369

@pano9000
Copy link

pano9000 commented Nov 9, 2023

have run into the same issue here, where I need to work with an API, that required the host header to be set for the Authentication to work, e.g.:
https://kiwihr.com/api/docs/authentication.doc.html

this used to work fine with previous versions, but now since I cannot overide the host header anymore, this fails with the built-in fetch, which is a bit of a bummer.

would be nice to have a way to do this with the built-in fetch in the future again

@paulrutter
Copy link

Following nodejs/undici#2369 (comment), using fetch is not advocated for backend development.

Undici.request is the advised approach.

@matthias-hertel-db
Copy link

using fetch is not advocated for backend development.

fetch API and behavior was defined as a API in end-user browser level JavaScript applications. For this use-case the host header must fit to the domain name used for DNS based IP retrieval. That closes some security concerns in combination with Script injection etc.

When you have to implement backend components like content inspection & security scanning that you face the situation that you like to pass the host header "as is" to a backend server(having an internal domain name).

We don't need 2 libraries just 2 different behaviors.

@gongfan99
Copy link

gongfan99 commented May 31, 2024

I have the similar issue. My backend server#1 needs to call the backend server#2 by IP/hostname since the server#2 is a multi-tenant. Not allowing Host header makes fetch() not suitable for the task.

Personally I am Ok to use undici.request() instead since it obviously allows the Host header.

However, server#1 is for Nextjs which patches fetch() to facilitate deduplication and request-side caching. If I use undici.request() in server#1, then I will lose deduplication and caching.

For de-dup, it is Ok since I do not use it that much.

For caching, it is also probably Ok because Nextjs' handling of caching does not make sense anyway: whether or not caching should not be decided by the request-side; the request-side should simply respect the Cache-Control header sent by the response-side.

So the question is, by default, does undici.request() respect the Cache-Control header just like a real browser? Or do I need to config it to make it behave that way? I tried to dive into the undici code but it is too deep for me :-)

@Netail
Copy link

Netail commented Jun 26, 2024

Facing the same issue for our local reverse proxy, since we switch from node-fetch to native fetch :(

E.g. in /etc/hosts you direct 2 custom domains to 127.0.0.1

127.0.0.1 local.google.nl local.google.be local.google.uk

All go through the same reverse proxy client, but based on the Host header we can determine on which domain the user is based.

@paulrutter it's weird to force backend people to use a different fetch package, just because of the Host override.
NodeJS is also largely used for backend... Would it be possible to remove this override @mcollina ?

@paulrutter
Copy link

See nodejs/undici#2369 (comment) for more details.

@Netail
Copy link

Netail commented Jun 26, 2024

See nodejs/undici#2369 (comment) for more details.

Aye I've read it, but it feels weird to add a library just because of the different Host header library. What about a check if it's in the browser or not? Only override the header if it's browser-side. And it is pretty confusing to see the Host header not working...

@paulrutter
Copy link

I agree with you on that, but got the same response several times, so decided to move to undici (which works fine, and is still the library behind fetch).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fetch Issues and PRs related to the Fetch API
Projects
None yet
Development

No branches or pull requests

8 participants