Add support for proxies in destination context

[watson]

If a proxy is used, the destination context will not be set on outgoing http spans because the URL cannot be successfully extracted from the request. For details see #1769.

[ebuildy]

Hello, any new on this one?

[astorm]

@ebuildy Thanks for checking in -- this isn't specifically on our radar as a pressing backlog item, but that could easily change :)

Would you be able to share some details (a reproduction, details about which proxy server you're using, etc.) about how this problem manifests for you?

[ebuildy]

@astorm we use APM in production, to monitore our nestJS based API.

HTTP proxy is configured via env. variable HTTP(S)_PROXY, this bug is filling our error log with

Could not set destination context: Invalid URL: http://10.110.0.15:2001https://XXXXXXXXXX\n

what I dont understand, is document span.name value is ok (= POST XXXXXXX) so URL parsing is working fine somewhere.

It becomes annoying for us, APM is generating more errors than the app it-self! but this is so useful,

BTW we have a platinium licence, should I report this bug as customer too?

thanks you

[astorm]

Thanks for the background context @ebuildy -- I found it useful!

It looks like the root cause of this bug is this issue here: Qard/http-request-to-url#1

I think @Qard is waiting to hear back from -- someone? -- that the proposed solution of defaulting to returning the final destination and having an option to return the proxy instead was an acceptable one or not. I'll chime in with my two cents -- if you do the same it could help move things forward.

BTW we have a platinium licence, should I report this bug as customer too?

Reporting a bug via formal channels can often help get it solved, or get the bug work prioritized -- so yes, definitely report :)
Qard/http-request-to-url#1 (comment)

[astorm]

@ebuildy we have a PR into the upstream repository that will move this one forward. Qard/http-request-to-url#2 Once that PR lands we can pull it into the agent and we'll see if it help your issue.

[astorm]

@ebuildy Quick update on this. We've released version 3.14 of the Node.js agent. This version of the agent changes up our strategy for extracting a request URL for outgoing spans. This change was made to solve a separate issue, but we believe it should also solve the issue where proxy URLs are jammed together with the actual URL.

We're going to close this issue out, but if you (or anyone) is still seeing a problem please let us know.

[Pluiesurlavitre]

Unfortunately, the issue is still present on version 3.14.0:

{"log.level":"error","@timestamp":"2021-04-22T09:06:34.752Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"Could not set destination context: Invalid URL: http://xx.xxx.xxx.xxx:2001https://the_url_im_requesting"}

Is there anything I can do to help with this ?

[Pluiesurlavitre]

Sorry to bump this @astorm - is there any way we can reopen this MR ?

[ebuildy]

Hello @astorm , just testing with :

{
  "dependencies": {
    "axios": "^0.21.1",
    "elastic-apm-node": "^3.16.0",
    "express": "^4.17.1"
  }
}

and this simple code source

const express = require('express')
const app = express()
const axios = require('axios')
const port = 3000

const apm = require('elastic-apm-node').start({
    serviceName: 'test-nodejs-tom',
    serverUrl: 'https://apm.dev-steam',
    verifyServerCert: false
  })

app.get('/', (req, res) => {
    axios.get('https://api.ipify.org?format=json', {
        proxy: {
            host: '10.100.XX.XX,
            port: 2001
          }
    })
    .then(function (response) {
        console.log(response);
        res.send(response.data)
    })
    .catch(function (error) {
        console.log(error);
        res.send(error)
    })
})

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`)
})

I still can see

{"log.level":"error","@timestamp":"2021-06-24T09:00:21.387Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"Could not set destination context: Invalid URL: http://10.100.X.X:2001https://api.ipify.org/?format=json"}

How can I fix this one please? this become really urgent now we have installed plat. licence and use elastic APM in all our products.

thanks you,

[ebuildy]

I see Qard dependency have been removed:

0be8103#diff-36acc683b9a592acebe63253770b5f944cc8caff0e49344f71a849a2568da62f

[astorm]

Thanks for letting us know @ebuildy and for the reproduction -- I'm taking a look right now.

[astorm]

OK -- I've been able to reproduce this behavior by creating a "single url" proxy server running on port 4000 with this program (has some cruft from copy/pasta)

      const express = require('express');
      const morgan = require("morgan");
      const { createProxyMiddleware } = require('http-proxy-middleware');
      const app = express();

      // Configuration
      const PORT = 4000;
      const HOST = "localhost";
      const API_SERVICE_URL = "https://jsonplaceholder.typicode.com";
      app.use(morgan('dev'));
      app.get('/info', (req, res, next) => {
        res.send('This is a proxy service which proxies to Billing and Account APIs.');
      });

      app.use('', (req, res, next) => {
        console.log("called the middleware")
        next()
      });

      app.use('/', createProxyMiddleware({
        target: 'https://www.google.com/',
        changeOrigin: true,
        pathRewrite: {
          [`^/test`]: '',
      },
      }));

      app.listen(PORT, HOST, () => {
        console.log(`Starting Proxy at ${HOST}:${PORT}`);
      });

and then using a modified version of @ebuildy's repro in a second instrumented application

app.get('/test-axios', function(req, res) {
  axios.get('http://www.google.com/',{
    proxy: {
        host: 'localhost',
        port: 4000
      }
    })
    .then(function (response) {
        console.log(response);
        res.send(response.data)
    })
    .catch(function (error) {
        console.log(error);
        res.send(error)
    })

})

In the above scenario, requesting http://localhost:3000/test-axios triggers a request to http://www.google.com proxied through http://localhost:4000/, and results in the following logged error

{"log.level":"error","@timestamp":"2021-06-24T17:46:36.501Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"Could not set destination context: Invalid URL: http://localhost:4000http://www.google.com/"}

[ebuildy]

Hello @astorm , thanks for the merge! When the next version will be available please? (more or less)