SQL query is not traced

[ceefour]

https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html#api-segmentdocuments-sql mentioned that the subsegment can/should contain sanitized_query, however TracingInterceptor never sends it:

aws-xray-sdk-java/aws-xray-recorder-sdk-sql-postgres/src/main/java/com/amazonaws/xray/sql/postgres/TracingInterceptor.java

Lines 159 to 162 in 01d52ae

	if (process && null != subsegment) {
	subsegment.putAllSql(additionalParams);
	subsegment.setNamespace(Namespace.REMOTE.toString());
	}

@jamesdbowman Is this a bug or is it intentional?

If it's indeed intentional, then please add a way to enable/disable this, most likely through environment variable or system property or AWSXRayRecorderBuilder. (since in Spring Boot, we simply use spring.datasource.tomcat.jdbc-interceptors=com.amazonaws.xray.sql.postgres.TracingInterceptor which means no direct bean property configuration)

I'm thinking that SQL statement should be included in the trace by default (but can be disabled for those who want it), because as it stands, TracingInterceptor is almost useless. I know each request performs a bunch of queries, and a few are slow, but which ones?

[ceefour]

After trying myself, some (long?) queries make this error:

com.amazonaws.xray.emitters.UDPEmitter   : Exception while sending segment over UDP.

java.io.IOException: Message too long (sendto failed)
        at java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.net.DatagramSocket.send(DatagramSocket.java:693)

https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html mentions that the largest segment size is 64 KB. I doubt that any of my queries are tens of kilobytes in size. Does this have something to do with UDPEmitter instead of X-Ray limits?

[ceefour]

Here's how I do it now, with UDP message size workaround:

// https://github.com/aws/aws-xray-sdk-java/issues/4
subsegment.putSql("sanitized_query", StringUtils.abbreviateMiddle(this.query, "…", 1000));

[jamesdbowman]

Hi ceefour,

Thanks for reaching out about the sanitized_query feature. We intentionally left this out of the SDK for until it can be added back in with a feature flag and sufficient sanitization logic. Pull requests are always welcome!

As for the long UDP message, please check the value of sudo sysctl net.inet.udp.maxdgram if applicable on your system. MacOS, for example, does not enable the maximum UDP packet size of ~65kb by default: sudo sysctl -w net.inet.udp.maxdgram=65535.

Closing this issue for now.

[ceefour]

@jamesdbowman

⟫ sudo sysctl net.inet.udp.maxdgram
sysctl: cannot stat /proc/sys/net/inet/udp/maxdgram: No such file or directory

Hmm? (Ubuntu 16.04)

What I have is

⟫ sudo sysctl net.core.rmem_max
net.core.rmem_max = 212992
⟫ sudo sysctl net.core.wmem_max
net.core.wmem_max = 212992

should be enough?

[jamesdbowman]

Hi @ceefour ,

The sysctl example provided above applies to macOS systems, if you're running Ubuntu 16.04 your system likely already supports the sending of ~65kb UDP packets.

You may also be interested in enabling subsegment streaming on your instance of AWSXRayRecorder. Subsegment streaming enables the recorder to break apart a large segment into multiple UDP packets. If your subsegments are individually under the ~65kb limit, modifying the DefaultStreamingStrategy used by your recorder instance will allow you to work around this packet size limitation. I would recommend constructing a new instance of DefaultStreamingStrategy with a low (single-digit) maxSegmentSize and modifying your recorder instance to use this strategy.

Hope this helps, thanks.

[ceefour]

@jamesdbowman I wonder why this burden has to be on the developer.

Shouldn't the XRay library automatically handle this on developer's behalf? (Unless in very rare specific instances)

The sole reason and purpose of using XRay is to monitor or diagnose app's problems. It would be ironic if the introduction of XRay requires the developer to have this deep of knowledge of it's internals to diagnose and monitor its own problems. :(

[jamesdbowman]

@ceefour I understand your concern.

It would be possible for the SDK to automatically apply subsegment streaming to large segments. However, the segment size in bytes is not known until the segment JSON document has been serialized to a string - a relatively expensive operation.

X-Ray SDKs are designed to be as minimally invasive to application resources as possible. It was determined that multiple serialization passes on an object of unknown size would not meet this design goal. So, by default, the X-Ray SDKs are not equipped to conditionally stream segments based on payload size.

It would be possible however to subclass com.amazonaws.xray.strategyStreamingStrategy or even com.amazonaws.xray.emitters.Emitter to enable such functionality, if desired.