Can the span be made unmodifiable once finished?

[yurishkuro]

One user reported the exception below, which can only happen if span.log() is called concurrently with the span being processed by the reporter (i.e. after it was finished).

Exception in thread "jaeger.RemoteReporter-QueueProcessor" java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
        at java.util.ArrayList$Itr.next(ArrayList.java:851)
        at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042)
        at com.uber.jaeger.reporters.protocols.JaegerThriftSpanConverter.buildLogs(JaegerThriftSpanConverter.java:71)
        at com.uber.jaeger.reporters.protocols.JaegerThriftSpanConverter.convertSpan(JaegerThriftSpanConverter.java:53)
        at com.uber.jaeger.senders.ThriftSender.append(ThriftSender.java:66)
        at com.uber.jaeger.reporters.RemoteReporter$AppendCommand.execute(RemoteReporter.java:142)
        at com.uber.jaeger.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:175)
        at java.lang.Thread.run(Thread.java:748)

cc @dray92

[quaff]

Exception in thread "jaeger.RemoteReporter-QueueProcessor" java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
	at java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1661)
	at java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1654)
	at io.jaegertracing.reporters.protocols.JaegerThriftSpanConverter.buildTags(JaegerThriftSpanConverter.java:92)
	at io.jaegertracing.reporters.protocols.JaegerThriftSpanConverter.convertSpan(JaegerThriftSpanConverter.java:52)
	at io.jaegertracing.senders.ThriftSender.append(ThriftSender.java:54)
	at io.jaegertracing.reporters.RemoteReporter$AppendCommand.execute(RemoteReporter.java:137)
	at io.jaegertracing.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:170)
	at java.lang.Thread.run(Thread.java:748)

[jpkrohling]

@quaff do you have a sample code that would trigger this condition?

[quaff]

@jpkrohling It happens occasionally when debugging and the break point in span is blocked and later resumed.

[bygui86]

In my case it happens on both sides of a Kafka broker: after 6 successfully sent messages the exception appear in the producer, after 3 successfully received messages same exception in the consumer. This causes the tracing info to be lost, luckily there is no impact on Kafka communications or other part of the applications for the moment.
I'm working with spring-boot, spring-cloud-stream with kafka, opentracing-spring-jaeger-cloud-starter and opentracing-kafka-spring.
@jpkrohling I will tidy up my example and post the link today or tomorrow.

[bygui86]

@jpkrohling here is my repo with the example described in my previous comment:
https://github.com/bygui86/spring-cloud-stream-kafka-jaeger

Under the folder logs you can find the specific exception and the logs of the services.
Under the folder screenshots you can find how jaeger looks like during a test.

Actually the exception is the same of @quaff , I'm not using uber libraries like @yurishkuro.

I hope this can help :)
I will continue my tests to find out something more ;)

[bygui86]

@pavolloffay After some tests I found a workaround to let everything work without any Exception on concurrency. I added just the following line in my application.properties:
opentracing.spring.cloud.log.enabled = false

I know that this is just a workaround, but meanwhile you guys are fixing the issue, we can continue working on the project using the same technologies.

Everything is committed in my repo posted in the previous comment.
Cheers :)