Expose configuration property for Cassandra's control queries timeout

[logicatmidod]

While trying to use spring-data-cassandra with 2.3.5.RELEASE when trying to connect to cassandra getting below error

Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S

I have configured these properties still don't know why timeout is considered to be PT2S

spring:
  profiles:
    active: mockdata,standalone
  data:
    cassandra:
      request:
        timeout: 120000
      connect-timeout-millis: 10000
      read-timeout-millis: 10000
      connect-timeout: 10000
      read-timeout: 10000
      pool:
        pool-timeout: 10000
      connection:
        connect-timeout: 120000
        init-query-timeout: 120000

Also tried to use DriverConfigLoaderBuilderCustomizer as specified here https://stackoverflow.com/questions/61332483/spring-boot-2-3-0-m4-cassandra-and-ssl but not able to overwrite the time
but cant see DriverConfigLoaderBuilderCustomizer Bean being invoked at all any config I am missing that needs to be done.

[snicoll]

@logicatmidod all those properties with -millis at the end do not exist. I can only guess you were trying to make this work. If you have an IDE that has Spring Boot integration it should do a much better job at hinting you to that.

The property that you're looking to tune for the exception above is spring.data.cassandra.request.timeout. I can see it being set but that exception seems to indicate that it hasn't been. It is impossible for me to figure out why that is with the information you've shared.

If you want support, please take the time to build a small sample that we can run to reproduce the problem. You can do so by attaching a zip to this issue or sharing a link to a GitHub repository.

[logicatmidod]

I solved this issue by setting

advanced.control-connection.timeout

by setting manually at Datastax Driver Option this property was not exposed at spring boot side.
spring.data.cassandra.request.timeout is already set initially when it was not set I was facing some other timeout error not this one

[snicoll]

Thanks for the feedback. Timeout for control queries, I see.

This is worth exposing I guess although it's not immediately obvious from the exception you've shared that advanced.control-connection.timeout is configuring this. Paging @adutra in case there's something else we should know.

[smakalias]

I solved this issue by setting
advanced.control-connection.timeout
by setting manually at Datastax Driver Option this property was not exposed at spring boot side.

Hi @logicatmidod , I'm facing the same issue when running migrations (we use https://github.com/patka/cassandra-migration) - the issue is really weird, it seems to occur randomly in PRD installations (either low-data-load or heavy-data-load) and I cannot replicate locally).

Could you please elaborate how did you actually set the advanced.control-connection.timeout value so I can rule out things?
Also, is there a clear way to know if this value is picked up or Spring Boot overrides it?

Thanks in advance for your response and help

[snicoll]

@smakalias this is described in the documentation, something like:

@Bean
DriverConfigLoaderBuilderCustomizer cassandraDriverCustomizer() {
    return (builder) -> builder.withDuration(
        DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, Duration.ofSeconds(10));
}

Spring Boot is not going to override anything you've specified yourself. You can also inject the DriverConfigLoader and look at its config. Going forward please ask questions on StackOverflow, as mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.

[adutra]

Sorry for missing the debate. I think it makes sense to expose the control connection timeout, although it remains something of an advanced setting.

@snicoll I would however raise your attention to the timeout defaults used in Spring Boot:

• connect timeout
• init query timeout
• request timeout
• etc.

Having such defaults hardcoded here is worrisome because they could get out of sync of the actual driver defaults. In fact it happened already: since driver 4.8 the init query timeout was raised to 5 seconds, see JAVA-2841 for details.

If it is possible to set the defaults by reading the default driver configuration, that would be better.

Hint: as I said elsewhere, the OptionsMap API could solve this, there is a driverDefaults() method that could come in handy here.

[wilkinsona]

Thanks for brining the misalignment to our attention, @adutra.

If it is possible to set the defaults by reading the default driver configuration, that would be better.

We can't do this without losing the default values in our configuration property metadata. This metadata is used by IDEs to provide auto-completion and property descriptions when editing application.properties and application.yaml files.

I've opened #24965 to add some tests to verify that things are in sync.

[rajeevshukla]

Hello,

I know this thread is closed, but I don't know if I should create a separate thread to discuss the same problem.
I am stuck to the same problem for a week, did lots of research/StackOverflow, and not able to resolve this issue.

I used below configuration properties below -

spring.data.cassandra.contact-points=xxxxxx
spring.data.cassandra.username=xxxx
spring.data.cassandra.password=xxxxx
spring.data.cassandra.keyspace-name=xxxx
spring.data.cassandra.port=9042
spring.data.cassandra.schema-action=NONE
spring.data.cassandra.local-datacenter=mydc
spring.data.cassandra.connection.connect-timeout=PT10S
spring.data.cassandra.connection.init-query-timeout=PT20S
spring.data.cassandra.request.timeout=PT10S

#checking to see if DataStax props can be picked from here -- I believe no. 

datastax-java-driver.basic.request.timeout = 10 seconds
datastax-java-driver.advanced.connection.init-query-timeout = 10 seconds
datastax-java-driver.advanced.control-connection.timeout = 10 seconds

Below is the configuration I used as suggested in the post -

@Bean
	DriverConfigLoaderBuilderCustomizer cassandraDriverCustomizer() {
		return (builder) -> builder.withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT,
				Duration.ofSeconds(30));
	}

But I still getting the same error
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S

I also tried different approached like creating custom CqlSessionFactoryBean

@Bean(name = "session")
	@Primary
	public CqlSessionFactoryBean cassandraSession() {
		CqlSessionFactoryBean factory = new CqlSessionFactoryBean();
		factory.setUsername(userName);
		factory.setPassword(password);
		factory.setPort(port);
		factory.setKeyspaceName(keyspaceName);
		factory.setContactPoints(contactPoints);
		factory.setLocalDatacenter(dataCenter);
		factory.setSessionBuilderConfigurer(getSessionBuilderConfigurer()); // my session builder configurer 
		return factory;
	}

// And provided my own SessionBuilder Configurer like below

protected SessionBuilderConfigurer getSessionBuilderConfigurer() {
  	return new SessionBuilderConfigurer() {

  		@Override
  		public CqlSessionBuilder configure(CqlSessionBuilder cqlSessionBuilder) {
  			ProgrammaticDriverConfigLoaderBuilder config = DriverConfigLoader.programmaticBuilder()
  					.withDuration(DefaultDriverOption.CONNECTION_INIT_QUERY_TIMEOUT, Duration.ofSeconds(30))
  					.withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true)
  					.withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(30))
  					.withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, Duration.ofSeconds(20));
  			return cqlSessionBuilder.withAuthCredentials(userName, password).withConfigLoader(config.build());
  		}
  	};
  }

It didn't work same error. Also, I excluded the Cassandra auto-configuration classes like suggested here on StackOverflow

I also tried to customize custom session builder like below to see if that can work -

@Bean
public CqlSessionBuilderCustomizer cqlSessionBuilderCustomizer() {
		return cqlSessionBuilder -> cqlSessionBuilder.withAuthCredentials(userName, password)
				.withConfigLoader(DriverConfigLoader.programmaticBuilder()
						.withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofMillis(15000))
						.withDuration(DefaultDriverOption.CONNECTION_INIT_QUERY_TIMEOUT, Duration.ofSeconds(30))
						.withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true)
						.withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(30))
						.withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, Duration.ofSeconds(20)).build());
	}

Still no luck.

Not only that I also added application.conf file as DataStax documentation suggested putting that on the classpath, even though that file is getting parsed (after making the syntactical mistake I got to know that it is being read). It didn't work.

datastax-java-driver {
 basic.request.timeout = 10 seconds
 advanced.connection.init-query-timeout = 10 seconds
 advanced.control-connection.timeout = 10 seconds
}

I also switched my spring boot version to 2.5.0.M3 to see property files works it does not. I spent 4-5 days on it. Any help would be appreciated.

I have pushed my project to my GitHub account. Please take a look here.

Thank you in advance.

[wilkinsona]

@rajeevshukla Thanks for getting in touch, but it feels like this is a question that would be better suited to Stack Overflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements. Feel free to update this issue with a link to the re-posted question (so that other people can find it) or open a new issue if you feel this is a genuine bug.

[rajeevshukla]

@wilkinsona Thank you. I have posted the same question on StackOverflow here. I feel like this is a genuine bug as application properties are still not in effect after upgrading my version to spring boot 2.5.0.M3. I still see values are not being overridden. But I would do that if don't get any response from stackoverflow.