Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent java.lang.IllegalStateException: Connection pool shut down on clean install #689

Closed
dannylamb opened this issue Aug 15, 2017 · 11 comments
Closed

Intermittent java.lang.IllegalStateException: Connection pool shut down on clean install #689

dannylamb opened this issue Aug 15, 2017 · 11 comments

Comments

@dannylamb
Copy link
Contributor

dannylamb commented Aug 15, 2017 •
edited
Loading

While testing the fixes required for #686, I ran into this every 10 or so times I set up Karaf with Api-X and Alpaca.

2017-08-15 11:55:43,966 | INFO  | pool-52-thread-3 | InitMgr                          | 124 - fcrepo-api-x-jena - 0.3.0.SNAPSHOT | Caught exception while initializing.  Trying again in 1000 ms
java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34)[54:org.apache.httpcomponents.httpcore:4.4.5]
	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:185)[54:org.apache.httpcomponents.httpcore:4.4.5]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:257)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.fcrepo.apix.registry.HttpClientFetcher$CloseableHttpClientProxy.doExecute(HttpClientFetcher.java:115)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)[53:org.apache.httpcomponents.httpclient:4.5.3]
	at org.fcrepo.apix.jena.impl.LdpContainerRegistry.exists(LdpContainerRegistry.java:280)
	at org.fcrepo.apix.jena.impl.LdpContainerRegistry.lambda$init$8(LdpContainerRegistry.java:131)
	at org.fcrepo.apix.jena.impl.InitMgr$1.lambda$$11(InitMgr.java:70)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

A quick googling leads me to believe that there's timing issues where Api-X needs to close connections, and that's shutting down the connection pool.

https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html#setConnectionManagerShared(boolean) seems to be a solution according to stack overflow https://stackoverflow.com/a/28646605.

@ajs6f Thoughts? I know we talked about whether or not we consider our custom http cllient to be shared when you were doing the initial work.
@ajs6f
Copy link

ajs6f commented Aug 15, 2017 •
edited
Loading

Hm. My first (really rough) thought is that we can triple-check that all CloseableHttpResponses are really getting closed (try-withresource FTW) and then that we might want to adjust any conneciton pooling. I'd really rather not be fooling around with manually closing connections or pools and we (in theory) shouldn't have to.

The HttpClient itself is definitely thread-safe and shareable-- that's part of its basic contract. Now, if we are mucking about with its underlying componentry (e.g. connection pools) we have to support that guarantee by doing what we are doing in a thread-safe shareable way...

@dannylamb
Copy link
Contributor Author

I'll file an issue with Api-X and see if we can sort this out.

@dannylamb dannylamb mentioned this issue Aug 15, 2017
Closed
@DiegoPino
Copy link
Contributor

PoolingHttpClientConnectionManager...

https://stackoverflow.com/questions/25889925/apache-poolinghttpclientconnectionmanager-throwing-illegal-state-exception ?

@dannylamb dannylamb mentioned this issue Aug 15, 2017
Closed
@ajs6f
Copy link

ajs6f commented Aug 15, 2017

In API-X mvn dependency:tree | grep httpcomponents gives 4.4.5 and 4.5.2, so not the bug. Possibly the code needs to be updated, which should probably come under the ticket @dannylamb filed. Otherwise, @DiegoPino are you thinking that the problem is in CLAW? Because that stacktrace is clearly from API-X.

@DiegoPino
Copy link
Contributor

@ajs6f yeah, the bug is prior 4.4, I see some settings and option to set the shared pool and i think it is something in API-X, but we(claw) are triggering it. So it is not our error, and agree with the sharing part

.setConnectionManagerShared(true)

@ajs6f
Copy link

ajs6f commented Aug 15, 2017

Cool, I will continue tracing through.

@DiegoPino
Copy link
Contributor

DiegoPino commented Aug 15, 2017 •
edited
Loading

@ajs6f is it https://github.com/fcrepo4-labs/fcrepo-api-x/blob/63022f47ec32e05e689e138220eee56aef713561/fcrepo-api-x-registry/src/main/java/org/fcrepo/apix/registry/HttpClientFactory.java#L117-L118? Can we tune the config there when we deploy CLAW + the shared client part?

(I´m on a meeting so distracted... sorry)

@ajs6f
Copy link

ajs6f commented Aug 15, 2017

@DiegoPino This is our injected client. I think we might need to adjust there.

@dannylamb
Copy link
Contributor Author

@DiegoPino Those were my initial thoughts too after reading the stack overflow page I posted in the issue description.

@DiegoPino
Copy link
Contributor

@dannylamb i missed your link, jajajaja. Same one i pasted...feel so silly 🤦‍♂️ so funny.

@dannylamb
Copy link
Contributor Author

@DiegoPino We're just on the same page, man. Literally. =)

ajs6f added a commit to ajs6f/Alpaca that referenced this issue Aug 15, 2017
@ajs6f
Islandora/documentation#689: Adding stronger pooling config for client
7c9194c
@ajs6f ajs6f mentioned this issue Aug 15, 2017
Merged
ajs6f added a commit to ajs6f/Alpaca that referenced this issue Aug 15, 2017
@ajs6f
Islandora/documentation#689: Adding stronger pooling config for client
bbbc864
ajs6f added a commit to ajs6f/Alpaca that referenced this issue Aug 15, 2017
@ajs6f
Islandora/documentation#689: Adding stronger pooling config for client
2e8a47c
dannylamb pushed a commit to Islandora/Alpaca that referenced this issue Aug 23, 2017
@ajs6f @dannylamb
Adding stronger pooling config for client (#42)
0906cc7 
* Islandora/documentation#689: Adding stronger pooling config for client

* Marking connection pool as shared to prevent premature shutdown
dannylamb pushed a commit to dannylamb/CLAW that referenced this issue Feb 8, 2018
@bryjbrown @DiegoPino
Adds check for invalid PIDs to islandora_object_load fails (Islandora…
9b37dcc 
…#689)

* Adds check for invalid PIDs to islandora_object_load fails

* Reordered invalid PID check to catch more invalidity cases

* Exploring the logical limits of Drupal Coding Style

* Removes watchdog logging when islandora_load_object gets a malformed PID
@littleaj littleaj mentioned this issue Mar 21, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ajs6f @DiegoPino @dannylamb