CLAW vagrant - created objects not being added to Fedora intermittently

[MarcusBarnes]

Noting some intermittent behavior I have seen on three different machines running CLAW vagrant in case others are experiencing the same behavior. What we are seeing intermittently is the following:

1. Create an object in the Drupal admin, for example, an Islandora Image.
2. Go to the Fedora4 interface http://localhost:8080/fcrepo/rest. Usually, you'll see the object there, but intermittently, the newly created objects from the Drupal side are not showing up there.
3. Going to ActiveMQ administrative interface, the newly created objects appear in the islandora-connector-broadcast queue, but are not processed further.

Any insight into what might be happening or how I may best provide additional details to help investigate the cause of this behavior? Thank you in advance.

[DiegoPino]

@MarcusBarnes do you have any fedora4 logs / Karaf logs around that you could share when that happens? If the queue is populated, then one possibility could (probably) that the resulting RDF (JSON-LD) is not being accepted by Fedora's API (we are doing all via PUT right @dannylamb @whikloj ?) which could mean some predicates or references fedora4 does not understand or that we are wrongly serializing. Could you share one of those Nodes? Also, it could be some issue with authentication, that some credentials get stuck? so many options with async.

[MarcusBarnes]

Thanks @DiegoPino. The next time this happens, I'll grab the Fedroa4 and Karaf logs.

[DiegoPino]

@MarcusBarnes thanks. It can be hard to debug for sure and more if it is intermittent. I would be very helpful for us if that ever happens again if you can share that with us. I feel that following the flow of data/ involved services and their logs can shine some light on that. And it could be an error of our code of course or some exceptional interaction between services also.

[dannylamb]

@MarcusBarnes If you hop into the Karaf console with /opt/karaf/bin/client and do a la | grep islandora inside it, are all of the bundles listed as 'Active'?

[dannylamb]

I've managed to narrow this down to

Bundle XXX is waiting for namespace handlers [http://camel.apache.org/schema/blueprint]

for

• fcrepo-api-x-routing
• fcrepo-api-x-loader
• fcrepo-api-x-listener
• fcrepo-indexing-triplestore
• islandora-indexing-fcrepo
• islandora-indexing-triplestore
• islandora-connector-broadcast

[dannylamb]

feature:uninstall fcrepo-api-x pulled the islandora features out of 'Grace Period' and into 'Active'.

feature:install fcrepo-api-x put everything back to 'Grace Period'. So there's something intermittent going on with our api-x install.

[dannylamb]

So I audited alpaca's install and found a few weird things.

First, I bumped activemq up to 5.14.4, so hopefully that'll fix the long standing issues we've been experiencing with it. If not, 5.15.0 is available and a possibility.

We were declaring activemq unneccessarily as hard dependencies when it's really optional. So that won't do anything for us now, but by removing them we're paving the way for other brokers.

I bumped fcrepo-camel-toolbox from 4.7.0 to 4.7.2.

And the big one I think is that we were allowing a version range for Camel, which I've locked down to just the version we're compiling against. I think the main issue was that we had that fuzzy range and there were two copies of camel-blueprint floating out there between alpaca and api-x, and intermittently karaf wasn't resolving to the right one. Hence the waiting on camel-blueprint messages I was getting from the logs.

I've tested this three or four times now, and will probably keep doing it on and off throughout the day to see if I can get the error to reproduce. If not, I'll issue a PR.

[dannylamb]

I'm occasionally running into #689 now, but will probably need to get that addressed in the Api-X codebase.

[dannylamb]

I've managed to slice the version bumps out of the other work it got tangled in with. I'm running a vagrant up now to see if this sorts it out once and for all. PR should be coming soon.

This should also help with Islandora-Devops/islandora-playbook#13

[jonathangreen]

I believe this is resolved by Islandora/Alpaca@2ea125c. Everyone involved happy if it's closed?

[DiegoPino]

+1 to close El El mar, 19 de sep. de 2017 a las 17:00, Jonathan Green < notifications@github.com> escribió:

I believe this is resolved by ***@***.*** <Islandora/Alpaca@2ea125c>. Everyone involved happy if it's closed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#686 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGn851oN4uLp1bEllsh7uUsdUiCq-64Lks5skCtMgaJpZM4Ow7Jf> .

-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)

[dannylamb]

Resolved via Islandora/Alpaca@2ea125c