ConnectionError when doing collection dump to Elasticsearch

[Garito]

Hi!
Think it's needed to ajust the bulk insertion size. When the elasticsearch is left behind by too much, the bulk insertion fails. Here is the trackback:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 490, in do_dump
    upsert_all(dm)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 474, in upsert_all
    dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 38, in wrapped
    reraise(new_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 32, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic_doc_manager.py", line 189, in bulk_upsert
    for ok, resp in responses:
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 118, in streaming_bulk
    raise e
ConnectionFailed: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))

Thanks

[Garito]

Seems that is not a matter of the optlog too big.
Now everytime I stop the cluster and restart it mongo-connector fails (with the error above)

Any clue? Thanks!

[Garito]

Anyone maintaining this repo?

[llvtt]

This looks like an error on the Elasticsearch end, since Mongo Connector can't even connect to Elasticsearch. Can you reach Elasticsearch with a web browser or using a tool like curl? You might have more luck getting a solution to this problem on the Elasticsearch Google Group.

[Garito]

Sorry but it only fails when the replicaset is not empty
So, now that I'm in a very early stage of the development and I haven't too much data I can delete the collection, restart the mongo-connector and it works ok

So, seems to be a mongo-connector issue, don't you think so?

[llvtt]

Mongo Connector doesn't establish a connection to Elasticsearch until it needs to do something. If there's no data in the replica set, then there's nothing for Mongo Connector to do. This explains why you only see this Exception after there's any data in MongoDB. The error message "connection refused" leads me to believe that there's some kind of network or configuration problem on or around the Elasticsearch end.

[Garito]

So how do you explain that after I delete the database all works ok?
Seems more to me that the mongo-connector try to follow up the optlog since the beginning not since the last change

[Garito]

Hi!
I really need some help here...

[llvtt]

@Garito
The connector keeps track of its progress in the oplog through the oplog timestamp file. However, it seems that your problem has to do with connecting to Elasticsearch. Can you please confirm that you can reach Elasticsearch from the machine on which you are running mongo-connector? If you cannot, the problem is more fundamental than Mongo Connector. Again, a ConnectionError is pretty generic; the folks over at Elasticsearch probably have a more detailed list of items to troubleshoot for this problem than I can, so I strongly encourage you to ask a question here: https://groups.google.com/forum/#!forum/elasticsearch

[Garito]

I confirm you 100%
Here is what I do:
I start all the services without any data It works just fine
I create the first document so the database and the collection is created and the index too. Everything is fine
I stop all the services
Restart them. mongo-connector fails with the connection error. No matter how many times I try: it fails
With the mongo-connector down but the rest of the services up, I delete the collection on the mongodb. Obviously the elasticsearch index is not deleted as expected
I stop all the services
And restart. All just fine, mongo-connector included
I create a document and it is replicated to elastic
I stop the services
And restart. Mongo-connector fails again
Repeat if you like, everything will work like explained

So I need to understand that I have no problem with the connections to my services
Do you agree?

[llvtt]

@Garito
How are you starting and restarting services? Are you using the Linux service script? Is it possible that mongo-connector is starting before Elasticsearch, so that it's attempting to connect before Elasticsearch is ready to accept connections? I.e.:

sudo service mongodb start
# Operations sent to MongoDB at this point cannot be sent to Elasticsearch yet,
# since it's not yet running.
sudo service mongo-connector start
sudo service elasticsearch start

If this is the problem, then you want something like this instead:

sudo service mongodb start
sudo service elasticsearch start
# Need to ensure that both MongoDB and Elasticsearch are ready to accept connections.
for attempt in `seq 60`; do if curl localhost:9200; then break; else sleep 1; fi; done
sudo service mongo-connector start

[Garito]

No, that's not the case for sure: I'm using docker with docker-compose and links from mongo-connector container to the mongo and the elastic ones so docker takes care to launch them in the correct order

[Garito]

Hi!
Any progress or idea about this issue?
Thanks!

[Garito]

Hi!
I've changed how Elasticsearch is configured to connect with the outside world and the error has changed to:

ConnectionFailed: ConnectionError(('Connection aborted.', BadStatusLine("''",))) caused by: ProtocolError(('Connection aborted.', BadStatusLine("''",)))

Any ideas?
Thanks!

[Garito]

Sorry was the same error (My need to solve this makes me see ghosts)

[Garito]

Hi!
I'm in the last stage of my infrastructure development, the only thing stops me going on is this issue
As I told here, I can reach the elasticsearch without problems from the mongo-connector container by doing wget http://es:9200 so there is no connection issues to my ES server

I really need some help here or change the mechanism for replication, so What is going to be?

[Garito]

I think I figured out what's happening:
I'm running my cluster in a docker infrastructure with docker-compose who takes care to lunch the containers in order to prevent linking problems
But when mongo-connector starts, if there is an optlog, as @llvtt point above, it trys to update the ES but this server is booting and it doesn't start the REST API so mongo-connector fails like a champ (as you could imagine)

So the question will become: is it possible to specify a timeout time to allow ES to launch before raise the error? If there is not, I must to add that docker is here to stay so would be a must have feature if anyone wants to use the connector. Perhaps an enhancement?

What do you think?

[llvtt]

@Garito That sounds like a plausible explanation to me. Try adding a sleep or something similar to this comment at the beginning of the mongo-connector init script/end of the elasticsearch init script.

[Garito]

Yeah, I saw that before. The problem with this is I'm using docker and since this issue will be a pattern I prefer to check first if docker has already solved it in a canonical way (I'll posted here if I find it before to close the issue if you are ok with that)

[llvtt]

Sounds good to me. If you let me know how you fix this in the end, I'll add
it to the FAQ.

On Thu, Jun 18, 2015 at 10:19 AM, Garito notifications@github.com wrote:

Yeah, I saw that before. The problem with this is I'm using docker and
since this issue will be a pattern I prefer to check first if docker has
already solved it in a canonical way (I'll posted here if I find it before
to close the issue if you are ok with that)

—
Reply to this email directly or view it on GitHub
#251 (comment)
.

[Garito]

It turns out I'm an idiot and this issue is as easy as add this to the docker-compose yml for the container:
restart: always
or if you are using only docker add --restart="always" and the mongo-connector container will retry to start up several times so ES will have time to start properly

Sorry for being a pain in the ass and thanks for your patience

[llvtt]

Glad you got it working!

On Fri, Jun 19, 2015 at 9:57 AM, Garito notifications@github.com wrote:

Closed #251 #251.

—
Reply to this email directly or view it on GitHub
#251 (comment)
.

[impanda-cookie]

Same issues.
if above solution can't be solve your question, you can try:

• decrease mongo-connector config.jsondocManagers.bulkSize ;
• reset mongo-connector config.json docManagers.args.clientOptions.timeout / docManagers.autoCommitInterval for a long time;
• upgrade elasticsearch env "ES_JAVA_OPTS=-Xms2g -Xms2g" or more ...

restart elasticsearch / mongo-connector