Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Starting api-umbrella FAIL #100

Closed
dmolina-ot opened this issue Feb 5, 2015 · 9 comments
Closed

Starting api-umbrella FAIL #100

dmolina-ot opened this issue Feb 5, 2015 · 9 comments
Milestone
v0.7

Comments

@dmolina-ot
Copy link

Hi!
I have problems starting API-Umbrella in Ubuntu 14.04. I think the problem is that mongo-db is not active when another modules (gatekeeper, distributed-rate-limits-sync, ...) need it.

I send you some relevant logs, if you need more, you can ask for them.

sudo /etc/init.d/api-umbrella start
Starting api-umbrella............... [FAIL]

Failed to start processes:
distributed-rate-limits-sync (FATAL - /opt/api-umbrella/var/log/supervisor/distributed-rate-limits-sync.log)
gatekeeper1 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper1.log)
gatekeeper2 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper2.log)
gatekeeper3 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper3.log)
gatekeeper4 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper4.log)
web-delayed-job (FATAL - /opt/api-umbrella/var/log/supervisor/web-delayed-job.log)

See /opt/api-umbrella/var/log/supervisor/supervisord_forever.log for more details

Stopping api-umbrella... [ OK ]
error: Forever detected script exited with code: 1
error: Script restart attempt #1
Starting api-umbrella............... [FAIL]

Failed to start processes:
distributed-rate-limits-sync (FATAL - /opt/api-umbrella/var/log/supervisor/distributed-rate-limits-sync.log)
gatekeeper1 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper1.log)
gatekeeper2 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper2.log)
gatekeeper3 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper3.log)
gatekeeper4 (FATAL - /opt/api-umbrella/var/log/supervisor/gatekeeper4.log)
log-processor (FATAL - /opt/api-umbrella/var/log/supervisor/log-processor.log)
web-delayed-job (FATAL - /opt/api-umbrella/var/log/supervisor/web-delayed-job.log)

See /opt/api-umbrella/var/log/supervisor/supervisord_forever.log for more details

Stopping api-umbrella... [ OK ]
error: Forever detected script exited with code: 1

supervisord_forever.log:
2015-02-05 09:36:43,435 CRIT Supervisor running as root (no user in config file)
2015-02-05 09:36:43,441 INFO RPC interface 'supervisor' initialized
2015-02-05 09:36:43,441 INFO RPC interface 'laforge' initialized
2015-02-05 09:36:43,442 CRIT Server 'inet_http_server' running without any HTTP authentication checking
Unlinking stale socket /opt/api-umbrella/var/run/supervisor.sock
2015-02-05 09:36:43,742 INFO RPC interface 'supervisor' initialized
2015-02-05 09:36:43,742 INFO RPC interface 'laforge' initialized
2015-02-05 09:36:43,743 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-02-05 09:36:43,743 INFO supervisord started with pid 5795
2015-02-05 09:36:43,744 INFO spawned: 'router-log-listener' with pid 5798
2015-02-05 09:36:43,745 INFO spawned: 'gatekeeper2' with pid 5799
2015-02-05 09:36:43,746 INFO spawned: 'gatekeeper3' with pid 5800
2015-02-05 09:36:43,747 INFO spawned: 'gatekeeper1' with pid 5801
2015-02-05 09:36:43,748 INFO spawned: 'gatekeeper4' with pid 5802
2015-02-05 09:36:43,750 INFO spawned: 'config-reloader' with pid 5803
2015-02-05 09:36:43,751 INFO spawned: 'varnishd' with pid 5805
2015-02-05 09:36:43,753 INFO spawned: 'web-delayed-job' with pid 5807
2015-02-05 09:36:43,754 INFO spawned: 'log-processor' with pid 5810
2015-02-05 09:36:43,756 INFO spawned: 'mongod' with pid 5811
2015-02-05 09:36:43,757 INFO spawned: 'redis' with pid 5813
2015-02-05 09:36:43,759 INFO spawned: 'distributed-rate-limits-sync' with pid 5815
2015-02-05 09:36:43,761 INFO spawned: 'web-nginx' with pid 5816
2015-02-05 09:36:43,762 INFO spawned: 'router-nginx' with pid 5817
2015-02-05 09:36:43,764 INFO spawned: 'varnishncsa' with pid 5818
2015-02-05 09:36:43,766 INFO spawned: 'elasticsearch' with pid 5819
2015-02-05 09:36:43,768 INFO spawned: 'web-puma' with pid 5820
2015-02-05 09:36:44,806 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:36:44,853 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:36:44,921 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:36:44,999 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:36:45,194 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:36:45,634 INFO exited: log-processor (exit status 1; not expected)
2015-02-05 09:36:45,896 INFO spawned: 'gatekeeper4' with pid 6139
2015-02-05 09:36:45,898 INFO spawned: 'distributed-rate-limits-sync' with pid 6140
2015-02-05 09:36:45,928 INFO spawned: 'gatekeeper2' with pid 6143
2015-02-05 09:36:46,029 INFO spawned: 'gatekeeper1' with pid 6145
2015-02-05 09:36:46,285 INFO spawned: 'gatekeeper3' with pid 6147
2015-02-05 09:36:46,699 INFO spawned: 'log-processor' with pid 6149
2015-02-05 09:36:46,769 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:36:46,824 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:36:47,079 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:36:47,166 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:36:47,332 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:36:47,577 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:36:47,997 INFO exited: log-processor (exit status 1; not expected)
2015-02-05 09:36:48,586 INFO spawned: 'web-delayed-job' with pid 6219
2015-02-05 09:36:48,808 INFO success: varnishd entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:36:48,808 INFO success: redis entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:36:48,809 INFO spawned: 'distributed-rate-limits-sync' with pid 6222
2015-02-05 09:36:48,809 INFO success: web-nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:36:48,809 INFO success: router-nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:36:48,809 INFO success: varnishncsa entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:36:48,900 INFO spawned: 'gatekeeper2' with pid 6224
2015-02-05 09:36:49,151 INFO spawned: 'gatekeeper4' with pid 6227
2015-02-05 09:36:49,178 INFO spawned: 'gatekeeper1' with pid 6229
2015-02-05 09:36:49,227 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:36:49,390 INFO spawned: 'gatekeeper3' with pid 6235
2015-02-05 09:36:49,615 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:36:49,713 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:36:49,881 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:36:50,068 INFO spawned: 'log-processor' with pid 6249
2015-02-05 09:36:50,235 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:36:51,465 INFO exited: log-processor (exit status 1; not expected)
2015-02-05 09:36:51,528 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:36:52,283 INFO spawned: 'distributed-rate-limits-sync' with pid 6284
2015-02-05 09:36:52,616 INFO spawned: 'gatekeeper2' with pid 6286
2015-02-05 09:36:52,638 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:36:52,724 INFO spawned: 'gatekeeper4' with pid 6292
2015-02-05 09:36:52,724 INFO gave up: distributed-rate-limits-sync entered FATAL state, too many start retries too quickly
2015-02-05 09:36:52,948 INFO spawned: 'gatekeeper1' with pid 6295
2015-02-05 09:36:53,127 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:36:53,174 INFO gave up: gatekeeper2 entered FATAL state, too many start retries too quickly
2015-02-05 09:36:53,251 INFO spawned: 'gatekeeper3' with pid 6305
2015-02-05 09:36:53,253 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:36:53,286 INFO gave up: gatekeeper4 entered FATAL state, too many start retries too quickly
2015-02-05 09:36:53,461 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:36:53,522 INFO gave up: gatekeeper1 entered FATAL state, too many start retries too quickly
2015-02-05 09:36:53,539 INFO spawned: 'web-delayed-job' with pid 6311
2015-02-05 09:36:53,755 INFO success: router-log-listener entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:36:53,755 INFO success: config-reloader entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:36:53,755 INFO success: mongod entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:36:53,771 INFO success: elasticsearch entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:36:53,771 INFO success: web-puma entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:36:53,775 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:36:53,873 INFO gave up: gatekeeper3 entered FATAL state, too many start retries too quickly
2015-02-05 09:36:54,477 INFO spawned: 'log-processor' with pid 6326
2015-02-05 09:36:55,610 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:36:58,612 INFO spawned: 'web-delayed-job' with pid 6349
2015-02-05 09:37:00,460 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:37:00,482 INFO gave up: web-delayed-job entered FATAL state, too many start retries too quickly
2015-02-05 09:37:04,551 INFO success: log-processor entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:37:09,273 CRIT Supervisor running as root (no user in config file)
2015-02-05 09:37:09,280 INFO RPC interface 'supervisor' initialized
2015-02-05 09:37:09,280 INFO RPC interface 'laforge' initialized
2015-02-05 09:37:09,280 CRIT Server 'inet_http_server' running without any HTTP authentication checking
Unlinking stale socket /opt/api-umbrella/var/run/supervisor.sock
2015-02-05 09:37:09,581 INFO RPC interface 'supervisor' initialized
2015-02-05 09:37:09,581 INFO RPC interface 'laforge' initialized
2015-02-05 09:37:09,581 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-02-05 09:37:09,581 INFO supervisord started with pid 6376
2015-02-05 09:37:09,583 INFO spawned: 'router-log-listener' with pid 6379
2015-02-05 09:37:09,584 INFO spawned: 'gatekeeper2' with pid 6380
2015-02-05 09:37:09,585 INFO spawned: 'gatekeeper3' with pid 6381
2015-02-05 09:37:09,586 INFO spawned: 'gatekeeper1' with pid 6382
2015-02-05 09:37:09,587 INFO spawned: 'gatekeeper4' with pid 6383
2015-02-05 09:37:09,589 INFO spawned: 'config-reloader' with pid 6384
2015-02-05 09:37:09,590 INFO spawned: 'varnishd' with pid 6386
2015-02-05 09:37:09,592 INFO spawned: 'web-delayed-job' with pid 6388
2015-02-05 09:37:09,593 INFO spawned: 'log-processor' with pid 6390
2015-02-05 09:37:09,595 INFO spawned: 'mongod' with pid 6393
2015-02-05 09:37:09,597 INFO spawned: 'redis' with pid 6394
2015-02-05 09:37:09,599 INFO spawned: 'distributed-rate-limits-sync' with pid 6395
2015-02-05 09:37:09,600 INFO spawned: 'web-nginx' with pid 6396
2015-02-05 09:37:09,602 INFO spawned: 'router-nginx' with pid 6397
2015-02-05 09:37:09,604 INFO spawned: 'varnishncsa' with pid 6398
2015-02-05 09:37:09,607 INFO spawned: 'elasticsearch' with pid 6402
2015-02-05 09:37:09,612 INFO spawned: 'web-puma' with pid 6404
2015-02-05 09:37:10,578 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:37:10,776 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:37:10,779 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:37:10,932 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:37:11,023 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:37:11,330 INFO exited: log-processor (exit status 1; not expected)
2015-02-05 09:37:11,596 INFO spawned: 'distributed-rate-limits-sync' with pid 6720
2015-02-05 09:37:11,848 INFO spawned: 'gatekeeper3' with pid 6722
2015-02-05 09:37:11,850 INFO spawned: 'gatekeeper4' with pid 6723
2015-02-05 09:37:11,984 INFO spawned: 'gatekeeper2' with pid 6726
2015-02-05 09:37:12,111 INFO spawned: 'gatekeeper1' with pid 6728
2015-02-05 09:37:12,233 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:37:12,378 INFO spawned: 'log-processor' with pid 6734
2015-02-05 09:37:12,869 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:37:13,028 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:37:13,094 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:37:13,152 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:37:13,591 INFO exited: log-processor (exit status 1; not expected)
2015-02-05 09:37:14,200 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:37:14,301 INFO spawned: 'distributed-rate-limits-sync' with pid 6759
2015-02-05 09:37:14,636 INFO success: varnishd entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:37:14,636 INFO success: redis entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:37:14,636 INFO success: web-nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:37:14,636 INFO success: router-nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:37:14,636 INFO success: varnishncsa entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2015-02-05 09:37:14,685 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:37:14,966 INFO spawned: 'gatekeeper3' with pid 6805
2015-02-05 09:37:15,400 INFO spawned: 'gatekeeper1' with pid 6807
2015-02-05 09:37:15,401 INFO spawned: 'gatekeeper4' with pid 6808
2015-02-05 09:37:15,402 INFO spawned: 'web-delayed-job' with pid 6809
2015-02-05 09:37:15,408 INFO spawned: 'gatekeeper2' with pid 6810
2015-02-05 09:37:15,467 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:37:15,632 INFO spawned: 'log-processor' with pid 6819
2015-02-05 09:37:16,174 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:37:16,288 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:37:16,288 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:37:16,616 INFO exited: log-processor (exit status 1; not expected)
2015-02-05 09:37:17,695 INFO spawned: 'distributed-rate-limits-sync' with pid 6858
2015-02-05 09:37:17,696 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:37:18,130 INFO exited: distributed-rate-limits-sync (exit status 1; not expected)
2015-02-05 09:37:18,135 INFO gave up: distributed-rate-limits-sync entered FATAL state, too many start retries too quickly
2015-02-05 09:37:18,566 INFO spawned: 'gatekeeper3' with pid 6868
2015-02-05 09:37:19,046 INFO exited: gatekeeper3 (exit status 1; not expected)
2015-02-05 09:37:19,115 INFO gave up: gatekeeper3 entered FATAL state, too many start retries too quickly
2015-02-05 09:37:19,226 INFO spawned: 'gatekeeper4' with pid 6874
2015-02-05 09:37:19,337 INFO spawned: 'gatekeeper2' with pid 6876
2015-02-05 09:37:19,338 INFO spawned: 'gatekeeper1' with pid 6877
2015-02-05 09:37:19,593 INFO success: router-log-listener entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:37:19,593 INFO success: config-reloader entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:37:19,694 INFO spawned: 'log-processor' with pid 6880
2015-02-05 09:37:19,694 INFO success: mongod entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:37:19,694 INFO success: elasticsearch entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:37:19,694 INFO success: web-puma entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-02-05 09:37:19,699 INFO spawned: 'web-delayed-job' with pid 6881
2015-02-05 09:37:19,756 INFO exited: gatekeeper4 (exit status 1; not expected)
2015-02-05 09:37:19,815 INFO gave up: gatekeeper4 entered FATAL state, too many start retries too quickly
2015-02-05 09:37:19,901 INFO exited: gatekeeper2 (exit status 1; not expected)
2015-02-05 09:37:19,928 INFO gave up: gatekeeper2 entered FATAL state, too many start retries too quickly
2015-02-05 09:37:20,471 INFO exited: gatekeeper1 (exit status 1; not expected)
2015-02-05 09:37:20,471 INFO gave up: gatekeeper1 entered FATAL state, too many start retries too quickly
2015-02-05 09:37:20,577 INFO exited: log-processor (exit status 1; not expected)
2015-02-05 09:37:20,589 INFO gave up: log-processor entered FATAL state, too many start retries too quickly
2015-02-05 09:37:21,780 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:37:24,855 INFO spawned: 'web-delayed-job' with pid 6921
2015-02-05 09:37:26,698 INFO exited: web-delayed-job (exit status 1; not expected)
2015-02-05 09:37:26,740 INFO gave up: web-delayed-job entered FATAL state, too many start retries too quickly

distributed-rate-limits-sync.log:
[Thu Feb 05 2015 09:36:44 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:44 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:36:46 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:46 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:36:49 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:49 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:36:52 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:52 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:10 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:10 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:12 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:12 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:14 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:14 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:18 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:18 GMT+0100 (CET)] ERROR Distributed rate limits sync connections error: [Error: failed to connect to [127.0.0.1:14001]]

gatekeeper1.log:
[Thu Feb 05 2015 09:36:44 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:44 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:36:47 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:47 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:36:49 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:49 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:36:53 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:36:53 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:11 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:11 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:13 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:13 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:16 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:16 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]
[Thu Feb 05 2015 09:37:19 GMT+0100 (CET)] ERROR Mongo event: disconnected undefined
[Thu Feb 05 2015 09:37:19 GMT+0100 (CET)] ERROR Gatekeeper worker connections error: [Error: failed to connect to [127.0.0.1:14001]]

mongodb.log:
2015-02-05T09:36:43.811+0100 [initandlisten] MongoDB starting : pid=5811 port=14001 dbpath=/opt/api-umbrella/var/db/mongodb 64-bit host=PR096
2015-02-05T09:36:43.811+0100 [initandlisten] db version v2.6.5
2015-02-05T09:36:43.811+0100 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
2015-02-05T09:36:43.811+0100 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-02-05T09:36:43.811+0100 [initandlisten] allocator: tcmalloc
2015-02-05T09:36:43.811+0100 [initandlisten] options: { config: "/opt/api-umbrella/etc/mongod.conf", net: { port: 14001 }, storage: { dbPath: "/opt/api-umbrella/var/db/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/opt/api-umbrella/var/log/mongod.log", quiet: true } }
2015-02-05T09:36:43.826+0100 [initandlisten] journal dir=/opt/api-umbrella/var/db/mongodb/journal
2015-02-05T09:36:43.826+0100 [initandlisten] recover begin
2015-02-05T09:36:43.826+0100 [initandlisten] recover lsn: 0
2015-02-05T09:36:43.826+0100 [initandlisten] recover /opt/api-umbrella/var/db/mongodb/journal/j._0
2015-02-05T09:36:43.833+0100 [initandlisten] recover cleaning up
2015-02-05T09:36:43.833+0100 [initandlisten] removeJournalFiles
2015-02-05T09:36:44.427+0100 [initandlisten] recover done
2015-02-05T09:36:44.427+0100 [initandlisten] preallocating a journal file /opt/api-umbrella/var/db/mongodb/journal/prealloc.0
2015-02-05T09:36:47.034+0100 [initandlisten] File Preallocator Progress: 713031680/1073741824 66%
2015-02-05T09:36:55.135+0100 [initandlisten] waiting for connections on port 14001
2015-02-05T09:37:09.636+0100 ***** SERVER RESTARTED *****
2015-02-05T09:37:09.651+0100 [initandlisten] MongoDB starting : pid=6393 port=14001 dbpath=/opt/api-umbrella/var/db/mongodb 64-bit host=PR096
2015-02-05T09:37:09.651+0100 [initandlisten] db version v2.6.5
2015-02-05T09:37:09.651+0100 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
2015-02-05T09:37:09.651+0100 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-02-05T09:37:09.651+0100 [initandlisten] allocator: tcmalloc
2015-02-05T09:37:09.651+0100 [initandlisten] options: { config: "/opt/api-umbrella/etc/mongod.conf", net: { port: 14001 }, storage: { dbPath: "/opt/api-umbrella/var/db/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/opt/api-umbrella/var/log/mongod.log", quiet: true } }
2015-02-05T09:37:09.661+0100 [initandlisten] journal dir=/opt/api-umbrella/var/db/mongodb/journal
2015-02-05T09:37:09.661+0100 [initandlisten] recover begin
2015-02-05T09:37:09.661+0100 [initandlisten] recover lsn: 0
2015-02-05T09:37:09.661+0100 [initandlisten] recover /opt/api-umbrella/var/db/mongodb/journal/j._0
2015-02-05T09:37:09.668+0100 [initandlisten] recover cleaning up
2015-02-05T09:37:09.668+0100 [initandlisten] removeJournalFiles
2015-02-05T09:37:09.893+0100 [initandlisten] recover done
2015-02-05T09:37:09.893+0100 [initandlisten] preallocating a journal file /opt/api-umbrella/var/db/mongodb/journal/prealloc.0
2015-02-05T09:37:12.062+0100 [initandlisten] File Preallocator Progress: 660602880/1073741824 61%
2015-02-05T09:37:15.400+0100 [initandlisten] File Preallocator Progress: 985661440/1073741824 91%
2015-02-05T09:37:20.730+0100 [initandlisten] waiting for connections on port 14001

Thanks
@darylrobbins
Copy link
Contributor

How did you install API Umbrella? From the official package or master?

If you're deploying the master, you are probably running into #98. The master branch is temporarily unstable as the developer is preparing for a new release this weekend.

If you're using the package, can you please provide your /etc/api-umbrella/api-umbrella.yml file? All you should require is:

web:
  admin:
    initial_superusers:
      - (your email here)

@darylrobbins
Copy link
Contributor

Also, if you are installing from the package, how much RAM do you have available? You are going to need around 2 Gb for a single node API Umbrella install.

@dmolina-ot
Copy link
Author

I installed API-Umbrella from official package. I have 8Gb of RAM!

/etc/api-umbrella/api-umbrella.yml:

# services:
# - general_db
# - log_db
# - router
# - web
web:
admin:
initial_superusers:
- dmolina@opentrends.net
- imi.apimanager.test@gmail.com
auth_strategies:
enabled:
- github
# - google
- persona
github:
client_id:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
client_secret:xxxxxxxxxxxxxxxxxxxxxxxxxx
# google:
# client_id:
# client_secret:
# elasticsearch:
# hosts:
# - http://127.0.0.1:14002
# mongodb:
# url: mongodb://127.0.0.1:14001/api_umbrella
# nginx:
# workers: 4
# gatekeeper:
# workers: 4

@dmolina-ot dmolina-ot reopened this Feb 5, 2015
@darylrobbins
Copy link
Contributor

@GUI Care to weigh in?

@dmolina-ot
Copy link
Author

What?

@GUI
Copy link
Member

GUI commented Feb 5, 2015

I believe this is a race condition on startup. It should be fixed in the upcoming v0.7 packages that should be released this weekend, but I'm unfortunately not sure if there's an easy workaround if you're running API Umbrella on a single box and you're consistently running into this right now.

Basically, what's happening is that if you're running all the services on one box, we're spinning up our API Umbrella processes and the MongoDB process at the same time. However, if MongoDB takes a long-ish time to startup (which may happen the first time due to the preallocator, and might vary depending on the speed of your disk or other variables), then the API Umbrella processes eventually die because they weren't able to connect to MongoDB in time. This subsequently halts the entire startup process.

I've only seen this happen sporadically, but this should be fixed in the upcoming release (now our processes wait a while for the MongoDB connection to come alive). If you'd like, I might be able to share a pre-release version of that package later today that you could test out.

Alternatively, one possible, but untested workaround with the current version is to let MongoDB finish it's pre-allocator business once, in which case I think it is much quicker to startup again. Waiting for the updated package might be the better bet, but if you really feel like messing around now, you could try these steps:

  1. Make sure API Umbrella is stopped

  2. In /etc/api-umbrella/api-umbrella.yml add the following lines to the bottom:

    services:
  - general_db

  3. Start API Umbrella: sudo /etc/init.d/api-umbrella start

  4. Assuming that succeeds, then shutdown API Umbrella: sudo /etc/init.d/api-umbrella stop

  5. In /etc/api-umbrella/api-umbrella.yml remove those lines you added above.

  6. Start API Umbrella: sudo /etc/init.d/api-umbrella start Does it successfully startup this time?

@GUI GUI added this to the v0.7 milestone Feb 5, 2015
@dmolina-ot
Copy link
Author

It's a race condition because it sometimes starts fine. Can I start the mongodb externally and then start-up API-Umbrella without mongodb?

How "services" configuration works?

# services:
# - general_db
# - log_db
# - router
# - web

Thanks

@dmolina-ot
Copy link
Author

workaround:

First start mongodb with:
sudo /opt/api-umbrella/embedded/bin/mongod --config /opt/api-umbrella/etc/mongod.conf

And then, start Api Umbrella, but with:

services:
# - general_db
- log_db
- web
- router

And all works fine!

@GUI
Copy link
Member

GUI commented Feb 10, 2015

The new v0.7 packages have been released which should hopefully fix a lot of these startup woes. See the changelog for details. I'm going to close this, but please open up a new ticket if you're still running into issues with the new version.

@GUI GUI closed this as completed Feb 10, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.7
Development

No branches or pull requests
3 participants
@GUI @darylrobbins @dmolina-ot