Incorrect work in emperor mode with php plugin

[dborisov]

Vassal configuration:

[uwsgi]
plugins = php
socket = /tmp/testuser_example.com.sock
processes = 4
cheaper = 1
idle = 60
max-requests = 1000
vacuum = true
chmod-socket = 666
master = true
master-as-root = true
chroot = /fpanel/data/testuser
;disable-logging = true
uid = 2002
gid = 2002
harakiri = 30
harakiri-verbose = true
php-index = index.php

Nginx configuration:

location ~ .php$ {
    include uwsgi_params;
    uwsgi_modifier1 14;
    uwsgi_pass unix:///tmp/testuser_example.com.sock;
    uwsgi_param DOCUMENT_ROOT   /home/testuser/example.com/www;
}

Then I start uwsgi and it work as expected

# uwsgi --emperor /etc/uwsgi/vassals/
*** Starting uWSGI 1.2.6 (64bit) on [Sun Oct 21 14:03:57 2012] ***
compiled with version: 4.4.6 20120305 (Red Hat 4.4.6-4) on 02 October 2012 22:08:05
detected number of CPU cores: 2
current working directory: /home/dborisov
detected binary path: /usr/sbin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
*** starting uWSGI Emperor ***
*** has_emperor mode detected (fd: 6) ***
[uWSGI] getting INI configuration from fpanel.ini
*** Starting uWSGI 1.2.6 (64bit) on [Sun Oct 21 14:03:58 2012] ***
compiled with version: 4.4.6 20120305 (Red Hat 4.4.6-4) on 02 October 2012 22:08:05
detected number of CPU cores: 2
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/sbin/uwsgi
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uwsgi socket 0 bound to UNIX address /tmp/testuser_example.com.sock fd 3
PHP 5.3.3 initialized
your server socket listen backlog is limited to 100 connections
*** Operational MODE: preforking ***
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 7126)
spawned uWSGI worker 1 (pid: 7127, cores: 1)

But when I change ini-file of vassal or simply touch it and uwsgi gracefilly reloads the config, I got an error

[emperor] reload the uwsgi instance fpanel.ini
Sun Oct 21 14:14:04 2012 - received message 1 from emperor
...gracefully killing workers...
binary reloading uWSGI...
chdir() to /etc/uwsgi/vassals
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (/tmp/testuser_example.com.sock)
running /usr/sbin/uwsgi
*** has_emperor mode detected (fd: 6) ***
[uWSGI] getting INI configuration from fpanel.ini
*** Starting uWSGI 1.2.6 (64bit) on [Sun Oct 21 14:14:04 2012] ***
compiled with version: 4.4.6 20120305 (Red Hat 4.4.6-4) on 02 October 2012 22:08:05
detected number of CPU cores: 2
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/sbin/uwsgi
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uwsgi socket 0 inherited UNIX address /tmp/testuser_example.com.sock fd 3
PHP 5.3.3 initialized
your server socket listen backlog is limited to 100 connections
*** Operational MODE: preforking ***
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 7126)
spawned uWSGI worker 1 (pid: 7139, cores: 1)
workers have been inactive for more than 60 seconds (1350814506-1350814445)
cheap mode enabled: waiting for socket connection...
Respawned uWSGI worker 1 (new pid: 7141)
realpath() of /home/testuser/example.com/www failed: No such file or directory [utils.c line 4302]
[pid: 7141|app: -1|req: -1/1] 192.168.1.102 () {44 vars in 813 bytes} [Sun Oct 21 14:15:16 2012] GET /index.php => generated 0 bytes in 0 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
announcing my loyalty to the Emperor...
*** vassal fpanel.ini is now loyal ***

Uwsgi thinks that this path is not in chroot, because I change DOCUMENT_ROOT var in nginx configuration. But without it any site in chroot will be broken.
Then I kill emperor process and start it again and it works perfectly until the next reload.
I think it is a bug.

PS: Without using chroot this problem is not presented. uwsgi 1.2.6. Sorry for my english

[prymitive]

Can You try without DOCUMENT_ROOT var in nginx and with those options in vassal config:

chdir = /home/testuser/example.com/www
php-docroot = /home/testuser/example.com/www

It should work with chrooted vassal. I have few php apps in chroot and they all work fine

[prymitive]

Sorry, I've confused --chroot with --namespace, those are two different options. Anyway --php-docroot should help with DOCUMENT_ROOT

[dborisov]

I've removed DOCUMENT_ROOT from nginx configuration and php-docroot = /home/testuser/example.com/www to vassal configuration. It is OK until reload of vassal. Problem is still presented. But now uwsgi return 404, not 500.

[emperor] reload the uwsgi instance fpanel.ini
Sun Oct 21 18:09:43 2012 - received message 1 from emperor
...gracefully killing workers...
Gracefully killing worker 1 (pid: 7451)...
binary reloading uWSGI...
chdir() to /etc/uwsgi/vassals
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (/tmp/testuser_example.com.sock)
running /usr/sbin/uwsgi
*** has_emperor mode detected (fd: 6) ***
[uWSGI] getting INI configuration from fpanel.ini
*** Starting uWSGI 1.2.6 (64bit) on [Sun Oct 21 18:09:44 2012] ***
compiled with version: 4.4.6 20120305 (Red Hat 4.4.6-4) on 02 October 2012 22:08:05
detected number of CPU cores: 2
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/sbin/uwsgi
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uwsgi socket 0 inherited UNIX address /tmp/testuser_example.com.sock fd 3
PHP 5.3.3 initialized
your server socket listen backlog is limited to 100 connections
*** Operational MODE: preforking ***
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 7450)
spawned uWSGI worker 1 (pid: 7454, cores: 1)
[pid: 7454|app: -1|req: -1/1] 192.168.1.102 () {44 vars in 804 bytes} [Sun Oct 21 18:09:51 2012] GET / => generated 0 bytes in 0 msecs (HTTP/1.1 404) 0 headers in 61 bytes (0 switches on core 0)
announcing my loyalty to the Emperor...
*** vassal fpanel.ini is now loyal ***

[prymitive]

Why do You have chroot = /fpanel/data/testuser when Your app lives inside /home/testuser/example.com/www ? What's inside of /fpanel/data/testuser ? Shouldn't You chroot to app to /home/testuser/example.com/www instead?

[dborisov]

Under /fpanel/data/testuser/ I have chrooted env for user.

ll /fpanel/data/testuser/
drwxr-xr-x 2 root root 4096 Окт 10 23:42 bin
drwxr-xr-x 2 root root 4096 Окт 10 23:42 dev
drwxr-xr-x 2 root root 4096 Окт 10 23:42 etc
drwxr-xr-x 3 root root 4096 Окт 10 23:42 home
drwxr-xr-x 2 root root 4096 Окт 10 23:42 lib64
drwxrwxrwx 2 root root 4096 Окт 20 18:48 tmp
drwxr-xr-x 6 root root 4096 Окт 10 23:42 usr

under /home/testuser/* user has their sites

ll /fpanel/data/testuser/home/testuser/
drwxr-xr-x 7 root     root   4096 Окт 11 09:01 example.com

I'll try to build uwsgi from source. May be it's a distro specific problem

[prymitive]

Try this options instead of chroot, this is how I run php apps and it works fine.

namespace = /fpanel/data/testuser/
chdir = /home/testuser/example.com/www
php-docroot = /home/testuser/example.com/www

[prymitive]

I tried to reproduce this issue but --chroot option works fine for me, the only thing I've noticed is that You don't have

chroot() to $chroot_path
*** Warning, on linux system you have to bind-mount the /proc fs in your chroot to get memory debug/report.

message in Your logs while I do. Try using latest uwsgi from source.

[prymitive]

OK, if I have master-as-root in my config than my vassal does not chroot(), please remove it from Your vassal ini and try again. Hopefully this time it will work ;)

[prymitive]

Working config (using http socket instead of nginx, just remove http and remap-modifier options and then uncomment socket to use it in nginx)

[uwsgi]
autoload = true
plugins-dir = /usr/lib/uwsgi/plugins
http = :8080
#socket = /tmp/testuser_example.com.sock
processes = 4
cheaper = 1
idle = 60
max-requests = 1000
vacuum = true
chmod-socket = 666
master = true
chroot = /sudoku/app-561
chdir = /home/app/www/public
plugins = php
remap-modifier = 14:0
php-docroot = /home/app/www/public
uid = www-data
gid = www-data
harakiri = 30
harakiri-verbose = true
php-index = index.php

[dborisov]

Yes! It works without master-as-root! But in this case chrooted user can get access to socket file. It is not very good.
Is it a bug or feature? =)

[prymitive]

First of all: every user will have access to socket file since You set chmod-socket = 666 ;)
I don't really see issue here, Your worker processes and nginx needs to have read-write access to socket, not just uWSGI master. What chrooted user are You referring to? If You mean the user with uid 2002, than just run uWSGI with dedicated user account (like www-data).

[dborisov]

chmod-socket = 666 is for test purposes =)
I will use TCP socket to fix this headache.
Thanks for your time!
uWSGI is cool=)