Admin API max request body size

[travisyates]

Kong version: 0.5.1
OS: Ubuntu 15.04

When attempting to submit PATCH requests for the ip-restriction plugin with a list of blacklist IPs (config.blacklist), the request begins failing somewhere around 10KB. The point of failure seems to vary slightly between cURL (~10.2k) and Postman (~10.8k). To construct the list of IPs, I took the CIDR blocks from Spamhaus's drop and edrop lists (http://www.spamhaus.org/drop/drop.txt and http://www.spamhaus.org/drop/edrop.txt). Below the threshold, the modifications take effect. Above the threshold, Kong returns {"message":"Cannot parse JSON body"}. I will submit captures of the output via email as they are lengthy and I do not have permission to post attachments in this repo.

[ahmadnassri]

@travisyates use a gist?

thanks for reporting.

[travisyates]

@ahmadnassri Here you go: https://gist.github.com/travisyates/452916d42af9add85ea4

[ahmadnassri]

awesome, thank you!

we'll look into this ASAP :)

[thibaultcha]

On mobile now but have you tried increasing the client_body_buffer_size? Otherwise nginx uses multiple buffers and thus the admin API is only receiving a chunk of the JSON, and cannot parse it correctly.

We could use client_body_in_single_buffer and a client_max_body_size to enforce this.

[subnetmarco]

Otherwise nginx uses multiple buffers and thus the admin API is only receiving a chunk of the JSON

If this was the case, the admin API should wait for all the chunks before processing the request.

[thibaultcha]

client_body_in_single_buffer

[subnetmarco]

Also related to leafo/lapis#179.

[subnetmarco]

@thibaultcha Setting client_body_in_single_buffer on; is not enough in my tests.

[travisyates]

It seems less than ideal, but setting client_body_buffer_size to 128k allow me to send in larger lists.

[thibaultcha]

Setting client_body_in_single_buffer on; is not enough in my tests.

With client_max_body_size and client_body_buffer_size set as well?

[subnetmarco]

client_body_buffer_size is the issue, if we increase it then it works, but it's an ugly hack. My idea is to set this number to a higher value, and wait for a lapis fix. It doesn't fix the root problem though.

Specifically this is the problem:

2015/11/09 13:47:05 [warn] 55464#0: *4 a client request body is buffered to a temporary file /Users/marco/git/kong/nginx_tmp/client_body_temp/0000000001, client: 127.0.0.1, server: , request: "POST /apis/hello.com/plugins/ HTTP/1.1", host: "localhost:8101"
2015/11/09 13:47:05 [error] 55464#0: *4 [lua] app.lua:113: handle_error(): /usr/local/share/lua/5.1/lapis/nginx.lua:118: requesty body in temp file not supported

stack traceback:
    [C]: in function 'get_post_args'
    /usr/local/share/lua/5.1/lapis/nginx.lua:118: in function 'fn'
    /usr/local/share/lua/5.1/lapis/nginx.lua:133: in function '__index'
    /usr/local/share/lua/5.1/lapis/application.lua:382: in function 'resolve'
    /usr/local/share/lua/5.1/lapis/application.lua:402: in function </usr/local/share/lua/5.1/lapis/application.lua:400>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.1/lapis/application.lua:400: in function 'dispatch'
    /usr/local/share/lua/5.1/lapis/nginx.lua:181: in function 'serve'
    content_by_lua(nginx.conf:137):9: in function <content_by_lua(nginx.conf:137):1>, client: 127.0.0.1, server: , request: "POST /apis/hello.com/plugins/ HTTP/1.1", host: "localhost:8101"

[thibaultcha]

It does fix the root problem. Nginx buffers its input. If you want a bigger input, you need to increase this value. That is the root problem to me, but not a problem, more like a chosen behavior Lapis needs to deal with.

Of course, what would be nicer is Lapis reading from the temporary file. I consider this more as a new feature in Lapis though. It would be nice to implement it.

[subnetmarco]

When the request body is too big, nginx only loads small chunks in memory (whose size equals the value of client_body_buffer_size) and saves them to a temporary file ready to be used later, which makes sense. Lapis is not able to handle this particular scenario. Increasing the buffer size will consume more memory and avoid caching the request body into a file, but that's just a workaround. Properly using ngx.req.get_body_file would fix the root problem.

It fixes the problem because it will make it work (as long as request_body <= client_body_buffer_size), but it doesn't fix the root problem.

I am not sure, but we may have the same problem in our plugins if the request is too big (!), because we don't use ngx.req.get_body_file anywhere.

[thibaultcha]

I am very well aware of all this, hence why I knew this was the problem from the start. Increasing this value for the Admin API does make sense. yes it is not ideal but it is required since this is how nginx works.

Like I said, it would be nice to implement reading the request body from the tmp file in Lapis.

Yes, the proxy does have the same problem (when plugins read the request body). Have been aware of it for a long time now. Again, Kong should not be opinionated on those values and they should be configurable by users depending on their use case.

[subnetmarco]

Yes, the proxy does have the same problem

I opened #701.

[thibaultcha]

Merged #702 for now. Implementing file reading in Lapis would be great in the future.