Allow for response caching when "Authorization" header is part of API backend configuration #281
Comments
This is part of 18F/api.data.gov#281 The bulk of this will be implemented (with tests) over in the router project, since that's where the caching stuff lives. But this allows us to distinguish between authorization headers we add due to api backend config versus externally sent authorization headers.
This is part of 18F/api.data.gov#281 This allows for the HTTP cache to work despite Authorization headers being present if the gatkeeper is what's responsible for adding the Authorization headers to the request during proxying (since by default, Authorization headers disables all caching).
|
This should be fixed and deployed to production. Caching is now allowed for any API backend that sets HTTP Basic auth as part of their proxy config. However, we still disallow caching if the user is sending the HTTP basic auth headers as part of their original request, so these changes should be safe if there are any APIs out there that were also doing doing their own user-facing HTTP basic auth stuff. @dnesting: Your benchmarks should now be seeing cached responses from api.data.gov. I had to roll out these changes in phases on our servers, so your benchmark results might be slightly in flux between 12:49AM ET - 1:05AM ET. You shouldn't have seen any outages, but please let me know if you happen to have seen anything. However, performance may have degraded slightly as we took servers out of rotation during this time period. Cached results should have started around 12:56AM ET, but things weren't totally settled with all nodes back in action until 1:05AM ET. Hopefully this caching will help out. |
This is part of 18F/api.data.gov#281 The associated changes are on the gatekeeper project's 6d2e6574998ff0478481d7ff7f0b327019006e05 commit.
While looking into some API caching things today, I discovered that api.data.gov is unable to cache API responses if the API backend is configured to use HTTP basic auth. This issue only crops up when an API backend is being protected by HTTP basic authentication (username/password), and api.data.gov is setup to add that basic auth to each request as it traverses our proxy layer. The basic flow looks something like this:
The problem is that whenever HTTP authentication headers are on a request, HTTP caching servers by default will refuse to cache any responses. This is done for safety reasons because you typically don't want your HTTP cache to cache any authenticated requests & responses, since then subsequent requests would bypass any authentication logic your app may implement. However, our situation is a little different since api.data.gov is acting as the gatekeeper, and this http basic auth isn't something the user is passing in (this is only to establish trust between the api.data.gov servers and the API backend). So in these cases, caching the responses should be perfectly valid and would be consistent with the rest of our APIs.
I think we only have a few API backend setups using this approach, but this is a problem, since this is a perfectly valid way to setup an API backend and yet it effectively disables the entire caching layer (in a pretty non-obvious way).
The text was updated successfully, but these errors were encountered: