I agree with the article, FastCGI is better than HTTP for these things.
Though I'd like to make another protocol known: Web Application Socket (WAS). I designed it 16 years ago at my dayjob because I thought FastCGI still wasn't good enough.
Instead of packing bulk data inside frames on the main socket, WAS has a control socket plus two pipes (raw request+response body). Both the WAS application and the web server can use splice() to operate on a pipe, for example. No framing needed. Also, requests are cancellable and the three file descriptors can always be recovered.
Over the years, we used WAS for many of our internal applications, and for our web hosting environment, I even wrote a PHP SAPI for WAS. Quite a large number of web sites operate with WAS internally.
As I understand it FastCGI doesn't handle websockets, which is a shame. It should be able to handle SSE though since that's effectively just a regular slow-loading/streaming HTTP response.
This is quite an interesting article for its omissions.
I remember the great FastCGI vs. SCGI vs. HTTP wars: I was founding a Web2.0 startup right at the time these technologies were gaining adoption, and so was responsible for setting up the frontend stack. HTTP won because of simplicity: instead of needing to introduce another protocol into your stack, you can just use HTTP, which you already needed to handle at the gateway. Now all sorts of complex network topologies became trivial: you could introduce multiple levels of reverse proxies if you ran out of capacity; you could have servers that specialized in authentication or session management or SSL termination or DDoS filtering or all the other cross-cutting concerns without them needing to know their position in the request chain; and you could use the same application servers for development, with a direct HTTP connection, as you did in production, where they'd sit behind a reverse proxy that handled SSL and authentication and abuse detection.
It also helped that nginx was lots faster than most FastCGI/SCGI modules of the time, and more robust. I'd initially setup my startup's stack as HTTP -> Lighttpd -> FastCGI -> Django, but it was way slower than just using nginx.
The use of HTTP was basically the web equivalent of the End-to-End Principle [1] for TCP/IP. It's the idea that the network and its protocols should be agnostic to what's being transmitted, and all application logic should be in nodes of the network that filter and redirect packets accordingly. This has been a very powerful principle and shouldn't be discarded lightly.
The observation the article makes is that for security, it's often better to follow the Principle of Least Privilege [2] rather than blindly passing information along. Allowlist your communications to only what you expect, so that you aren't unwittingly contributing to a compromise elsewhere in the network.
And the article is highlighting - not explicitly, but it's there - the tension between these two principles. E2E gives you flexibility, but with flexibility comes the potential for someone to use that flexibility to cause harm. PoLP gives you security, but at the cost of inflexibility, where your system can only do what you designed it to do and cannot easily adapt to new requirements.
It makes a lot of sense. Most large organizations are collections of independent teams, many of whom don't communicate with each other other than sending quarterly OKRs and status updates back to their VP. The E2E principle is what allows them to each do their thing, agnostic to what the other servers handling the request are doing, and then let higher levels of the organization reconfigure and provision the system based on the needs of the moment.
Large organizations have a well-known pattern for how to handle this tension between the E2E principle and the PoLP. It's a firewall. As per the E2E principle, this is a node in the system, usually placed near the outside, which is responsible for inspecting and sanitizing every request that enters the system. The input is untrusted external requests that may have arbitrary binary data. The output is the particular subset of HTTP that form valid requests for the server, sanitized to a minimal grammar and now trusted because you reject every packet that wasn't a well-formed request for your particular service. As an added bonus, now you can collect stats on who is sending these malformed requests, which lets you do things like DDoS protection or calling their ISP or contacting the FBI.
The article even admits this: the right solution to untrusted headers is to strip out everything you aren't explicitly expecting at the reverse proxy. If you didn't know True-Client-IP exists, don't pass it on. Allowlist and block everything by default, don't blocklist and allow everything by default.
Putting security-critical logic into proxies is a violation of the End-to-End Principle, not an example of it. That doesn't mean it's a bad thing; as ragall notes, the End-to-End Principle doesn't make sense here.
You're correct that if the proxy removes all unknown headers, you're safe (with HTTP/2). But that sounds extremely inconvenient - before your application can use a new header, you have to talk to the team who runs the proxy. And popular reverse proxy software doesn't do that by default (and according to masfuerte, nginx doesn't even support it) so it remains a huge footgun for the unwary. All completely avoided with FastCGI.
What I dislike about nginx is ... the documentation. I find it virtually useless because of that.
Sadly httpd went the way of "let's make the configuration difficult"; I abandoned it when they suddenly changed the configuration format. I could have adjusted, but I switched to lighttpd (and also, past that point I let ruby autogenerate any configuration format, so technically I could return to httpd, but I don't want to - I think people who develop webservers, need to think about forcing people to adjust to any new format. If there is a "simple" decision to willy-nilly switch the configuration format, perhaps enable e. g. yaml-configuration in ADDITION, so that we don't have to go through new if-clause config statements suddenly).
The HTTP semantics are useful for anyone developing a web app but the wire protocol of HTTP itself is awful. Multiplexing didn’t arrive until HTTP 2.0 for example, so using HTTP for communication between a reverse proxy and a backend is very wasteful. Google for example has long wrapped HTTP into their own Stubby protocol between their web frontends and application frontends (which would usually be called backends); it’s much faster and more featureful than using the HTTP wire protocol.
Won't argue with that, but it's a classic example of "Worse is better" [1]. It was simple and "good enough". Being ubiquitous is often more important than being efficient.
Most of the arguments for using HTTP reverse proxying over FastCGI or SCGI came down to ubiquity. It let you do things (like connect directly to your app servers with a web browser) that you couldn't do with FastCGI.
Most of the stuff I've done for reverse proxies has been pretty straightforward and just using the stuff built into Nginx, but I have to admit that it wouldn't have even occurred to me to use FastCGI if I needed something more elaborate.
I used FastCGI a bit about ten years ago to "convert" some C++ code I wrote to work on the web, but admittedly I haven't used it much since then.
Also, embedded servers are now much much much more popular. Stuff an HTTP server directly into your application and do whatever you gotta do without gateways.
That is way! Unfortunately, sometimes you have to do path-based routing to different backends, and now you're back to needing a proxy between your clients and your applications.
This is the way only if you're operating in a trusted environment (eg. homelab, intranet) or you're sticking CloudFlare or some other "reverse proxy as a service" in front of it. If you expose an embedded HTTP app server directly to the Internet you're almost guaranteed to get pwned, as the Internet has now become an extremely hostile place.
Indeed. I'm sure that someone will butt in with "it's just a bad implementation!" but the whole bit about allowlisting communications will cause flashbacks in those of us who had all our PUT requests just quit working on an IIS server.
I'd love for CGI to be updated, kind of merging what works and not really caring about what does not work. Getting a .cgi file to work on Linux is really easy. Naturally you get more leverage with e. g. rails, but there is also a lot more complexity and I really hate intrinsic complexity.
I agree with the article, FastCGI is better than HTTP for these things.
Though I'd like to make another protocol known: Web Application Socket (WAS). I designed it 16 years ago at my dayjob because I thought FastCGI still wasn't good enough.
Instead of packing bulk data inside frames on the main socket, WAS has a control socket plus two pipes (raw request+response body). Both the WAS application and the web server can use splice() to operate on a pipe, for example. No framing needed. Also, requests are cancellable and the three file descriptors can always be recovered.
Over the years, we used WAS for many of our internal applications, and for our web hosting environment, I even wrote a PHP SAPI for WAS. Quite a large number of web sites operate with WAS internally.
It's all open source:
- library: https://github.com/CM4all/libwas - documentation: https://libwas.readthedocs.io/en/latest/ - non-blocking library: https://github.com/CM4all/libcommon/tree/master/src/was/asyn... - our web server: https://github.com/CM4all/beng-proxy - WebDAV: https://github.com/CM4all/davos - PHP fork with WAS SAPI: https://github.com/CM4all/php-src
As I understand it FastCGI doesn't handle websockets, which is a shame. It should be able to handle SSE though since that's effectively just a regular slow-loading/streaming HTTP response.
This is quite an interesting article for its omissions.
I remember the great FastCGI vs. SCGI vs. HTTP wars: I was founding a Web2.0 startup right at the time these technologies were gaining adoption, and so was responsible for setting up the frontend stack. HTTP won because of simplicity: instead of needing to introduce another protocol into your stack, you can just use HTTP, which you already needed to handle at the gateway. Now all sorts of complex network topologies became trivial: you could introduce multiple levels of reverse proxies if you ran out of capacity; you could have servers that specialized in authentication or session management or SSL termination or DDoS filtering or all the other cross-cutting concerns without them needing to know their position in the request chain; and you could use the same application servers for development, with a direct HTTP connection, as you did in production, where they'd sit behind a reverse proxy that handled SSL and authentication and abuse detection.
It also helped that nginx was lots faster than most FastCGI/SCGI modules of the time, and more robust. I'd initially setup my startup's stack as HTTP -> Lighttpd -> FastCGI -> Django, but it was way slower than just using nginx.
The use of HTTP was basically the web equivalent of the End-to-End Principle [1] for TCP/IP. It's the idea that the network and its protocols should be agnostic to what's being transmitted, and all application logic should be in nodes of the network that filter and redirect packets accordingly. This has been a very powerful principle and shouldn't be discarded lightly.
The observation the article makes is that for security, it's often better to follow the Principle of Least Privilege [2] rather than blindly passing information along. Allowlist your communications to only what you expect, so that you aren't unwittingly contributing to a compromise elsewhere in the network.
And the article is highlighting - not explicitly, but it's there - the tension between these two principles. E2E gives you flexibility, but with flexibility comes the potential for someone to use that flexibility to cause harm. PoLP gives you security, but at the cost of inflexibility, where your system can only do what you designed it to do and cannot easily adapt to new requirements.
[1] https://en.wikipedia.org/wiki/End-to-end_principle
[2] https://en.wikipedia.org/wiki/Principle_of_least_privilege
The end-to-end principle within a datacenter makes little sense, and as shown in the article, ends up enabling insecure behaviour.
It makes a lot of sense. Most large organizations are collections of independent teams, many of whom don't communicate with each other other than sending quarterly OKRs and status updates back to their VP. The E2E principle is what allows them to each do their thing, agnostic to what the other servers handling the request are doing, and then let higher levels of the organization reconfigure and provision the system based on the needs of the moment.
Large organizations have a well-known pattern for how to handle this tension between the E2E principle and the PoLP. It's a firewall. As per the E2E principle, this is a node in the system, usually placed near the outside, which is responsible for inspecting and sanitizing every request that enters the system. The input is untrusted external requests that may have arbitrary binary data. The output is the particular subset of HTTP that form valid requests for the server, sanitized to a minimal grammar and now trusted because you reject every packet that wasn't a well-formed request for your particular service. As an added bonus, now you can collect stats on who is sending these malformed requests, which lets you do things like DDoS protection or calling their ISP or contacting the FBI.
The article even admits this: the right solution to untrusted headers is to strip out everything you aren't explicitly expecting at the reverse proxy. If you didn't know True-Client-IP exists, don't pass it on. Allowlist and block everything by default, don't blocklist and allow everything by default.
Putting security-critical logic into proxies is a violation of the End-to-End Principle, not an example of it. That doesn't mean it's a bad thing; as ragall notes, the End-to-End Principle doesn't make sense here.
You're correct that if the proxy removes all unknown headers, you're safe (with HTTP/2). But that sounds extremely inconvenient - before your application can use a new header, you have to talk to the team who runs the proxy. And popular reverse proxy software doesn't do that by default (and according to masfuerte, nginx doesn't even support it) so it remains a huge footgun for the unwary. All completely avoided with FastCGI.
Can you recommend a reverse proxy that supports white-listing of headers? nginx doesn't seem to.
What I dislike about nginx is ... the documentation. I find it virtually useless because of that.
Sadly httpd went the way of "let's make the configuration difficult"; I abandoned it when they suddenly changed the configuration format. I could have adjusted, but I switched to lighttpd (and also, past that point I let ruby autogenerate any configuration format, so technically I could return to httpd, but I don't want to - I think people who develop webservers, need to think about forcing people to adjust to any new format. If there is a "simple" decision to willy-nilly switch the configuration format, perhaps enable e. g. yaml-configuration in ADDITION, so that we don't have to go through new if-clause config statements suddenly).
The HTTP semantics are useful for anyone developing a web app but the wire protocol of HTTP itself is awful. Multiplexing didn’t arrive until HTTP 2.0 for example, so using HTTP for communication between a reverse proxy and a backend is very wasteful. Google for example has long wrapped HTTP into their own Stubby protocol between their web frontends and application frontends (which would usually be called backends); it’s much faster and more featureful than using the HTTP wire protocol.
Won't argue with that, but it's a classic example of "Worse is better" [1]. It was simple and "good enough". Being ubiquitous is often more important than being efficient.
Most of the arguments for using HTTP reverse proxying over FastCGI or SCGI came down to ubiquity. It let you do things (like connect directly to your app servers with a web browser) that you couldn't do with FastCGI.
[1] https://dreamsongs.com/RiseOfWorseIsBetter.html
Interesting.
Most of the stuff I've done for reverse proxies has been pretty straightforward and just using the stuff built into Nginx, but I have to admit that it wouldn't have even occurred to me to use FastCGI if I needed something more elaborate.
I used FastCGI a bit about ten years ago to "convert" some C++ code I wrote to work on the web, but admittedly I haven't used it much since then.
Also, embedded servers are now much much much more popular. Stuff an HTTP server directly into your application and do whatever you gotta do without gateways.
That is way! Unfortunately, sometimes you have to do path-based routing to different backends, and now you're back to needing a proxy between your clients and your applications.
This is the way only if you're operating in a trusted environment (eg. homelab, intranet) or you're sticking CloudFlare or some other "reverse proxy as a service" in front of it. If you expose an embedded HTTP app server directly to the Internet you're almost guaranteed to get pwned, as the Internet has now become an extremely hostile place.
Go's embedded HTTP server can handle it just fine: https://blog.gopheracademy.com/advent-2016/exposing-go-on-th...
The PHP/Apache configuration that is distributed in the Red Hat family is "FastCGI Process Manager" (FPM).
I don't know if anything else in the RHEL distributions use FastCGI.
What you're looking for is mod_proxy_fcgi, not FPM. It's included in Fedora's httpd-core package; I don't know about RHEL: https://packages.fedoraproject.org/pkgs/httpd/httpd-core/fed...
I'm not looking for anything. I use this now, and it works.
I don't really know anything about the FastCGI.
I've fought many battles with perl + windows + apache + FastCGI in a previous life. No thank you.
Indeed. I'm sure that someone will butt in with "it's just a bad implementation!" but the whole bit about allowlisting communications will cause flashbacks in those of us who had all our PUT requests just quit working on an IIS server.
I'd love for CGI to be updated, kind of merging what works and not really caring about what does not work. Getting a .cgi file to work on Linux is really easy. Naturally you get more leverage with e. g. rails, but there is also a lot more complexity and I really hate intrinsic complexity.