Jan 162015

After an eon of fighting the code, I managed to get this to build. It’s been since December 23, 2014 that the PPAs have needed updating, and it turns out I made a failure during updates which caused things to not build. That’s resolved, so now it’s updated.

The NGINX Mainline PPAs are now updated with NGINX 1.7.9. The nginx-lua and nginx-cache-purge modules were updated in order to address build failures. The remaining set of packaging remains the same.

Below are the changes which come with NGINX 1.7.9 (from the upstream NGINX changelog):

Changes with nginx 1.7.9                                         23 Dec 2014

    *) Feature: variables support in the "proxy_cache", "fastcgi_cache",
       "scgi_cache", and "uwsgi_cache" directives.

    *) Feature: variables support in the "expires" directive.

    *) Feature: loading of secret keys from hardware tokens with OpenSSL
       Thanks to Dmitrii Pichulin.

    *) Feature: the "autoindex_format" directive.

    *) Bugfix: cache revalidation is now only used for responses with 200
       and 206 status codes.
       Thanks to Piotr Sikora.

    *) Bugfix: the "TE" client request header line was passed to backends
       while proxying.

    *) Bugfix: the "proxy_pass", "fastcgi_pass", "scgi_pass", and
       "uwsgi_pass" directives might not work correctly inside the "if" and
       "limit_except" blocks.

    *) Bugfix: the "proxy_store" directive with the "on" parameter was
       ignored if the "proxy_store" directive with an explicitly specified
       file path was used on a previous level.

    *) Bugfix: nginx could not be built with BoringSSL.
       Thanks to Lukas Tribus.
Dec 182014

This post serves as a notice regarding the BREACH vulnerability and NGINX.

For Ubuntu, Debian, and the PPA users: If you are on 1.6.2-5 (or 1.7.8 from the PPAs), the default configuration has GZIP compression enabled, which means it does not mitigate BREACH on your sites by default. You need to look into whether you are actually impacted by BREACH, and if you are consider mitigation steps.

What is it?

Unlke CRIME, which attacks TLS/SPDY compression and is mitigated by disabling SSL compression, BREACH attacks HTTP responses. These are compressed using the common HTTP compression, which is much more common than TLS-level compression. This allows essentially the same attack demonstrated by Duong and Rizzo, but without relying on TLS-level compression (as they anticipated).

BREACH is a category of vulnerabilities and not a specific instance affecting a specific piece of software. To be vulnerable, a web application must:

  • Be served from a server that uses HTTP-level compression
  • Reflect user-input in HTTP response bodies
  • Reflect a secret (such as a CSRF token) in HTTP response bodies

Additionally, while not strictly a requirement, the attack is helped greatly by responses that remain mostly the same (modulo the attacker’s guess). This is because the difference in size of the responses measured by the attacker can be quite small. Any noise in the side-channel makes the attack more difficult (though not impossible).

It is important to note that the attack is agnostic to the version of TLS/SSL, and does not require TLS-layer compression. Additionally, the attack works against any cipher suite. Against a stream cipher, the attack is simpler; the difference in sizes across response bodies is much more granular in this case. If a block cipher is used, additional work must be done to align the output to the cipher text blocks.

How practical is it?

The BREACH attack can be exploited with just a few thousand requests, and can be executed in under a minute. The number of requests required will depend on the secret size. The power of the attack comes from the fact that it allows guessing a secret one character at a time.

Am I affected?

If you have an HTTP response body that meets all the following conditions, you might be vulnerable:

  • Compression – Your page is served with HTTP compression enabled (GZIP / DEFLATE)
  • User Data – Your page reflects user data via query string parameters, POST …
  • A Secret – Your application page serves Personally Identifiable Information (PII), a CSRF token, sensitive data …


NOTE: The Breach Attack Information Site offers several tactics for mitigating the attack. Unfortunately, they are unaware of a clean, effective, practical solution to the problem. Some of these mitigations are more practical and a single change can cover entire apps, while others are page specific.

The mitigations are ordered by effectiveness (not by their practicality – as this may differ from one application to another).

  1. Disabling HTTP compression
  2. Separating secrets from user input
  3. Randomizing secrets per request
  4. Masking secrets (effectively randomizing by XORing with a random secret per request)
  5. Protecting vulnerable pages with CSRF
  6. Length hiding (by adding random number of bytes to the responses)
  7. Rate-limiting the requests.

Whichever mitigation you choose, it is strongly recommended you also monitor your traffic to detect attempted attacks.

Mitigation Tactics and Practicality

Unfortunately, the practicality of the listed mitigation tactics is widely varied. Practicality is determined by the application you are working with, and in a lot of cases it is not possible to just disable GZIP compression outright due to the size of what’s being served.

This blog post will cover and describe in varying detail three mitigation methods: Disabling HTTP Compression, Randomizing secrets per request, and Length Hiding (using this site as a reference for the descriptions here).

Disabling HTTP Compression

This is the simplest and most effective mitigation tactic, but is ultimately not the most wieldy mitigation tactic, as there is a chance your application actually requires GZIP compression. If this is the case, then you should not use this mitigation option, when GZIP compression is needed in your environment. However, if your application and use case does not necessitate the requirement of GZIP compression, this is easily fixed.

To disable GZIP globally on your NGINX instances, in nginx.conf, add this code to the http block: gzip off;.

To disable GZIP specifically in your sites and not globally, follow the same instructions for globally disabling GZIP, but add it to your server block in your sites’ specific configurations instead.

If you are using NGINX from the Ubuntu or Debian repositories, or the NGINX PPAs, you should check your /etc/nginx.conf file to see if it has gzip on; and you should comment this out or change it to gzip off;.

However, if disabling GZIP compression is not an option for your sites, then consider looking into other mitigation methods.

Randomizing secrets per request or masking secrets

Unfortunately, this one is the least descriptive here. Secret handling is handled on an application level and not an NGINX level. If you have the capability to modify your application, you should modify it to randomize the secrets with each request, or mask the secrets. If this is not an option, then consider using another method of mitigation.

Length hiding

Length hiding can be done by nginx, however it is not currently available in the NGINX packages in Ubuntu, Debian, or the PPAs.

It can be done on the application side, but it is easier to update an nginx configuration than to modify and deploy an application when you need to enable or disable this in a production environment. A Length Hiding Filter Module has been made by Nulab, and it adds randomly generated HTML comments to the end of an HTML response to hide correct length and make it difficult for attackers to guess secret information.

An example of such a comment added by the module is as follows:

<!-- random-length HTML comment: E9pigGJlQjh2gyuwkn1TbRI974ct3ba5bFe7LngZKERr29banTtCizBftbMk0mgRq8N1ltPtxgY -->

NOTE: To use this method, until there is any packaging available that uses this module or includes it, you will need to compile NGINX from the source tarballs.

To enable this module, you will need to compile NGINX from source and add the module. Then, add the length_hiding directive to the server,http, or location blocks in your configuration with this line: length_hiding on;

Special Packaging of NGINX PPA with Length Hiding Enabled

I am currently working on building NGINX stable and mainline with the Length Hiding module included in all variants of the package which have SSL enabled. This will eventually be available in separate PPAs for the stable and mainline PPAs.

Until then, I strongly suggest that you look into whether you can operate without GZIP compression enabled, or look into one of the other methods of mitigating this issue.

Dec 162014

This weekend, the NGINX PPAs were updated.

Stable PPA: Packaging resynced with Debian 1.6.2-5 to get some fixes and version updates for the third-party modules into the package.

Mainline PPA:

  • Updated verison to 1.7.8.
  • Module updates:
    • Lua module updated to 0.9.13 full from upstream. (Update needed to fix a Fail To Build issue)
    • Cache purge module updated to 2.2 from upstream. (Updated to fix a segmentation fault issue)

NGINX in Ubuntu, PPAs, and Debian: naxsi packages dropped as of September 26, 2014 (repost)

 nginx, NGINX Mainline PPA, NGINX Stable PPA  Comments Off on NGINX in Ubuntu, PPAs, and Debian: naxsi packages dropped as of September 26, 2014 (repost)
Dec 092014

Back in April, I upstreamed (that is, reported a bug to Debian) regarding the `nginx-naxsi` packages. The initial bug I upstreamed was about the outdated naxsi version in the naxsi packages. (see this bug in Ubuntu and the related bug in Debian)

The last update on the Debian bug is on September 10, 2014. That update says the following, and was made by Christos Trochalakis:

After discussing it with the fellow maintainers we have decided that it is
better to remove the nginx-naxsi package before jessie is freezed.

Packaging naxsi is not trivial and, unfortunately, none of the maintainers uses
it. That’s the reason nginx-naxsi is not in a good shape and we are not feeling
comfortable to release and support it.

We are sorry for any inconvenience caused.

I asked what the expected timeline was for the packages being dropped. In a response from Christos today, September 15, 2014, it was said:

It ‘ll get merged and released (1.6.1-3) by the end of the month.

Update (December 9, 2014): In Ubuntu, these changes will not make it into 14.10, but are now included in 15.04 as part of nginx 1.6.2-4ubuntu1.  As such, the naxsi variant of nginx is no longer included starting with 15.04.  Community support for the naxsi variant is also limited as well.

In the PPAs, the naxsi packages will be dropped with stable 1.6.2-2+precise0 +trusty0 +utopic0 and mainline 1.7.5-2+precise0 +trusty0 +utopic0.

In Debian, these changes have been applied as 1.6.2-2.


(Originally published on September 26, 2014 @ 12:34)

NGINX Webserver Admins: Don’t Use SSLv3 in Your SSL-Enabled Sites!

 nginx, NGINX PPA, Server Packages, Ubuntu, Uncategorized  Comments Off on NGINX Webserver Admins: Don’t Use SSLv3 in Your SSL-Enabled Sites!
Oct 242014

The SSLv3 “POODLE” Vulnerability.

Most of us are aware of the recent protocol flaw vulnerability in SSLv3. Officially designated CVE-2014-3566, it is more commonly referred to as the “POODLE” (Padding Oracle On Downgraded Legacy Encryption) vulnerability.

The vulnerability is a result of a flaw in the way that the (now old) SSLv3 protocol behaves and operates. There is a Ubuntu-specific question on the POODLE vulnerability on Ask Ubuntu (link) which answers common questions on it. There is also a more general question on the POODLE vulnerability on the Information Security Stack Exchange site (link) with more general details on the POODLE vulnerability. If you would like more details, you should refer to those sites, or read the OpenSSL Whitepaper on the POODLE vulnerability (link).

As this is a protocol flaw in SSLv3, ALL implementations of SSLv3 are affected, so the only way to truly protect against POODLE is to disable SSLv3 protocol support in your web application, whether it be software you write, or hosted by a web server.

Disable SSLv3 in nginx:

Since the recommendation is to no longer use SSLv3, the simplest thing to do is disable SSLv3 for your site. In nginx, this is very simple to achieve.

Typically, one would have SSL enabled on their site with the following protocols line or similar if using the example in the default-shipped configuration files (in latest Debian or the NGINX PPAs, prior to the latest updates that happened in the past week or so):
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

To resolve this issue and disable SSLv3 support, we merely need to use the following instead to use only TLS:
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

Note that on really old implementations of OpenSSL, you won’t be able to get TLSv1.1 and TLSv1.2, so at the very least you can just have TLSv1 on the ssl_protocols line. You should probably consider updating to a more recent version of OpenSSL, though, because of other risks/issues in OpenSSL.

Update OpenSSL to get TLS_FALLBACK_SCSV Support:

More importantly than just disabling SSLv3, you should definitely update your OpenSSL, or whatever SSL implementation you use, to receive support for TLS_FALLBACK_SCSV. There is an attack vector that would make you vulnerable to POODLE by starting a TLS session, but then falling back to SSLv3, and then open you to the POODLE vulnerability. By updating, and then having the use of TLS_FALLBACK_SCSV, you will be protecting yourself from protocol downgrading attacks which would also make you vulnerable to POODLE.

Ubuntu Users:


Fortunately for all users of Ubuntu, the OpenSSL packages were updated to protect against SSL downgrade attacks. This is detailed in “USN-2385-1: OpenSSL vulnerabilities” (link). Simply running sudo apt-get update with the security repositories enabled should get you the OpenSSL update to address this.

nginx from the Ubuntu Repositories:

Due to the vulnerability, and Debian already having these changes done, I was able to get in a last-minute update (courtesy of the Ubuntu Security Team and the Ubuntu Release Team), into the nginx package for the Utopic (14.10) release, which happened officially yesterday (October 23, 2014). In Utopic, the nginx package’s default config does NOT have SSLv3 on the ssl_protocols line. All other supported versions of Ubuntu do not have this change (this means that Precise and Trusty are both affected).

PPA Users:

Of course, many users of Ubuntu and nginx like the newer features of the latest nginx Stable or Mainline releases. This is why the nginx PPAs exist. Originally maintained by some of the Debian maintainers of the nginx package, I’ve taken primary responsibility of updating the nginx packages, and keeping them in sync (as close as I can) to the Debian nginx packaging.

As of today (October 24, 2014), both the Stable and Mainline PPAs have been updated to be in sync with the latest Debian packaging of the nginx package. This includes the removal of SSLv3 from the default ssl_protocols line.

Debian Users:


Fortunately, like Ubuntu, Debian has also updated the OpenSSL packages to protect against SSL downgrade attacks. This is detailed in “DSA-3053-1 openssl — security update” (link). Like in Ubuntu, this can be fixed by running sudo apt-get update or similar to update your packages.

nginx in the Debian Repositories:

If you are on Debian Unstable, you are in luck. The Debian package in Unstable has this change in it already.

If you are on Debian Testing or Debian Stable or Debian Old Stable, you’re unfortunately out of luck, this change isn’t in those versions of the package yet. You can easily do the aforementioned changes, though, and fix your configs to disable SSLv3.

Sep 092014

Due to changes in NGINX, there are now changes done to the default configurations which will break fastcgi sites.

NGINX in Debian, and as such, the PPAs, were previously shipping different configuration files which differed from NGINX itself. The Debian package has now synced with the nginx configurations upstream, and as such, certain very different changes have happened.

This is the massively-detailed NEWS entry in Debian for these changes:

nginx-common (1.6.1-2) unstable; urgency=medium

As of nginx-1.6.1-2 we have synced all configuration files with upstream and
we plan to keep them in sync from now on.

Unfortunately that might break existing configuration for some users. Please
check the matrix below for more information:

File Changes
koi-win whitespace
koi-utf whitespace
mime-types whitespace, changed js/rss mime type,
minor other changes & additions
scgi_params whitespace, added HTTPS
uwsgi_params whitespace, added HTTPS, removed UWSGI_SCHEME
fastcgi_params whitespace, removed SCRIPT_FILENAME
fastcgi.conf new upstream configuration file

Fastcgi configuration issues

nginx shipped a modified `fastcgi_params`, which declared `SCRIPT_FILENAME`
fastcgi_param. This line has now been removed. From now on we are also
shipping fastcgi.conf from the upstream repository, which includes a sane
`SCRIPT_FILENAME` parameter value.

So, if you are using fastcgi_params, you can try switching to fastcgi.conf
or manually set the relevant params.

You might also want to read the documentation section before proceeding.

section: $fastcgi_script_name variable.

You will need to change fastcgi_params to the fastcgi.conf file, or manually set the relevant parameters in your site configs, in order to make things work again.

(This is introduced in the PPAs as 1.6.1-2+precise0 +trusty0 and +utopic0. This is also introduced in the PPAs as 1.7.4-1+precise0 +trusty0 and +utopic0. THis will also be in Ubuntu in versions after 1.6.1-2)