NGINX: Mixing and Matching Ubuntu Repositories (and NGINX team PPAs on Launchpad) or Debian Repositories with Upstream Repositories will result in problems.

 Debian, nginx, NGINX, NGINX PPA, Server Packages, Ubuntu  Comments Off on NGINX: Mixing and Matching Ubuntu Repositories (and NGINX team PPAs on Launchpad) or Debian Repositories with Upstream Repositories will result in problems.
Apr 042015
 

We’ve seen this before, but we see it too frequently. People want the latest NGINX version. So they use the upstream nginx.org repository to get it. They try and install, and you immediately get conflicts.

In Ubuntu, people then file bugs on this thinking it’s an Ubuntu issue (such as this bug here). Problem is, it’s not an Ubuntu bug. Nor is it a bug in the PPAs I maintain. Nor is it a bug in Debian. It’s a problem that arises when you mix the upstream repositories and either Ubuntu, Debian, or the Ubuntu PPAs, and assuming you can ‘upgrade’ cleanly with the upstream repositories.

Basically, this is what it comes down to:

For Debian, if you’re depending on third party modules, you should stick to Debian’s packaging and wait. For Ubuntu, you should use the PPAs which I maintain (under the nginx team on Launchpad) if you want latest software and features, based off of Debian’s packaging decisions.

If you want/depend on NAXSI though, you have no choice but to recompile NGINX with NAXSI yourself, in order to get it working in a sane way. Neither Debian, nor Ubuntu, nor the PPAs have naxsi in the builds anymore.

If none of those apply to you, you will have to purge all nginx binaries from your computer, and all nginx packages and configuration files with this command before installing from the upstream repository: sudo apt-get purge nginx nginx-doc nginx-common (This should also purge the other dependent packages as well)

But, if you’re curious why you can’t mix the repositories, this post explains it from my perspective. Here’s a breakdown of why you cannot mix repositories such as this, and the problems you run into.


Problem 1: Debian / Ubuntu / NGINX PPAs (maintained by yours truly) have flavors; nginx upstream does not.

And by flavors, I mean nginx-light, nginx-full, nginx-extras, nginx-naxsi (up until 1.6.2-2), and nginx-core (Ubuntu only, since Ubuntu 14.04). Each of these flavors contains a different set of modules, based on demand originating in Debian or the community (and ultimately implemented in Debian). I will not go into the differences here, however you can go to here and read my answer to the question for more details.

As a result of how NGINX modules are currently incorporated into the program’s binaries, it is absolutely critical to separate out the configuration files and default sample files and locations so that you can switch between flavors (and upgrade between versions between Ubuntu/Debian/PPAs) without issues and conflict between configuration files. This requires the introduction of a package called nginx-common – a package which contains files and other items that are common to all versions of the nginx flavors in those versions of the source package. This nginx-common is wholly the brainchild of Debian’s work, and inherited in Ubuntu and the PPAs I maintain.

The problem is: NGINX upstream does not ship ‘flavors’. They enable all the default modules that are shipped in the nginx upstream ‘core’ code, and do not include any third party modules, nor do they discriminate the modules to enable (to make ‘light’ builds, or ‘full’ builds). As such, the nginx upstream package is a single solitary ‘nginx’ package. It does not separate out configuration files, nor does it separate out the binaries.


Problem 2: NGINX upstream doesn’t have third party modules.

Now, I know what you’re thinking: “Why would Debian/Ubuntu include third-party modules in the packaging?” Turns out, in Debian, the demand for such ‘third party extensions’ was so high, that they decided to include the modules. Such modules include the nginx-lua module, the nginx-perl module, and even the NAXSI modules for NGINX (which were in nginx-naxsi up until 1.6.2-2).

The problem: These are third party modules, maintained separately from the NGINX code base itself. They’re shipped as part of some of Debian’s packages due to demand, but they in and of themselves can cause packaging issues and conflicts, to the point where it requires updating the modules’ code with each new release to fix issues in those modules. This in and of itself lends to ‘maintainability’ problems. This is why the nginx-naxsi flavor was dropped from NGINX in Debian and Ubuntu as of package version/revision 1.6.2-2. To fix even simple bugs in the nginx-naxsi flavor (and its related packages), the entire NAXSI module needed to be removed and replaced with the latest upstream revisions of the code and plugin. This means that to even fix bugs in how the NAXSI rules were handled (in order to match the actual rule formats that needed to be in place for whitelists and such), you’d have to do a replace of the entire NAXSI module in the nginx-naxsi flavor. In Ubuntu, this would break the ‘Stable Release Update’ in that new features would be added to the package that could break things, old features could be removed, and it would go beyond the ‘nitpick fix’ that’d be needed for a Stable Release Update, such that the system would never be fixed.

As a result, third party modules have to be maintained and updated with almost every code update from NGINX upstream. For NGINX Mainline, the Lua module needed to be updated three times for build failure fixes in the PPAs. Since a lot of these third party modules (such as NAXSI or Lua) are in demand by the community, but not available in the NGINX Upstream repository (or in the case of the NAXSI release, even the PPAs nowadays), you should not mix repositories, as you will lose those modules, or lose some of the modules and gain others.


NGINX in Ubuntu Vivid: If upgrading to Vivid on a 32bit i386 platform, consider upgrading to 64bit amd64 platform in the process!

 nginx, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA, Server Packages, Ubuntu  Comments Off on NGINX in Ubuntu Vivid: If upgrading to Vivid on a 32bit i386 platform, consider upgrading to 64bit amd64 platform in the process!
Apr 042015
 

The latest in updates done to Debian and Ubuntu’s nginx packaging has changed slightly the compilation of the nginx package, namely that two new hardening features have been enabled in the compiling: making the executables Position Independent, and activating immediate binding.

There’s a small problem, here, however. In amd64 (64-bit), Position Independent Executables work fine. However, there is a performance impact that will be noticeable in higher-performance-requiring uses of the nginx executables in 32-bit i386 platforms.

As such, it is highly recommended that if you are planning on upgrading a 32-bit i386 server running nginx with Ubuntu Utopic to Ubuntu Vivid, and your applications that are running via nginx require much higher performance demands (small, static sites don’t necessarily count), then you should strongly consider upgrading to a 64bit amd64 platform, rather than sticking with a 32bit i386 platform due to the performance hit that will be caused as a result of the Position Independent Executable compilation option.

This will affect Ubuntu Vivid (all nginx flavors) and will in future also affect the nginx PPAs. (It has not yet been implemented in the PPAs as of yet, however it will likely end up there in the future.).

NGINX 1.7.11 Now Available in PPA

 nginx, NGINX Mainline PPA, NGINX PPA, Server Packages, Ubuntu  Comments Off on NGINX 1.7.11 Now Available in PPA
Mar 252015
 

The NGINX Mainline PPA has been updated with NGINX Mainline version 1.7.11. It includes builds for Ubuntu Precise, Ubuntu Trusty, Ubuntu Utopic, and Ubuntu Vivid, and the i386, amd64, and armhf architectures.

The following is the list of changes from NGINX upstream:

Changes with nginx 1.7.11                                        24 Mar 2015

    *) Change: the "sendfile" parameter of the "aio" directive is
       deprecated; now nginx automatically uses AIO to pre-load data for
       sendfile if both "aio" and "sendfile" directives are used.

    *) Feature: experimental thread pools support.

    *) Feature: the "proxy_request_buffering", "fastcgi_request_buffering",
       "scgi_request_buffering", and "uwsgi_request_buffering" directives.

    *) Feature: request body filters experimental API.

    *) Feature: client SSL certificates support in mail proxy.
       Thanks to Sven Peter, Franck Levionnois, and Filipe Da Silva.

    *) Feature: startup speedup when using the "hash ... consistent"
       directive in the upstream block.
       Thanks to Wai Keen Woon.

    *) Feature: debug logging into a cyclic memory buffer.

    *) Bugfix: in hash table handling.
       Thanks to Chris West.

    *) Bugfix: in the "proxy_cache_revalidate" directive.

    *) Bugfix: SSL connections might hang if deferred accept or the
       "proxy_protocol" parameter of the "listen" directive were used.
       Thanks to James Hamlin.

    *) Bugfix: the $upstream_response_time variable might contain a wrong
       value if the "image_filter" directive was used.

    *) Bugfix: in integer overflow handling.
       Thanks to Régis Leroy.

    *) Bugfix: it was not possible to enable SSLv3 with LibreSSL.

    *) Bugfix: the "ignoring stale global SSL error ... called a function
       you should not call" alerts appeared in logs when using LibreSSL.

    *) Bugfix: certificates specified by the "ssl_client_certificate" and
       "ssl_trusted_certificate" directives were inadvertently used to
       automatically construct certificate chains.
Feb 242015
 

After some minor debates with others, the NGINX Stable and Mainline PPAs have been updated to include builds for the armhf architecture. This means that individuals running Ubuntu Precise 12.04, Trusty 14.04, Utopic 14.10, or Vivid 15.04 (although I have no idea why you’d be using this version in production) on armv7 architecture (which is armhf architecture) will be able to add the PPA and install the NGINX packages as if they were on a standard 64bit or 32bit server.

Shoutout to William Grant for helping to get the two staging PPAs I use for building the packages set up with ARM builds. Didn’t take much to do, but each little bit of assistance to move the PPAs forward towards the modern era helps, so thanks, William Grant for your assistance in turning on ARM builds for the PPAs.

Jan 162015
 

After an eon of fighting the code, I managed to get this to build. It’s been since December 23, 2014 that the PPAs have needed updating, and it turns out I made a failure during updates which caused things to not build. That’s resolved, so now it’s updated.

The NGINX Mainline PPAs are now updated with NGINX 1.7.9. The nginx-lua and nginx-cache-purge modules were updated in order to address build failures. The remaining set of packaging remains the same.

Below are the changes which come with NGINX 1.7.9 (from the upstream NGINX changelog):

Changes with nginx 1.7.9                                         23 Dec 2014

    *) Feature: variables support in the "proxy_cache", "fastcgi_cache",
       "scgi_cache", and "uwsgi_cache" directives.

    *) Feature: variables support in the "expires" directive.

    *) Feature: loading of secret keys from hardware tokens with OpenSSL
       engines.
       Thanks to Dmitrii Pichulin.

    *) Feature: the "autoindex_format" directive.

    *) Bugfix: cache revalidation is now only used for responses with 200
       and 206 status codes.
       Thanks to Piotr Sikora.

    *) Bugfix: the "TE" client request header line was passed to backends
       while proxying.

    *) Bugfix: the "proxy_pass", "fastcgi_pass", "scgi_pass", and
       "uwsgi_pass" directives might not work correctly inside the "if" and
       "limit_except" blocks.

    *) Bugfix: the "proxy_store" directive with the "on" parameter was
       ignored if the "proxy_store" directive with an explicitly specified
       file path was used on a previous level.

    *) Bugfix: nginx could not be built with BoringSSL.
       Thanks to Lukas Tribus.
Dec 182014
 

This post serves as a notice regarding the BREACH vulnerability and NGINX.

For Ubuntu, Debian, and the PPA users: If you are on 1.6.2-5 (or 1.7.8 from the PPAs), the default configuration has GZIP compression enabled, which means it does not mitigate BREACH on your sites by default. You need to look into whether you are actually impacted by BREACH, and if you are consider mitigation steps.


What is it?

Unlke CRIME, which attacks TLS/SPDY compression and is mitigated by disabling SSL compression, BREACH attacks HTTP responses. These are compressed using the common HTTP compression, which is much more common than TLS-level compression. This allows essentially the same attack demonstrated by Duong and Rizzo, but without relying on TLS-level compression (as they anticipated).

BREACH is a category of vulnerabilities and not a specific instance affecting a specific piece of software. To be vulnerable, a web application must:

  • Be served from a server that uses HTTP-level compression
  • Reflect user-input in HTTP response bodies
  • Reflect a secret (such as a CSRF token) in HTTP response bodies

Additionally, while not strictly a requirement, the attack is helped greatly by responses that remain mostly the same (modulo the attacker’s guess). This is because the difference in size of the responses measured by the attacker can be quite small. Any noise in the side-channel makes the attack more difficult (though not impossible).

It is important to note that the attack is agnostic to the version of TLS/SSL, and does not require TLS-layer compression. Additionally, the attack works against any cipher suite. Against a stream cipher, the attack is simpler; the difference in sizes across response bodies is much more granular in this case. If a block cipher is used, additional work must be done to align the output to the cipher text blocks.

How practical is it?

The BREACH attack can be exploited with just a few thousand requests, and can be executed in under a minute. The number of requests required will depend on the secret size. The power of the attack comes from the fact that it allows guessing a secret one character at a time.

Am I affected?

If you have an HTTP response body that meets all the following conditions, you might be vulnerable:

  • Compression – Your page is served with HTTP compression enabled (GZIP / DEFLATE)
  • User Data – Your page reflects user data via query string parameters, POST …
  • A Secret – Your application page serves Personally Identifiable Information (PII), a CSRF token, sensitive data …

Mitigations

NOTE: The Breach Attack Information Site offers several tactics for mitigating the attack. Unfortunately, they are unaware of a clean, effective, practical solution to the problem. Some of these mitigations are more practical and a single change can cover entire apps, while others are page specific.

The mitigations are ordered by effectiveness (not by their practicality – as this may differ from one application to another).

  1. Disabling HTTP compression
  2. Separating secrets from user input
  3. Randomizing secrets per request
  4. Masking secrets (effectively randomizing by XORing with a random secret per request)
  5. Protecting vulnerable pages with CSRF
  6. Length hiding (by adding random number of bytes to the responses)
  7. Rate-limiting the requests.

Whichever mitigation you choose, it is strongly recommended you also monitor your traffic to detect attempted attacks.


Mitigation Tactics and Practicality

Unfortunately, the practicality of the listed mitigation tactics is widely varied. Practicality is determined by the application you are working with, and in a lot of cases it is not possible to just disable GZIP compression outright due to the size of what’s being served.

This blog post will cover and describe in varying detail three mitigation methods: Disabling HTTP Compression, Randomizing secrets per request, and Length Hiding (using this site as a reference for the descriptions here).

Disabling HTTP Compression

This is the simplest and most effective mitigation tactic, but is ultimately not the most wieldy mitigation tactic, as there is a chance your application actually requires GZIP compression. If this is the case, then you should not use this mitigation option, when GZIP compression is needed in your environment. However, if your application and use case does not necessitate the requirement of GZIP compression, this is easily fixed.

To disable GZIP globally on your NGINX instances, in nginx.conf, add this code to the http block: gzip off;.

To disable GZIP specifically in your sites and not globally, follow the same instructions for globally disabling GZIP, but add it to your server block in your sites’ specific configurations instead.

If you are using NGINX from the Ubuntu or Debian repositories, or the NGINX PPAs, you should check your /etc/nginx.conf file to see if it has gzip on; and you should comment this out or change it to gzip off;.

However, if disabling GZIP compression is not an option for your sites, then consider looking into other mitigation methods.

Randomizing secrets per request or masking secrets

Unfortunately, this one is the least descriptive here. Secret handling is handled on an application level and not an NGINX level. If you have the capability to modify your application, you should modify it to randomize the secrets with each request, or mask the secrets. If this is not an option, then consider using another method of mitigation.

Length hiding

Length hiding can be done by nginx, however it is not currently available in the NGINX packages in Ubuntu, Debian, or the PPAs.

It can be done on the application side, but it is easier to update an nginx configuration than to modify and deploy an application when you need to enable or disable this in a production environment. A Length Hiding Filter Module has been made by Nulab, and it adds randomly generated HTML comments to the end of an HTML response to hide correct length and make it difficult for attackers to guess secret information.

An example of such a comment added by the module is as follows:

<!-- random-length HTML comment: E9pigGJlQjh2gyuwkn1TbRI974ct3ba5bFe7LngZKERr29banTtCizBftbMk0mgRq8N1ltPtxgY -->

NOTE: To use this method, until there is any packaging available that uses this module or includes it, you will need to compile NGINX from the source tarballs.

To enable this module, you will need to compile NGINX from source and add the module. Then, add the length_hiding directive to the server,http, or location blocks in your configuration with this line: length_hiding on;


Special Packaging of NGINX PPA with Length Hiding Enabled

I am currently working on building NGINX stable and mainline with the Length Hiding module included in all variants of the package which have SSL enabled. This will eventually be available in separate PPAs for the stable and mainline PPAs.

Until then, I strongly suggest that you look into whether you can operate without GZIP compression enabled, or look into one of the other methods of mitigating this issue.

Dec 162014
 

This weekend, the NGINX PPAs were updated.


Stable PPA: Packaging resynced with Debian 1.6.2-5 to get some fixes and version updates for the third-party modules into the package.


Mainline PPA:

  • Updated verison to 1.7.8.
  • Module updates:
    • Lua module updated to 0.9.13 full from upstream. (Update needed to fix a Fail To Build issue)
    • Cache purge module updated to 2.2 from upstream. (Updated to fix a segmentation fault issue)

NGINX and Ubuntu: Using Flavors Other than `nginx-core`

 nginx, Server Packages, Ubuntu  Comments Off on NGINX and Ubuntu: Using Flavors Other than `nginx-core`
Dec 132014
 

Today, a bug was filed reporting that nginx-full and nginx-common were different versions.  Naturally I went hunting, and found this not to be the case.  (The bug is here)

It looks like unless you enable Universe, you won’t be able to see the updated versions for nginx-common and nginx-full, and there will be version mismatch errors. You’ll see the older version(s) in Main, though, because Main ships with nginx-core and nginx-common specifically, and not the other flavors

So, here’s a little tidbit for NGINX users on Ubuntu: Make sure you have the following repositories enabled if you want to use a specific flavor of nginx:

  • nginx-core: Main
  • nginx-full: Universe
  • nginx-light: Universe
  • nginx-extras: Universe
  • nginx-naxsi (Obsolete, removed with Ubuntu 15.04): Universe

NGINX in Ubuntu, PPAs, and Debian: naxsi packages dropped as of September 26, 2014 (repost)

 nginx, NGINX Mainline PPA, NGINX Stable PPA  Comments Off on NGINX in Ubuntu, PPAs, and Debian: naxsi packages dropped as of September 26, 2014 (repost)
Dec 092014
 

Back in April, I upstreamed (that is, reported a bug to Debian) regarding the `nginx-naxsi` packages. The initial bug I upstreamed was about the outdated naxsi version in the naxsi packages. (see this bug in Ubuntu and the related bug in Debian)

The last update on the Debian bug is on September 10, 2014. That update says the following, and was made by Christos Trochalakis:

After discussing it with the fellow maintainers we have decided that it is
better to remove the nginx-naxsi package before jessie is freezed.

Packaging naxsi is not trivial and, unfortunately, none of the maintainers uses
it. That’s the reason nginx-naxsi is not in a good shape and we are not feeling
comfortable to release and support it.

We are sorry for any inconvenience caused.

I asked what the expected timeline was for the packages being dropped. In a response from Christos today, September 15, 2014, it was said:

It ‘ll get merged and released (1.6.1-3) by the end of the month.


Update (December 9, 2014): In Ubuntu, these changes will not make it into 14.10, but are now included in 15.04 as part of nginx 1.6.2-4ubuntu1.  As such, the naxsi variant of nginx is no longer included starting with 15.04.  Community support for the naxsi variant is also limited as well.

In the PPAs, the naxsi packages will be dropped with stable 1.6.2-2+precise0 +trusty0 +utopic0 and mainline 1.7.5-2+precise0 +trusty0 +utopic0.

In Debian, these changes have been applied as 1.6.2-2.

 

(Originally published on September 26, 2014 @ 12:34)

NGINX Webserver Admins: Don’t Use SSLv3 in Your SSL-Enabled Sites!

 nginx, NGINX PPA, Server Packages, Ubuntu, Uncategorized  Comments Off on NGINX Webserver Admins: Don’t Use SSLv3 in Your SSL-Enabled Sites!
Oct 242014
 

The SSLv3 “POODLE” Vulnerability.

Most of us are aware of the recent protocol flaw vulnerability in SSLv3. Officially designated CVE-2014-3566, it is more commonly referred to as the “POODLE” (Padding Oracle On Downgraded Legacy Encryption) vulnerability.

The vulnerability is a result of a flaw in the way that the (now old) SSLv3 protocol behaves and operates. There is a Ubuntu-specific question on the POODLE vulnerability on Ask Ubuntu (link) which answers common questions on it. There is also a more general question on the POODLE vulnerability on the Information Security Stack Exchange site (link) with more general details on the POODLE vulnerability. If you would like more details, you should refer to those sites, or read the OpenSSL Whitepaper on the POODLE vulnerability (link).

As this is a protocol flaw in SSLv3, ALL implementations of SSLv3 are affected, so the only way to truly protect against POODLE is to disable SSLv3 protocol support in your web application, whether it be software you write, or hosted by a web server.


Disable SSLv3 in nginx:

Since the recommendation is to no longer use SSLv3, the simplest thing to do is disable SSLv3 for your site. In nginx, this is very simple to achieve.

Typically, one would have SSL enabled on their site with the following protocols line or similar if using the example in the default-shipped configuration files (in latest Debian or the NGINX PPAs, prior to the latest updates that happened in the past week or so):
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

To resolve this issue and disable SSLv3 support, we merely need to use the following instead to use only TLS:
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

Note that on really old implementations of OpenSSL, you won’t be able to get TLSv1.1 and TLSv1.2, so at the very least you can just have TLSv1 on the ssl_protocols line. You should probably consider updating to a more recent version of OpenSSL, though, because of other risks/issues in OpenSSL.


Update OpenSSL to get TLS_FALLBACK_SCSV Support:

More importantly than just disabling SSLv3, you should definitely update your OpenSSL, or whatever SSL implementation you use, to receive support for TLS_FALLBACK_SCSV. There is an attack vector that would make you vulnerable to POODLE by starting a TLS session, but then falling back to SSLv3, and then open you to the POODLE vulnerability. By updating, and then having the use of TLS_FALLBACK_SCSV, you will be protecting yourself from protocol downgrading attacks which would also make you vulnerable to POODLE.


Ubuntu Users:

OpenSSL:

Fortunately for all users of Ubuntu, the OpenSSL packages were updated to protect against SSL downgrade attacks. This is detailed in “USN-2385-1: OpenSSL vulnerabilities” (link). Simply running sudo apt-get update with the security repositories enabled should get you the OpenSSL update to address this.

nginx from the Ubuntu Repositories:

Due to the vulnerability, and Debian already having these changes done, I was able to get in a last-minute update (courtesy of the Ubuntu Security Team and the Ubuntu Release Team), into the nginx package for the Utopic (14.10) release, which happened officially yesterday (October 23, 2014). In Utopic, the nginx package’s default config does NOT have SSLv3 on the ssl_protocols line. All other supported versions of Ubuntu do not have this change (this means that Precise and Trusty are both affected).

PPA Users:

Of course, many users of Ubuntu and nginx like the newer features of the latest nginx Stable or Mainline releases. This is why the nginx PPAs exist. Originally maintained by some of the Debian maintainers of the nginx package, I’ve taken primary responsibility of updating the nginx packages, and keeping them in sync (as close as I can) to the Debian nginx packaging.

As of today (October 24, 2014), both the Stable and Mainline PPAs have been updated to be in sync with the latest Debian packaging of the nginx package. This includes the removal of SSLv3 from the default ssl_protocols line.


Debian Users:

OpenSSL:

Fortunately, like Ubuntu, Debian has also updated the OpenSSL packages to protect against SSL downgrade attacks. This is detailed in “DSA-3053-1 openssl — security update” (link). Like in Ubuntu, this can be fixed by running sudo apt-get update or similar to update your packages.

nginx in the Debian Repositories:

If you are on Debian Unstable, you are in luck. The Debian package in Unstable has this change in it already.

If you are on Debian Testing or Debian Stable or Debian Old Stable, you’re unfortunately out of luck, this change isn’t in those versions of the package yet. You can easily do the aforementioned changes, though, and fix your configs to disable SSLv3.