Oct 132016
 

After a long enough time, I’ve finally got builds working again! NGINX PPAs now have updated builds pending.

  • Stable PPA: Resync 1.10.1 packages with Debian, incorporate 1.10.1-3 into the PPAs. (Includes dynamic modules as exist in Unstable)
  • Mainline PPA: Resync basic packaging with Debian, using 1.10.1 as a base for Mainline packaging. Bump Mainline version to 1.11.5 for the PPA.

These updates *do* have a resync with Debian on the packaging, which should address some issues, and also a bump in Mainline to the latest release there.

As of October 13, 2016, at 10:54 AM (UTC-04:00), these are not yet available in the primary Stable or Mainline PPAs, and exist in the staging repository. I’m waiting on the builds to finish running and uploading first, because I can’t copy them over until that’s done.

As of October 13, 2016, at 11:04 AM (UTC-04:00), the primary PPAs have had the process of copying the packages from the staging repositories into the main PPAs. They should be available soon for use.


That said, these updates being overdue as they are, I will have to make some decisions here. And these decisions are effective as of now, mostly for Precise and older Ubuntu releases using the PPAs.

  • Ubuntu Precise will continue to get Stable PPA updates until April 2017. After April 2017, Precise will no longer be supported in the PPA.
  • Ubuntu Precise will no longer receive Mainline PPA updates effective October 15, 2016. We saw this before when we tried to backport newer NGINX to older Lucid releases long ago. The trouble with supporting old releases is multi-fold, but with regards to NGINX and Precise the two primary issues are supporting the build dependencies which continue to evolve as newer versions are available, and the timeline for Ubuntu to End Of Life the old Precise release. Precise is scheduled to go End of Life in April 2017. People still using Precise should be upgrading to Trusty or later at this point. Given this timeline of support, and the build dependencies issues, it will become far too difficult to maintain Mainline for Precise. (If an update to Mainline includes Security content, then an update will be made to Precise; however, no other updates will happen to Precise, so go and upgrade your Precise servers sooner rather than later!)

This may inconvenience some people using Precise, but unfortunately it’s getting too difficult to maintain NGINX for ancient releases.


Speaking of old releases, the PPAs are getting a cleanup too. Vivid and Wily packages, both releases now EOL, will be having their packages removed shortly.

NGINX Mainline PPA: 1.11.2 is being built and released; Ubuntu Wily EOL: No new NGINX versions in PPAs for Wily.

 NGINX, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA, Ubuntu  Comments Off on NGINX Mainline PPA: 1.11.2 is being built and released; Ubuntu Wily EOL: No new NGINX versions in PPAs for Wily.
Jul 112016
 

Been a while since I posted about NGINX on my blog.

Anyways, good news. NGINX 1.11.2 has been uploaded to the staging PPA, and is in the process of being built. If there’s no issues with the builds, then I’ll push the packages to the main Mainline PPA when they’re completed.

NGINX 1.11.2 includes a few new features, but also a bunch of bugfixes:

Changes with nginx 1.11.2                                        05 Jul 2016

    *) Change: now nginx always uses internal MD5 and SHA1 implementations;
       the --with-md5 and --with-sha1 configure options were canceled.

    *) Feature: variables support in the stream module.

    *) Feature: the ngx_stream_map_module.

    *) Feature: the ngx_stream_return_module.

    *) Feature: a port can be specified in the "proxy_bind", "fastcgi_bind",
       "memcached_bind", "scgi_bind", and "uwsgi_bind" directives.

    *) Feature: now nginx uses the IP_BIND_ADDRESS_NO_PORT socket option
       when available.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/2 and the "proxy_request_buffering" directive.

    *) Bugfix: the "Content-Length" request header line was always added to
       requests passed to backends, including requests without body, when
       using HTTP/2.

    *) Bugfix: "http request count is zero" alerts might appear in logs when
       using HTTP/2.

    *) Bugfix: unnecessary buffering might occur when using the "sub_filter"
       directive; the issue had appeared in 1.9.4.

All in all this is a good thing.

However, for Ubuntu Wily 15.10 server users, who use the Mainline PPA, this is the last update for the Mainline PPA for Ubuntu Wily. Ubuntu Wily goes End of Life on July 28, 2016. This means it will no longer be supported by Ubuntu upstream, and will receive no new security updates, bug fix updates, etc. on that date. With the EOL date being so close, this is the last upload to the Mainline PPA for Ubuntu Wily. (This also holds true for the Stable PPA – there will be no new Wily updates except for security updates that may happen between now and July 28th)

Apr 122016
 

Hello again! NGINX 1.9.14 is now available in Ubuntu Xenial. There’s quite a few things we should make known to everyone who uses nginx in Ubuntu, with php5-fpm currently!


HTTP/2 is now enabled

Yes, HTTP/2 is now enabled for nginx-core, nginx-full, and nginx-extras in Ubuntu Xenial. Add http2 to your SSL listener line in your server blocks, and HTTP/2 will be enabled for that port and site.

For HTTP/2 on non-Xenial Ubuntu releases, you can use the Mainline PPA for Wily and later. Anything before Wily does not have full HTTP/2 support, and very likely will not be usable to get HTTP/2 working as intended.


Ubuntu Xenial ships php7.0-fpm, and not php5-fpm, and this will break existing site configurations

The Ubuntu Xenial packages for nginx have already been updated for this change, pointing to php7.0-fpm instead of php5-fpm.

However, users who have existing site configurations will not benefit from these changes. They must manually apply the changes.

Effectively, this is what a default setup uses to interface with the default php5-fpm setup on Ubuntu versions before Xenial, passing all PHP processing to the php5-fpm backend. This is from the default configuration file, but it’s still similar for all PHP passing:

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
    
        # With php5-cgi alone:
        #fastcgi_pass 127.0.0.1:9000;
        # With php5-fpm:
        fastcgi_pass unix:/var/run/php5-fpm.sock;
    }

In Ubuntu Xenial, the TCP listener for php7.0-cgi will be unchanged, however for php7.0-fpm, it will be necessary to update the configuration to look like this for existing site configurations:

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
    
        # With php7.0-cgi alone:
        #fastcgi_pass 127.0.0.1:9000;
        # With php7.0-fpm:
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    }

This will prevent HTTP 502 Bad Gateway errors, and will use the updated php7.0-fpm instead of the php5-fpm packages.

(If for some reason you still want to have php5-fpm under Xenial, you will not be able to get support from Ubuntu for this; you will need to use a PPA. I explain this on a different post on my blog.)

NGINX PPA Cleanup

 NGINX, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA  Comments Off on NGINX PPA Cleanup
Feb 052016
 

The NGINX PPAs have had some cleanup done to them today.

Previously, the PPAs kept the ‘older’ package versions in them for now-EOL releases (this included keeping ancient versions for Maverick, Natty, Oneiric, Quantal, Raring, Saucy, and Utopic). This was decided upon in order to prevent people from seeing 404 errors on PPA checking. We also included a large list of “Final Version” items for each Ubuntu release, stating there would be no more updates for that release, but keeping the ancient packages in place for installation.

Looking back on this, this is a bad thing for multiple reasons. Firstly, it means people in ‘older releases’ can still use the PPA for that release. This means security-holed versions of NGINX could still be used. Secondly, it implies that we still ‘support’ the use of older releases of Ubuntu in the PPAs. This has the security connotation that we are OK with people using no-longer-updated releases, which in turn have their own security holes.

So, today, in an effort to discourage the use of ancient Ubuntu versions which get no security updates or support anymore, I’ve made changes to the way that the PPAs will operate going forward: Unless a release recently went End of Life, versions of the nginx package in the PPAs for older Ubuntu releases are no longer going to be kept, and will be deleted a week after the version goes End of Life.

Therefore, as of today, I have deleted all the packages in the NGINX PPAs (both Stable and Mainline, in both staging and release PPAs) for the following releases of Ubuntu:

  • Maverick (10.10)
  • Natty (11.04)
  • Oneiric (11.10)
  • Quantal (12.10)
  • Raring (13.04)
  • Saucy (13.10)
  • Utopic (15.04)

People still using ancient versions of NGINX or Ubuntu are strongly recommended to upgrade to get continued support and security/bug fixes.

Nginx 1.9.3 in PPAs, and retiring of Utopic Uploads for both PPAs

 NGINX, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA  Comments Off on Nginx 1.9.3 in PPAs, and retiring of Utopic Uploads for both PPAs
Jul 222015
 

The latest Nginx Mainline version, 1.9.3, is now available in the Mainline PPA (link).


With this 1.9.3 upload to the PPAs, we are hereby retiring the Utopic release from both the NGINX Stable and NGINX Mainline PPAs. The Ubuntu Utopic 14.10 release EOLs tomorrow, July 23rd, 2015. We are not planning any additional uploads to affect Utopic, and are hereby considering those releases “disabled” for uploads and building. Packages as they exist in the PPA will continue to exist, but will not receive updates for Utopic.

NGINX in Ubuntu Vivid: If upgrading to Vivid on a 32bit i386 platform, consider upgrading to 64bit amd64 platform in the process!

 nginx, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA, Server Packages, Ubuntu  Comments Off on NGINX in Ubuntu Vivid: If upgrading to Vivid on a 32bit i386 platform, consider upgrading to 64bit amd64 platform in the process!
Apr 042015
 

The latest in updates done to Debian and Ubuntu’s nginx packaging has changed slightly the compilation of the nginx package, namely that two new hardening features have been enabled in the compiling: making the executables Position Independent, and activating immediate binding.

There’s a small problem, here, however. In amd64 (64-bit), Position Independent Executables work fine. However, there is a performance impact that will be noticeable in higher-performance-requiring uses of the nginx executables in 32-bit i386 platforms.

As such, it is highly recommended that if you are planning on upgrading a 32-bit i386 server running nginx with Ubuntu Utopic to Ubuntu Vivid, and your applications that are running via nginx require much higher performance demands (small, static sites don’t necessarily count), then you should strongly consider upgrading to a 64bit amd64 platform, rather than sticking with a 32bit i386 platform due to the performance hit that will be caused as a result of the Position Independent Executable compilation option.

This will affect Ubuntu Vivid (all nginx flavors) and will in future also affect the nginx PPAs. (It has not yet been implemented in the PPAs as of yet, however it will likely end up there in the future.).

Feb 242015
 

After some minor debates with others, the NGINX Stable and Mainline PPAs have been updated to include builds for the armhf architecture. This means that individuals running Ubuntu Precise 12.04, Trusty 14.04, Utopic 14.10, or Vivid 15.04 (although I have no idea why you’d be using this version in production) on armv7 architecture (which is armhf architecture) will be able to add the PPA and install the NGINX packages as if they were on a standard 64bit or 32bit server.

Shoutout to William Grant for helping to get the two staging PPAs I use for building the packages set up with ARM builds. Didn’t take much to do, but each little bit of assistance to move the PPAs forward towards the modern era helps, so thanks, William Grant for your assistance in turning on ARM builds for the PPAs.

Dec 182014
 

This post serves as a notice regarding the BREACH vulnerability and NGINX.

For Ubuntu, Debian, and the PPA users: If you are on 1.6.2-5 (or 1.7.8 from the PPAs), the default configuration has GZIP compression enabled, which means it does not mitigate BREACH on your sites by default. You need to look into whether you are actually impacted by BREACH, and if you are consider mitigation steps.


What is it?

Unlke CRIME, which attacks TLS/SPDY compression and is mitigated by disabling SSL compression, BREACH attacks HTTP responses. These are compressed using the common HTTP compression, which is much more common than TLS-level compression. This allows essentially the same attack demonstrated by Duong and Rizzo, but without relying on TLS-level compression (as they anticipated).

BREACH is a category of vulnerabilities and not a specific instance affecting a specific piece of software. To be vulnerable, a web application must:

  • Be served from a server that uses HTTP-level compression
  • Reflect user-input in HTTP response bodies
  • Reflect a secret (such as a CSRF token) in HTTP response bodies

Additionally, while not strictly a requirement, the attack is helped greatly by responses that remain mostly the same (modulo the attacker’s guess). This is because the difference in size of the responses measured by the attacker can be quite small. Any noise in the side-channel makes the attack more difficult (though not impossible).

It is important to note that the attack is agnostic to the version of TLS/SSL, and does not require TLS-layer compression. Additionally, the attack works against any cipher suite. Against a stream cipher, the attack is simpler; the difference in sizes across response bodies is much more granular in this case. If a block cipher is used, additional work must be done to align the output to the cipher text blocks.

How practical is it?

The BREACH attack can be exploited with just a few thousand requests, and can be executed in under a minute. The number of requests required will depend on the secret size. The power of the attack comes from the fact that it allows guessing a secret one character at a time.

Am I affected?

If you have an HTTP response body that meets all the following conditions, you might be vulnerable:

  • Compression – Your page is served with HTTP compression enabled (GZIP / DEFLATE)
  • User Data – Your page reflects user data via query string parameters, POST …
  • A Secret – Your application page serves Personally Identifiable Information (PII), a CSRF token, sensitive data …

Mitigations

NOTE: The Breach Attack Information Site offers several tactics for mitigating the attack. Unfortunately, they are unaware of a clean, effective, practical solution to the problem. Some of these mitigations are more practical and a single change can cover entire apps, while others are page specific.

The mitigations are ordered by effectiveness (not by their practicality – as this may differ from one application to another).

  1. Disabling HTTP compression
  2. Separating secrets from user input
  3. Randomizing secrets per request
  4. Masking secrets (effectively randomizing by XORing with a random secret per request)
  5. Protecting vulnerable pages with CSRF
  6. Length hiding (by adding random number of bytes to the responses)
  7. Rate-limiting the requests.

Whichever mitigation you choose, it is strongly recommended you also monitor your traffic to detect attempted attacks.


Mitigation Tactics and Practicality

Unfortunately, the practicality of the listed mitigation tactics is widely varied. Practicality is determined by the application you are working with, and in a lot of cases it is not possible to just disable GZIP compression outright due to the size of what’s being served.

This blog post will cover and describe in varying detail three mitigation methods: Disabling HTTP Compression, Randomizing secrets per request, and Length Hiding (using this site as a reference for the descriptions here).

Disabling HTTP Compression

This is the simplest and most effective mitigation tactic, but is ultimately not the most wieldy mitigation tactic, as there is a chance your application actually requires GZIP compression. If this is the case, then you should not use this mitigation option, when GZIP compression is needed in your environment. However, if your application and use case does not necessitate the requirement of GZIP compression, this is easily fixed.

To disable GZIP globally on your NGINX instances, in nginx.conf, add this code to the http block: gzip off;.

To disable GZIP specifically in your sites and not globally, follow the same instructions for globally disabling GZIP, but add it to your server block in your sites’ specific configurations instead.

If you are using NGINX from the Ubuntu or Debian repositories, or the NGINX PPAs, you should check your /etc/nginx.conf file to see if it has gzip on; and you should comment this out or change it to gzip off;.

However, if disabling GZIP compression is not an option for your sites, then consider looking into other mitigation methods.

Randomizing secrets per request or masking secrets

Unfortunately, this one is the least descriptive here. Secret handling is handled on an application level and not an NGINX level. If you have the capability to modify your application, you should modify it to randomize the secrets with each request, or mask the secrets. If this is not an option, then consider using another method of mitigation.

Length hiding

Length hiding can be done by nginx, however it is not currently available in the NGINX packages in Ubuntu, Debian, or the PPAs.

It can be done on the application side, but it is easier to update an nginx configuration than to modify and deploy an application when you need to enable or disable this in a production environment. A Length Hiding Filter Module has been made by Nulab, and it adds randomly generated HTML comments to the end of an HTML response to hide correct length and make it difficult for attackers to guess secret information.

An example of such a comment added by the module is as follows:

<!-- random-length HTML comment: E9pigGJlQjh2gyuwkn1TbRI974ct3ba5bFe7LngZKERr29banTtCizBftbMk0mgRq8N1ltPtxgY -->

NOTE: To use this method, until there is any packaging available that uses this module or includes it, you will need to compile NGINX from the source tarballs.

To enable this module, you will need to compile NGINX from source and add the module. Then, add the length_hiding directive to the server,http, or location blocks in your configuration with this line: length_hiding on;


Special Packaging of NGINX PPA with Length Hiding Enabled

I am currently working on building NGINX stable and mainline with the Length Hiding module included in all variants of the package which have SSL enabled. This will eventually be available in separate PPAs for the stable and mainline PPAs.

Until then, I strongly suggest that you look into whether you can operate without GZIP compression enabled, or look into one of the other methods of mitigating this issue.

Dec 162014
 

This weekend, the NGINX PPAs were updated.


Stable PPA: Packaging resynced with Debian 1.6.2-5 to get some fixes and version updates for the third-party modules into the package.


Mainline PPA:

  • Updated verison to 1.7.8.
  • Module updates:
    • Lua module updated to 0.9.13 full from upstream. (Update needed to fix a Fail To Build issue)
    • Cache purge module updated to 2.2 from upstream. (Updated to fix a segmentation fault issue)

NGINX in Ubuntu, PPAs, and Debian: naxsi packages dropped as of September 26, 2014 (repost)

 nginx, NGINX Mainline PPA, NGINX Stable PPA  Comments Off on NGINX in Ubuntu, PPAs, and Debian: naxsi packages dropped as of September 26, 2014 (repost)
Dec 092014
 

Back in April, I upstreamed (that is, reported a bug to Debian) regarding the `nginx-naxsi` packages. The initial bug I upstreamed was about the outdated naxsi version in the naxsi packages. (see this bug in Ubuntu and the related bug in Debian)

The last update on the Debian bug is on September 10, 2014. That update says the following, and was made by Christos Trochalakis:

After discussing it with the fellow maintainers we have decided that it is
better to remove the nginx-naxsi package before jessie is freezed.

Packaging naxsi is not trivial and, unfortunately, none of the maintainers uses
it. That’s the reason nginx-naxsi is not in a good shape and we are not feeling
comfortable to release and support it.

We are sorry for any inconvenience caused.

I asked what the expected timeline was for the packages being dropped. In a response from Christos today, September 15, 2014, it was said:

It ‘ll get merged and released (1.6.1-3) by the end of the month.


Update (December 9, 2014): In Ubuntu, these changes will not make it into 14.10, but are now included in 15.04 as part of nginx 1.6.2-4ubuntu1.  As such, the naxsi variant of nginx is no longer included starting with 15.04.  Community support for the naxsi variant is also limited as well.

In the PPAs, the naxsi packages will be dropped with stable 1.6.2-2+precise0 +trusty0 +utopic0 and mainline 1.7.5-2+precise0 +trusty0 +utopic0.

In Debian, these changes have been applied as 1.6.2-2.

 

(Originally published on September 26, 2014 @ 12:34)