Changes for NGINX Mainline 1.9.6+ Packages in PPAs

 nginx, NGINX Mainline PPA, NGINX PPA, Server Packages, Ubuntu  Comments Off on Changes for NGINX Mainline 1.9.6+ Packages in PPAs
Nov 022015
 

Good day to everyone!

This post is to announce that there are some changes being introduced in the NGINX 1.9.6 packages maintained by Thomas Ward in the Mainline PPA.

1.9.5: SPDY module removed, replaced with HTTP/2 module
The SPDY module was dropped as of nginx 1.9.5. It was replaced with the HTTP/2 module which is considered ‘experimental’. This is reflected in the packages, however 1.9.5 was ever uploaded to the PPAs due to issues in the package which made it less secure.

1.9.6: HTTP/2 bugfixes and other changes
1.9.6 introduced a bunch of bugfixes, and even more bugfixes to HTTP/2. HTTP/2 had multiple issues, but is still considered an “experimental” module. The issues resolved so far make this one OK to upload and use.

Flavor Changes
As a result of the changes to modules, there will be a few changes to the flavors shipped in the PPAs.

Firstly, the SPDY module is no longer included in either nginx-full or nginx-extras.

Secondly, the HTTP/2 module is only loaded in nginx-extras, due to its experimental nature. If you need HTTP/2 you will need to use the nginx-extras version.

Landscape and Gitlab on the same server: Headaches, and thoughts.

 GitLab, Landscape, NGINX, Ubuntu  Comments Off on Landscape and Gitlab on the same server: Headaches, and thoughts.
Aug 262015
 

This is a mini case study, or rather a report from me, on how difficult it can be to run multiple services from the same server. Especially when they listen on similar ports for different aspects. In this post, I examine the headaches of making two things work on the same server: GitLab (via their Omnibus .deb packages), and Landscape (Canonical’s systems management tool).

I am not an expert on either of the software I listed, but what I do know I will state here.

The Software

Landscape

Many of you have probably heard of Landscape, Canonical’s systems management tool for the Ubuntu operating system. Some of you probably know about how we can deploy Landscape standalone for our own personal use with 10 Virtual and 10 Physical machines managed by Landscape, via Juju, or manually.

Most of my systems/servers are Ubuntu, and I have enough that makes management by one individual a headache. In the workplace, we have an entire infrastructure set up for a specific set of applications, all on an Ubuntu base, and a similar headache in managing them all one at a time. For me, discovering Landscape Dedicated Server, the setup yourself, makes management FAR easier. Landscape has a dependency on Apache

GitLab

GitLab is almost like GitHub in a sense. It provides a web interface for working with code, via the Git Version Control System. Github and GitLab are both very useful, but for those of us wanting the same interface in only one organization, or for personal use, and not trusting the Cloud hosts like GitHub or GitLab’s cloud, we can run it via their Omnibus package, which is Gitlab pre-packaged for different distributions (Ubuntu included!)

It includes its own copy of nginx for serving content, and uses Unicorn for the Ruby components. It listens on both port 80 and 8080, initially, per the gitlab configuration file which rewrites and modifies all the other configurations for Gitlab, which includes both of those servers.

The tricky parts

But then, I ran into a dilemma on my own personal setup of it: What happens if you need Landscape and multiple other sites run from the same server, some parts with SSL, some without? Throw into the mix that I am not an Apache person, and part of the dilemma appears.

1: Port 8080.

There’s a conflict between these two softwares. Part of Landscape (I believe the appserver part) and part of GitLab (it’s Unicorn server, which handles the Ruby-to-nginx interface both try and bind to port 8080.

2: Conflicting Web Servers on Same Web Ports

Landscape relies on Apache. GitLab relies on its own-shipped nginx. Both are set by default to listen on port 80. Landscape’s Apache config also listens on HTTPS.

These configurations, out of the box by default, have a very evil problem: both try to bind to port 80, so they don’t work together on the same server.

My solution

Firstly, some information. The nginx bundled as part of GitLab is not easily configured for additional sites. It’s not very friendly to be a ‘reverse proxy’ handler. Secondly, I am not an Apache person. Sure, you may be able to get Apache to work as the ‘reverse proxy’, but it is unwieldy for me to do that, as I’m an nginx guy.

These steps also needed to be done with Landscape turned off. (That’s as easy as running sudo lsctl stop)

1: Solve the Port 8080 conflict

Given that Landscape is something by Canonical, I chose to not mess with it. Instead, we can mess with GitLab to make it bind Unicorn to a different port.

What we have to do with GitLab is tell its Unicorn to listen on a different IP/Port combination. These two lines in the default configuration file control it (the file is located at /etc/gitlab/gitlab.rb in the Omnibus packages):

# unicorn['listen'] = '127.0.0.1'
# unicorn['port'] = 8080

These are commented out by default. The default binding is to bind to 127.0.0.1:8080. We can easily change GitLab’s configurations though, by editing the file, and uncommenting both lines. We have to uncomment both because otherwise it tries to bind to the specified port, but also *:8080 (which breaks Landscape’s services). After making those changes, we now run sudo gitlab-ctl reconfigure and it redoes its configurations and makes everything adapt to those changes we just made.

2: Solve the web server problem

As I said above, I’m an nginx guy. I also discovered revising the GitLab nginx server to do this is a painful thing, so I did an ingenious thing.

First up: Apache.

I set the Apache bindports to be something else. In this case, I revised /etc/apache2/ports.conf to be the following:

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen 10080


Listen 10443


Listen 10443

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Now, I went into the sites-enabled configuration for Landscape, and also changed the bindports accordingly – the HTTP listener on Port 80 now listens on 10080, and the SSL listener on Port 443 now listens on 10443 instead.

Second: GitLab.

This one’s easier, since we simply edit /etc/gitlab/gitlab.rb, and modify the following lines:

#nginx['listen_addresses'] = ['127.0.0.1']
#nginx['listen_port'] = 80

First, we uncomment the lines. And then, we change the 'listen_port' item to be whatever we want. I chose 20080. Then sudo gitlab-ctl reconfigure will apply those changes.

Finally, a reverse proxy server to handle everything.

Behold, we introduce a third web server: nginx, 1.8.0, from the NGINX Stable PPA.

This works by default because we already changed all the important bindhosts for services. Now the headache: we have to configure this nginx to do what we want.

Here’s a caveat: I prefer to run things behind HTTPS, with SSL. To do this, and to achieve it with multiple domains, I have a few wildcard certs. You’ll have to modify the configurations that I specify to set them up to use YOUR SSL certs. Otherwise, though, the configurations will be identical.

I prefer to use different site configuration files for each site, so we’ll do that. Also note that you will need to put in real values where I say DOMAIN.TLD and such, same for SSL certs and keys.

First, the catch-all for catching other domains NOT hosted on the server, placed in /etc/nginx/sites-available/catchall:

server {
listen 80 default_server;

server_name _;

return 406; # HTTP 406 is "Not Acceptable". 404 is "Not Found", 410 is "Gone", I chose 406.
}

Second, a snippet file with the configuration to be imported in all the later configs, with reverse proxy configurations and proxy-related settings and headers, put into /etc/nginx/snippets/proxy.settings.snippet:


proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

Third, the reverse-proxy configuration for Landscape, which is fairly annoying and took me multiple tries to get working right, placed in /etc/nginx/sites-available/landscape_reverseproxy. Don’t forget that Landscape needs SSL for parts of it, so you can’t skip SSL here:


server {
listen 443 ssl;

server_name landscape.DOMAIN.TLD;

ssl_certificate PATH_TO_SSL_CERTIFICATE; ##### PUT REAL VALUES HERE!
ssl_certificate_key PATH_TO_SSL_CERTIFICATE_KEY; ##### PUT REAL VALUES HERE

# These are courtesy of https://cipherli.st, minus a few things.
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
proxy_pass https://127.0.0.1:10443/;
}

location /message-system {
proxy_pass https://127.0.0.1:10443/;
}
}

server {
listen 80;
server_name landscape.DOMAIN.TLD;

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
return 301 https://landscape.DOMAIN.TLD$request_uri;
}

location /ping {
proxy_pass http://127.0.0.1:10080/;
}
}

Forth, the reverse-proxy configuration for GitLab, which was not as hard to make working. Remember, I put this behind SSL, so I have SSL configurations here. I’m including comments for what to put if you want to NOT have SSL:

# If you don't want to have the SSL listener, you don't need this first server block
server {
listen 80;
server_name gitlab.DOMAIN.TLD

# We just send all HTTP traffic over to HTTPS here.
return 302 https://gitlab.DOMAIN.TLD$request_uri;
}

server {
listen 443 ssl;
# If you want to have this listen on HTTP instead of HTTPS,
# uncomment the below line, and comment out the other listen line.
#listen 80;
server_name gitlab.DOMAIN.TLD;

# If you're not using HTTPS, remove from here to the line saying
# "Stop SSL Remove" below
ssl_certificate /etc/ssl/hellnet.io/hellnet.io.chained.pem;
ssl_certificate_key /etc/ssl/hellnet.io/hellnet.io.key;

ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
# Stop SSL Remove

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
proxy_pass http://127.0.0.1:20080/;
}
}

System specifications considerations

Landscape is not light on resources. It takes about a gig of RAM to run safely, from what I’ve observed, but 2GB is more recommended.

GitLab recommends AT LEAST 2GB of RAM. It uses at least that, so you should have 3GB for this at the minimum.

Running both demands just over 3GB of RAM. You can run it on a 4GB box, but it’s better to have double that space just in case, especially if Landscape and Gitlab both get heavy use. I run it on an 8GB converted desktop, which is now a Linux server but used to be a Desktop.

Nginx 1.9.3 in PPAs, and retiring of Utopic Uploads for both PPAs

 NGINX, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA  Comments Off on Nginx 1.9.3 in PPAs, and retiring of Utopic Uploads for both PPAs
Jul 222015
 

The latest Nginx Mainline version, 1.9.3, is now available in the Mainline PPA (link).


With this 1.9.3 upload to the PPAs, we are hereby retiring the Utopic release from both the NGINX Stable and NGINX Mainline PPAs. The Ubuntu Utopic 14.10 release EOLs tomorrow, July 23rd, 2015. We are not planning any additional uploads to affect Utopic, and are hereby considering those releases “disabled” for uploads and building. Packages as they exist in the PPA will continue to exist, but will not receive updates for Utopic.

NGINX: Mixing and Matching Ubuntu Repositories (and NGINX team PPAs on Launchpad) or Debian Repositories with Upstream Repositories will result in problems.

 Debian, nginx, NGINX, NGINX PPA, Server Packages, Ubuntu  Comments Off on NGINX: Mixing and Matching Ubuntu Repositories (and NGINX team PPAs on Launchpad) or Debian Repositories with Upstream Repositories will result in problems.
Apr 042015
 

We’ve seen this before, but we see it too frequently. People want the latest NGINX version. So they use the upstream nginx.org repository to get it. They try and install, and you immediately get conflicts.

In Ubuntu, people then file bugs on this thinking it’s an Ubuntu issue (such as this bug here). Problem is, it’s not an Ubuntu bug. Nor is it a bug in the PPAs I maintain. Nor is it a bug in Debian. It’s a problem that arises when you mix the upstream repositories and either Ubuntu, Debian, or the Ubuntu PPAs, and assuming you can ‘upgrade’ cleanly with the upstream repositories.

Basically, this is what it comes down to:

For Debian, if you’re depending on third party modules, you should stick to Debian’s packaging and wait. For Ubuntu, you should use the PPAs which I maintain (under the nginx team on Launchpad) if you want latest software and features, based off of Debian’s packaging decisions.

If you want/depend on NAXSI though, you have no choice but to recompile NGINX with NAXSI yourself, in order to get it working in a sane way. Neither Debian, nor Ubuntu, nor the PPAs have naxsi in the builds anymore.

If none of those apply to you, you will have to purge all nginx binaries from your computer, and all nginx packages and configuration files with this command before installing from the upstream repository: sudo apt-get purge nginx nginx-doc nginx-common (This should also purge the other dependent packages as well)

But, if you’re curious why you can’t mix the repositories, this post explains it from my perspective. Here’s a breakdown of why you cannot mix repositories such as this, and the problems you run into.


Problem 1: Debian / Ubuntu / NGINX PPAs (maintained by yours truly) have flavors; nginx upstream does not.

And by flavors, I mean nginx-light, nginx-full, nginx-extras, nginx-naxsi (up until 1.6.2-2), and nginx-core (Ubuntu only, since Ubuntu 14.04). Each of these flavors contains a different set of modules, based on demand originating in Debian or the community (and ultimately implemented in Debian). I will not go into the differences here, however you can go to here and read my answer to the question for more details.

As a result of how NGINX modules are currently incorporated into the program’s binaries, it is absolutely critical to separate out the configuration files and default sample files and locations so that you can switch between flavors (and upgrade between versions between Ubuntu/Debian/PPAs) without issues and conflict between configuration files. This requires the introduction of a package called nginx-common – a package which contains files and other items that are common to all versions of the nginx flavors in those versions of the source package. This nginx-common is wholly the brainchild of Debian’s work, and inherited in Ubuntu and the PPAs I maintain.

The problem is: NGINX upstream does not ship ‘flavors’. They enable all the default modules that are shipped in the nginx upstream ‘core’ code, and do not include any third party modules, nor do they discriminate the modules to enable (to make ‘light’ builds, or ‘full’ builds). As such, the nginx upstream package is a single solitary ‘nginx’ package. It does not separate out configuration files, nor does it separate out the binaries.


Problem 2: NGINX upstream doesn’t have third party modules.

Now, I know what you’re thinking: “Why would Debian/Ubuntu include third-party modules in the packaging?” Turns out, in Debian, the demand for such ‘third party extensions’ was so high, that they decided to include the modules. Such modules include the nginx-lua module, the nginx-perl module, and even the NAXSI modules for NGINX (which were in nginx-naxsi up until 1.6.2-2).

The problem: These are third party modules, maintained separately from the NGINX code base itself. They’re shipped as part of some of Debian’s packages due to demand, but they in and of themselves can cause packaging issues and conflicts, to the point where it requires updating the modules’ code with each new release to fix issues in those modules. This in and of itself lends to ‘maintainability’ problems. This is why the nginx-naxsi flavor was dropped from NGINX in Debian and Ubuntu as of package version/revision 1.6.2-2. To fix even simple bugs in the nginx-naxsi flavor (and its related packages), the entire NAXSI module needed to be removed and replaced with the latest upstream revisions of the code and plugin. This means that to even fix bugs in how the NAXSI rules were handled (in order to match the actual rule formats that needed to be in place for whitelists and such), you’d have to do a replace of the entire NAXSI module in the nginx-naxsi flavor. In Ubuntu, this would break the ‘Stable Release Update’ in that new features would be added to the package that could break things, old features could be removed, and it would go beyond the ‘nitpick fix’ that’d be needed for a Stable Release Update, such that the system would never be fixed.

As a result, third party modules have to be maintained and updated with almost every code update from NGINX upstream. For NGINX Mainline, the Lua module needed to be updated three times for build failure fixes in the PPAs. Since a lot of these third party modules (such as NAXSI or Lua) are in demand by the community, but not available in the NGINX Upstream repository (or in the case of the NAXSI release, even the PPAs nowadays), you should not mix repositories, as you will lose those modules, or lose some of the modules and gain others.


NGINX in Ubuntu Vivid: If upgrading to Vivid on a 32bit i386 platform, consider upgrading to 64bit amd64 platform in the process!

 nginx, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA, Server Packages, Ubuntu  Comments Off on NGINX in Ubuntu Vivid: If upgrading to Vivid on a 32bit i386 platform, consider upgrading to 64bit amd64 platform in the process!
Apr 042015
 

The latest in updates done to Debian and Ubuntu’s nginx packaging has changed slightly the compilation of the nginx package, namely that two new hardening features have been enabled in the compiling: making the executables Position Independent, and activating immediate binding.

There’s a small problem, here, however. In amd64 (64-bit), Position Independent Executables work fine. However, there is a performance impact that will be noticeable in higher-performance-requiring uses of the nginx executables in 32-bit i386 platforms.

As such, it is highly recommended that if you are planning on upgrading a 32-bit i386 server running nginx with Ubuntu Utopic to Ubuntu Vivid, and your applications that are running via nginx require much higher performance demands (small, static sites don’t necessarily count), then you should strongly consider upgrading to a 64bit amd64 platform, rather than sticking with a 32bit i386 platform due to the performance hit that will be caused as a result of the Position Independent Executable compilation option.

This will affect Ubuntu Vivid (all nginx flavors) and will in future also affect the nginx PPAs. (It has not yet been implemented in the PPAs as of yet, however it will likely end up there in the future.).

NGINX 1.7.11 Now Available in PPA

 nginx, NGINX Mainline PPA, NGINX PPA, Server Packages, Ubuntu  Comments Off on NGINX 1.7.11 Now Available in PPA
Mar 252015
 

The NGINX Mainline PPA has been updated with NGINX Mainline version 1.7.11. It includes builds for Ubuntu Precise, Ubuntu Trusty, Ubuntu Utopic, and Ubuntu Vivid, and the i386, amd64, and armhf architectures.

The following is the list of changes from NGINX upstream:

Changes with nginx 1.7.11                                        24 Mar 2015

    *) Change: the "sendfile" parameter of the "aio" directive is
       deprecated; now nginx automatically uses AIO to pre-load data for
       sendfile if both "aio" and "sendfile" directives are used.

    *) Feature: experimental thread pools support.

    *) Feature: the "proxy_request_buffering", "fastcgi_request_buffering",
       "scgi_request_buffering", and "uwsgi_request_buffering" directives.

    *) Feature: request body filters experimental API.

    *) Feature: client SSL certificates support in mail proxy.
       Thanks to Sven Peter, Franck Levionnois, and Filipe Da Silva.

    *) Feature: startup speedup when using the "hash ... consistent"
       directive in the upstream block.
       Thanks to Wai Keen Woon.

    *) Feature: debug logging into a cyclic memory buffer.

    *) Bugfix: in hash table handling.
       Thanks to Chris West.

    *) Bugfix: in the "proxy_cache_revalidate" directive.

    *) Bugfix: SSL connections might hang if deferred accept or the
       "proxy_protocol" parameter of the "listen" directive were used.
       Thanks to James Hamlin.

    *) Bugfix: the $upstream_response_time variable might contain a wrong
       value if the "image_filter" directive was used.

    *) Bugfix: in integer overflow handling.
       Thanks to RĂ©gis Leroy.

    *) Bugfix: it was not possible to enable SSLv3 with LibreSSL.

    *) Bugfix: the "ignoring stale global SSL error ... called a function
       you should not call" alerts appeared in logs when using LibreSSL.

    *) Bugfix: certificates specified by the "ssl_client_certificate" and
       "ssl_trusted_certificate" directives were inadvertently used to
       automatically construct certificate chains.
Feb 242015
 

After some minor debates with others, the NGINX Stable and Mainline PPAs have been updated to include builds for the armhf architecture. This means that individuals running Ubuntu Precise 12.04, Trusty 14.04, Utopic 14.10, or Vivid 15.04 (although I have no idea why you’d be using this version in production) on armv7 architecture (which is armhf architecture) will be able to add the PPA and install the NGINX packages as if they were on a standard 64bit or 32bit server.

Shoutout to William Grant for helping to get the two staging PPAs I use for building the packages set up with ARM builds. Didn’t take much to do, but each little bit of assistance to move the PPAs forward towards the modern era helps, so thanks, William Grant for your assistance in turning on ARM builds for the PPAs.

Jan 162015
 

After an eon of fighting the code, I managed to get this to build. It’s been since December 23, 2014 that the PPAs have needed updating, and it turns out I made a failure during updates which caused things to not build. That’s resolved, so now it’s updated.

The NGINX Mainline PPAs are now updated with NGINX 1.7.9. The nginx-lua and nginx-cache-purge modules were updated in order to address build failures. The remaining set of packaging remains the same.

Below are the changes which come with NGINX 1.7.9 (from the upstream NGINX changelog):

Changes with nginx 1.7.9                                         23 Dec 2014

    *) Feature: variables support in the "proxy_cache", "fastcgi_cache",
       "scgi_cache", and "uwsgi_cache" directives.

    *) Feature: variables support in the "expires" directive.

    *) Feature: loading of secret keys from hardware tokens with OpenSSL
       engines.
       Thanks to Dmitrii Pichulin.

    *) Feature: the "autoindex_format" directive.

    *) Bugfix: cache revalidation is now only used for responses with 200
       and 206 status codes.
       Thanks to Piotr Sikora.

    *) Bugfix: the "TE" client request header line was passed to backends
       while proxying.

    *) Bugfix: the "proxy_pass", "fastcgi_pass", "scgi_pass", and
       "uwsgi_pass" directives might not work correctly inside the "if" and
       "limit_except" blocks.

    *) Bugfix: the "proxy_store" directive with the "on" parameter was
       ignored if the "proxy_store" directive with an explicitly specified
       file path was used on a previous level.

    *) Bugfix: nginx could not be built with BoringSSL.
       Thanks to Lukas Tribus.
Dec 182014
 

This post serves as a notice regarding the BREACH vulnerability and NGINX.

For Ubuntu, Debian, and the PPA users: If you are on 1.6.2-5 (or 1.7.8 from the PPAs), the default configuration has GZIP compression enabled, which means it does not mitigate BREACH on your sites by default. You need to look into whether you are actually impacted by BREACH, and if you are consider mitigation steps.


What is it?

Unlke CRIME, which attacks TLS/SPDY compression and is mitigated by disabling SSL compression, BREACH attacks HTTP responses. These are compressed using the common HTTP compression, which is much more common than TLS-level compression. This allows essentially the same attack demonstrated by Duong and Rizzo, but without relying on TLS-level compression (as they anticipated).

BREACH is a category of vulnerabilities and not a specific instance affecting a specific piece of software. To be vulnerable, a web application must:

  • Be served from a server that uses HTTP-level compression
  • Reflect user-input in HTTP response bodies
  • Reflect a secret (such as a CSRF token) in HTTP response bodies

Additionally, while not strictly a requirement, the attack is helped greatly by responses that remain mostly the same (modulo the attacker’s guess). This is because the difference in size of the responses measured by the attacker can be quite small. Any noise in the side-channel makes the attack more difficult (though not impossible).

It is important to note that the attack is agnostic to the version of TLS/SSL, and does not require TLS-layer compression. Additionally, the attack works against any cipher suite. Against a stream cipher, the attack is simpler; the difference in sizes across response bodies is much more granular in this case. If a block cipher is used, additional work must be done to align the output to the cipher text blocks.

How practical is it?

The BREACH attack can be exploited with just a few thousand requests, and can be executed in under a minute. The number of requests required will depend on the secret size. The power of the attack comes from the fact that it allows guessing a secret one character at a time.

Am I affected?

If you have an HTTP response body that meets all the following conditions, you might be vulnerable:

  • Compression – Your page is served with HTTP compression enabled (GZIP / DEFLATE)
  • User Data – Your page reflects user data via query string parameters, POST …
  • A Secret – Your application page serves Personally Identifiable Information (PII), a CSRF token, sensitive data …

Mitigations

NOTE: The Breach Attack Information Site offers several tactics for mitigating the attack. Unfortunately, they are unaware of a clean, effective, practical solution to the problem. Some of these mitigations are more practical and a single change can cover entire apps, while others are page specific.

The mitigations are ordered by effectiveness (not by their practicality – as this may differ from one application to another).

  1. Disabling HTTP compression
  2. Separating secrets from user input
  3. Randomizing secrets per request
  4. Masking secrets (effectively randomizing by XORing with a random secret per request)
  5. Protecting vulnerable pages with CSRF
  6. Length hiding (by adding random number of bytes to the responses)
  7. Rate-limiting the requests.

Whichever mitigation you choose, it is strongly recommended you also monitor your traffic to detect attempted attacks.


Mitigation Tactics and Practicality

Unfortunately, the practicality of the listed mitigation tactics is widely varied. Practicality is determined by the application you are working with, and in a lot of cases it is not possible to just disable GZIP compression outright due to the size of what’s being served.

This blog post will cover and describe in varying detail three mitigation methods: Disabling HTTP Compression, Randomizing secrets per request, and Length Hiding (using this site as a reference for the descriptions here).

Disabling HTTP Compression

This is the simplest and most effective mitigation tactic, but is ultimately not the most wieldy mitigation tactic, as there is a chance your application actually requires GZIP compression. If this is the case, then you should not use this mitigation option, when GZIP compression is needed in your environment. However, if your application and use case does not necessitate the requirement of GZIP compression, this is easily fixed.

To disable GZIP globally on your NGINX instances, in nginx.conf, add this code to the http block: gzip off;.

To disable GZIP specifically in your sites and not globally, follow the same instructions for globally disabling GZIP, but add it to your server block in your sites’ specific configurations instead.

If you are using NGINX from the Ubuntu or Debian repositories, or the NGINX PPAs, you should check your /etc/nginx.conf file to see if it has gzip on; and you should comment this out or change it to gzip off;.

However, if disabling GZIP compression is not an option for your sites, then consider looking into other mitigation methods.

Randomizing secrets per request or masking secrets

Unfortunately, this one is the least descriptive here. Secret handling is handled on an application level and not an NGINX level. If you have the capability to modify your application, you should modify it to randomize the secrets with each request, or mask the secrets. If this is not an option, then consider using another method of mitigation.

Length hiding

Length hiding can be done by nginx, however it is not currently available in the NGINX packages in Ubuntu, Debian, or the PPAs.

It can be done on the application side, but it is easier to update an nginx configuration than to modify and deploy an application when you need to enable or disable this in a production environment. A Length Hiding Filter Module has been made by Nulab, and it adds randomly generated HTML comments to the end of an HTML response to hide correct length and make it difficult for attackers to guess secret information.

An example of such a comment added by the module is as follows:

<!-- random-length HTML comment: E9pigGJlQjh2gyuwkn1TbRI974ct3ba5bFe7LngZKERr29banTtCizBftbMk0mgRq8N1ltPtxgY -->

NOTE: To use this method, until there is any packaging available that uses this module or includes it, you will need to compile NGINX from the source tarballs.

To enable this module, you will need to compile NGINX from source and add the module. Then, add the length_hiding directive to the server,http, or location blocks in your configuration with this line: length_hiding on;


Special Packaging of NGINX PPA with Length Hiding Enabled

I am currently working on building NGINX stable and mainline with the Length Hiding module included in all variants of the package which have SSL enabled. This will eventually be available in separate PPAs for the stable and mainline PPAs.

Until then, I strongly suggest that you look into whether you can operate without GZIP compression enabled, or look into one of the other methods of mitigating this issue.

Dec 162014
 

This weekend, the NGINX PPAs were updated.


Stable PPA: Packaging resynced with Debian 1.6.2-5 to get some fixes and version updates for the third-party modules into the package.


Mainline PPA:

  • Updated verison to 1.7.8.
  • Module updates:
    • Lua module updated to 0.9.13 full from upstream. (Update needed to fix a Fail To Build issue)
    • Cache purge module updated to 2.2 from upstream. (Updated to fix a segmentation fault issue)