Oct 132016
 

After a long enough time, I’ve finally got builds working again! NGINX PPAs now have updated builds pending.

  • Stable PPA: Resync 1.10.1 packages with Debian, incorporate 1.10.1-3 into the PPAs. (Includes dynamic modules as exist in Unstable)
  • Mainline PPA: Resync basic packaging with Debian, using 1.10.1 as a base for Mainline packaging. Bump Mainline version to 1.11.5 for the PPA.

These updates *do* have a resync with Debian on the packaging, which should address some issues, and also a bump in Mainline to the latest release there.

As of October 13, 2016, at 10:54 AM (UTC-04:00), these are not yet available in the primary Stable or Mainline PPAs, and exist in the staging repository. I’m waiting on the builds to finish running and uploading first, because I can’t copy them over until that’s done.

As of October 13, 2016, at 11:04 AM (UTC-04:00), the primary PPAs have had the process of copying the packages from the staging repositories into the main PPAs. They should be available soon for use.


That said, these updates being overdue as they are, I will have to make some decisions here. And these decisions are effective as of now, mostly for Precise and older Ubuntu releases using the PPAs.

  • Ubuntu Precise will continue to get Stable PPA updates until April 2017. After April 2017, Precise will no longer be supported in the PPA.
  • Ubuntu Precise will no longer receive Mainline PPA updates effective October 15, 2016. We saw this before when we tried to backport newer NGINX to older Lucid releases long ago. The trouble with supporting old releases is multi-fold, but with regards to NGINX and Precise the two primary issues are supporting the build dependencies which continue to evolve as newer versions are available, and the timeline for Ubuntu to End Of Life the old Precise release. Precise is scheduled to go End of Life in April 2017. People still using Precise should be upgrading to Trusty or later at this point. Given this timeline of support, and the build dependencies issues, it will become far too difficult to maintain Mainline for Precise. (If an update to Mainline includes Security content, then an update will be made to Precise; however, no other updates will happen to Precise, so go and upgrade your Precise servers sooner rather than later!)

This may inconvenience some people using Precise, but unfortunately it’s getting too difficult to maintain NGINX for ancient releases.


Speaking of old releases, the PPAs are getting a cleanup too. Vivid and Wily packages, both releases now EOL, will be having their packages removed shortly.

The Road Ahead for NGINX in Ubuntu

 nginx, NGINX, Server Packages, Ubuntu  Comments Off on The Road Ahead for NGINX in Ubuntu
Jul 112016
 

Hello, everyone! Two blog posts and a flurry of tweets in a day, what the heck has gotten into me?

Some fun things have happened in the last development cycle leading up to Xenial for nginx! Let’s recap a couple of the big things that’re ‘great’ happenings:

  • NGINX 1.9.x was accepted into Xenial during the development process.
  • Later in the dev cycle, we were given the ACK by the Security Team to enable the HTTP/2 module (yay, HTTP/2 support!)
  • Close to the end, that was also updated to 1.10.x post-release to get us onto a Stable version for the duration of the LTS! Yay, an LTS with a Stable version!

All in all, a good dev cycle for getting NGINX into the Ubuntu repositories! Now, we look ahead to the future.


First, a note about Wily. The NGINX PPAs will no longer get any Wily updates, as of today. This close to the End of Life date of Wily, I can’t guarantee there’ll be any updates beyond security-critical ones prompting such updates, given the EOL date of Wily being in a couple weeks.

This means, for the most part, that bugs which are against the Wily package in Ubuntu also get less scrutiny as we focus on the future. Any such Wily-filed bugs will need to be confirmed in another release of an equal or newer version (basically, Xenial or later) before I poke at them or another person pokes at them (this doesn’t prevent the community from submitting patches though). This also means people on Wily boxes who want to get continued NGINX support should upgrade to Xenial because I can’t guarantee they’ll get updates as they wish. And once Wily goes EOL, they get nothing.


Secondly, the road ahead. Up in Debian, they’re starting to test builds against the next OpenSSL version (1.1.0). Unfortunately, NGINX Stable 1.10.x doesn’t build. After poking upstream, I’ve learned there is a fix for this… but for NGINX Mainline… and it won’t be backported to 1.10.x. This is a little bit of a headache, for a couple reasons.

  1. NGINX Stable 1.10.x is not going to be able to be supported at some point in the future in Ubuntu, because it won’t have OpenSSL support.
  2. To get NGINX Mainline as the version in NGINX, I need to merge in the quite-evil Debian ‘dynamic modules’ support.
  3. Further, to get NGINX Mainline into Ubuntu during a development cycle, I need to go and pull in from Debian Experimental, and then build test against the older OpenSSL to make sure nothing dies off.

The big issues of this are mostly that we don’t know the full timeline of OpenSSL 1.1.0 being released in Debian. I have assurances from the Ubuntu Security Team, however, that OpenSSL 1.1.0 will not be included until packages don’t Fail to Build from Source (FTBFS) against it. Which means that I don’t have to act on this immediately.

The additional headache added to this list though is that, while I merge in Dynamic Module Support, it is not 100% ‘supported’ yet in Debian, and it won’t be totally supported in a sane way for packages which ship third-party modules. There has been discussion threads on some third-party modules packaging their modules to work as a dynamic module for Ubuntu Universe / Debian. This is a double-edged sword. Not only do I have to worry about NGINX updates, but I will have to start making sure all the dynamic modules get rebuilt for each upload. I’ll be working to try and find a better solution to this, but this will preclude updates to things getting done at times, given the signature-based approach to dynamic modules that exists currently. We’ll work through this, though, at some point, and make it more supportable in the future.

——

Just wanted to give you all some insights into the future of NGINX, and the headaches I will have to work through, for Ubuntu’s packages going forward.

NGINX Mainline PPA: 1.11.2 is being built and released; Ubuntu Wily EOL: No new NGINX versions in PPAs for Wily.

 NGINX, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA, Ubuntu  Comments Off on NGINX Mainline PPA: 1.11.2 is being built and released; Ubuntu Wily EOL: No new NGINX versions in PPAs for Wily.
Jul 112016
 

Been a while since I posted about NGINX on my blog.

Anyways, good news. NGINX 1.11.2 has been uploaded to the staging PPA, and is in the process of being built. If there’s no issues with the builds, then I’ll push the packages to the main Mainline PPA when they’re completed.

NGINX 1.11.2 includes a few new features, but also a bunch of bugfixes:

Changes with nginx 1.11.2                                        05 Jul 2016

    *) Change: now nginx always uses internal MD5 and SHA1 implementations;
       the --with-md5 and --with-sha1 configure options were canceled.

    *) Feature: variables support in the stream module.

    *) Feature: the ngx_stream_map_module.

    *) Feature: the ngx_stream_return_module.

    *) Feature: a port can be specified in the "proxy_bind", "fastcgi_bind",
       "memcached_bind", "scgi_bind", and "uwsgi_bind" directives.

    *) Feature: now nginx uses the IP_BIND_ADDRESS_NO_PORT socket option
       when available.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/2 and the "proxy_request_buffering" directive.

    *) Bugfix: the "Content-Length" request header line was always added to
       requests passed to backends, including requests without body, when
       using HTTP/2.

    *) Bugfix: "http request count is zero" alerts might appear in logs when
       using HTTP/2.

    *) Bugfix: unnecessary buffering might occur when using the "sub_filter"
       directive; the issue had appeared in 1.9.4.

All in all this is a good thing.

However, for Ubuntu Wily 15.10 server users, who use the Mainline PPA, this is the last update for the Mainline PPA for Ubuntu Wily. Ubuntu Wily goes End of Life on July 28, 2016. This means it will no longer be supported by Ubuntu upstream, and will receive no new security updates, bug fix updates, etc. on that date. With the EOL date being so close, this is the last upload to the Mainline PPA for Ubuntu Wily. (This also holds true for the Stable PPA – there will be no new Wily updates except for security updates that may happen between now and July 28th)

Apr 122016
 

Hello again! NGINX 1.9.14 is now available in Ubuntu Xenial. There’s quite a few things we should make known to everyone who uses nginx in Ubuntu, with php5-fpm currently!


HTTP/2 is now enabled

Yes, HTTP/2 is now enabled for nginx-core, nginx-full, and nginx-extras in Ubuntu Xenial. Add http2 to your SSL listener line in your server blocks, and HTTP/2 will be enabled for that port and site.

For HTTP/2 on non-Xenial Ubuntu releases, you can use the Mainline PPA for Wily and later. Anything before Wily does not have full HTTP/2 support, and very likely will not be usable to get HTTP/2 working as intended.


Ubuntu Xenial ships php7.0-fpm, and not php5-fpm, and this will break existing site configurations

The Ubuntu Xenial packages for nginx have already been updated for this change, pointing to php7.0-fpm instead of php5-fpm.

However, users who have existing site configurations will not benefit from these changes. They must manually apply the changes.

Effectively, this is what a default setup uses to interface with the default php5-fpm setup on Ubuntu versions before Xenial, passing all PHP processing to the php5-fpm backend. This is from the default configuration file, but it’s still similar for all PHP passing:

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
    
        # With php5-cgi alone:
        #fastcgi_pass 127.0.0.1:9000;
        # With php5-fpm:
        fastcgi_pass unix:/var/run/php5-fpm.sock;
    }

In Ubuntu Xenial, the TCP listener for php7.0-cgi will be unchanged, however for php7.0-fpm, it will be necessary to update the configuration to look like this for existing site configurations:

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
    
        # With php7.0-cgi alone:
        #fastcgi_pass 127.0.0.1:9000;
        # With php7.0-fpm:
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    }

This will prevent HTTP 502 Bad Gateway errors, and will use the updated php7.0-fpm instead of the php5-fpm packages.

(If for some reason you still want to have php5-fpm under Xenial, you will not be able to get support from Ubuntu for this; you will need to use a PPA. I explain this on a different post on my blog.)

NGINX PPA Cleanup

 NGINX, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA  Comments Off on NGINX PPA Cleanup
Feb 052016
 

The NGINX PPAs have had some cleanup done to them today.

Previously, the PPAs kept the ‘older’ package versions in them for now-EOL releases (this included keeping ancient versions for Maverick, Natty, Oneiric, Quantal, Raring, Saucy, and Utopic). This was decided upon in order to prevent people from seeing 404 errors on PPA checking. We also included a large list of “Final Version” items for each Ubuntu release, stating there would be no more updates for that release, but keeping the ancient packages in place for installation.

Looking back on this, this is a bad thing for multiple reasons. Firstly, it means people in ‘older releases’ can still use the PPA for that release. This means security-holed versions of NGINX could still be used. Secondly, it implies that we still ‘support’ the use of older releases of Ubuntu in the PPAs. This has the security connotation that we are OK with people using no-longer-updated releases, which in turn have their own security holes.

So, today, in an effort to discourage the use of ancient Ubuntu versions which get no security updates or support anymore, I’ve made changes to the way that the PPAs will operate going forward: Unless a release recently went End of Life, versions of the nginx package in the PPAs for older Ubuntu releases are no longer going to be kept, and will be deleted a week after the version goes End of Life.

Therefore, as of today, I have deleted all the packages in the NGINX PPAs (both Stable and Mainline, in both staging and release PPAs) for the following releases of Ubuntu:

  • Maverick (10.10)
  • Natty (11.04)
  • Oneiric (11.10)
  • Quantal (12.10)
  • Raring (13.04)
  • Saucy (13.10)
  • Utopic (15.04)

People still using ancient versions of NGINX or Ubuntu are strongly recommended to upgrade to get continued support and security/bug fixes.

Landscape and Gitlab on the same server: Headaches, and thoughts.

 GitLab, Landscape, NGINX, Ubuntu  Comments Off on Landscape and Gitlab on the same server: Headaches, and thoughts.
Aug 262015
 

This is a mini case study, or rather a report from me, on how difficult it can be to run multiple services from the same server. Especially when they listen on similar ports for different aspects. In this post, I examine the headaches of making two things work on the same server: GitLab (via their Omnibus .deb packages), and Landscape (Canonical’s systems management tool).

I am not an expert on either of the software I listed, but what I do know I will state here.

The Software

Landscape

Many of you have probably heard of Landscape, Canonical’s systems management tool for the Ubuntu operating system. Some of you probably know about how we can deploy Landscape standalone for our own personal use with 10 Virtual and 10 Physical machines managed by Landscape, via Juju, or manually.

Most of my systems/servers are Ubuntu, and I have enough that makes management by one individual a headache. In the workplace, we have an entire infrastructure set up for a specific set of applications, all on an Ubuntu base, and a similar headache in managing them all one at a time. For me, discovering Landscape Dedicated Server, the setup yourself, makes management FAR easier. Landscape has a dependency on Apache

GitLab

GitLab is almost like GitHub in a sense. It provides a web interface for working with code, via the Git Version Control System. Github and GitLab are both very useful, but for those of us wanting the same interface in only one organization, or for personal use, and not trusting the Cloud hosts like GitHub or GitLab’s cloud, we can run it via their Omnibus package, which is Gitlab pre-packaged for different distributions (Ubuntu included!)

It includes its own copy of nginx for serving content, and uses Unicorn for the Ruby components. It listens on both port 80 and 8080, initially, per the gitlab configuration file which rewrites and modifies all the other configurations for Gitlab, which includes both of those servers.

The tricky parts

But then, I ran into a dilemma on my own personal setup of it: What happens if you need Landscape and multiple other sites run from the same server, some parts with SSL, some without? Throw into the mix that I am not an Apache person, and part of the dilemma appears.

1: Port 8080.

There’s a conflict between these two softwares. Part of Landscape (I believe the appserver part) and part of GitLab (it’s Unicorn server, which handles the Ruby-to-nginx interface both try and bind to port 8080.

2: Conflicting Web Servers on Same Web Ports

Landscape relies on Apache. GitLab relies on its own-shipped nginx. Both are set by default to listen on port 80. Landscape’s Apache config also listens on HTTPS.

These configurations, out of the box by default, have a very evil problem: both try to bind to port 80, so they don’t work together on the same server.

My solution

Firstly, some information. The nginx bundled as part of GitLab is not easily configured for additional sites. It’s not very friendly to be a ‘reverse proxy’ handler. Secondly, I am not an Apache person. Sure, you may be able to get Apache to work as the ‘reverse proxy’, but it is unwieldy for me to do that, as I’m an nginx guy.

These steps also needed to be done with Landscape turned off. (That’s as easy as running sudo lsctl stop)

1: Solve the Port 8080 conflict

Given that Landscape is something by Canonical, I chose to not mess with it. Instead, we can mess with GitLab to make it bind Unicorn to a different port.

What we have to do with GitLab is tell its Unicorn to listen on a different IP/Port combination. These two lines in the default configuration file control it (the file is located at /etc/gitlab/gitlab.rb in the Omnibus packages):

# unicorn['listen'] = '127.0.0.1'
# unicorn['port'] = 8080

These are commented out by default. The default binding is to bind to 127.0.0.1:8080. We can easily change GitLab’s configurations though, by editing the file, and uncommenting both lines. We have to uncomment both because otherwise it tries to bind to the specified port, but also *:8080 (which breaks Landscape’s services). After making those changes, we now run sudo gitlab-ctl reconfigure and it redoes its configurations and makes everything adapt to those changes we just made.

2: Solve the web server problem

As I said above, I’m an nginx guy. I also discovered revising the GitLab nginx server to do this is a painful thing, so I did an ingenious thing.

First up: Apache.

I set the Apache bindports to be something else. In this case, I revised /etc/apache2/ports.conf to be the following:

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen 10080


Listen 10443


Listen 10443

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Now, I went into the sites-enabled configuration for Landscape, and also changed the bindports accordingly – the HTTP listener on Port 80 now listens on 10080, and the SSL listener on Port 443 now listens on 10443 instead.

Second: GitLab.

This one’s easier, since we simply edit /etc/gitlab/gitlab.rb, and modify the following lines:

#nginx['listen_addresses'] = ['127.0.0.1']
#nginx['listen_port'] = 80

First, we uncomment the lines. And then, we change the 'listen_port' item to be whatever we want. I chose 20080. Then sudo gitlab-ctl reconfigure will apply those changes.

Finally, a reverse proxy server to handle everything.

Behold, we introduce a third web server: nginx, 1.8.0, from the NGINX Stable PPA.

This works by default because we already changed all the important bindhosts for services. Now the headache: we have to configure this nginx to do what we want.

Here’s a caveat: I prefer to run things behind HTTPS, with SSL. To do this, and to achieve it with multiple domains, I have a few wildcard certs. You’ll have to modify the configurations that I specify to set them up to use YOUR SSL certs. Otherwise, though, the configurations will be identical.

I prefer to use different site configuration files for each site, so we’ll do that. Also note that you will need to put in real values where I say DOMAIN.TLD and such, same for SSL certs and keys.

First, the catch-all for catching other domains NOT hosted on the server, placed in /etc/nginx/sites-available/catchall:

server {
listen 80 default_server;

server_name _;

return 406; # HTTP 406 is "Not Acceptable". 404 is "Not Found", 410 is "Gone", I chose 406.
}

Second, a snippet file with the configuration to be imported in all the later configs, with reverse proxy configurations and proxy-related settings and headers, put into /etc/nginx/snippets/proxy.settings.snippet:


proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

Third, the reverse-proxy configuration for Landscape, which is fairly annoying and took me multiple tries to get working right, placed in /etc/nginx/sites-available/landscape_reverseproxy. Don’t forget that Landscape needs SSL for parts of it, so you can’t skip SSL here:


server {
listen 443 ssl;

server_name landscape.DOMAIN.TLD;

ssl_certificate PATH_TO_SSL_CERTIFICATE; ##### PUT REAL VALUES HERE!
ssl_certificate_key PATH_TO_SSL_CERTIFICATE_KEY; ##### PUT REAL VALUES HERE

# These are courtesy of https://cipherli.st, minus a few things.
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
proxy_pass https://127.0.0.1:10443/;
}

location /message-system {
proxy_pass https://127.0.0.1:10443/;
}
}

server {
listen 80;
server_name landscape.DOMAIN.TLD;

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
return 301 https://landscape.DOMAIN.TLD$request_uri;
}

location /ping {
proxy_pass http://127.0.0.1:10080/;
}
}

Forth, the reverse-proxy configuration for GitLab, which was not as hard to make working. Remember, I put this behind SSL, so I have SSL configurations here. I’m including comments for what to put if you want to NOT have SSL:

# If you don't want to have the SSL listener, you don't need this first server block
server {
listen 80;
server_name gitlab.DOMAIN.TLD

# We just send all HTTP traffic over to HTTPS here.
return 302 https://gitlab.DOMAIN.TLD$request_uri;
}

server {
listen 443 ssl;
# If you want to have this listen on HTTP instead of HTTPS,
# uncomment the below line, and comment out the other listen line.
#listen 80;
server_name gitlab.DOMAIN.TLD;

# If you're not using HTTPS, remove from here to the line saying
# "Stop SSL Remove" below
ssl_certificate /etc/ssl/hellnet.io/hellnet.io.chained.pem;
ssl_certificate_key /etc/ssl/hellnet.io/hellnet.io.key;

ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
# Stop SSL Remove

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
proxy_pass http://127.0.0.1:20080/;
}
}

System specifications considerations

Landscape is not light on resources. It takes about a gig of RAM to run safely, from what I’ve observed, but 2GB is more recommended.

GitLab recommends AT LEAST 2GB of RAM. It uses at least that, so you should have 3GB for this at the minimum.

Running both demands just over 3GB of RAM. You can run it on a 4GB box, but it’s better to have double that space just in case, especially if Landscape and Gitlab both get heavy use. I run it on an 8GB converted desktop, which is now a Linux server but used to be a Desktop.

Nginx 1.9.3 in PPAs, and retiring of Utopic Uploads for both PPAs

 NGINX, NGINX Mainline PPA, NGINX PPA, NGINX Stable PPA  Comments Off on Nginx 1.9.3 in PPAs, and retiring of Utopic Uploads for both PPAs
Jul 222015
 

The latest Nginx Mainline version, 1.9.3, is now available in the Mainline PPA (link).


With this 1.9.3 upload to the PPAs, we are hereby retiring the Utopic release from both the NGINX Stable and NGINX Mainline PPAs. The Ubuntu Utopic 14.10 release EOLs tomorrow, July 23rd, 2015. We are not planning any additional uploads to affect Utopic, and are hereby considering those releases “disabled” for uploads and building. Packages as they exist in the PPA will continue to exist, but will not receive updates for Utopic.

NGINX: Mixing and Matching Ubuntu Repositories (and NGINX team PPAs on Launchpad) or Debian Repositories with Upstream Repositories will result in problems.

 Debian, nginx, NGINX, NGINX PPA, Server Packages, Ubuntu  Comments Off on NGINX: Mixing and Matching Ubuntu Repositories (and NGINX team PPAs on Launchpad) or Debian Repositories with Upstream Repositories will result in problems.
Apr 042015
 

We’ve seen this before, but we see it too frequently. People want the latest NGINX version. So they use the upstream nginx.org repository to get it. They try and install, and you immediately get conflicts.

In Ubuntu, people then file bugs on this thinking it’s an Ubuntu issue (such as this bug here). Problem is, it’s not an Ubuntu bug. Nor is it a bug in the PPAs I maintain. Nor is it a bug in Debian. It’s a problem that arises when you mix the upstream repositories and either Ubuntu, Debian, or the Ubuntu PPAs, and assuming you can ‘upgrade’ cleanly with the upstream repositories.

Basically, this is what it comes down to:

For Debian, if you’re depending on third party modules, you should stick to Debian’s packaging and wait. For Ubuntu, you should use the PPAs which I maintain (under the nginx team on Launchpad) if you want latest software and features, based off of Debian’s packaging decisions.

If you want/depend on NAXSI though, you have no choice but to recompile NGINX with NAXSI yourself, in order to get it working in a sane way. Neither Debian, nor Ubuntu, nor the PPAs have naxsi in the builds anymore.

If none of those apply to you, you will have to purge all nginx binaries from your computer, and all nginx packages and configuration files with this command before installing from the upstream repository: sudo apt-get purge nginx nginx-doc nginx-common (This should also purge the other dependent packages as well)

But, if you’re curious why you can’t mix the repositories, this post explains it from my perspective. Here’s a breakdown of why you cannot mix repositories such as this, and the problems you run into.


Problem 1: Debian / Ubuntu / NGINX PPAs (maintained by yours truly) have flavors; nginx upstream does not.

And by flavors, I mean nginx-light, nginx-full, nginx-extras, nginx-naxsi (up until 1.6.2-2), and nginx-core (Ubuntu only, since Ubuntu 14.04). Each of these flavors contains a different set of modules, based on demand originating in Debian or the community (and ultimately implemented in Debian). I will not go into the differences here, however you can go to here and read my answer to the question for more details.

As a result of how NGINX modules are currently incorporated into the program’s binaries, it is absolutely critical to separate out the configuration files and default sample files and locations so that you can switch between flavors (and upgrade between versions between Ubuntu/Debian/PPAs) without issues and conflict between configuration files. This requires the introduction of a package called nginx-common – a package which contains files and other items that are common to all versions of the nginx flavors in those versions of the source package. This nginx-common is wholly the brainchild of Debian’s work, and inherited in Ubuntu and the PPAs I maintain.

The problem is: NGINX upstream does not ship ‘flavors’. They enable all the default modules that are shipped in the nginx upstream ‘core’ code, and do not include any third party modules, nor do they discriminate the modules to enable (to make ‘light’ builds, or ‘full’ builds). As such, the nginx upstream package is a single solitary ‘nginx’ package. It does not separate out configuration files, nor does it separate out the binaries.


Problem 2: NGINX upstream doesn’t have third party modules.

Now, I know what you’re thinking: “Why would Debian/Ubuntu include third-party modules in the packaging?” Turns out, in Debian, the demand for such ‘third party extensions’ was so high, that they decided to include the modules. Such modules include the nginx-lua module, the nginx-perl module, and even the NAXSI modules for NGINX (which were in nginx-naxsi up until 1.6.2-2).

The problem: These are third party modules, maintained separately from the NGINX code base itself. They’re shipped as part of some of Debian’s packages due to demand, but they in and of themselves can cause packaging issues and conflicts, to the point where it requires updating the modules’ code with each new release to fix issues in those modules. This in and of itself lends to ‘maintainability’ problems. This is why the nginx-naxsi flavor was dropped from NGINX in Debian and Ubuntu as of package version/revision 1.6.2-2. To fix even simple bugs in the nginx-naxsi flavor (and its related packages), the entire NAXSI module needed to be removed and replaced with the latest upstream revisions of the code and plugin. This means that to even fix bugs in how the NAXSI rules were handled (in order to match the actual rule formats that needed to be in place for whitelists and such), you’d have to do a replace of the entire NAXSI module in the nginx-naxsi flavor. In Ubuntu, this would break the ‘Stable Release Update’ in that new features would be added to the package that could break things, old features could be removed, and it would go beyond the ‘nitpick fix’ that’d be needed for a Stable Release Update, such that the system would never be fixed.

As a result, third party modules have to be maintained and updated with almost every code update from NGINX upstream. For NGINX Mainline, the Lua module needed to be updated three times for build failure fixes in the PPAs. Since a lot of these third party modules (such as NAXSI or Lua) are in demand by the community, but not available in the NGINX Upstream repository (or in the case of the NAXSI release, even the PPAs nowadays), you should not mix repositories, as you will lose those modules, or lose some of the modules and gain others.