Landscape and Gitlab on the same server: Headaches, and thoughts.

 GitLab, Landscape, NGINX, Ubuntu  Comments Off on Landscape and Gitlab on the same server: Headaches, and thoughts.
Aug 262015
 

This is a mini case study, or rather a report from me, on how difficult it can be to run multiple services from the same server. Especially when they listen on similar ports for different aspects. In this post, I examine the headaches of making two things work on the same server: GitLab (via their Omnibus .deb packages), and Landscape (Canonical’s systems management tool).

I am not an expert on either of the software I listed, but what I do know I will state here.

The Software

Landscape

Many of you have probably heard of Landscape, Canonical’s systems management tool for the Ubuntu operating system. Some of you probably know about how we can deploy Landscape standalone for our own personal use with 10 Virtual and 10 Physical machines managed by Landscape, via Juju, or manually.

Most of my systems/servers are Ubuntu, and I have enough that makes management by one individual a headache. In the workplace, we have an entire infrastructure set up for a specific set of applications, all on an Ubuntu base, and a similar headache in managing them all one at a time. For me, discovering Landscape Dedicated Server, the setup yourself, makes management FAR easier. Landscape has a dependency on Apache

GitLab

GitLab is almost like GitHub in a sense. It provides a web interface for working with code, via the Git Version Control System. Github and GitLab are both very useful, but for those of us wanting the same interface in only one organization, or for personal use, and not trusting the Cloud hosts like GitHub or GitLab’s cloud, we can run it via their Omnibus package, which is Gitlab pre-packaged for different distributions (Ubuntu included!)

It includes its own copy of nginx for serving content, and uses Unicorn for the Ruby components. It listens on both port 80 and 8080, initially, per the gitlab configuration file which rewrites and modifies all the other configurations for Gitlab, which includes both of those servers.

The tricky parts

But then, I ran into a dilemma on my own personal setup of it: What happens if you need Landscape and multiple other sites run from the same server, some parts with SSL, some without? Throw into the mix that I am not an Apache person, and part of the dilemma appears.

1: Port 8080.

There’s a conflict between these two softwares. Part of Landscape (I believe the appserver part) and part of GitLab (it’s Unicorn server, which handles the Ruby-to-nginx interface both try and bind to port 8080.

2: Conflicting Web Servers on Same Web Ports

Landscape relies on Apache. GitLab relies on its own-shipped nginx. Both are set by default to listen on port 80. Landscape’s Apache config also listens on HTTPS.

These configurations, out of the box by default, have a very evil problem: both try to bind to port 80, so they don’t work together on the same server.

My solution

Firstly, some information. The nginx bundled as part of GitLab is not easily configured for additional sites. It’s not very friendly to be a ‘reverse proxy’ handler. Secondly, I am not an Apache person. Sure, you may be able to get Apache to work as the ‘reverse proxy’, but it is unwieldy for me to do that, as I’m an nginx guy.

These steps also needed to be done with Landscape turned off. (That’s as easy as running sudo lsctl stop)

1: Solve the Port 8080 conflict

Given that Landscape is something by Canonical, I chose to not mess with it. Instead, we can mess with GitLab to make it bind Unicorn to a different port.

What we have to do with GitLab is tell its Unicorn to listen on a different IP/Port combination. These two lines in the default configuration file control it (the file is located at /etc/gitlab/gitlab.rb in the Omnibus packages):

# unicorn['listen'] = '127.0.0.1'
# unicorn['port'] = 8080

These are commented out by default. The default binding is to bind to 127.0.0.1:8080. We can easily change GitLab’s configurations though, by editing the file, and uncommenting both lines. We have to uncomment both because otherwise it tries to bind to the specified port, but also *:8080 (which breaks Landscape’s services). After making those changes, we now run sudo gitlab-ctl reconfigure and it redoes its configurations and makes everything adapt to those changes we just made.

2: Solve the web server problem

As I said above, I’m an nginx guy. I also discovered revising the GitLab nginx server to do this is a painful thing, so I did an ingenious thing.

First up: Apache.

I set the Apache bindports to be something else. In this case, I revised /etc/apache2/ports.conf to be the following:

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen 10080


Listen 10443


Listen 10443

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Now, I went into the sites-enabled configuration for Landscape, and also changed the bindports accordingly – the HTTP listener on Port 80 now listens on 10080, and the SSL listener on Port 443 now listens on 10443 instead.

Second: GitLab.

This one’s easier, since we simply edit /etc/gitlab/gitlab.rb, and modify the following lines:

#nginx['listen_addresses'] = ['127.0.0.1']
#nginx['listen_port'] = 80

First, we uncomment the lines. And then, we change the 'listen_port' item to be whatever we want. I chose 20080. Then sudo gitlab-ctl reconfigure will apply those changes.

Finally, a reverse proxy server to handle everything.

Behold, we introduce a third web server: nginx, 1.8.0, from the NGINX Stable PPA.

This works by default because we already changed all the important bindhosts for services. Now the headache: we have to configure this nginx to do what we want.

Here’s a caveat: I prefer to run things behind HTTPS, with SSL. To do this, and to achieve it with multiple domains, I have a few wildcard certs. You’ll have to modify the configurations that I specify to set them up to use YOUR SSL certs. Otherwise, though, the configurations will be identical.

I prefer to use different site configuration files for each site, so we’ll do that. Also note that you will need to put in real values where I say DOMAIN.TLD and such, same for SSL certs and keys.

First, the catch-all for catching other domains NOT hosted on the server, placed in /etc/nginx/sites-available/catchall:

server {
listen 80 default_server;

server_name _;

return 406; # HTTP 406 is "Not Acceptable". 404 is "Not Found", 410 is "Gone", I chose 406.
}

Second, a snippet file with the configuration to be imported in all the later configs, with reverse proxy configurations and proxy-related settings and headers, put into /etc/nginx/snippets/proxy.settings.snippet:


proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

Third, the reverse-proxy configuration for Landscape, which is fairly annoying and took me multiple tries to get working right, placed in /etc/nginx/sites-available/landscape_reverseproxy. Don’t forget that Landscape needs SSL for parts of it, so you can’t skip SSL here:


server {
listen 443 ssl;

server_name landscape.DOMAIN.TLD;

ssl_certificate PATH_TO_SSL_CERTIFICATE; ##### PUT REAL VALUES HERE!
ssl_certificate_key PATH_TO_SSL_CERTIFICATE_KEY; ##### PUT REAL VALUES HERE

# These are courtesy of https://cipherli.st, minus a few things.
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
proxy_pass https://127.0.0.1:10443/;
}

location /message-system {
proxy_pass https://127.0.0.1:10443/;
}
}

server {
listen 80;
server_name landscape.DOMAIN.TLD;

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
return 301 https://landscape.DOMAIN.TLD$request_uri;
}

location /ping {
proxy_pass http://127.0.0.1:10080/;
}
}

Forth, the reverse-proxy configuration for GitLab, which was not as hard to make working. Remember, I put this behind SSL, so I have SSL configurations here. I’m including comments for what to put if you want to NOT have SSL:

# If you don't want to have the SSL listener, you don't need this first server block
server {
listen 80;
server_name gitlab.DOMAIN.TLD

# We just send all HTTP traffic over to HTTPS here.
return 302 https://gitlab.DOMAIN.TLD$request_uri;
}

server {
listen 443 ssl;
# If you want to have this listen on HTTP instead of HTTPS,
# uncomment the below line, and comment out the other listen line.
#listen 80;
server_name gitlab.DOMAIN.TLD;

# If you're not using HTTPS, remove from here to the line saying
# "Stop SSL Remove" below
ssl_certificate /etc/ssl/hellnet.io/hellnet.io.chained.pem;
ssl_certificate_key /etc/ssl/hellnet.io/hellnet.io.key;

ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
# Stop SSL Remove

include /etc/nginx/snippets/proxy.settings.snippet;

location / {
proxy_pass http://127.0.0.1:20080/;
}
}

System specifications considerations

Landscape is not light on resources. It takes about a gig of RAM to run safely, from what I’ve observed, but 2GB is more recommended.

GitLab recommends AT LEAST 2GB of RAM. It uses at least that, so you should have 3GB for this at the minimum.

Running both demands just over 3GB of RAM. You can run it on a 4GB box, but it’s better to have double that space just in case, especially if Landscape and Gitlab both get heavy use. I run it on an 8GB converted desktop, which is now a Linux server but used to be a Desktop.