My Secret for Multiple Python Environments without Ubuntu Packages for them: `pyenv`, and `virtualenv`.

 Uncategorized  Comments Off on My Secret for Multiple Python Environments without Ubuntu Packages for them: `pyenv`, and `virtualenv`.
Dec 042017

Been a while since I last posted, heh. Been busy, unfortunately.

In any case, I ran into multiple problems recently. In testing my code for some Python scripts I was writing, they functioned fine in Python 3.5, but failed on a newer system running Python 3.6.3 (but worked somehow on 3.6.1). I ran into a small headache, because my 16.04 system only has Python 3.5 available on it.

So, I had four choices:

  1. Use multiple systems (virtual or bare metal) or containers to get newer Ubuntu versions, which could be a headache.
  2. Upgrade my own system to a newer version (which I don’t want to do, because I track LTSes for my regular-use laptop)
  3. Try and make multiple Python versions on my system in a way that doesn’t clobber the system packages, or
  4. Python virtualenv and pyenv.

Now, fortunately for me, I’m a power user. I could have done most of these options.

I could have done option 1, because my computer can handle amazing things like VMs, or I could have used LXD to install a newer OS image and used that for testing. However, that requires me to duplicate whatever I’m working on and run it separately or run its tests separately, which can cause some… headaches. Mainly because I can’t use PyCharm IDE to run everything locally and test, or use its debugger for the newer version.

Option 2 is a no-go – my preference is to stick to LTSes for my system, and not use the cutting-edge because of problems that usually arise.

Option 3 is painful because I have to compile each version I need from source. Granted, I’d have to do the same for any newer unpackaged version, or install versions made by others which could be malicious.

This leaves option 4 – use virtualenv for custom project directories and execution environments, or for PyCharm IDE, set up a pyenv with a given Python version, and point PyCharm IDE at that. This option works because everything runs in userspace for these environments and doesn’t require me to do much messing around with my system.

Now, you’re probably wondering a couple of things: what are virtualenv and pyenv, and why should I use it. I’ll address both. I’ll also address how to get all this set up.

What is virtualenv and pyenv, and what do I use them for?

This is a two part question. Put simply, pyenv provides a method for creating Python installations and environments in a local directory. Usually, within the user’s home directory, in a .pyenv folder, with a fairly complicated setup. You can also create environments based on standard Python versions and use those environments as-is. This allows you to create multiple Python setups for multiple Python versions. I mentioned that pyenv is mostly deprecated. It’s been replaced with, essentially, `python3

That said, there’s also a headache – if you want to install a specific program that needs to be its own new item via pip3 it can cause some issues. Especially if you want to not have to alter your pristine environment. A prime example of this is the PostgreSQL administration tool called pgadmin4 because it needs a newer Python version than what Ubuntu 16.04 has. For these cases (with Python 3.0 and above only), I also pair pyenv with the standard python3-virtualenv software. I used virtualenv to create a pgadmin4 environment based on Python 3.6.3. This allowed me to make a pristine environment for pgadmin4 to modify. This has its own ups and downs, but sorta worked for what I need.

However, for my other programs and scripts I develop, I only need the standard Python 3.6.3 environment. To that end, I will only explain the use of pyenv here.

Installing pyenv, and making it work.

Now, this is the fun part. I started with the instructions here, which is a pretty good guide for 15.10. It’s also valid here, so I’ll bring you the relevant steps.

  1. Install dependencies.
    sudo apt-get update && sudo apt-get upgrade sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev git
  2. Download the pyenv script.
    curl -L | bash
  3. Update your .bash_profile or .bashrc file with the following:
    export PATH="~/.pyenv/bin:$PATH"
    eval "$(pyenv init -)"
    eval "$(pyenv virtualenv-init -)"
  4. If you want to make this work instantly without restarting your terminal, run source ~/.bashrc. Otherwise, restart your Terminal or SSH session.

That’s really all to make that work. However, there’s a lot of usage parts of relevance here.

Install a different Python version with pyenv

This is the easy part. It’ll download and compile Python versions from the source tarballs from Python upstream. It’s useful because you can get multiple Python environments.

Let’s start by installing an oddball Python version, like Python 2.5 (specifically, 2.5.6, the last version of the 2.5.x series) which is out of date. Just run pyenv install 2.5.6, let it download, and then compile and install it.

But, how do I use it?

Well, let’s assume we’re in my computer, where the python3 executable on the system itself is actually 3.5.2. I need 3.6.3. I ran pyenv install 3.6.3 and it installed. Now, how do I use it?

Well, that is the usefulness of the pyenv software. I can change the local Python environment temporarily. By running pyenv local 3.6.3, the soystem will put me into the 3.6.3 directory. This is useful, because if I run python3 after executing pyenv local 3.6.3, the Python 3 version is 3.6.3 for that execution session. And if you want to switch back to the system’s 3.5.2 version, just run pyenv local system to switch back to the standard System version of Python, which is 3.5.2 on a 16.04 system.

You mentioned PyCharm IDE… how do I use this with pyenv?

Well thankfully, we don’t actually have to do any of the pyenv software via PyCharm IDE. PyCharm is smart enough to be able to detect obvious locations such as those in the pyenv system’s installation locations, and can offer them as usable environments. You just have to adjust the environment settings for PyCharm for the given project (or execution or runtime configuration) to use the alternate Python version as the runtime environment. This is fairly straightforward, but as I prefer to not develop in odd systems, I just have to point PyCharm to the local Python environment. Which is not as easy, because I ahve to add it into PyCharm. For any given PyCharm project, there’s a “Project Interpreter” option setting. You can choose the specific interpreter to use.

Which means you can choose the Python version to test and program against. When you go into there, there is a small gear icon next to the interpreter selection dropdown. Go into there, choose “More…”, and if you’re lucky, you’ll see the Python version in there. If not, we just have to add it. It’ll be in /home/$USER/.pyenv/versions/VERSION/bin/pythonX.Y, where you replace $USER with your username, VERSION with the version number you installed, and X.Y with the major and minor version numbers (for example, 3.6 for 3.6.3. Then PyCharm can work with that version.

It’ll be a bit more complex than this, of course, but this is the basics. downtime – May 03, 2016

 Uncategorized  Comments Off on downtime – May 03, 2016
May 012016 may have temporary downtime on May 3rd, 2016 for system updates on the underlying server.

Downtime is not a guarantee, but should downtime occur, it is not predicted to be down for more than 30 minutes.

If you have any questions about this downtime, please contact Thomas Ward with those questions.

Ubuntu Xenial: Adding php5.6 to Xenial

 Uncategorized  Comments Off on Ubuntu Xenial: Adding php5.6 to Xenial
Apr 122016

Ubuntu Xenial will not ship php5 at all.

The only way to get continued php5 access is to use a PPA, specifically Ondřej Surý’s PPA for co-installable php5 and php7.0. However, this is not supported by the Ubuntu Server Team or the Ubuntu Security Team, and you accept the risks therein of using PPAs for getting php5.

The packages are *not* named php5 but are instead named php5.6.

So, to add php5.6-fpm to Xenial, you would do something like this to add the PPA, update, and then also install php5.6-fpm and dependencies:

sudo apt-get install python-software-properties
sudo LC_ALL=en_US.UTF-8 add-apt-repository ppa:ondrej/php
sudo apt-get update
sudo apt-get install php5.6-fpm

(Note that I have not tested this; this is, however, supposedly usable based on user experience data gathered on Ask Ubuntu by myself.)

This should be a similar process for any of the other php5.6 packages you would need. However, you do NOT need to re-add the PPA if it’s already on your system. now operational

 Uncategorized  Comments Off on now operational
Feb 052016

Thanks to a joint effort between Thomas Ward ( providing server power and disk space and Simon Quigley ( providing the drive behind the project, the Ubuntu Wisconsin LoCo Team’s Planet feed aggregation site, here, is now operational.

Please contact Simon Quigley for information about this feed aggregator (and how to get added to it), or the Ubuntu Wisconsin LoCo Team.

If there is a technical issue with this site, or if the site is not responding for some reason, please contact Thomas Ward.

NGINX Webserver Admins: Don’t Use SSLv3 in Your SSL-Enabled Sites!

 nginx, NGINX PPA, Server Packages, Ubuntu, Uncategorized  Comments Off on NGINX Webserver Admins: Don’t Use SSLv3 in Your SSL-Enabled Sites!
Oct 242014

The SSLv3 “POODLE” Vulnerability.

Most of us are aware of the recent protocol flaw vulnerability in SSLv3. Officially designated CVE-2014-3566, it is more commonly referred to as the “POODLE” (Padding Oracle On Downgraded Legacy Encryption) vulnerability.

The vulnerability is a result of a flaw in the way that the (now old) SSLv3 protocol behaves and operates. There is a Ubuntu-specific question on the POODLE vulnerability on Ask Ubuntu (link) which answers common questions on it. There is also a more general question on the POODLE vulnerability on the Information Security Stack Exchange site (link) with more general details on the POODLE vulnerability. If you would like more details, you should refer to those sites, or read the OpenSSL Whitepaper on the POODLE vulnerability (link).

As this is a protocol flaw in SSLv3, ALL implementations of SSLv3 are affected, so the only way to truly protect against POODLE is to disable SSLv3 protocol support in your web application, whether it be software you write, or hosted by a web server.

Disable SSLv3 in nginx:

Since the recommendation is to no longer use SSLv3, the simplest thing to do is disable SSLv3 for your site. In nginx, this is very simple to achieve.

Typically, one would have SSL enabled on their site with the following protocols line or similar if using the example in the default-shipped configuration files (in latest Debian or the NGINX PPAs, prior to the latest updates that happened in the past week or so):
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

To resolve this issue and disable SSLv3 support, we merely need to use the following instead to use only TLS:
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

Note that on really old implementations of OpenSSL, you won’t be able to get TLSv1.1 and TLSv1.2, so at the very least you can just have TLSv1 on the ssl_protocols line. You should probably consider updating to a more recent version of OpenSSL, though, because of other risks/issues in OpenSSL.

Update OpenSSL to get TLS_FALLBACK_SCSV Support:

More importantly than just disabling SSLv3, you should definitely update your OpenSSL, or whatever SSL implementation you use, to receive support for TLS_FALLBACK_SCSV. There is an attack vector that would make you vulnerable to POODLE by starting a TLS session, but then falling back to SSLv3, and then open you to the POODLE vulnerability. By updating, and then having the use of TLS_FALLBACK_SCSV, you will be protecting yourself from protocol downgrading attacks which would also make you vulnerable to POODLE.

Ubuntu Users:


Fortunately for all users of Ubuntu, the OpenSSL packages were updated to protect against SSL downgrade attacks. This is detailed in “USN-2385-1: OpenSSL vulnerabilities” (link). Simply running sudo apt-get update with the security repositories enabled should get you the OpenSSL update to address this.

nginx from the Ubuntu Repositories:

Due to the vulnerability, and Debian already having these changes done, I was able to get in a last-minute update (courtesy of the Ubuntu Security Team and the Ubuntu Release Team), into the nginx package for the Utopic (14.10) release, which happened officially yesterday (October 23, 2014). In Utopic, the nginx package’s default config does NOT have SSLv3 on the ssl_protocols line. All other supported versions of Ubuntu do not have this change (this means that Precise and Trusty are both affected).

PPA Users:

Of course, many users of Ubuntu and nginx like the newer features of the latest nginx Stable or Mainline releases. This is why the nginx PPAs exist. Originally maintained by some of the Debian maintainers of the nginx package, I’ve taken primary responsibility of updating the nginx packages, and keeping them in sync (as close as I can) to the Debian nginx packaging.

As of today (October 24, 2014), both the Stable and Mainline PPAs have been updated to be in sync with the latest Debian packaging of the nginx package. This includes the removal of SSLv3 from the default ssl_protocols line.

Debian Users:


Fortunately, like Ubuntu, Debian has also updated the OpenSSL packages to protect against SSL downgrade attacks. This is detailed in “DSA-3053-1 openssl — security update” (link). Like in Ubuntu, this can be fixed by running sudo apt-get update or similar to update your packages.

nginx in the Debian Repositories:

If you are on Debian Unstable, you are in luck. The Debian package in Unstable has this change in it already.

If you are on Debian Testing or Debian Stable or Debian Old Stable, you’re unfortunately out of luck, this change isn’t in those versions of the package yet. You can easily do the aforementioned changes, though, and fix your configs to disable SSLv3.

nginx-core is now in Ubuntu Trusty 14.04 Main!

 Uncategorized  Comments Off on nginx-core is now in Ubuntu Trusty 14.04 Main!
Mar 132014

Thanks to the efforts of myself and others, we have been able to get NGINX into the Ubuntu Main repositories for Trusty 14.04!

Having said this, none of the already-established flavors of nginx are included in Ubuntu Main (nginx-light, nginx-full, nginx-extras, and nginx-naxsi). The Ubuntu Security Team has said that the third-party modules are wildly different in coding and therefore cannot be supported.

To that end, we created a package called nginx-core which has been included in the Main repository. This package contains only the modules that ship with the stock nginx tarball. We do not include any third-party modules with this package, just the modules that come from NGINX upstream.

Thanks to everyone on the MIR an Security teams for all their help in getting nginx into Main!

A Minimal IPTables Setup: WebServer, MySQL, PHP. (LAMP, for example)

 iptables, Ubuntu, Uncategorized  Comments Off on A Minimal IPTables Setup: WebServer, MySQL, PHP. (LAMP, for example)
Aug 032012

A few weeks ago, someone posted on Ask Ubuntu asking for a minimal IPtables setup for LAMP servers.

As you can guess by the only answer there, I posted the following iptables commands for this:
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p udp --dport 80 -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-unreachable

The above is extraordinarily minimal. Time to go through this, and figure out what I should have included, and what I intentionally left out.

First, we’ll look at the first command, iptables -A INPUT -i lo -j ACCEPT. This adds a rule which states that “Any traffic originating on the loopback interface, or localhost, is allowed.”.

The next rule, iptables -A INPUT -m conntrack –ctstate ESTABLISHED,RELATED -j ACCEPT, uses a specialized module, called ‘conntrack’, which tracks connection states. This rule states that pre-established connections, and related connections, are accepted and not blocked.

The next rule, iptables -A INPUT -p tcp –dport 22 -j ACCEPT, accepts all traffic on port 22 (for default SSH installations and setups).

The next two rules, iptables -A INPUT -p tcp –dport 80 -j ACCEPT and iptables -A INPUT -p udp –dport 80 -j ACCEPT, accept HTTP traffic on Port 80. Since HTTP traffic is both TCP and UDP, you need to have both.

The last rule in the above set, iptables -A INPUT -j REJECT –reject-with icmp-host-unreachable, uses a specialized target chain called “REJECT” which rejects the packets. The specific ICMP packet that will be used to reject the packet is “host unreachable” which terminates the networking connection to the server.

Now apparently, there are a few things I intentionally left out. Unless your server specifically hosts MySQL databases for external servers, you don’t need to allow MySQL traffic. Therefore, I did not add a rule to allow traffic from outside of the server itself related to MySQL (local mysql traffic is handled via the loopback rule that we first added). PHP has no reason to listen externally for requests, so it listens only on (in PHP 5.3.x, on Ubuntu Precise), or on the UNIX socket /var/run/php-fpm.sock (or similar, in PHP 5.4.x, on Ubuntu Quantal and later). Therefore, since PHP traffic will only be local, it’ll also be covered by that first rule, so no rule is needed for the PHP traffic.

I intentionally left out filtering of ICMP packets, such as pings. For an absolutely minimal setup of IPTables, you probably won’t want people to send you large pings (Pings of Death), or other ICMP packets outside of local traffic. I did not add ICMP filtering rules because I generally to not accept ICMP packets such as pings from external networks. I have a specialized program that establishes a TCP connection via HTTP and Curl automatically from my servers to one central server on an Amazon EC2 instance. The nginx access.log therefore will actually list those connections, and since I run that connection script every two minutes, its a semi-effective way of determining if the server infrastructure is online or not. If it’s not, i’ve got a nag-script that emails me if one misses a scheduled check-in several times, or if it hasnt responded in the past hour.