Monthly Archives: February 2010

Deploying a Pylons App to Production, Step-by-Step (Part 2 of 2)

In Part 1 of this tutorial, I described how to prepare Nginx (along with Apache) to serve a production Pylons app. In this article, I walk you through packaging and installing your application.

Packaging Your Pylons App

The official Pylons book offers a lengthy section on how to package your application. However, I feel that it there is a bit too much focus on how to prepare it for distribution. I think the bulk of beginning Pylons developers aren’t all that interested in putting their application on PyPI for the world at large to download (not yet, anyways). Therefore I’m going to distill the information into what you need to know to package you app for deployment into your own production environment.

Just as if we were going to post our project on PyPI, we’ll need to package it up into an .egg (which isn’t much more than a compressed tarball with installation metadata) so it can be installed using easy_install. The file that we’re most interested in is in your project root. Here is an example from one of my projects:
    from setuptools import setup, find_packages
except ImportError:
    from ez_setup import use_setuptools
    from setuptools import setup, find_packages

    description='My app which is most awesome',
    author='Your Name',
    package_data={'myawesomeapp': ['i18n/*/LC_MESSAGES/*.mo']},
    #message_extractors={'myawesomeapp': [
    #        ('**.py', 'python', None),
    #        ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),
    #        ('public/**', 'ignore', None)]},
    paster_plugins=['PasteScript', 'Pylons'],
    main = myawesomeapp.config.middleware:make_app

    main = pylons.util:PylonsInstaller

You’ll want to modify the obvious fields regarding your application information, plus figure out your version number. Since we’ll be pulling your source from Subversion, the version number will automatically be appended with the build number, so don’t worry too much about what to set as the version number. The Pylons book has a good overview.

The “install_requires” section contains a list of your project dependencies. These will automatically be pulled down from PyPI by setup tools when your application is installed. In this example, I have five dependencies other than the standard Pylons stuff. Your project will vary, but the dependency that you must list here is “flup”. flup is the WSGI module that we will be using with FastCGI and Nginx.

Once you’ve modified your, go ahead and check your project into your SVN repo one last time before we deploy it.

Creating production.ini

My production.ini file does not exist in the SVN repo. It’s contained separately on the production server and is copied over each time I deploy an updated version of an app. It’s very similar to the development.ini file, but has several important differences:

# myapp - Pylons development environment configuration
# The %(here)s variable will be replaced with the parent directory of this file
# Uncomment and replace with the address which should receive any error reports
#email_to =
smtp_server = localhost
error_email_from = paste@localhost

#use = egg:Paste#http
use = egg:Flup#fcgi_thread
host = localhost
port = 9000

use = egg:myapp
full_stack = true
static_files = true

DSN = dbname='ap' user='ap_dbo' host='my_db_server'

cache_dir = %(here)s/data
beaker.session.key = myapp
beaker.session.secret = somesecret

# If you'd like to fine-tune the individual locations of the cache data dirs
# for the Cache data, or the Session saves, un-comment the desired settings
# here:
beaker.cache.data_dir = %(here)s/data/cache
beaker.session.data_dir = %(here)s/data/sessions

# SQLAlchemy database URL
sqlalchemy.url = sqlite:///%(here)s/development.db

# Debug mode will enable the interactive debugging tool, allowing ANYONE to
# execute malicious code after an exception is raised.
set debug = false

# Logging configuration
keys = root, routes, myapp, sqlalchemy

keys = console

keys = generic

level = INFO
handlers = console

level = INFO
handlers =
qualname = routes.middleware
# "level = DEBUG" logs the route matched and routing variables.

level = DEBUG
handlers =
qualname = myapp

level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither.  (Recommended for production systems.)

class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

Line 13 is commented out and replaced with line 14. This instructs Paster to use WSGI/FastCGI via flup rather than answer the HTTP requests directly. Line 16 must match the TCP port specified on line 41 of nginx.conf (as discussed in Part 1 of this tutorial). Finally, line 41 disables the interactive debugger.

Deploying to Production

Before we can install our application, we need to install a virtual environment using the same process that was used when you first began developing your application. (I’m assuming that you used python-virtualenv here; if you’re using Buildout, set up your environment similarly).

cd /path/to/my/project/
tar zxfv virtualenv-1.4.5.tar.gz
cp  virtualenv-1.4.5/ ./
rm -r virtualenv-1.4.5
rm virtualenv-1.4.5.tar.gz
python --python=/usr/local/bin/python2.6 /path/to/my/project/env

In above example I’m using the “–python” option to specify that I want to use Python 2.6 in the virtual environment since that is the version I used to develop my app. If I hadn’t made the distinction, I would have ended up with the default interpreter on this server, Python 2.4

Now that we have the HTTP server installed and configured, our application package set up, and our virtual environment ready to go, it’s time to deploy! Here’s what we need to do:

  1. Stop our application if it’s running (e.g., if we’re upgrading an existing app)
  2. Checkout the HEAD revision from our SVN repo into a temporary directory on the production server.
  3. Use setup tools to package the app into an .egg
  4. Use easy_install to install/upgrade our app and install the dependencies we listed in
  5. Clean up our temp files
  6. Copy over our production.ini
  7. Start/restart the application

Luckly, I’ve created a shell script to automate this process for me. I would not recommend that you try to use this script verbatim. It’s not very smart, does zero error checking, and is tailored for my environment. I will update this post when I have the time to make it a bit more flexible and bulletproof. But it should be a good starting point:


# Save current directory
pushd . 

# Kill our current server if it's running
OLD_PID=`pgrep -f ${PROJECT_NAME}`
if [ "$?" -ne "1" ]
   kill ${OLD_PID}

# Remove any previous versions of our app
rm -R ${PROJECT_DIR}/env/lib/python2.6/site-packages/${PROJECT_NAME}*.egg

# Create our temp dir for the SVN checkout
mkdir $SVN_DIR

# Checkout the HEAD version from the repo
svn co ${SVN_REPO} $SVN_DIR --username $SVN_USERNAME --password $SVN_PWD

# Create our .egg from the SVN repo
${PROJECT_DIR}/env/bin/python2.6 bdist_egg
EGG_FULL=`ls ${SVN_DIR}/dist/*.egg`
EGG=`basename ${EGG_FULL}`

# Install/Upgrade application using our new .egg
${PROJECT_DIR}/env/bin/easy_install -U $EGG_FULL

# Copy over the production.ini
cp -f ${PROJECT_DIR}/deploy/production.ini ${PROJECT_DIR}/env/lib/python2.6/site-packages/$EGG/

# Recreate our sym link
rm ${PROJECT_DIR}/app
ln -s ${PROJECT_DIR}/env/lib/python2.6/site-packages/$EGG ${PROJECT_DIR}/app

# Nuke temp SVN dir
rm -R ${SVN_DIR}

# Jump into our app directory and restart the production daemon
cd ${PROJECT_DIR}/app
${PROJECT_DIR}/env/bin/python2.6 ${PROJECT_DIR}/env/bin/paster serve production.ini --daemon

# Restore original directory

And if you haven’t already done so, now start Nginx:

/etc/init.d/nginx start

Congrats! Your Pylons app is now deployed in production! The most challenging part about customizing this shell script for your use will probably be the SVN portion. I am using SVN over SSH which works very well, but does require a bit of configuration (which is beyond the scope of this article).

Conclusion / Caveats

There are a few things that I still need to address yet about this process:

  • I am having issues using OpenID and Authkit under this configuration with one of my Pylons apps. I believe it’s related to the Nginx FastCGI configuration. I need to spend some time developing a deeper understanding of FastCGI and Nginx and then will update my config files appropriately.
  • Paster will not start at system boot. This can be fixed rather easily by creating an init.d script for it. I have not yet done so, but there’s already at least one floating around on the web that will do the trick.
  • I have not set up logging in the production.ini file. This is important if you care about how well your app is running once in production. I will likely be making these changes ASAP.
  • needs quite a bit of work. I will likely rework it the next time I have a new Pylons app that needs to be deployed.

Having said that though, I’m finding this setup to be a joy to work with. I simply check-in my latest revision of my app, SSH into the production server, and run Bam! New version online in about 20 seconds.


Deploying a Pylons App to Production, Step-by-Step (Part 1 of 2)

I think there’s a very good reason why the “Deployment” chapter in the official Pylons book is listed in section entitled “Expert Pylons.” Deploying a Pylons app into a production environment can be a real hair-puller for the folks coming from the PHP or ASP/ASP.Net worlds.

Like most everything else about Pylons, you have a lot of choices when it comes to production deployment. The goal of this article is to take you through one particular process, step-by-step. This isn’t your only option, and (depending on your needs) may not even be the best choice for you. However, it’s worked well for me. The components that I will be using are:

  • CentOS 5.2
  • Apache 2 / Nginx 0.7.65
  • FastCGI / flup
  • Pylons 0.9.7
  • Subversion

Preparing the Production Environment

Installing Nginx

Nginx (“engine x”) is a lightweight HTTP server. Until this project, I’d never touched it. However, I decided to try it after I managed to bork my Apache install trying to recompile it with FastCGI support. After spending two hours fixing that, I was loathe to continue poking at the httpd beast with a stick. Additionally, I knew that reddit serves about 200M pageviews/mo. using Nginx in front of a Pylons app, so it made sense to at least explore it as another path.

I was very pleasantly surprised with the outcome. Even though I compiled from source and had zero prior experience with it, getting Nginx working was actually the easiest part of this project. It simply needs to be downloaded, compiled, and installed:

cd /var/src
tar zxvf nginx-0.7.65.tar.gz
cd nginx-0.7.65
make && make install

Finally, let’s add a new non-privileged account and adjust the file ownership:

adduser nginx
chown -R nginx:nginx /usr/local/nginx

Tada! You now have a working installation of nginx installed in /usr/local/nginx. If the config command yells at you, you probably need to install some dependencies. On one of the servers I use, it complained about not being able to find PCRE, however yum install pcre-devel quickly solved that problem.

You’ll most likely want Nginx to start and stop when the server boots/shutsdown, so go ahead and add the appropriate init.d script:

cd /etc/init.d/
mv nginx_initd.txt nginx
chmod +x nginx
chkconfig --add nginx
chkconfig --level 345 nginx on

If you’ve modified the default install location (by running ./configure --prefix=/my/new/path when you configured nginx), be sure to modify the init.d script appropriately. We’ll want to configure Nginx before we start it, but first we’ll deal with Apache…

Dealing with Apache

If you’re able to remove Apache from your existing server, or you’re building a box specifically for your Pylons app, you’re in an enviable position. Feel free to skip this section. However, for the rest of us wanting to run Nginx and Apache on the same server to maintain existing applications/websites, there are a couple of hoops that need to be jumped through first. You have three basic options when choosing to go this route:

  1. Add another public IP address to your server. Configure Apache not to listen on that new address and use it exclusively with Nginx.
  2. Proxy the Apache requests using Nginx.
  3. Proxy the Nginx requests using Apache.

Option one is certainly the most simple and is the preferable choice of the three. However, adding an additional IP may not an option for a lot of people. I personally run almost all my stuff on Amazon EC2 these days. The only way for me to get an additional IP address is by launching an additional instance and I’d rather not pay for that unless necessary.

Configuring Nginx to pass through (proxy) requests for my legacy sites to Apache is certainly doable. But being that a goal of mine is to leave the existing environment as untouched as possible, it probably makes the most sense to configure Nginx to run behind Apache rather than vice-versa. Fortunately, this isn’t difficult to configure. At the bottom of the /etc/httpd/conf/httpd.conf file add the following:


        ProxyPass / http://localhost:8080/
        ProxyPassReverse / http://localhost:8080/
        ProxyPreserveHost On
        ErrorLog /path/to/your/error_log
        CustomLog "|/usr/sbin/rotatelogs /path/to/your/access_logs/www.%Y%m%d.log 84500" combined

Depending on the complexity of your httpd.conf, it might be better to create this config in a seperate file and put it in the Apache config includes directory (/etc/httpd/conf.d). Lines #4 and #5 contain the TCP port number that you will be configuring Nginx to listen on. You may pick almost any port that you wish, but since you’ll be running Nginx under a non-privileged (non-root) user account for security reasons, the port has to be >1024. Feel free to change the “ErrorLog”, “CustomLog”, “ServerName”, and “ServerAlias” lines to suit your needs. Once the config is modified, give Apache a heads-up with a /etc/init.d/httpd reload command.

Configuring Nginx

Now that we know which port Apache is expecting Nginx to be listening at, we can finish the Nginx configuration. The file that we’re interested in is the main config file, /usr/local/nginx/conf/nginx.conf:

user nginx;
worker_processes  1;

error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/;

events {
    worker_connections  1024;

http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       8080;

        location / {
             include /usr/local/nginx/conf/fastcgi.conf;
             fastcgi_index index;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;


As previously mentioned, this is the first time that I’ve used Nginx so its config file is largely the default. The lines that I’ve modified/added are #1, #35-36, and #39-41. The port number specified in line #35 should match the port you chose in the Apache ProxyPass configuration. Or, if you’re not using Apache in front of Nginx (or have the two daemons bound to separate IP addresses), you’ll probably want to listen on the standard port 80. Line #41 is the host/port used by FastCGI to communicate with your Pylons app. Again, you may chose almost any arbitrary port, so long as it matches the value in your project’s production.ini (but we’ll get to that topic in Part 2).


At this point, you should have your production environment ready to go and awaiting your Pylons application. In Part 2 of this tutorial I will discuss how to package up the application for deployment directly from your Subversion repository and then fire it up in a live environment.


Adding a “disable” feature to the Ajax.Autocompleter

Scriptalicious’s Ajax.Autocompleter control is pretty cool. It allows a developer to add a drop-down auto-completion feature to an input box with one line of Javascript and a single div tag in the HTML. However (surprisingly), it doesn’t provide a means to turn off the autocompletion feature once it’s attached it to the target text box. There’s no method or property to disable the feature and setting the object to null has no effect (because it doesn’t affect the observers, but we’ll get to that…)

Line #8 doesn’t work as expected:

var autocomplete;

if( $('search_box') ) {
    autocomplete = new Ajax.Autocompleter("search_box", "choices", "/Autocomplete", {'frequency': 0.2, 'minChars': 3});

function disableAutocomplete() {
    autocomplete = null;

The top Google result for this issue talks about patching’s “controls.js”. However, I rejected this solution because I typically load the libraries using Google’s JS API rather than serving them off my web server. Via Google the libraries load faster because they’re served off a CDN. Plus, I don’t have to pay for the bandwidth and it’s just generally more convenient.

Another solution I found is a ticket in RoR’s Trac system. Rather than hacking the source file, this used class inheritance to extend the functionality of the base Ajax.Autocompleter. This looked great because I could continue to use Google’s JS API. The bad news is that the ticket (and solution) was written over three years ago. It did not work with v1.8.x or prototype.js v1.6.1.

The primary reason why this class was broken was that the methods in which the prototype library keeps track of Event observers changed considerably between versions 1.5, 1.6, and 1.6.1. Long story short: I ended up rewriting the second solution to work with prototype.js 1.6.1 and v1.8.3.

My new Ajax.ToggleableAutocompleter class:

Ajax.ToggleableAutocompleter= Class.create();
Object.extend(Object.extend(Ajax.ToggleableAutocompleter.prototype, Autocompleter.Base.prototype), {
  initialize: function(element, update, url, options) {
    this.baseInitialize(element, update, options);
    this.options.asynchronous  = true;
    this.options.onComplete    = this.onComplete.bind(this);
    this.options.defaultParams = this.options.parameters || null;
    this.url                   = url;
    this.blurHandler           = new Array();
    this.keydownHandler        = new Array();
    this.keypressHandler       = new Array();    
  disable: function() {
    var el = this.element;

    if (this.blurHandler.length==0 && this.keydownHandler.length==0 && this.keypressHandler.length==0) {

    this.blurHandler.each( function(handler) {
      el.stopObserving('blur', handler);

    this.keydownHandler.each( function(handler) {
      el.stopObserving('keydown', handler);
    this.keypressHandler.each( function(handler) {
      el.stopObserving('keypress', handler);
  enable: function() {
    var ele=this.element;
    for (var i = 0; i < this.blurHandler.length; i++) {
      Event.observe(ele, "blur", this.blurHandler[i]);
    for (var i = 0; i < this.keydownHandler.length; i++) {
      Event.observe(ele, "keydown", this.keydownHandler[i]);
    for (var i = 0; i < this.keypressHandler.length; i++) {
      Event.observe(ele, "keypress", this.keypressHandler[i]);
  onComplete: function(request) {
  getUpdatedChoices: function() {

    var entry = encodeURIComponent(this.options.paramName) + '=' +

    this.options.parameters = this.options.callback ?
      this.options.callback(this.element, entry) : entry;

      this.options.parameters += '&' + this.options.defaultParams;

    new Ajax.Request(this.url, this.options);
  _registerHandlers: function() {
    var o = this;
    if ( !o.element.getStorage().get('prototype_event_registry').size() > 0 ) return;
    with( o.element.getStorage().get('prototype_event_registry') ) {
       var blurEvents = get('blur');
       var keypressEvents = get('keypress');
       var keydownEvents = get('keydown');
    if( blurEvents ) blurEvents.each(function(e){ 
    if( keypressEvents ) keypressEvents.each(function(e){
    if( keydownEvents ) keydownEvents.each(function(e){

To use it:

var autocomplete;

if( $('search_box') ) {
    autocomplete = new Ajax.ToggleableAutocompleter("search_box", "choices", "/Autocomplete", {'frequency': 0.2, 'minChars': 3});

function disableAutocomplete() {

function enableAutocomplete() {