Running Django with FCGI and lighttpd

Diese Dokumentation ist für einen grösseren Kreis als nur .de gedacht, daher das ganze in NeuwestfälischEnglisch. Sorry.

Update: I maintain the actually descriptions now in my trac system. See the FCGI+lighty description for Django. There are different ways to run Django on your machine. One way is only for development: use the django-admin.py runserver command as documented in the tutorial. The builtin server isn't good for production use, though. The other option is running it with mod_python. This is currently the preferred method to run Django. This posting is here to document a third way: running Django behind lighttpd with FCGI.

First you need to install the needed packages. Fetch them from their respective download address and install them or use preinstalled packages if your system provides those. You will need the following stuff:

  • [Django][2] itself - currently fetched from SVN. Follow the setup instructions or use python setup.py install.
  • [Flup][3] - a package of different ways to run WSGI applications. I use the threaded WSGIServer in this documentation.
  • [lighttpd][4] itself of course. You need to compile at least the fastcgi, the rewrite and the accesslog module, usually they are compiled with the system.

First after installing ligthttpd you need to create a lighttpd config file. The configfile given here is tailored after my own paths - you will need to change them to your own situation. This config file activates a server on port 8000 on localhost - just like the runserver command would do. But this server is a production quality server with multiple FCGI processes spawned and a very fast media delivery.

   # lighttpd configuration file
   # 
   ############ Options you really have to take care of ####################

server.modules = ( "mod_rewrite", "mod_fastcgi", "mod_accesslog" )

server.document-root = "/home/gb/public_html/" server.indexfiles = ( "index.html", "index.htm", "default.htm" )

these settings attch the server to the same ip and port as runserver would do

server.errorlog = "/home/gb/log/lighttpd-error.log" accesslog.filename = "/home/gb/log/lighttpd-access.log"

fastcgi.server = (
"/myproject-admin.fcgi" => (
"admin" => (
"socket" => "/tmp/myproject-admin.socket",
"bin-path" => "/home/gb/public_html/myproject-admin.fcgi",
"min-procs" => 1,
"max-procs" => 1 ) ),
"/myproject.fcgi" => (
"polls" => (
"socket" => "/tmp/myproject.socket",
"bin-path" => "/home/gb/public_html/myproject.fcgi" ) ) )

url.rewrite = (
"^(/admin/.*)$" =>
"/myproject-admin.fcgi$1",
"^(/polls/.*)$" =>
"/myproject.fcgi$1" )

This config file will start only one FCGI handler for your admin stuff and the default number of handlers (each one multithreaded!) for your own site. You can finetune these settings with the usual ligthttpd FCGI settings, even make use of external FCGI spawning and offloading of FCGI processes to a distributed FCGI cluster! Admin media files need to go into your lighttpd document root.

The config works by translating all standard URLs to be handled by the FCGI script for each settings file - to add more applications to the system you would only duplicate the rewrite rule for the /polls/ line and change that to choices or whatever your module is named. The next step would be to create the .fcgi scripts. Here are the two I am using:

   #!/bin/sh
   # this is myproject.fcgi - put it into your docroot

export DJANGOSETTINGSMODULE=myprojects.settings.main

/home/gb/bin/django-fcgi.py

   #!/bin/sh
   # this is myproject-admin.fcgi - put it into your docroot

export DJANGOSETTINGSMODULE=myprojects.settings.admin

/home/gb/bin/django-fcgi.py

These two files only make use of a django-fcgi.py script. This is not part of the Django distribution (not yet - maybe they will incorporate it) and it's source is given here:

   #!/usr/bin/python2.3

def main(): from flup.server.fcgi import WSGIServer from django.core.handlers.wsgi import WSGIHandler WSGIServer(WSGIHandler()).run()

if name == 'main': main()

As you can see it's rather simple. It uses the threaded WSGIServer from the fcgi-module, but you could as easily use the forked server - but as the lighttpd already does preforking, I think there isn't much use with forking at the FCGI level. This script should be somewhere in your path or just reference it with fully qualified path as I do. Now you have all parts togehter. I put my lighttpd config into /home/gb/etc/lighttpd.conf, the .fcgi scripts into /home/gb/public_html and the django-fcgi.py into /home/gb/bin. Then I can start the whole mess with /usr/local/sbin/lighttpd -f etc/lighttpd.conf. This starts the server, preforkes all FCGI handlers and detaches from the tty to become a proper daemon. The nice thing: this will not run under some special system account but under your normal user account, so your own file restrictions apply. lighttpd+FCGI is quite powerfull and should give you a very nice and very fast option for running Django applications.

Problems:
  • under heavy load some FCGI processes segfault. I first suspected the fcgi library, but after a bit of fiddling (core debugging) I found out it's actually the psycopg on my system that segfaults. So you might have more luck (unless you run Debian Sarge, too)
  • Performance behind a front apache isn't what I would have expected. A lighttpd with front apache and 5 backend FCGI processes only achieves 36 requests per second on my machine while the django-admin.py runserver achieves 45 requests per second! (still faster than mod_python via apache2: only 27 requests per second)
Updates:
  • the separation of the two FCGI scripts didn't work right. Now I don't match only on the .fcgi extension but on the script name, that way /admin/ really uses the myproject-admin.fcgi and /polls/ really uses the myproject.fcgi.
  • I have [another document online][6] that goes into more details with regard to load distribution

tags: Django, Programmierung, Python, Sysadmin, Texte

Jan Kneschke July 27, 2005, 9:59 a.m.

If the FastCGI backend causes problems you might checkout the SCGI module: mod_scgi shipped with lighttpd since 1.3.14.

hugo July 27, 2005, 10:27 a.m.

I found out that it's not the FastCGI backend but the psycopg database adapter and multithreading. And since the codebase of Flup is based around the same server architecture for every protocol it's doubtfull that switching the protocol will change much with regard to my problem. I can circumvent it by not using the Flup threading code (setting maxSpare and maxThreads to 1) and using spawn-fcgi or the lighttpd builtin fcgi spawner instead.

Loevborg Sept. 6, 2005, 1:23 a.m.

With a recent version of lighttpd, you can disable the check-local option so that you don't need to create an empty physical file for your virtual .fcgi URL to work (cf. the documentation)

yejun March 5, 2006, 2:31 a.m.

Actually there is no need to using a shell file to change environment settings, since mod_fastcgi can change environment.

"bin-environment" => (
"DJANGO_SETTINGS_MODULE" => "dj.settings",
"PYTHONPATH" => "/var/www" )

notes: if an environment value already exists, it will not be replaced by lighttpd. unset that value before starting lighttpd.



Clinton May 23, 2006, 12:04 a.m.

Your Python code should be:

if __name__ == '__main__':

instead of:

if name == 'main':

Other than that minor glitch, a very informative and helpful article. Thanks!

Alexey Blinov Nov. 21, 2007, 10:44 a.m.

@Clinton
i'm sure that double underscore was eaten by markdown filter ;) Look at that bold name and 'main'