django - 15.7.2005 - 9.2.2006

Django Templates are not limited

shannon -jj behrens thinks that Django template language is limited - because it doesn't have functions with parameters to do html snippet reuse. Of course the official - and simplified - answer to this is, that Djangos template language is that simple by design, so that it can easily be learned by non-programmers (as often designers aren't necessarily programmers). This is a quite good reasoning, but I think it's a bit too simplified.

So here is the longer - more complete - answer to this accusition: the Django template language isn't limited at all. Yes, I know that the "include" and "block" tags aren't parameterizable and so aren't often that useful for more complex situations (at least if you don't want to end in namespace hell due to passing some template-globals in the context).

So what should you do if you notice that your templates would need more complex code? One way would be to precompute the data in the view function and pass it on via the context to the template - that way the template has the ready data and can directly present it.

But what to do if you can't precompute, because you are using generic views? You could wrap your generic view with your own code and call the original generic view in that function with the modified context. That way you have the same benefit as above - youre templates have the data readily available. If you have many view functions that all need the same context enrichment, you can write your wrapper as a decorator - and just decorate the generic views and use those decorated functions in your urlpatterns.

But what if even wrapping isn't the answer? Shouldn't there be some way to do more complex code without all that wrapping? Sure there is! The answer are custom template tags. This might sound like a bit of overkill, but believe me, writing some template tags isn't really that hard. There is documentation on using and extending the template system in python

An even easier way to write your own tags is to use the "simple_tag" or "inclusion_tag" helpers in django.template.Library. Those functions allow to build simple tags very easily - the inclusion tag will base it's output on some template snippet, so you can see it as a template function with paramerters. A lot of usage of custom templates is in the contrib/admin stuff.

The main problem with the newer stuff in the code is, there is documentation missing for it. Hopefully that will be solved over time. But please, if the next time someone tries to tell you that the Django Template Language is to primitive, don't believe him. The Django Template Language is easy to grasp for non-programmers - but it's very extensible for Python programmers. And you extend it in the language you like - in Python.

Django Paste - Ian is starting to integrate Django with paste (and paste deploy). I for one will most definitely try to support that, so his list of related tickets is already down by one. Paste deploy might even be taken as the future default FCGI/SCGI solution - because it uses the same FLUP lib, it is as capable as my scripts, but due to the structure of Paste, installation should be much easier (and might even be standard in the future with Python hosters).

Jacobian.org : Django performance tips - Jacob, one of the Django Core-Devs, writes about performance tuning for Django applications. Strongly aligns with my experiences.

setting user passwords in admin

A rather ugly - but still useful - monkeypatch:

# monkey-patch for auth.users
from django.models.auth import User

def user_pre_save(self):
 if not self.password.startswith('sha1$'):
 self.set_password(self.password)

User._pre_save = user_pre_save

Put this into your model file (or somewhere else that is loaded early on) and you will be able to set passwords in the admin by entering clear text passwords. If the password starts with 'sha1$', it is seen as already encrypted and nothing happens. If it doesn't start with that string, it is converted using the standard Django function for password encryption.

No, this isn't something that should go into core - it's far too ugly for that. But at least it allows you to set passwords through the admin, without requiring the user to calculate the actual password hash.

Blog Move

Well, here it is - I'm finally moving my weblog here to the new software - no more PHP for my main blog. Right now, both systems are running separately, I'm just synchronizing the content to the new blog. In the next few days, however, I will install a redirector here that redirects all important URLs to the new system. Most comments are transferred, only the comments on blogmarks are lost, the new software no longer has a separate page for links where comments could be placed - it doesn't make sense anyway, anyone who wants to discuss the links should use the contact options of the linked page.

Otherwise, the new system is of course completely created with Django - finally everything in Python. That was also the main reason. Moreover, the ever-increasing PageRank, all the many links and the - for my expectations huge - traffic became increasingly unsettling, something had to be done about that. And the simplest solution is still to change the domain.

Oh, by the way, feeds are also redirected, but if you want, you can already subscribe to the new feed at the new address.

If you notice anything about the new system, either write here or over there in the comments (where it then works). I have tested almost everything, but errors still creep in from time to time ...

Again something from the crafting front

Content-type: matter-transport/sentient-life-form - for those who want to get a taste of where my blog is headed. Not quite finished yet, some bugs in my software, a few things waiting for patches in Django, but overall I'm already quite satisfied.

JobControl - Django Projects - Trac - a simple job control system for Django, with which you can set up background jobs.

Some things annoy me terribly

For example, if umlauts are not processed cleanly - as with pre_populate_from in Django. Therefore, I no longer use this in my CMSProject, but simply fill the slug in _pre_save. And then let a corresponding routine run there. Although this is not really perfect, but at least usable ...

And yes, this is a test post for the function to create a slug from a title with umlauts.

A Test Framework for Django

DjangoTesting is part of my DjangoStuff project and is the start of a testing framework for Django, modeled after the testing framework that Ruby on Rails provides. Currently only model tests are implemented, request/response tests are planned.

The testing framework is built solely on unittest and django, so you don't need additional modules (besides my DjangoStuff project, of course). It provides python-based fixture notations (fixtures are just python classes with attributes in a DATA subclass) and a basic command line utility to make use of those tests and fixtures.

Tests and fixtures are stored in applications and projects, so you can have application specific tests (especially usefull with generic applications) and project-level tests that will integrate stuff over several applications.

I think a good testing framework would really be important for Django applications, especially for applications that should be shared between projects. But I do think that a good testing framework needs some banging on, too - so I started it as a small subproject on my own. But if it grows into something useful, I will opt for inclusion into Django trunk.

Case/When/Otherwise for Django

If you have any evil plans for a switch statement for Django (hia rjwittams! ), you might want to look into my TagLib. There is a case/when/otherwise statement in there. It's quite easy to use:

{% case variable %}
{% when "value1" %}
{% endwhen %}
{% when "value2" %}
{% endwhen %}
{% otherwise %}
{% endotherwise %}
{% endcase %}

The reason for the tag structure is that the django template parser only looks for parameterless block-closing tags in the parsefor function and so you can't just pull an easy one like this:

{% if condition %}
{% elif condition %}
{% else %}
{% endif %}

You would have to copy over much from the template parser to get a parsefor that looks for a token with a tag and parameters to close the current block.

So I opted for the scoped tags approach where the "case" tag only sets up a context variable "case" and populates it with a dictionary with "value" and "fired" - with the latter one a trigger that can be fired by any "when" tag to prevent other "when" tags or the "otherwise" tag to fire themselves. A bit ugly, but working.

Adhoc-Organization in CM-Systems

Adhoc organization is what I named the basic design decisions for my new content management system (blog system, personal wiki, digital image shoebox - whatever). It's coming along nicely, even though up to now I only used it as a sample application to make use of my little tools from the DjangoStuff pseudo-project. And it still is one of the best ways to see how tagging or searching or the new calendar tag or other stuff is used.

But it's coming along so good that I think I will be able to change over some sites in the near future. The basic design decisions are somewhat documented in the linked document in my trac-wiki. The main objective for me is to get something that I can use as easy for image presentation as for text presentation and that allows me to really integrate both parts. So that articles really can consist of a multitude of media and text.

It's quite fun to work on a project where you tear down the model and rebuild part of it from time to time, or make major refactoring decisions that leave you with a broken heap of python-bullshit for a while

"Fitting on" some framework

How do you know wether a framework fits your style of thinking? It's not as if you could just look into a mirror wether it suits you nicely, you need other ways to decide that. One way to decide it is productivity - how fast you get your project up and running.

But does that really tell you the whole story? What if the project would have been something completely different? Did you just hit the sweet spot of the framework? Where you just lucky?

One way to decide wether some framework, language or tool fits my style of working for me is to look at the basic abstractions this tool gives me. And to look how I can use them and how naturally they fit my thinking - do I stumble on problems, not immediately knowing what abstraction to use, what tool to pull? Or do things just fall in place?

I discovered quite early on that I am a bit uncommon in programming, in that I don't build my own abstractions and try to translate from them into what the language or framework gives me, but that I start to think directly in the abstractions and syntaxes given to me - but only if they match my way.

So that's for me the ultimate measurement of wether a framework really fits into my thinking: checking from time to time wether I try to do translations or wether stuff just flows. Reaching "the flow" is what it's all about for me nowadays.

So how does Django match up? Quite nicely. It really gives me what I need in most cases, there are only very few areas where "the flow" is broken, where I need to think around problems, start to do translations. One area is special behaviour of entry fields - this curently is done in Django with parameterized instances of predefined field classes. There is no really nice way to do subclassing, you end up copying code from other parts of the django source - definitely breaking "the flow".

But most other parts just fall into place: middleware for global management of the request-response span. Template loaders for - well - template loading (yes, it's not a big deal - but being able to write your own template loader really is helpfull). The urlpatterns -hey, that's really a cool idea, because of it's absolutely loose couplying you don't even try to model your urls after your code structure, but tend to design them. And that's how it should be.

Models just powerfull enough to really move the model-related functionality there (although the class MODULE stuff will make it even nicer, especially the kind of ugly module_globals thingy). It would be cool if model classes would support mixin classes, so that abstract apps could provide stuff that just would be referenced by users to add functionality. But you can solve many of those problems with generated classes - thanks to python introspection (although you need to know a bit about djangos model-magic).

Most complex stuff tends to go into template tags and generic views - my CMS project currently only has 3 view functions of it's own, the rest is abstracted away into generic views (for searching and tagging). Template tags could be a bit easier to write, especially the parser is too primitive - a library of helper functions for easy deconstructing the tag string would be good (hey, maybe I write one, the basics arealready in my SVN repository).

Template filters are a big of an ugly duckling - they don't see the request context, so they can't do much more than just take the incoming object and some constant parameters. I think they should get the context passed in, so that they could be a bit smarter, if needed (like allowing filters to resolve a parameter against the context).

Generic views are quite nice, too - even though I don't use the predefined ones that often. The main reason is that more often than not I end up in wrapping the generic views in some code that modifies their behaviour - and then it's quite often simpler to just roll my own. But they are great for first starts into areas, just tack them into your project and funcitonality is available. You can allways exchange them with your own view functions if you discover that you need to.

And the admin, the one thing that makes Django stand out in the crowd? In my first play-projects I loved it, in later ones I didn't use it (the Gallery doesn't need it), but with the CMS project I did the first one that makes really heavy use of it. And I have to say, I like it. It should get a bit more flexibility (the new_admin branch might help there, as it moves more stuff into templates, so they can be overridden), but overall it's really cool and usefull.

Two things, though, are definitely needed for the admin: full transaction support bound to request-response (ticket #9 in the django trac), because changing stuff and ending up with inconsistent tables is no fun. Like getting an exception because something broke in repr , so the log entry isn't written, but the object is written. Of course you don't notice it, go back, send again, end up with two objects and still no log message ...

The other thing that is needed: basic hooks for object-based authentication. Not a full blown ACL or anything like that, just some really simple hooks from the admin to the model that the user can define to tell the admin wether some object should be editable or should only be shown readonly. The main problem with the current solution is, it only handles full tables - you can't even tell the admin that some user can only work on the current site and can't change objects of other sites (my CMS project makes heavy use of the multi-site capability in Django - one admin server should manage multiple sites in one admin interface).

But all in all webapp building with Django is real fun. It's not only productive to me, it just feels natural to do things the Django way. So, yes, Django fits my thinking style. Seems to have hit home right on.

Markdown for Django

Django already includes a markdown filter (in contrib.markup), but I nonetheless rolled my own Markdown for Django mini-app. The main benefits are link integration with django models (by using generic model queries and get absolute url), a documentation generic view that handles language switching and a nice macro facility for markdown. Macros are a usefull way to extend markdown by writing Django template snippets that are called whenever the users calls the makro in his markdown source.

It was formerly part of the CMS project, but I think it's usefull in it's own and so much better put into the stuff pseudo-project.

generic search service for Django

If your Django application needs searching capabilities, you can roll your own. Or you can use my generic search view. This provides a parser for queries and a search machinery that is suiteable for moderate database sizes. It provides an extensible google-like syntax.

The main problem is that django doesn't support OR query combinations and that it doesn't support "icontainsnot" queries. So the search engine does multiple selects to get one query. It starts with the longest search word and goes down in size from that result set, restricting it from one step to the next. But since it needs to keep the last result set in memory (at least the list of IDs), if your database contains too much rows, this might pose problems to your server (especially if the users do silly queries that produce large resultsets).

Maybe in future this will learn some optimizations to make it work better with larger databases, but it's quite fine as a search engine for your blog or standard content management systems.

very simple view functions

Sometimes you have a bunch of really simple view functions in your django project (yes, this is for your, bitprophet! ). View functions that are actually not more than just a render to response call - take a template, take some data from the request, stuff them in and render a response. It's rather boring to write them down and it breaks the DRY principle. So what to do? Write your own generic view.

from django.core.extensions \
 import render_to_response

def simple_view(request, template, **kwargs):
 return render_to_response(
 template, kwargs)

That's all. A simple and nice view function that just does that - render a template. It even can be fed with context variables from the urlpattern. Use it like this in your urlconf:

urlpatterns = patterns('',
(r'^page/(?P<arg>.*)/$', 'cool.simple_view',
 {'template': 'app/mytemplate'}),
)

That way a /page/foo/ view would be routed to the 'app/mytemplate' template with a context that just includes the variable 'arg' with the value 'foo'. And you never will need to write those simple_view functions again. For extra spices you could throw in a context_instance = DjangoContext(request) into the render to response call to even get the authenticated user and stuff like that from the request.

Module Hacking for Django

Django itself constructs model modules dynamically from your model classes. That's what I used in my first take at the abstract tagging application. Now I found a better way in the current version - I can modify the dynamic module myself quite easily, generate a dynamic model class and push that into the model module. What it actually does is just mimicking what happens when Python defines a class - most stuff is done by the meta.ModelBase metaclass in Django, anyway. I only had to add some module hacking stuff. Python introspection rules!

What this gives you is a much cleaner interface to create the tagrelation class for your model - just a function call, no silly subclassing or superfluous assignment. Everything happens as by magic.

It is magic.

Tagging with Django

Since the question about how to do tagging with Django shows up quite often, I have written a small solution to this problem: AbstractTagging. This is a generic application and generic views that give you a very simple solution to add tagging to any model you have in your django apps. It's currently used by me in my CMS project. The source is in the stuff project.

It was a bit weird to build, because I had to dynamically construct a base class you can subclass in your models - this is because of the magic in django.core.meta, where model classes are turned into modules. But the result is quite nice, I think.

Using Django as a CMS

I am currently reworking one of my sites - Content-type: matter-transport/sentient-life-form. It was a Apache::MiniWiki based site before and is now in the transition to being a Django based site. The idea of the code for that site is to build a CMS based on Django that fully uses the Django admin. So the users should be able to do all management only in the admin, while the site itself behaves a bit like a Wiki. Autolinking, Autoediting of missing pages, Editlinks, Versioning (currently missing in the source) - all that should be done based on tools the Django admin provides.

This isn't for a full-blown site, though - the linked site is allmost empty, I never put much up there. It's more a project to dig deeper into the Django admin to see what it's like to work in it - so I know about that stuff when I start to build real projects.

The code itself is freely available - and there is already a nice thing in it. It's a template loader who pulls stuff from the database instead from the filesystem or from python eggs. It's "ticketed" at Django under #633, so it might make it into django.contrib some day.

Django i18n status

I worked a bit more on the i18n stuff in django today and finally switched my gallery to the i18n branch. You can now see the strings on that site with either english or german settings. Other languages get english output (so if you are in Germany and still see english strings - check the language settings in your browser, wether German is defined with higher priority than English).

The code works quite nice and I think I will give it a week or so to settle and than start to put finishing touches to it - like adding much more translation hooks to the django source.

IRC Logger update

The IRC logger is working fine, but I wasn't happy with the dependence on muh - so I wrote my own little logger bot in python, based on irclib. Does work fine and does only what I want it to do - logging. I allways feel a bit queasy when IRC bots have command structures and stuff like that and I actually don't need any of those ...

So now the project is mostly complete - just use the django admin to add channels to your database, point the logger bot to some IRC host and see how it joins channels and starts logging.

Oh, there are still things to do - for example the bot needs to rescan the list of channels so it notices newly added channels and leaves deleted channels (and maybe I should add channel activation/deactivation so I can switch off channels for some time without losing the archives), but for now it just logs #django and for that it's good enough.

IRC logger for #django

There now is a IRC logger for #django on freenode.net. Ever since the loglibrary broke down, I thought about rolling my own. So I started to build the stuff needed for an IRC logger. The interface itself is written with django, of course

As allways, the source is available in my trac instance. It's currently only running in #django, but it can easily be extended to other channels. And it stores log lines in a database, so I will be able to add searching and stuff like that. It already has a calendar view on the logs.

The logging itself is done with muh - a nice IRC proxy that allows logging to named pipes. Then there is a script fetch.py that pulls the lines from the named pipe and stores them in the database. The last part is the django-based viewer for those logs.

Update: the logging is now done with a dedicated IRC logger written in python. It's in the source tree as loggerbot.py.

I could make use of generic views in Django, only that I needed to parameterize them. I did that with a wrapper function that moves stuff from the keywords of a view function to the extra_lookup args and extra context keywords of generic views. You can see the code in the repository.

The rest is just standard Django stuff: generic templates (that make use of the cool regroup tag) and some custom tags for the calendar and the user colorization. A bit of model hackery and that's it, actually. Nice and simple. Took only a few hours to bang together, and that includes playing with muh and named pipes ...

i18n and django

Jacob did set up branch commit rights for me and a branch for i18n stuff. So I worked today on the ideas in the patch on ticket 65 by nesh. I did write the stuff mostly from scratch because I wanted some things a bit differently and now it is available for testing.

So first on how you can use the i18n stuff with your django checkout. You need to have a current svn trunk checkout and go to the root of your checkout and issue the following command:

svn switch http://code.djangoproject.com/svn/django/branches/i18n/

After that you should have a tree with my patches applied. I currently only translated very little stuff to make the patch and changes as small as possible, but I already added a german translation file with stuf for the admin index and the isAlphaNumeric validator. I think I will add some more stuff to the translations soon.

The patch only addresses the translation part - other things like date formatting, number formatting, timezone handling should go into different patches to make each one of them as small as possible. The translation object (that's the beast that is responsible for turning strings into their new form) is built on request. This gives us the chance to look at various places that might help in deciding what language to present to the user. The code starts by looking into the session for a django_language variable. If that isn't found, it looks into the cookies for a django_lanaguage cookie. If that isn't found, too, it digs at the HTTP headers. It looks for the Http-accept-language header and splits that up by languages and sorts them by preference. It will use the first language (ordered by preference, highest preferences first) found in the django messagefile repository. If none of those languages can be found, it will ultimately fall back to the default translation object that is defined by the LANGUAGE_CODE setting in your settings file.Message files can be stored in three different places: the django project message files are stored in the django.conf package in a locale subdirectory. This is much like the admin_media and admin template directories. The locale subdirectory is structured as is typical with locale storage: one subdirectory per language and a LC MESSAGES directory in there. The language domain for django message files is allways django . The next place where django looks for message files is in the project - if you have a locale directory in your project, you can store additional message files there. The third place is the application - you can have a locale directory besides your apps views directory. All locale directories are structured the same.

A translation object for a given language is actually a concatenation of four translation objects: first the application translation object. This will have a fallback to the project translation. That in turn will fallback to the global translation object which will fallback to the translation object for the default language. That way higher levels can override translations from lower levels and applications can provide their own translations.

The application for the translations is actually discovered by module introspection - it uses the view func to call on a URL to discover what application carries this view func and uses that to look for local translations. There are two tools provided to manage translations: make-messages.py and compile-messages.py. Both tools can be called in either the root of the django svn tree or in the project directory or the application directory. make-messages.py will scan the current directory and everything below that for strings to translate and will create a django.po file in the locale directory for the given language. compile-messages.py will just turn all .po files into .mo files. Adding translations is easy. In python code you just surround strings (only string constants!) with ('...') or ("..."). That will mark those strings for translation so that make-messages.py can pull them out and write them to the .po files. And it will translate the string on runtime, using the current translation object as discovered from the request. In templates there is the template tag {% i18n ('....') %} - same syntax as with python code, only you need to wrap it as a template tag. Those strings will be pulled from the .html files into the .po, too. The i18n tag supports string interpolation from the context: {% i18n ('blah %(blubb) s blubber') %} would first translate the string and then interpolate the context variable blubb into the translated result.

A hint: when writing strings to translate, don't use positional parameters for interpolation (the %s stuff) but use named parameters (%(blah) s) instead. That way people building translations can reorder the string without breaking your code - some languages have different orders from english.

Using the translations is easy, too: you just need to set your default language in the LANGUAGE_CODE setting and add the django.middleware.locale.LocaleMiddleware to your middleware setting. You need to put it to the top - especially in the admin it should come before the AdminUserRequired middleware - but it needs to be after the SessionMiddleware, if you use that.

That's it for a start, play with it and tell me when something goes wrong. Best place to tell is on the ticket in the django trac.

Routes for Python

Very interesting: Ben Bangert has ported Routes to Python. Routes is the core of the mapping of URLs to functions and back that is used in Ruby on Rails. So a general mechanism with which a Python object can be determined from a URL and a URL can be determined from a Python object - flexibly configurable.

Could also be interesting as an element in Django, as an alternative to the current URL pattern system. The current system elegantly provides the function to be called from a URL - but unfortunately there is not the same way back from the object or the function back to the URL.

In addition, Routes could also be interesting within WSGI projects - it elegantly solves a small sub-area and in such an abstract way that it should harmonize well with things like Python Paste(Ian Bickings Meta-Framework for WSGI applications).

DjangoScgi - Django Projects - Trac

Django with Apache and SCGI and Django with Apache and FCGI are two reworked documentations on how to get Django running with both FCGI and SCGI under Apache. I use the same parts as with my previous howtos, only that now SCGI is supported, too.

My gallery is currently running the Apache+SCGI setup, it's quite nice. The configuration in Apache is much nicer and cleaner than with the FCGI setup.

Since 2007 the links in this page didn't work any more, so I removed them.

Django Gallery Status

Again news from my gallery project - it's making good progress, even if some of my latest stuff isn't directly visible. I did a lot of reworking in the code and a lot of admin interface changes. Things in the source that might be interesting for other djangonistas:

• I still have a fully filesystem based gallery - but I added a database stored cache. So now when contents is pulled from the filesystem, the code directly checks with it's database cache and updates that accordingly. This led to a heavy rework of the code so that now actually all stuff is driven by model objects - the cache entries are just part of the django model. Makes code much easier and allows the next two changes.
• I added an automatic sitemap for galleries. First versions traversed the filesystem, but now it just uses the database cache to draw a folder hierarchy with thumbnails.
• I added RSS support. The main gallery selection references a RSS feed over all galleries and the folder and image views within a gallery reference a RSS feed only for images within that gallery. This makes use of the RSS framework of Django.
• the view that showed gallery folders and images was reworked to make much more use of django-like stuff: instead of lazy closures that are passed in to the template, I now use custom templates that help to reduce the code of the view drastically (ok, the code is moved to the taglib, but that's much better decoupling than before).

The result: the gallery is much faster, I have RSS up and running and I have automatic sitemaps. The code itself is much easier as it is more model-driven - the former clash between picturefolder model stuff on the one side and FSObject instances on the other side ist gone - and cache maintenance is automatic. And the XMLRPC integration is much faster, too. All in all some very useful internal changes.

Other changes are in the management interface where you now not only have the AJAX-stuff to change object names and hidden state, but can do that from a filemanager, too. That filemanager will be extended for simple management functions like moving, deleting etc., too. It already is able to create new folders.

News from the Gallery project

News from my Django Gallery project: it's coming along nicely. If you want sample code for AJAX with Django or XML-RPC with Django - look into it's source. I implemented parts of the blogger API, metaWeblog API and MoveableType API - just enough so that Photon can post pictures to my gallery. AJAX is used for logged in users to change the title of pictures and folders by just clicking the title and for activating the toolbox on the backside of pictures. Additionally the gallery includes the usual PIL stuff like thumbnailing - I am especially proud of the flipped corner look for folder thumbnails Other stuff worth looking into might be the usage of lazy evaluation to push stuff into the template without precalculating it - it's only calculated when the template actually uses it. And interesting might be the abstraction of the filesystem based content - only the base path is stored in the database model, the rest of the gallery is in the filesystem (and can so be easily managed with FTP, SSH or directly on the unix shell).

Additionally you might look into it for handling of authentication - user registration isn't yet done, but will follow some day. As will comments and RSS - but I have to write that, first. And I am starting to write documentation - docstrings in the code and documentation pages in the wiki.

DjangoGallery - sample app with sample installation

Mal wieder englisch, da auch für #django interessant I have uploaded the source and made available the repository and a trac instance for my django projects. The first one that's there is the DjangoGallery - that's what I am using at viele-bunte-bilder.de. I have written a first take at an installation instruction that shows how to get the gallery running on your own site - you could even integrate it into your own project (although you will have to do some minor source changes - mostly replacing the projectname "gallery" by whatever your project is called).

The application isn't fully done, yet. There are many plans on how to extend it, as it will be my main motiviation to remove all other gallery software I am running (I already replaced PHP Gallery and now I am targeting my old mod_perl stuff and last one will be attacking my Wordpress based gallery), so be warned that it will change over time.

If you find bugs, feel free to file tickets. You can file enhancement wishes, too - but since the main target of this project will be to replace my own other gallery projects, it's doubtfull that I will do much work besides what is needed to reach that goal. At least for now - there's no limit to what can be done to the code after I phased out all PHP and Perl code

Since 2007 these links are non-functional, so I removed them.

first Django application live

So, my first Django application is live - not finished yet, but at the moment already so good that it's better than the old PHP mess that was running before. And it's about my image gallery at viele-bunte-bilder.de. At the moment there is user registration and everything related to it - but I'll probably implement that later. Or maybe not. Let's see.

In any case, it's already quite nice - I can upload pictures directly from iPhoto again (which was the most important advantage of the old gallery software) and the files are in the filesystem, not in a database - which was also quite an important point.

The software itself is of course available - anyone who wants to browse, I have set up a Trac instance with my Django experiments.

Since 2007 nothing has been online.

Django has taken an important step for the release 1.0: anonymous sessions. Sessions were previously tied to user login in Django, but now it also works without registration. Much nicer than creating thousands of cookies for the user.

CRUD with Django

Create, Read, Update, Delete - the standard functions of classic interfaces - can be easily assembled with Django. For this, there are Generic Views. On Postneo there is now a CRUD Tutorial, which shows how simple such interfaces can be put together with Django.

A comparison of Django with Rails is a rather good comparison of Rails and Django. However, comparing two systems that address quite different topics is naturally quite difficult - but the article at least tries to position the two frameworks against each other.

Django, Apache and FCGI

In Django, lighttpd and FCGI, second take I described a method how to run Django with FCGI behind a lighttpd installation. I did run the Django FCGIs as standalone servers so that you can run them under different users than the webserver. This document will give you the needed information to do the same with Apache 1.3.

Update: I maintain my descriptions now in my trac system. See the Apache+FCGI description for Django.

Update: I changed from using unix sockets to using tcp sockets in the description. The reason is that unix sockets need write access from both processes - webserver and FCGI server - and that's a bit hard to setup right, sometimes. tcp sockets are only a tad bit slower but much easier to set up.

First the main question some might ask: why Apache 1.3? The answer is simple: many people still have Apache 1.3 running as their main server and can't easily upgrade to Apache 2.0 - for example if they run large codebases in mod perl or mod python they will run into troubles with migrating because Apache 2.0 will require mod perl2 or mod python2 and both are not fully compatible with older versions. And even though lighttpd is a fantastic webserver, if you already run Apache 1.3 there might just not be the need for another webserver.

So what do you need - besides the python and django stuff - for Apache 1.3 with FastCGI? Just the mod rewrite module and mod fastcgi module installed, that's all. Both should come with your systems distribution. You will still need all the python stuff I listed in the lighttpd article.

mod_fastcgi is a bit quirky in it's installation, I had to play a bit around with it. There are a few pitfalls I can think of:

• the specification of the socket can't be an absolute path but must be a relative path with respect to the FastCgiIpcDir
• the specification of the FCGI itself (even though it's purely virtual) must be in a fully qualified form with respect to the document root you want to use. If you use a relative path, it will be relative to the document root of the default virtual host - and that's most surely not the document root you will use if you want to set up a virtual host with the FCGI.
• the FCGI itself can't be defined within a virtual host - it must be defined in the main server config. That's where the relative addressing problem comes into play.
• the socket file must be both readable and writeable by the FCGI user and the Apache user. Usually you do this by changing the socket file to group writeable and changing the group of that socket file to a group where both the user and the apache are members of.

Now here is the config snippet you have to add to your httpd.conf. I use the same directories as with the lighttpd sample, you most surely will have to adapt that to your situation.

 FastCgiExternalServer /home/gb/work/myproject/publichtml/admin.fcgi -host 127.0.0.1:8000
FastCgiExternalServer /home/gb/work/myproject/publichtml/main.fcgi -host 127.0.0.1:8001

 <VirtualHost *> ServerAdmin gb@bofh.ms
 Servername www.example.com
 ErrorLog /home/gb/work/myproject/logs/django-error.log
 CustomLog /home/gb/work/myproject/logs/django-access.log combined
 DocumentRoot /home/gb/work/myproject/public_html
 RewriteEngine On
 RewriteRule ^(/admin/.)$ /admin.fcgi$1 [L]
 RewriteRule ^(/main/.)$ /main.fcgi$1 [L]
 </VirtualHost> ```

You have to allow the webserver write access to the logs directory, so you might want to use a different location for them - possibly in `/var/log/apache/ `or whereever your apache puts it's logs. The FastCgiExternalServer directives must be outside of the virtual host definitions, but must point to files within the virtual hosts document root. But those files needn't (and probably shouldn't) exist in the filesystem, they are purely virtual. The given setup reflects the setup I did for the lighttpd scenario.

Now restart your apache, start your django-fcgi.py and you should be able to access your django application. Keep in mind to copy the admin_media files over to the document root, otherwise your admin will look very ugly.

django-fcgi.py --settings=myproject.settings.main --host=127.0.0.1 --port=8000 --daemon django-fcgi.py --settings=myproject.settings.admin --host=127.0.0.1 --port=8001 --daemon

Have fun.

Again something new with Django

There's always news, but this time there's a very interesting feature again: the inspectdb command delivers all the tables and fields from a PostgreSQL database in the format of a Python data model. Additionally, foreign keys are also found if they are stored in the database. Very practical if you need to build an interface for an existing database, you save a lot of typing work.

Writing a Simple Filesystem Browser with Django

Dieser Artikel ist mal wieder in Englisch, da er auch für die Leute auf #django interessant sein könnte. This posting will show how to build a very simple filesystem browser with Django. This filesystem browser behaves mostly like a static webserver that allows directory traversal. The only speciality is that you can use the Django admin to define filesystems that are mounted into the namespace of the Django server. This is just to demonstrate how a Django application can make use of different data sources besides the database, it's not really meant to serve static content (although with added authentication it could come in quite handy for restricted static content!).

Even though the application makes very simple security checks on passed in filenames, you shouldn't run this on a public server - I didn't do any security tests and there might be buttloads of bad things in there that might expose your private data to the world. You have been warned. We start as usual by creating the filesystems application with the django-admin.py startapp filesystems command. Just do it like you did with your polls application in the first tutorial. Just as an orientation, this is how the myproject directory does look like on my development machine:

.
|-- apps
| |-- filesystems
| | |-- models
| | |-- urls
| | `-- views
| `-- polls
| |-- models
| |-- urls
| `-- views
|-- public_html
| `-- admin_media
| |-- css
| |-- img
| | `-- admin
| `-- js
| `-- admin
|-- settings
| `-- urls
`-- templates
 `-- filesystems

After creating the infrastructure, we start by building the model. The model for the filesystems is very simple - just a name for the filesystem and a path where the files are actually stored. So here it is, the model:

 from django.core import meta

class Filesystem(meta.Model):

fields = ( meta.CharField('name', 'Name', maxlength=64), meta.CharField('path', 'Path', maxlength=200), )

def repr(self): return self.name

def get_absolute_url(self): return '/files/%s/' % self.name

def isdir(self, path): import os p = os.path.realpath(os.path.join(self.path, path)) if not p.startswith(self.path): raise ValueError(path) return os.path.isdir(p)

def files(self, path=''): import os import mimetypes p = os.path.realpath(os.path.join(self.path, path)) if not p.startswith(self.path): raise ValueError(path) l = os.listdir(p) if path: l.insert(0, '..') return [(f, os.path.isdir(os.path.join(p, f)), mimetypes.guess_type(f)[0] or 'application/octetstream') for f in l]

def file(self, path): import os import mimetypes p = os.path.realpath(os.path.join(self.path, path)) if p.startswith(self.path): (t, e) = mimetypes.guess_type(p) return (p, t or 'application/octetstream') else: raise ValueError(path)

admin = meta.Admin( fields = ( (None, {'fields': ('name', 'path')}), ), list_display = ('name', 'path'), search_fields = ('name', 'path'), ordering = ['name'], )

As you can see, the model and the admin is rather boring. What is interesting, though, are the additional methods isdir , files and file . isdir just checks wether a given path below the filesystem is a directory or not. files returns the files of the given path below the filesystems base path and file returns the real pathname and the mimetype of a given file below the filesystems base path. All three methods check for validity of the passed in path - if the resulting path isn't below the filesystems base path, a ValueError is thrown. This is to make sure that nobody uses .. in the path name to break out of the defined filesystem area. So the model includes special methods you can use to access the filesystems content itself, without caring for how to do that in your views. It's job of the model to know about such stuff.

The next part of your little filesystem browser will be the URL configuration. It's rather simple, it consists of the line in settings/urls/main.py and the myproject.apps.filesystems.urls.filesystems module. Fist the line in the main urls module:

 from django.conf.urls.defaults import *

urlpatterns = patterns('',
 (r'^files/', include('myproject.apps.filesystems.urls.filesystems')),
 )

Next the filesystems own urls module:

 from django.conf.urls.defaults import *

urlpatterns = patterns('myproject.apps.filesystems.views.filesystems',
 (r'^$', 'index'),
 (r'^(?P<filesystem_name>.*?)/(?P<path>.*)$', 'directory'),
 )

You can now add the application to the main settings file so you don't forget to do that later on. Just look for the INSTALLED_APPS setting and add the filebrowser:

 INSTALLED_APPS = (
 'myproject.apps.polls',
 'myproject.apps.filesystems'
 )

One part is still missing: the views. This module defines the externally reachable methods we defined in the urlmapper. So we need two methods, index and directory . The second one actually doesn't work only with directories - if it get's passed a file, it just presents the contents of that file with the right mimetype. The view makes use of the methods defined in the model to access actual filesystem contents. Here is the source for the views module:

 from django.core import template_loader
 from django.core.extensions import DjangoContext as Context
 from django.core.exceptions import Http404
 from django.models.filesystems import filesystems
 from django.utils.httpwrappers import HttpResponse

def index(request):
 fslist = filesystems.getlist(orderby=['name'])
 t = templateloader.gettemplate('filesystems/index')
 c = Context(request, {
 'fslist': fslist,
 })
 return HttpResponse(t.render(c))

def directory(request, filesystem_name, path):
 import os
 try:
 fs = filesystems.getobject(name exact=filesystemname)
 if fs.isdir(path):
 files = fs.files(path)
 tpl = templateloader.gettemplate('filesystems/directory')
 c = Context(request, {
 'dlist': [f for (f, d, t) in files if d],
 'flist': [{'name':f, 'type':t} for (f, d, t) in files if not d],
 'path': path,
 'fs': fs,
 })
 return HttpResponse(tpl.render(c))
 else:
 (f, mimetype) = fs.file(path)
 return HttpResponse(open(f).read(), mimetype=mimetype)
 except ValueError: raise Http404
 except filesystems.FilesystemDoesNotExist: raise Http404
 except IOError: raise Http404

See how the elements of the directory pattern are passed in as parameters to the directory method - the filesystem name is used to find the right filesystem and the path is used to access content below that filesystems base path. Mimetypes are discovered using the mimetypes module from the python distribution, btw.

The last part of our little tutorial are the templates. We need two templates - one for the index of the defined filesystems and one for the content of some path below some filesystem. We don't need a template for the files content - file content is delivered raw. So first the main index template:

{% if fslist %}
<h1>defined filesystems</h1> <ul> {% for fs in fslist %}
<li><a href="{{ fs.get_absolute_url }}">{{ fs.name }}</a></li> {% endfor %}
</ul> {% else %}
<p>Sorry, no filesystems have been defined.</p> {% endif %}

The other template is the directory template that shows contents of a path below the filesystems base path:

 {% if dlist or flist %}
 <h1>Files in //{{ fs.name }}/{{ path }}</h1> <ul> {% for d in dlist %}
 <li> <a href="{{ fs.getabsoluteurl }}{{ path }}{{ d }}/" >{{ d }}</a> </li> {% endfor %}
 {% for f in flist %}
 <li> <a href="{{ fs.getabsoluteurl }}{{ path }}{{ f.name }}" >{{ f.name }}</a> ({{ f.type }})</li> {% endfor %}
 </ul> {% endif %}

Both templates need to be stored somewhere in your TEMPLATE PATH. I have set up a path in the TEMPLATE PATH with the name of the application: filesystems . In there I stored the files as index.html and directory.html . Of course you normally would build a base template for the site and extend that in your normal templates. And you would add a 404.html to handle 404 errors. But that's left as an exercise to the reader.After you start up your development server for your admin (don't forget to set DJANGO SETTINGS MODULE accordingly!) you can add a filesystem to your database (you did do django-admin.py install filesystems sometime in between? No? Do it now, before you start your server). Now stop the admin server, change your DJANGO SETTINGS MODULE and start the main settings server. Now you can surf to http://localhost:8000/files/(at least if you did set up your URLs and server like I do) and browse the files in your filesystem. That's it. Wasn't very complicated, right? Django is really simple to use

Bodies in the Basement

Every software has them - some skeletons in the closet that start to stink when you find them. Django unfortunately too. And that is the handling of Unicode. The automatically generated Admin in Django always sends XHTML and utf-8 to the browser. The browsers therefore also send utf-8 back. But there are browsers that use a slightly different format for the data to be sent in such cases - the so-called multipart format. This is used because it is the only guaranteed method in HTTP-POST where you can send a character set along.

Unfortunately, Django parses these multipart-POSTs with the email module from Python. This then diligently produces Unicode strings from the parts marked as utf-8. Which is correct in itself - only in the Django source code there are str() calls scattered everywhere in the source code. And these then naturally crash when they are given unicode with characters above chr(128) in them.

I took a look at the source code, the most realistic approach would be to make sure in Django that Unicode results are then converted back to utf-8, so that only normal Python strings are used internally. That works so far, but then there are still problems with some databases that recognize utf-8 content when saving and then produce Unicode again when reading the content - SQLite is such a database.

Well, this won't be easy to fix. I've already tried it, it's a pretty nasty topic and unfortunately not at all considered in Django - and therefore it crashes at all corners and edges. Let's see if I can't come up with something useful after all ...

What I also noticed: Django sends the Content-type only via a meta tag with http-equiv. That's a nasty hack, it would be much better if the Content-type were set correctly as a header, then nothing can go wrong if, for example, Apache wants to add a Default-Charset. And the browsers would also react in a much more reproducible way.

In any case, this is again the typical case of American programmers. They like to tell you that you just need to switch to Unicode and utf-8 when you report your character encoding problems, but I have never seen software from an American programmer that handled Unicode correctly ...

Otherwise, there are still one or two hinges in Django - especially annoying because they are not documented, but easy to solve: the default time zone in Django is America/Chicago. You just have to write a variable TIME_ZONE with 'Europe/Berlin' as value in your settings file and apply a small patch so that Django can handle the '-' as time zone separator. Oh man, when Americans write software ...

Somehow, my motivation to take a closer look at Ruby on Rails is increasing at the moment, after all, it was Danes who started it and they should at least get such things right (if only that nice automatic administration part of Django were not - that's exactly what I would be after. Why hasn't anyone built something like that for ROR, damn ...)

Update: I have attached a patch to the corresponding ticket for the Unicode problem (just scroll to the very bottom) that at least gets the problem somewhat under control - provided you don't use SQLite, because SQLite always returns Unicode strings and they then cause trouble again. But at least with PostgreSQL, umlauts now work in Django. The solution is not really perfect, but at least it can be brought in with only a little code change. A real solution would probably require larger code changes.

Another patch is attached to the ticket for the time zone problem, with the patch you can then also use TIME_ZONE = 'Europe/Berlin' to get the time specifications, for example in the change history, in the correct time zone.

In such moments you wish you had commit rights to Django, to be able to put such quite manageable patches in yourself

Another update: Adrian was in the chat yesterday and today and the problems with Unicode are largely gone. Only with SQLite there is still trouble, but I already have the patch finished. And the time zone issue is also fixed in the SVN. And he has started unit tests. Very useful, if you can then test the whole framework cleanly in the long run after a patch ...

Django, lighttpd and FCGI, second take

In my first take at this stuff I gave a sample on how to run django projects behind lighttpd with simple FCGI scripts integrated with the server. I will elaborate a bit on this stuff, with a way to combine lighttpd and Django that gives much more flexibility in distributing Django applications over machines. This is especially important if you expect high loads on your servers. Of course you should make use of the Django caching middleware, but there are times when even that is not enough and the only solution is to throw more hardware at the problem.

Update: I maintain my descriptions now in my trac system. See the lighty+FCGI description for Django.

Caveat: since Django is very new software, I don't have production experiences with it. So this is more from a theoretical standpoint, incorporating knowledge I gained with running production systems for several larger portals. In the end it doesn't matter much what your software is - it only matters how you can distribute it over your server farm.

To follow this documentation, you will need the following packages and files installed on your system:

• [Django][2] itself - currently fetched from SVN. Follow the setup instructions or use python setup.py install .
• [Flup][3] - a package of different ways to run WSGI applications. I use the threaded WSGIServer in this documentation.
• [lighttpd][4] itself of course. You need to compile at least the fastcgi, the rewrite and the accesslog module, usually they are compiled with the system.
• [Eunuchs][5] - only needed if you are using Python 2.3, because Flup uses socketpair in the preforked servers and that is only available starting with Python 2.4
• [django-fcgi.py][6] - my FCGI server script, might some day be part of the Django distribution, but for now just fetch it here. Put this script somewhere in your $PATH, for example /usr/local/bin and make it executable.
• If the above doesn't work for any reason (maybe your system doesn't support socketpair and so can't use the preforked server), you can fetch [django-fcgi-threaded.py][7] - an alternative that uses the threading server with all it's problems. I use it for example on Mac OS X for development.

Before we start, let's talk a bit about server architecture, python and heavy load. The still preferred Installation of Django is behind Apache2 with mod python2. mod python2 is a quite powerfull extension to Apache that integrates a full Python interpreter (or even many interpreters with distinguished namespaces) into the Apache process. This allows Python to control many aspects of the server. But it has a drawback: if the only use is to pass on requests from users to the application, it's quite an overkill: every Apache process or thread will incorporate a full python interpreter with stack, heap and all loaded modules. Apache processes get a bit fat that way.

Another drawback: Apache is one of the most flexible servers out there, but it's a resource hog when compared to small servers like lighttpd. And - due to the architecture of Apache modules - mod_python will run the full application in the security context of the web server. Two things you don't often like with production environments.

So a natural approach is to use lighter HTTP servers and put your application behind those - using the HTTP server itself only for media serving, and using FastCGI to pass on requests from the user to your application. Sometimes you put that small HTTP server behind an Apache front that only uses mod proxy (either directly or via mod rewrite) to proxy requests to your applications webserver - and believe it or not, this is actually a lot faster than serving the application with Apache directly!

The second pitfall is Python itself. Python has a quite nice threading library. So it would be ideal to build your application as a threaded server - because threads use much less resources than processes. But this will bite you, because of one special feature of Python: the GIL. The dreaded global interpreter lock. This isn't an issue if your application is 100% Python - the GIL only kicks in when internal functions are used, or when C extensions are used. Too bad that allmost all DBAPI libraries use at least some database client code that makes use of a C extension - you start a SQL command and the threading will be disabled until the call returns. No multiple queries running ...

So the better option is to use some forking server, because that way the GIL won't kick in. This allows a forking server to make efficient use of multiple processors in your machine - and so be much faster in the long run, despite the overhead of processes vs. threads.

For this documentation I take a three-layer-approach for distributing the software: the front will be your trusted Apache, just proxying all stuff out to your project specific lighttpd. The lighttpd will have access to your projects document root and wil pass on special requests to your FCGI server. The FCGI server itself will be able to run on a different machine, if that's needed for load distribution. It will use a preforked server because of the threading problem in Python and will be able to make use of multiprocessor machines.

I won't talk much about the first layer, because you can easily set that up yourself. Just proxy stuff out to the machine where your lighttpd is running (in my case usually the Apache runs on different machines than the applications). Look it up in the mod_proxy documentation, usually it's just ProxyPass and ProxyPassReverse.

The second layer is more interesting. lighttpd is a bit weird in the configuration of FCGI stuff - you need FCGI scripts in the filesystem and need to hook those up to your FCGI server process. The FCGI scripts actually don't need to contain any content - they just need to be in the file system.

So we start with your Django project directory. Just put a directory public html in there. That's the place where you put your media files, for example the admin media directory. This directory will be the document root for your project server. Be sure only to put files in there that don't contain private data - private data like configs and modules better stay in places not accessible by the webserver. Next set up a lighttpd config file. You only will use the rewrite and the fastcgi modules. No need to keep an access log, that one will be written by your first layer, your apache server. In my case the project is in /home/gb/work/myproject - you will need to change that to your own situation. Store the following content as /home/gb/work/myproject/lighttpd.conf

 server.modules = ( "mod_rewrite", "mod_fastcgi" )
 server.document-root = "/home/gb/work/myproject/public_html"
 server.indexfiles = ( "index.html", "index.htm" )
 server.port = 8000
 server.bind = "127.0.0.1"
 server.errorlog = "/home/gb/work/myproject/error.log"

fastcgi.server = (
"/main.fcgi" => (
"main" => (
"socket" => "/home/gb/work/myproject/main.socket"
 )
 ),
"/admin.fcgi" => (
"admin" => (
"socket" => "/home/gb/work/myproject/admin.socket"
 )
 )
 )

url.rewrite = (
"^(/admin/.*)$" => "/admin.fcgi$1",
"^(/polls/.*)$" => "/main.fcgi$1"
 )

mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "audio/x-wav",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".conf" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv"
 )

I bind the lighttpd only to the localhost interface because in my test setting the lighttpd runs on the same host as the Apache server. In multi server settings you will bind to the public interface of your lighttpd servers, of course. The FCGI scripts communicate via sockets in this setting, because in this test setting I only use one server for everything. If your machines would be distributed, you would use the "host" and "port" settings instead of the "socket" setting to connect to FCGI servers on different machines. And you would add multiple entries for the "main" stuff, to distribute the load of the application over several machines. Look it up in the lighttpd documentation what options you will have.

I set up two FCGI servers for this - one for the admin settings and one for the main settings. All applications will be redirected through the main settings FCGI and all admin requests will be routed to the admin server. That's done with the two rewrite rules - you will need to add a rewrite rule for every application you are using.

Since lighttpd needs the FCGI scripts to exist to pass along the PATH_INFO to the FastCGI, you will need to touch the following files: /home/gb/work/myprojectg/public_html/admin.fcgi ``/home/gb/work/myprojectg/public_html/main.fcgi

They don't need to contain any code, they just need to be listed in the directory. Starting with lighttpd 1.3.16 (at the time of this writing only in svn) you will be able to run without the stub files for the .fcgi - you just add "check-local" => "disable" to the two FCGI settings. Then the local files are not needed. So if you want to extend this config file, you just have to keep some very basic rules in mind:

• every settings file needs it's own .fcgi handler
• every .fcgi needs to be touched in the filesystem - this might go away in a future version of lighttpd, but for now it is needed
• load distribution is done on .fcgi level - add multiple servers or sockets to distribute the load over several FCGI servers
• every application needs a rewrite rule that connects the application with the .fcgi handler

Now we have to start the FCGI servers. That's actually quite simple, just use the provided django-fcgi.py script as follows:

 django-fcgi.py --settings=myproject.work.main
 --socket=/home/gb/work/myproject/main.socket
 --minspare=5 --maxspare=10 --maxchildren=100
 --daemon

django-fcgi.py --settings=myproject.work.admin
 --socket=/home/gb/work/myproject/admin.socket
 --maxspare=2 --daemon

Those two commands will start two FCGI server processes that use the given sockets to communicate. The admin server will only use two processes - this is because often the admin server isn't the server with the many hits, that's the main server. So the main server get's a higher-than-default setting for spare processes and maximum child processes. Of course this is just an example - tune it to your needs.

The last step is to start your lighttpd with your configuration file: lighttpd -f /home/gb/work/myproject/lighttpd.conf

That's it. If you now access either the lighttpd directly at http://localhost:8000/polls/ or through your front apache, you should see your application output. At least if everything went right and I didn't make too much errors.

There are days when my computer hates me

For example, when I play with Flup and instead of the threaded server I want to use a forked server. And I realize that the latter requires the socketpair function, which unfortunately is only available from Python 2.4, which is available on Debian Sarge, but for Python 2.4 there is no Psycopg in Debian Sarge - which in turn is a prerequisite for Django and PostgreSQL, which is why I am dealing with FastCGI in the first place. Installing Psycopg itself is no fun, as you not only need the PostgreSQL headers that are normally installed, but also a few internal headers - so in principle a build tree. And then you also need the egenix-mx-base headers, which you can only get for Python 2.3, so you would have to install that yourself as well. Backports from the next Debian version don't work either, as they are just switching to PostgreSQL 8.0 and Sarge is still using 7.4 and I didn't want to upgrade the whole system right away. And so you go in circles and feel a bit cheated by all the dependencies and version conflicts.

And what do you do as a solution, because the threaded server unfortunately only produces segfaults in Psycopg? You take the threaded server, forbid it to thread and start it via the spawn-fcgi from lighttpd, or directly from lighttpd. But that's somehow stupid again, because then there are always 3 threads per FCGI server, two of which just stand in the process list and do nothing. And all this just because mod python2 (which is needed for Django) requires Apache2, which in turn requires mod perl2, which is incompatible with the old mod perl, which is why a whole bunch of my sites wouldn't work anymore if I switched to Apache2. Which I don't want to do anyway, because Apache2 with mod python is damn slow. And once again I feel cheated. I really should have looked for a more meaningful job.

If you didn't understand anything: doesn't matter, it's technology, it's not important, I just wanted to say that.

Running Django with FCGI and lighttpd

Diese Dokumentation ist für einen grösseren Kreis als nur .de gedacht, daher das ganze in Neuwestfälisch Englisch. Sorry. Update: I maintain the actually descriptions now in my trac system. See the FCGI+lighty description for Django. There are different ways to run Django on your machine. One way is only for development: use the django-admin.py runserver command as documented in the tutorial. The builtin server isn't good for production use, though. The other option is running it with mod_python. This is currently the preferred method to run Django. This posting is here to document a third way: running Django behind lighttpd with FCGI.

First you need to install the needed packages. Fetch them from their respective download address and install them or use preinstalled packages if your system provides those. You will need the following stuff:

• [Django][2] itself - currently fetched from SVN. Follow the setup instructions or use python setup.py install .
• [Flup][3] - a package of different ways to run WSGI applications. I use the threaded WSGIServer in this documentation.
• [lighttpd][4] itself of course. You need to compile at least the fastcgi, the rewrite and the accesslog module, usually they are compiled with the system.

First after installing ligthttpd you need to create a lighttpd config file. The configfile given here is tailored after my own paths - you will need to change them to your own situation. This config file activates a server on port 8000 on localhost - just like the runserver command would do. But this server is a production quality server with multiple FCGI processes spawned and a very fast media delivery.

 # lighttpd configuration file
 #
 ############ Options you really have to take care of ####################

server.modules = ( "mod_rewrite", "mod_fastcgi", "mod_accesslog" )

server.document-root = "/home/gb/public_html/"
 server.indexfiles = ( "index.html", "index.htm", "default.htm" )

 these settings attch the server to the same ip and port as runserver would do

server.errorlog = "/home/gb/log/lighttpd-error.log"
 accesslog.filename = "/home/gb/log/lighttpd-access.log"

fastcgi.server = (
"/myproject-admin.fcgi" => (
"admin" => (
"socket" => "/tmp/myproject-admin.socket",
"bin-path" => "/home/gb/public_html/myproject-admin.fcgi",
"min-procs" => 1,
"max-procs" => 1
 )
 ),
"/myproject.fcgi" => (
"polls" => (
"socket" => "/tmp/myproject.socket",
"bin-path" => "/home/gb/public_html/myproject.fcgi"
 )
 )
 )

url.rewrite = (
"^(/admin/.*)$" => "/myproject-admin.fcgi$1",
"^(/polls/.*)$" => "/myproject.fcgi$1"
 )

This config file will start only one FCGI handler for your admin stuff and the default number of handlers (each one multithreaded!) for your own site. You can finetune these settings with the usual ligthttpd FCGI settings, even make use of external FCGI spawning and offloading of FCGI processes to a distributed FCGI cluster! Admin media files need to go into your lighttpd document root.

The config works by translating all standard URLs to be handled by the FCGI script for each settings file - to add more applications to the system you would only duplicate the rewrite rule for the /polls/ line and change that to choices or whatever your module is named. The next step would be to create the .fcgi scripts. Here are the two I am using:

 #!/bin/sh
 # this is myproject.fcgi - put it into your docroot

export DJANGOSETTINGSMODULE=myprojects.settings.main

/home/gb/bin/django-fcgi.py

 #!/bin/sh
 # this is myproject-admin.fcgi - put it into your docroot

export DJANGOSETTINGSMODULE=myprojects.settings.admin

/home/gb/bin/django-fcgi.py

These two files only make use of a django-fcgi.py script. This is not part of the Django distribution (not yet - maybe they will incorporate it) and it's source is given here:

 #!/usr/bin/python2.3

def main():
 from flup.server.fcgi import WSGIServer
 from django.core.handlers.wsgi import WSGIHandler
 WSGIServer(WSGIHandler()).run()

if name == 'main':
 main()

As you can see it's rather simple. It uses the threaded WSGIServer from the fcgi-module, but you could as easily use the forked server - but as the lighttpd already does preforking, I think there isn't much use with forking at the FCGI level. This script should be somewhere in your path or just reference it with fully qualified path as I do. Now you have all parts togehter. I put my lighttpd config into /home/gb/etc/lighttpd.conf , the .fcgi scripts into /home/gb/public_html and the django-fcgi.py into /home/gb/bin . Then I can start the whole mess with /usr/local/sbin/lighttpd -f etc/lighttpd.conf . This starts the server, preforkes all FCGI handlers and detaches from the tty to become a proper daemon. The nice thing: this will not run under some special system account but under your normal user account, so your own file restrictions apply. lighttpd+FCGI is quite powerfull and should give you a very nice and very fast option for running Django applications. Problems:

• under heavy load some FCGI processes segfault. I first suspected the fcgi library, but after a bit of fiddling (core debugging) I found out it's actually the psycopg on my system that segfaults. So you might have more luck (unless you run Debian Sarge, too)

• Performance behind a front apache isn't what I would have expected. A lighttpd with front apache and 5 backend FCGI processes only achieves 36 requests per second on my machine while the django-admin.py runserver achieves 45 requests per second! (still faster than mod_python via apache2: only 27 requests per second) Updates:

• the separation of the two FCGI scripts didn't work right. Now I don't match only on the .fcgi extension but on the script name, that way /admin/ really uses the myproject-admin.fcgi and /polls/ really uses the myproject.fcgi.

• I have [another document online][6] that goes into more details with regard to load distribution

Django Again

Django - the upcoming web framework for Python - now has SQLite 3 support. This makes setting up a development environment for Django projects extremely simple: you just need Python 2.3 or Python 2.4, SQLite3, and PySQLite2. On a Mac, everything is already there except PySQLite2, which you can get from www.pysqlite.org and install using sudo python setup.py install. And you're ready to start with Django and work through the tutorials. No Apache needed, no PostgreSQL (though it's the nicest of all SQL databases, it's sometimes overkill for a development environment on a notebook), and especially not psycopg - whose installation unfortunately requires almost a full PostgreSQL source tree. So there's no excuse for Pythonistas not to get involved with Django.

First Django Tutorials Online

The Django programmers start with the tutorials. The first tutorial primarily deals with creating the database model and the basic code for the objects to be managed, and the second tutorial deals with the automatically generated administration interface. Very nice, all of it.

The system is of course strongly focused on content creation and management - but still general enough so that it can also be used for differently structured content. The entire administration is automatically created from the object model and some hints, so it always aligns with the real data in the system. And the default look is also quite appealing.

Server integration is done simply via mod python - so via Apache. Which is also an advantage, as mod python offers very high performance right out of the box. And for more demanding cases, there's the caching in Django. I must say, what I've seen of Django so far, I like it very much.

An important note is missing in the installation instructions: Apache2 is mandatory, and therefore also ModPython in the corresponding version. However, Mac OS X only provides Apache 1.3, and many other servers also only have the 1.3 Apache available, so Django still has a real drawback here.

By the way, if you want to upgrade from Apache to Apache2 on Debian: if mod perl is in use, forget it. The mod perl2 for Apache2 in Debian Sarge is complete garbage - as if the API changes in mod perl2 compared to the old mod perl weren't annoying enough. In principle, you can no longer get Perl modules to run so easily with it.

Update: By the way, there is currently a lot of activity in the Subversion for Django to eliminate the requirement for Apache. A simple development server is already included, so in the future you will no longer need Apache for initial experiments. And you could also set up the deployment on other legs in the long run - for example, FCGI behind lighttpd.

Update 2: The third tutorial is out and deals with the view for the visitor. They have a pretty intense pace right now with Django.

Django - new web framework for Python

Another web framework for Python, this time with the bold name Django. I am skeptical about yet another web framework - there are already plenty, and I must admit that I have contributed to one or another - but this one offers some interesting approaches.

On the one hand, it addresses similar solutions like Ruby on Rails - but does not mention Ruby on Rails at all. That's already positive; lately, one almost gets the impression that Python programmers are panicking because of ROR and think that everything must only be oriented towards it.

On the other hand, Django offers automatically generated backend pages. This is something I really like and what I find so nice about Zope, for example - you immediately have a way to play with the actual data, even before the actual frontend is ready. Very practical, especially in the initial development phase.

Some of the other ideas are also quite funny - for example, the mapping of URLs to handlers in the Python code via regular expressions. Reminds a bit of mod_rewrite in Apache (where, with such solutions, the question of prioritization of overlapping regular expressions always remains). And an integrated object-relational manager is not bad either, even if you can of course just as well fall back on finished solutions there. And the fact that the developers have already thought about the need for efficient cache systems and then rely on memcached is also nice - many projects die at some point from the load, simply because caching was not thought of in time.

The template language, however, looks a bit unusual and somehow I wonder why there must be almost as many of them as there are web frameworks.