Archive 21.7.2005 - 2.8.2005

The Illusive setdefaultencoding

Ian Bicking mentioned a nice trick in his article about setdefaultencoding: simply reload the sys module with reload(sys) to make setdefaultencoding available.

setdefaultencoding is used to set the default encoding for bytestrings. Normally, this is ASCII, but it can be changed to iso-8859-1 or utf-8 - if you have setdefaultencoding at all. Unfortunately, it is deleted when the Python runtime environment starts - because the Python developers want to patronize the users again.

reload(sys) is of course something that does not necessarily inspire confidence - sys is after all not an unimportant module. But in my experiments it has worked so far and it definitely helps with the whole Unicode problem if you can give your programs a different encoding as standard.

It would be nicer if setdefaultencoding were not deleted in the standard distribution. Of course, this can also be achieved by patching site.py, but that is not better than reloading sys ...

(Un)trusted platform Apple?

Since it's currently fashionable to explain that one switched when Apple uses TPA - or whatever it might be called in the future: first wait and see. See what Apple does and how - there are always rumors beforehand.

If TPA is actually included: Linux can also be a usable system, even if the interfaces are quite sick (although current XFCE versions don't look that bad) and if there is no more PPC in Apple hardware and you put Linux on it: you can also buy your notebook from IBM. They have nice devices that also work very well under Linux.

And last but not least: just because new Apple hardware is different, it doesn't change the already purchased hardware - and Apple-typically, this usually lasts a few years longer. And under Linux, some Macs even run faster than under OS X.

Where Cease and Desist Notices and Anticipatory Obedience Can Lead

Just what the FFII noticed: Nutzwerk shut down FFII.org (a bit more info as usual at Heise).

Although the corresponding IP address was consistently reachable, the DNS provider registered as the technical contact for the domain FFII.org complied with Nutzwerk's request and shut down the domain FFII.org on the previous Friday afternoon. The name resolution of FFII.org and corresponding subdomains did not work temporarily. After an intervention by the FFII, the DNS provider reactivated the domain that same evening and wants to ask Nutzwerk for a clarification of the demand, according to FFII board member Hartmut Pilch to Golem.de.

A real dilemma: service providers want to protect themselves and unfortunately the Telemedia Act makes life difficult for these service providers: if they are notified of content that constitutes a legal violation, they must remove this content immediately. But how can one assess whether content constitutes a legal violation? Especially when it comes to things like at Nutzwerk - where critical reporting by the company is defined as a legal violation?

In the end, this gives companies a means of censorship without giving service providers (and of course the website operators themselves!) reasonable means of defense. How, for example, is a smaller provider supposed to protect itself from cease-and-desist letters with absurd claim amounts - as the music industry particularly likes to use? Legal protection insurance doesn't help here.

No wonder that some providers see preemptive obedience as the right strategy in such cases - they lose at most the customer they shut down, perhaps a bit of negative press, but taking on a fight against a company with exaggerated ideas, they can't win much.

If you then sit on the board of a privately operated provider like me, you start to wonder what the actual goal of these legislative changes in the context of the Telemedia Act really was ...

Effects of Genetically Modified Rapeseed and Co.

At Isotopp I found a pointer to information about the British evaluation project on genetically modified seeds. Shocking, what effects this has, for example, on related wild plants and what this will mean for us in the long run. But of course, it's all sooo safe and sooo important for us - it's really just about the revenue for the genetic engineering companies, not about what consumers really want.

But of course, every critic of genetically modified seeds is dismissed as a crank by the economy and its henchmen (such as Clement and other politicians).

Dave's new OPML editor with blog

I'm currently playing around with Dave Winer's OPML Editor, which he now uses for his blog. It looks quite fun and has a lot of features. My OPML Blog has collected some of the insights I've gained from it. I certainly won't switch over just like that - that would be Quark, which is not necessarily my target software. But it's fun to play with something completely different again.

Unfortunately, the OPML Editor has inherited some of the ailments of Radio Userland and Frontier, especially the handling of umlauts is not really smooth (I would like to have consistent UTF-8 support finally) and the runtime behavior is better than in Radio, but it still occasionally consumes too much CPU.

The concept of rendered outlines does have a certain charm. However, many parts of the rendering are not really accessible to normal users - you can edit the ancient table layout and make something else out of it, but the OPMLs are implemented with the internal OPML renderer and the HTML fragments are not so easy to change - and thus, for example, changing the language is quite cumbersome, as is the complete removal of layout tables.

More will certainly appear on the OPML blog from time to time, here I will write at most a few conclusions.

HEW Cyclassics 2005

A really nice race - it's always amazing how even a nearly quarter-hour lead isn't enough to bring home a victory in this race. It's also always amazing how the commentators every time say that the main field starts too late - by now they should know.

The Quickstep action was great - taking the victory and the podium positions away from the Fassa Bortolo guys after their sprint build-up is really super. Even if I would of course have preferred to see Zabel up front.

Merkelnix is also cramping

Just so no one thinks only the SPD has brainless slogans to shout into the world: "Make work possible in Germany again" is the reason why they want to increase the value-added tax:

Union Chancellor candidate Angela Merkel defended the planned increase in value-added tax by the CDU and CSU. The Union wants to achieve the goal of reducing labor costs with this, she said in an interview with "Bericht aus Berlin". It is about "making work possible in Germany again and thus enabling social security," the CDU chairwoman continued.

Sorry, but how incredibly stupid is that? Social security through an increase in value-added tax, which hits the hardest those who cannot further reduce their consumption because it already only consists of staple foods and other expenses necessary for survival?

Software Patents - Commentary in the NY Times

The NY Times asks why Bill Gates wants 3,000 new patents and finds a massive siege of the patent office with mountains of software patents, which are often just trivial patents (like the cited patent for adding/removing spaces in documents). The commentator makes a demand in the comment (after considering whether Microsoft should not simply have all the patents it already has revoked):

Perhaps that is going too far. Certainly, we should go through the lot and reinstate the occasional invention embodied in hardware. But patent protection for software? No. Not for Microsoft, nor for anyone else.

And this from the country that has had software patents for a long time and that is repeatedly cited by software patent proponents in the EU as a reason for a necessary worldwide harmonization.

No, software patents are also not popular there and not really useful. Dan Bricklin, known to some as the father of VisiCalc, also thinks so:

Mr. Bricklin, who has started several software companies and defensively acquired a few software patents along the way, says he, too, would cheer the abolition of software patents, which he sees as the bane of small software companies. "The number of patents you can run into with a small product is immense," he said. As for Microsoft's aggressive accumulation in recent years, he asked, "Isn't Microsoft the poster child of success without software patents?"

And why is Microsoft doing this now? The manager responsible gives a reason, as only a business administrator could come up with, it's that stupid:

"We realized we were underpatenting," Mr. Smith explained. The company had seen studies showing that other information technology companies filed about two patents for every $1 million spent on research and development. If Microsoft was spending $6 billion to $7.5 billion annually on its R&D, it would need to file at least 3,000 applications to keep up with the Joneses.

Ok, the idea of patent applications alone being oriented towards numbers from the industry is absurd, but how stupid do you have to be to draw a connection between the number of patents and revenue in the field of research and development?

The NY Times also draws a parallel to the pharmaceutical industry, which - at least according to its own statements - is happy to get a patent for a drug when it invests 20 million in research (which is already critical enough, as can be seen in the fight against AIDS in Africa).

And the fallout is also well summarized in the NY Times:

Last year at a public briefing, Kevin R. Johnson, Microsoft's group vice president for worldwide sales, spoke pointedly of "intellectual property risk" that corporate customers should take into account when comparing software vendors. On the one side, Microsoft has an overflowing war chest and bulging patent portfolio, ready to fight - or cross-license with - any plaintiff who accuses it of patent infringement. On the other are the open-source developers, without war chest, without patents of their own to use as bargaining chips and without the financial means to indemnify their customers.

The question of what Jefferson (the founder of the US patent system) would say about what is now being patented is quite justified. In his sense - which was actually more about protecting real inventive genius from exploitation by corporations - this is definitely not the case.

Election Campaign, Election Agony ...

Münte on the Roll: Linkspartei "politically and legally absurd". I have the impression, however, that he would help the SPD more by giving them a program that appeals to ordinary citizens again, instead of simply stirring up fear of the left and spreading defamation (sorry, but the combination of PDS and WASG in the open list may be strange, but legally flawless - claiming otherwise is simply defamation).

If the SPD cannot achieve more in the election campaign than just to blow the horn of the Union parties against the alleged danger from the left, the SPD will simply put itself out of the political game - with such nonsense, you don't win an election. If the SPD is nothing more than a union with a red tie, it can stay away from me ...

Writing a Simple Filesystem Browser with Django

Dieser Artikel ist mal wieder in Englisch, da er auch für die Leute auf #django interessant sein könnte. This posting will show how to build a very simple filesystem browser with Django. This filesystem browser behaves mostly like a static webserver that allows directory traversal. The only speciality is that you can use the Django admin to define filesystems that are mounted into the namespace of the Django server. This is just to demonstrate how a Django application can make use of different data sources besides the database, it's not really meant to serve static content (although with added authentication it could come in quite handy for restricted static content!).

Even though the application makes very simple security checks on passed in filenames, you shouldn't run this on a public server - I didn't do any security tests and there might be buttloads of bad things in there that might expose your private data to the world. You have been warned. We start as usual by creating the filesystems application with the django-admin.py startapp filesystems command. Just do it like you did with your polls application in the first tutorial. Just as an orientation, this is how the myproject directory does look like on my development machine:

.
|-- apps
| |-- filesystems
| | |-- models
| | |-- urls
| | `-- views
| `-- polls
| |-- models
| |-- urls
| `-- views
|-- public_html
| `-- admin_media
| |-- css
| |-- img
| | `-- admin
| `-- js
| `-- admin
|-- settings
| `-- urls
`-- templates
 `-- filesystems

After creating the infrastructure, we start by building the model. The model for the filesystems is very simple - just a name for the filesystem and a path where the files are actually stored. So here it is, the model:

 from django.core import meta

class Filesystem(meta.Model):

fields = ( meta.CharField('name', 'Name', maxlength=64), meta.CharField('path', 'Path', maxlength=200), )

def repr(self): return self.name

def get_absolute_url(self): return '/files/%s/' % self.name

def isdir(self, path): import os p = os.path.realpath(os.path.join(self.path, path)) if not p.startswith(self.path): raise ValueError(path) return os.path.isdir(p)

def files(self, path=''): import os import mimetypes p = os.path.realpath(os.path.join(self.path, path)) if not p.startswith(self.path): raise ValueError(path) l = os.listdir(p) if path: l.insert(0, '..') return [(f, os.path.isdir(os.path.join(p, f)), mimetypes.guess_type(f)[0] or 'application/octetstream') for f in l]

def file(self, path): import os import mimetypes p = os.path.realpath(os.path.join(self.path, path)) if p.startswith(self.path): (t, e) = mimetypes.guess_type(p) return (p, t or 'application/octetstream') else: raise ValueError(path)

admin = meta.Admin( fields = ( (None, {'fields': ('name', 'path')}), ), list_display = ('name', 'path'), search_fields = ('name', 'path'), ordering = ['name'], )

As you can see, the model and the admin is rather boring. What is interesting, though, are the additional methods isdir , files and file . isdir just checks wether a given path below the filesystem is a directory or not. files returns the files of the given path below the filesystems base path and file returns the real pathname and the mimetype of a given file below the filesystems base path. All three methods check for validity of the passed in path - if the resulting path isn't below the filesystems base path, a ValueError is thrown. This is to make sure that nobody uses .. in the path name to break out of the defined filesystem area. So the model includes special methods you can use to access the filesystems content itself, without caring for how to do that in your views. It's job of the model to know about such stuff.

The next part of your little filesystem browser will be the URL configuration. It's rather simple, it consists of the line in settings/urls/main.py and the myproject.apps.filesystems.urls.filesystems module. Fist the line in the main urls module:

 from django.conf.urls.defaults import *

urlpatterns = patterns('',
 (r'^files/', include('myproject.apps.filesystems.urls.filesystems')),
 )

Next the filesystems own urls module:

 from django.conf.urls.defaults import *

urlpatterns = patterns('myproject.apps.filesystems.views.filesystems',
 (r'^$', 'index'),
 (r'^(?P<filesystem_name>.*?)/(?P<path>.*)$', 'directory'),
 )

You can now add the application to the main settings file so you don't forget to do that later on. Just look for the INSTALLED_APPS setting and add the filebrowser:

 INSTALLED_APPS = (
 'myproject.apps.polls',
 'myproject.apps.filesystems'
 )

One part is still missing: the views. This module defines the externally reachable methods we defined in the urlmapper. So we need two methods, index and directory . The second one actually doesn't work only with directories - if it get's passed a file, it just presents the contents of that file with the right mimetype. The view makes use of the methods defined in the model to access actual filesystem contents. Here is the source for the views module:

 from django.core import template_loader
 from django.core.extensions import DjangoContext as Context
 from django.core.exceptions import Http404
 from django.models.filesystems import filesystems
 from django.utils.httpwrappers import HttpResponse

def index(request):
 fslist = filesystems.getlist(orderby=['name'])
 t = templateloader.gettemplate('filesystems/index')
 c = Context(request, {
 'fslist': fslist,
 })
 return HttpResponse(t.render(c))

def directory(request, filesystem_name, path):
 import os
 try:
 fs = filesystems.getobject(name exact=filesystemname)
 if fs.isdir(path):
 files = fs.files(path)
 tpl = templateloader.gettemplate('filesystems/directory')
 c = Context(request, {
 'dlist': [f for (f, d, t) in files if d],
 'flist': [{'name':f, 'type':t} for (f, d, t) in files if not d],
 'path': path,
 'fs': fs,
 })
 return HttpResponse(tpl.render(c))
 else:
 (f, mimetype) = fs.file(path)
 return HttpResponse(open(f).read(), mimetype=mimetype)
 except ValueError: raise Http404
 except filesystems.FilesystemDoesNotExist: raise Http404
 except IOError: raise Http404

See how the elements of the directory pattern are passed in as parameters to the directory method - the filesystem name is used to find the right filesystem and the path is used to access content below that filesystems base path. Mimetypes are discovered using the mimetypes module from the python distribution, btw.

The last part of our little tutorial are the templates. We need two templates - one for the index of the defined filesystems and one for the content of some path below some filesystem. We don't need a template for the files content - file content is delivered raw. So first the main index template:

{% if fslist %}
<h1>defined filesystems</h1> <ul> {% for fs in fslist %}
<li><a href="{{ fs.get_absolute_url }}">{{ fs.name }}</a></li> {% endfor %}
</ul> {% else %}
<p>Sorry, no filesystems have been defined.</p> {% endif %}

The other template is the directory template that shows contents of a path below the filesystems base path:

 {% if dlist or flist %}
 <h1>Files in //{{ fs.name }}/{{ path }}</h1> <ul> {% for d in dlist %}
 <li> <a href="{{ fs.getabsoluteurl }}{{ path }}{{ d }}/" >{{ d }}</a> </li> {% endfor %}
 {% for f in flist %}
 <li> <a href="{{ fs.getabsoluteurl }}{{ path }}{{ f.name }}" >{{ f.name }}</a> ({{ f.type }})</li> {% endfor %}
 </ul> {% endif %}

Both templates need to be stored somewhere in your TEMPLATE PATH. I have set up a path in the TEMPLATE PATH with the name of the application: filesystems . In there I stored the files as index.html and directory.html . Of course you normally would build a base template for the site and extend that in your normal templates. And you would add a 404.html to handle 404 errors. But that's left as an exercise to the reader.After you start up your development server for your admin (don't forget to set DJANGO SETTINGS MODULE accordingly!) you can add a filesystem to your database (you did do django-admin.py install filesystems sometime in between? No? Do it now, before you start your server). Now stop the admin server, change your DJANGO SETTINGS MODULE and start the main settings server. Now you can surf to http://localhost:8000/files/(at least if you did set up your URLs and server like I do) and browse the files in your filesystem. That's it. Wasn't very complicated, right? Django is really simple to use

Zerospan seems to be a P2P software with encryption and Bonjour (ex-Rendevouz, ex-Zeroconf) integration. I'm not quite getting it, as the download contains no documentation and the wiki with the documentation is currently broken, so I'll just blogmark it to check it out later.

Training as a low-wage sector

What lies behind the DIHK's demand for halving the basic apprentice salary and flexibilizing working hours becomes clear when you look at quotes from the DIHK chairman:

"My proposal is to introduce a nationwide basic remuneration of 270 euros," he told the newspaper "Die Welt". He justified his initiative by saying that this would allow more apprenticeships to be financed. "An apprenticeship remuneration of up to 800 euros is simply too high for many businesses."

"Working hours must be better adapted to the needs of the industries." It makes no sense that a 17-year-old restaurant specialist has to leave at 10 p.m. "if all the tables are still occupied."

This is simply about having cheap labor, but not about ensuring proper training. But these demands are not new.

And what the German economy thinks of training can be seen in the fact that the number of training positions has again decreased by 10% compared to the previous year - and thus young people have again been left without training positions, despite all the promises of the economy. Without a non-training fee for larger companies, this will not change either. But complaining that there are no trained skilled workers, the economy can do quite famously ...

Beckstein on the Roll

No idea what the herb is called that he takes, but it leaves severe brain damage: Beckstein wants German Guantanamo. Apart from the fact that he also wants to shoot suspects in the head and pack foreigners into camps because potential terrorists must not be allowed to run around freely (politicians like him, who are completely crazy, are not only allowed to run around freely but also to express their opinions freely), he is also constitutionally hostile:

Beckstein also criticized the ruling of the Federal Constitutional Court, which in a ruling on Lower Saxony legislation had demanded clear limitations on preventive telephone surveillance. The balance between security and freedom of interests must be re-evaluated, said Beckstein: "That the intimate sphere of terrorists should be protected is for me hardly bearable."

I'm sorry, Mr. Beckstein, but you have failed the test. Because the Constitutional Court does not explicitly protect the intimate sphere of terrorists - but the intimate sphere of citizens. And this is listed in the Basic Law as a protected asset.

Why is someone like this not observed by the Federal Office for the Protection of the Constitution? His hostility to the constitution is really documented multiple times ...

Novell will go for SCO's throat

And their considerations on the legal situation would - if they were to hold up in court - really deliver a significant blow to SCO.

The whole SCO-Linux movie is quite exciting, but quite honestly: the lengths between the action scenes are a bit exaggerated.

Pluto out or a new one in?

Astronomers have found a clump of mud outside Pluto's orbit that is at least as large, and likely even significantly larger than Pluto - Planet or Not, Pluto Now Has Far-Out Rival:

Astronomers announced yesterday that they had found a lump of rock and ice that was larger than Pluto and the farthest known object in the solar system. The discovery will probably rekindle debate over the definition of "planet" and whether Pluto still merits the designation.

Now it's about whether Pluto loses its planet status, or the new one also becomes a planet.

Who wants to work with PostgreSQL and Frontier, simply install the PostgreSQL Extension for Frontier. For Mac and Windows.

On Dealing with Security

Under ISS takes action against publication of Cisco vulnerability talk you can find a description of how Cisco and ISS envision security: massive interference with the freedom of expression of a speaker at the Black-Hat conference. Okay, he was a former ISS employee and probably used information he shouldn't have published - but it's exactly this ridiculous secrecy that undermines security - because attackers will gain this knowledge sooner or later - if security vulnerabilities exist, they will be found sooner or later. If someone reports about it publicly, at least you can defend yourself and take countermeasures. If the publication is suppressed, the end user is ultimately the victim - who has no chance to protect themselves - and even in an emergency by switching to another router manufacturer.

Therefore, it is indeed the case: neither ISS nor Cisco make a good impression in the public eye. On the contrary, their censorship attempts are actually only another argument in future product decisions to decide against Cisco - because you can obviously not trust their security statements.

Who believed that ISO time specifications are just YYYY-MM-TT HH:MM:SS.HS, forget it: International standard date and time notation. Of course, it's an ISO standard ...

Bodies in the Basement

Every software has them - some skeletons in the closet that start to stink when you find them. Django unfortunately too. And that is the handling of Unicode. The automatically generated Admin in Django always sends XHTML and utf-8 to the browser. The browsers therefore also send utf-8 back. But there are browsers that use a slightly different format for the data to be sent in such cases - the so-called multipart format. This is used because it is the only guaranteed method in HTTP-POST where you can send a character set along.

Unfortunately, Django parses these multipart-POSTs with the email module from Python. This then diligently produces Unicode strings from the parts marked as utf-8. Which is correct in itself - only in the Django source code there are str() calls scattered everywhere in the source code. And these then naturally crash when they are given unicode with characters above chr(128) in them.

I took a look at the source code, the most realistic approach would be to make sure in Django that Unicode results are then converted back to utf-8, so that only normal Python strings are used internally. That works so far, but then there are still problems with some databases that recognize utf-8 content when saving and then produce Unicode again when reading the content - SQLite is such a database.

Well, this won't be easy to fix. I've already tried it, it's a pretty nasty topic and unfortunately not at all considered in Django - and therefore it crashes at all corners and edges. Let's see if I can't come up with something useful after all ...

What I also noticed: Django sends the Content-type only via a meta tag with http-equiv. That's a nasty hack, it would be much better if the Content-type were set correctly as a header, then nothing can go wrong if, for example, Apache wants to add a Default-Charset. And the browsers would also react in a much more reproducible way.

In any case, this is again the typical case of American programmers. They like to tell you that you just need to switch to Unicode and utf-8 when you report your character encoding problems, but I have never seen software from an American programmer that handled Unicode correctly ...

Otherwise, there are still one or two hinges in Django - especially annoying because they are not documented, but easy to solve: the default time zone in Django is America/Chicago. You just have to write a variable TIME_ZONE with 'Europe/Berlin' as value in your settings file and apply a small patch so that Django can handle the '-' as time zone separator. Oh man, when Americans write software ...

Somehow, my motivation to take a closer look at Ruby on Rails is increasing at the moment, after all, it was Danes who started it and they should at least get such things right (if only that nice automatic administration part of Django were not - that's exactly what I would be after. Why hasn't anyone built something like that for ROR, damn ...)

Update: I have attached a patch to the corresponding ticket for the Unicode problem (just scroll to the very bottom) that at least gets the problem somewhat under control - provided you don't use SQLite, because SQLite always returns Unicode strings and they then cause trouble again. But at least with PostgreSQL, umlauts now work in Django. The solution is not really perfect, but at least it can be brought in with only a little code change. A real solution would probably require larger code changes.

Another patch is attached to the ticket for the time zone problem, with the patch you can then also use TIME_ZONE = 'Europe/Berlin' to get the time specifications, for example in the change history, in the correct time zone.

In such moments you wish you had commit rights to Django, to be able to put such quite manageable patches in yourself

Another update: Adrian was in the chat yesterday and today and the problems with Unicode are largely gone. Only with SQLite there is still trouble, but I already have the patch finished. And the time zone issue is also fixed in the SVN. And he has started unit tests. Very useful, if you can then test the whole framework cleanly in the long run after a patch ...

Liability for Links after the Heise Judgment

After this interview with WDR, the following applies: "Anyone who sets such a link is in trouble":

You really have to be very careful. Due to these new rulings, you have to think: Who am I linking to? In the past, as a private individual, you would say: 'Come on, I'll put a hundred links one after the other' and be quite proud. Today, you really have to consider whether the person you are linking to is really trustworthy. You also have to check these links at regular intervals and see what is happening on the linked page.

Which - if it were actually the case - would factually mean the end of privately operated information offerings in the short or long term, as no one can check all their links. I have almost 5000 articles in my blog, which I certainly won't be able to check to see if there is something somewhere that offends someone.

And thus, this ruling has driven another nail into the coffin of the Internet, simply because judges repeatedly rate the alleged rights of rights extortionists higher than free speech and free reporting.

If you, like me, find yourself in a situation where you don't like the Unicode strings in PySQLite2 and need UTF-8 byte strings: PysqliteFactories are the solution here, not converters. Because converters would have to be registered for every variation of varchar that is in use - the row factories, on the other hand, are quite agnostic and practical. And if you already use your own cursor class: simply set this as the cursor factory, which then assigns a row factory to the instance with self.row_factory.

Sysadmins Day

Lisa9 shows how to properly pay tribute to a sysadmin! Even DAU-friendly with illustrated instructions

Abridged guide to HTTP Caching is a description of the most important caching headers in HTTP and how they should be used.

JSAN is what CPAN is for Perl - a central directory and download area for JavaScript sources and packages.

Linux-VServer is a kernel patch and a set of utilities that enable running a series of virtual Linux boxes on a base machine, with resources strongly isolated from each other. Chroot on steroids, or most comparable to BSD Jails. Interesting for hosting projects where virtual root servers are required. It's even included in the current Debian.

Tor Network Status provides an overview of exit nodes in the Tor network with traffic information, allowed ports, and IP data. Nice. (found via the Rabenhorst)

typo is a blog software for Ruby on Rails with seemingly already quite extensive features. Specifically also with good caching (produces static pages) for high-traffic sites, where parts are then kept dynamically via JavaScript. Sounds like I'll take a closer look at it when my ROR book arrives ...

The Earth is flat after all - nonsense. Sing against it!

Django, lighttpd and FCGI, second take

In my first take at this stuff I gave a sample on how to run django projects behind lighttpd with simple FCGI scripts integrated with the server. I will elaborate a bit on this stuff, with a way to combine lighttpd and Django that gives much more flexibility in distributing Django applications over machines. This is especially important if you expect high loads on your servers. Of course you should make use of the Django caching middleware, but there are times when even that is not enough and the only solution is to throw more hardware at the problem.

Update: I maintain my descriptions now in my trac system. See the lighty+FCGI description for Django.

Caveat: since Django is very new software, I don't have production experiences with it. So this is more from a theoretical standpoint, incorporating knowledge I gained with running production systems for several larger portals. In the end it doesn't matter much what your software is - it only matters how you can distribute it over your server farm.

To follow this documentation, you will need the following packages and files installed on your system:

• [Django][2] itself - currently fetched from SVN. Follow the setup instructions or use python setup.py install .
• [Flup][3] - a package of different ways to run WSGI applications. I use the threaded WSGIServer in this documentation.
• [lighttpd][4] itself of course. You need to compile at least the fastcgi, the rewrite and the accesslog module, usually they are compiled with the system.
• [Eunuchs][5] - only needed if you are using Python 2.3, because Flup uses socketpair in the preforked servers and that is only available starting with Python 2.4
• [django-fcgi.py][6] - my FCGI server script, might some day be part of the Django distribution, but for now just fetch it here. Put this script somewhere in your $PATH, for example /usr/local/bin and make it executable.
• If the above doesn't work for any reason (maybe your system doesn't support socketpair and so can't use the preforked server), you can fetch [django-fcgi-threaded.py][7] - an alternative that uses the threading server with all it's problems. I use it for example on Mac OS X for development.

Before we start, let's talk a bit about server architecture, python and heavy load. The still preferred Installation of Django is behind Apache2 with mod python2. mod python2 is a quite powerfull extension to Apache that integrates a full Python interpreter (or even many interpreters with distinguished namespaces) into the Apache process. This allows Python to control many aspects of the server. But it has a drawback: if the only use is to pass on requests from users to the application, it's quite an overkill: every Apache process or thread will incorporate a full python interpreter with stack, heap and all loaded modules. Apache processes get a bit fat that way.

Another drawback: Apache is one of the most flexible servers out there, but it's a resource hog when compared to small servers like lighttpd. And - due to the architecture of Apache modules - mod_python will run the full application in the security context of the web server. Two things you don't often like with production environments.

So a natural approach is to use lighter HTTP servers and put your application behind those - using the HTTP server itself only for media serving, and using FastCGI to pass on requests from the user to your application. Sometimes you put that small HTTP server behind an Apache front that only uses mod proxy (either directly or via mod rewrite) to proxy requests to your applications webserver - and believe it or not, this is actually a lot faster than serving the application with Apache directly!

The second pitfall is Python itself. Python has a quite nice threading library. So it would be ideal to build your application as a threaded server - because threads use much less resources than processes. But this will bite you, because of one special feature of Python: the GIL. The dreaded global interpreter lock. This isn't an issue if your application is 100% Python - the GIL only kicks in when internal functions are used, or when C extensions are used. Too bad that allmost all DBAPI libraries use at least some database client code that makes use of a C extension - you start a SQL command and the threading will be disabled until the call returns. No multiple queries running ...

So the better option is to use some forking server, because that way the GIL won't kick in. This allows a forking server to make efficient use of multiple processors in your machine - and so be much faster in the long run, despite the overhead of processes vs. threads.

For this documentation I take a three-layer-approach for distributing the software: the front will be your trusted Apache, just proxying all stuff out to your project specific lighttpd. The lighttpd will have access to your projects document root and wil pass on special requests to your FCGI server. The FCGI server itself will be able to run on a different machine, if that's needed for load distribution. It will use a preforked server because of the threading problem in Python and will be able to make use of multiprocessor machines.

I won't talk much about the first layer, because you can easily set that up yourself. Just proxy stuff out to the machine where your lighttpd is running (in my case usually the Apache runs on different machines than the applications). Look it up in the mod_proxy documentation, usually it's just ProxyPass and ProxyPassReverse.

The second layer is more interesting. lighttpd is a bit weird in the configuration of FCGI stuff - you need FCGI scripts in the filesystem and need to hook those up to your FCGI server process. The FCGI scripts actually don't need to contain any content - they just need to be in the file system.

So we start with your Django project directory. Just put a directory public html in there. That's the place where you put your media files, for example the admin media directory. This directory will be the document root for your project server. Be sure only to put files in there that don't contain private data - private data like configs and modules better stay in places not accessible by the webserver. Next set up a lighttpd config file. You only will use the rewrite and the fastcgi modules. No need to keep an access log, that one will be written by your first layer, your apache server. In my case the project is in /home/gb/work/myproject - you will need to change that to your own situation. Store the following content as /home/gb/work/myproject/lighttpd.conf

 server.modules = ( "mod_rewrite", "mod_fastcgi" )
 server.document-root = "/home/gb/work/myproject/public_html"
 server.indexfiles = ( "index.html", "index.htm" )
 server.port = 8000
 server.bind = "127.0.0.1"
 server.errorlog = "/home/gb/work/myproject/error.log"

fastcgi.server = (
"/main.fcgi" => (
"main" => (
"socket" => "/home/gb/work/myproject/main.socket"
 )
 ),
"/admin.fcgi" => (
"admin" => (
"socket" => "/home/gb/work/myproject/admin.socket"
 )
 )
 )

url.rewrite = (
"^(/admin/.*)$" => "/admin.fcgi$1",
"^(/polls/.*)$" => "/main.fcgi$1"
 )

mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "audio/x-wav",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".conf" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv"
 )

I bind the lighttpd only to the localhost interface because in my test setting the lighttpd runs on the same host as the Apache server. In multi server settings you will bind to the public interface of your lighttpd servers, of course. The FCGI scripts communicate via sockets in this setting, because in this test setting I only use one server for everything. If your machines would be distributed, you would use the "host" and "port" settings instead of the "socket" setting to connect to FCGI servers on different machines. And you would add multiple entries for the "main" stuff, to distribute the load of the application over several machines. Look it up in the lighttpd documentation what options you will have.

I set up two FCGI servers for this - one for the admin settings and one for the main settings. All applications will be redirected through the main settings FCGI and all admin requests will be routed to the admin server. That's done with the two rewrite rules - you will need to add a rewrite rule for every application you are using.

Since lighttpd needs the FCGI scripts to exist to pass along the PATH_INFO to the FastCGI, you will need to touch the following files: /home/gb/work/myprojectg/public_html/admin.fcgi ``/home/gb/work/myprojectg/public_html/main.fcgi

They don't need to contain any code, they just need to be listed in the directory. Starting with lighttpd 1.3.16 (at the time of this writing only in svn) you will be able to run without the stub files for the .fcgi - you just add "check-local" => "disable" to the two FCGI settings. Then the local files are not needed. So if you want to extend this config file, you just have to keep some very basic rules in mind:

• every settings file needs it's own .fcgi handler
• every .fcgi needs to be touched in the filesystem - this might go away in a future version of lighttpd, but for now it is needed
• load distribution is done on .fcgi level - add multiple servers or sockets to distribute the load over several FCGI servers
• every application needs a rewrite rule that connects the application with the .fcgi handler

Now we have to start the FCGI servers. That's actually quite simple, just use the provided django-fcgi.py script as follows:

 django-fcgi.py --settings=myproject.work.main
 --socket=/home/gb/work/myproject/main.socket
 --minspare=5 --maxspare=10 --maxchildren=100
 --daemon

django-fcgi.py --settings=myproject.work.admin
 --socket=/home/gb/work/myproject/admin.socket
 --maxspare=2 --daemon

Those two commands will start two FCGI server processes that use the given sockets to communicate. The admin server will only use two processes - this is because often the admin server isn't the server with the many hits, that's the main server. So the main server get's a higher-than-default setting for spare processes and maximum child processes. Of course this is just an example - tune it to your needs.

The last step is to start your lighttpd with your configuration file: lighttpd -f /home/gb/work/myproject/lighttpd.conf

That's it. If you now access either the lighttpd directly at http://localhost:8000/polls/ or through your front apache, you should see your application output. At least if everything went right and I didn't make too much errors.

Eunuchs provides a few functions that were not yet available in Python 2.3. Specifically, socketpair and recvmsg/sendmsg are very important - for server programming with preforked servers, for example.

Another Questionnaire

From Lisa Sonnabend's mailbox:

As part of my thesis at the Institute for Communication Studies and Media Research at Ludwig-Maximilians-Universität München, I am conducting a survey among weblog readers on the topic of "Credibility of Weblogs". This is one of the first scientific studies on the phenomenon of weblogs in Germany.

It is to be clarified whether weblogs are already considered as trustworthy and competent in certain points such as fairness or impartiality as traditional media – and in which points there are great differences. In addition, the survey examines which special qualities weblogs have from the perspective of the recipients and what importance they have for the users. Based on the results, a prognosis about the future importance of weblogs in the media landscape is to be made.

And I should put the link in the blog so that people can participate. Well, I'll do that, even though I wonder what competence and credibility the old media are supposed to have.

Here goes the questionnaire

Preventive telephone surveillance is not

The Constitutional Court declares preventive telephone surveillance null and void. Primarily affects Lower Saxony, but similar situations exist in other federal states. Good when something is finally put in the way of the whole surveillance fetishism.

There are days when my computer hates me

For example, when I play with Flup and instead of the threaded server I want to use a forked server. And I realize that the latter requires the socketpair function, which unfortunately is only available from Python 2.4, which is available on Debian Sarge, but for Python 2.4 there is no Psycopg in Debian Sarge - which in turn is a prerequisite for Django and PostgreSQL, which is why I am dealing with FastCGI in the first place. Installing Psycopg itself is no fun, as you not only need the PostgreSQL headers that are normally installed, but also a few internal headers - so in principle a build tree. And then you also need the egenix-mx-base headers, which you can only get for Python 2.3, so you would have to install that yourself as well. Backports from the next Debian version don't work either, as they are just switching to PostgreSQL 8.0 and Sarge is still using 7.4 and I didn't want to upgrade the whole system right away. And so you go in circles and feel a bit cheated by all the dependencies and version conflicts.

And what do you do as a solution, because the threaded server unfortunately only produces segfaults in Psycopg? You take the threaded server, forbid it to thread and start it via the spawn-fcgi from lighttpd, or directly from lighttpd. But that's somehow stupid again, because then there are always 3 threads per FCGI server, two of which just stand in the process list and do nothing. And all this just because mod python2 (which is needed for Django) requires Apache2, which in turn requires mod perl2, which is incompatible with the old mod perl, which is why a whole bunch of my sites wouldn't work anymore if I switched to Apache2. Which I don't want to do anyway, because Apache2 with mod python is damn slow. And once again I feel cheated. I really should have looked for a more meaningful job.

If you didn't understand anything: doesn't matter, it's technology, it's not important, I just wanted to say that.

Higher-Order Perl is a book (currently in paper form, but it is supposed to be freely available online soon) that deals with higher-order functions and Perl - could be quite interesting, Perl offers a lot of features hidden under all those curly braces and other special characters ...

Running Django with FCGI and lighttpd

Diese Dokumentation ist für einen grösseren Kreis als nur .de gedacht, daher das ganze in Neuwestfälisch Englisch. Sorry. Update: I maintain the actually descriptions now in my trac system. See the FCGI+lighty description for Django. There are different ways to run Django on your machine. One way is only for development: use the django-admin.py runserver command as documented in the tutorial. The builtin server isn't good for production use, though. The other option is running it with mod_python. This is currently the preferred method to run Django. This posting is here to document a third way: running Django behind lighttpd with FCGI.

First you need to install the needed packages. Fetch them from their respective download address and install them or use preinstalled packages if your system provides those. You will need the following stuff:

• [Django][2] itself - currently fetched from SVN. Follow the setup instructions or use python setup.py install .
• [Flup][3] - a package of different ways to run WSGI applications. I use the threaded WSGIServer in this documentation.
• [lighttpd][4] itself of course. You need to compile at least the fastcgi, the rewrite and the accesslog module, usually they are compiled with the system.

First after installing ligthttpd you need to create a lighttpd config file. The configfile given here is tailored after my own paths - you will need to change them to your own situation. This config file activates a server on port 8000 on localhost - just like the runserver command would do. But this server is a production quality server with multiple FCGI processes spawned and a very fast media delivery.

 # lighttpd configuration file
 #
 ############ Options you really have to take care of ####################

server.modules = ( "mod_rewrite", "mod_fastcgi", "mod_accesslog" )

server.document-root = "/home/gb/public_html/"
 server.indexfiles = ( "index.html", "index.htm", "default.htm" )

 these settings attch the server to the same ip and port as runserver would do

server.errorlog = "/home/gb/log/lighttpd-error.log"
 accesslog.filename = "/home/gb/log/lighttpd-access.log"

fastcgi.server = (
"/myproject-admin.fcgi" => (
"admin" => (
"socket" => "/tmp/myproject-admin.socket",
"bin-path" => "/home/gb/public_html/myproject-admin.fcgi",
"min-procs" => 1,
"max-procs" => 1
 )
 ),
"/myproject.fcgi" => (
"polls" => (
"socket" => "/tmp/myproject.socket",
"bin-path" => "/home/gb/public_html/myproject.fcgi"
 )
 )
 )

url.rewrite = (
"^(/admin/.*)$" => "/myproject-admin.fcgi$1",
"^(/polls/.*)$" => "/myproject.fcgi$1"
 )

This config file will start only one FCGI handler for your admin stuff and the default number of handlers (each one multithreaded!) for your own site. You can finetune these settings with the usual ligthttpd FCGI settings, even make use of external FCGI spawning and offloading of FCGI processes to a distributed FCGI cluster! Admin media files need to go into your lighttpd document root.

The config works by translating all standard URLs to be handled by the FCGI script for each settings file - to add more applications to the system you would only duplicate the rewrite rule for the /polls/ line and change that to choices or whatever your module is named. The next step would be to create the .fcgi scripts. Here are the two I am using:

 #!/bin/sh
 # this is myproject.fcgi - put it into your docroot

export DJANGOSETTINGSMODULE=myprojects.settings.main

/home/gb/bin/django-fcgi.py

 #!/bin/sh
 # this is myproject-admin.fcgi - put it into your docroot

export DJANGOSETTINGSMODULE=myprojects.settings.admin

/home/gb/bin/django-fcgi.py

These two files only make use of a django-fcgi.py script. This is not part of the Django distribution (not yet - maybe they will incorporate it) and it's source is given here:

 #!/usr/bin/python2.3

def main():
 from flup.server.fcgi import WSGIServer
 from django.core.handlers.wsgi import WSGIHandler
 WSGIServer(WSGIHandler()).run()

if name == 'main':
 main()

As you can see it's rather simple. It uses the threaded WSGIServer from the fcgi-module, but you could as easily use the forked server - but as the lighttpd already does preforking, I think there isn't much use with forking at the FCGI level. This script should be somewhere in your path or just reference it with fully qualified path as I do. Now you have all parts togehter. I put my lighttpd config into /home/gb/etc/lighttpd.conf , the .fcgi scripts into /home/gb/public_html and the django-fcgi.py into /home/gb/bin . Then I can start the whole mess with /usr/local/sbin/lighttpd -f etc/lighttpd.conf . This starts the server, preforkes all FCGI handlers and detaches from the tty to become a proper daemon. The nice thing: this will not run under some special system account but under your normal user account, so your own file restrictions apply. lighttpd+FCGI is quite powerfull and should give you a very nice and very fast option for running Django applications. Problems:

• under heavy load some FCGI processes segfault. I first suspected the fcgi library, but after a bit of fiddling (core debugging) I found out it's actually the psycopg on my system that segfaults. So you might have more luck (unless you run Debian Sarge, too)

• Performance behind a front apache isn't what I would have expected. A lighttpd with front apache and 5 backend FCGI processes only achieves 36 requests per second on my machine while the django-admin.py runserver achieves 45 requests per second! (still faster than mod_python via apache2: only 27 requests per second) Updates:

• the separation of the two FCGI scripts didn't work right. Now I don't match only on the .fcgi extension but on the script name, that way /admin/ really uses the myproject-admin.fcgi and /polls/ really uses the myproject.fcgi.

• I have [another document online][6] that goes into more details with regard to load distribution

flup: random Python WSGI stuff - a collection of WSGI server adapters for FCGI, SCGI and Apache Jakarta 1.3 protocols as well as a few WSGI middlewares for authentication, compression and error handling.

Stupid Patents the One Hundred and Eleventh

Whenever you think you've seen the most idiotic patent, something even more stupid is guaranteed to come along - Microsoft wants to patent smileys:

The patent application published on Thursday describes a method for encoding self-created emoticon images as strings that can be embedded in text messages.

Leonardo is a CMS with blog and wiki modules in Python. Currently quite simple as CGI, but it should be migrated to WSGI and Paste and could then be quite interesting as general CMS components in a WSGI solution.

Python Paste is a meta-framework - a framework for creating new web frameworks based on WSGI. Many interesting middleware modules and a reimplementation of WebWare based on WSGI.

Scotland Yard shoots innocent people

By now it's clear that the person shot by British police was innocent. And yet: Scotland Yard defends headshot practice. The former Scotland Yard chief:

"There is only one sure way to stop a suicide bomber who is convinced of his mission - to destroy his brain immediately, completely," he wrote in the Sunday newspaper "News of the World". "That means shooting him in the head with destructive force, killing him instantly."

One must keep in mind that these were plainclothes police officers - a panic reaction when a group of armed people in civilian clothes are after you is, of course, preprogrammed. To use this as justification for a targeted headshot is more than just cynicism.

Executions on suspicion as a response to terrorism - this turns the alleged protection of society itself into terrorism and a danger to every citizen. The inhumanity of the police representatives in defending this practice - Police chief 'sorry' over death - is simply revolting.

Seaside is a flexible and very interesting web framework in Smalltalk. I have already linked to tutorials about it, but not the framework itself - at least not at its new address. Runs on Squeak and Visual Works - and through their wide availability on almost everything that can be called a computer and has a TCP/IP connection to the outside world.

Mandatory insurance sponsorship by the state?

Foreign Minister Fischer advocates for a mandatory Riester pension - naturally, the reader wonders what the Foreign Minister has to do with it, but never mind:

Federal Foreign Minister Joschka Fischer has spoken out in favor of introducing a mandatory private pension insurance. "The pension system must be affordable. I wish that the Riester pension would finally be introduced in a binding manner and that we encourage people to private provision."

Well, well. Encouraging private provision. To this end, a mandatory participation in the biggest insurance scam of all time - the Riester pension, whose returns are modest and whose payment guarantees are more than questionable.

One could, of course, simply choose a model of a citizen's insurance, in which everyone pays into the state social insurance schemes, without leaving a loophole for higher earners to escape - especially within the framework of great ideas such as the Ich-AG and increased self-employment, the social insurance system is further hollowed out. But that would be an intelligent solution. The state social insurances have the advantage that they are subject to certain rules by the constitution - and the state must ensure that the corresponding services are provided.

Instead, the private insurance industry is further sponsored and, according to Fischer's ideas, even with a mandatory requirement for citizens. Yes, that brings growth, that makes sense. That citizens are simply being taken advantage of and many models are pure extortion, and no payout security is given in any way, we simply ignore for now.

One thing is certain: with the idiocy of our politicians, there will soon be a lot to earn in the insurance industry. Which will presumably have nothing better to do than to gamble our then private forced pensions on the stock market and sell them to hedge funds.

Crazy Vinokourov ...

... snatches the victory from the sprinter teams right in front of their noses in Paris. Nobody really expected that. First, he still gets two seconds of goodwill in the fight for the overall classification and the 5th place and then he simply doesn't want to fall in line and leave the field to the sprinters. Classy move, I love such surprises.

Otto will foam again

Now he probably has to rant against the EU, as the EU Commission insists on the independence of data protection authorities:

The EU Commission has initiated a breach of contract procedure against Germany for disregarding the EU Data Protection Directive. It criticizes that the supervision of privacy protection in this country is in the hands of the state. The "current organization of the supervisory bodies responsible for monitoring data processing in the non-public sector" is "not compatible with Community law," according to a letter from the Directorate-General for Justice, Freedom and Security available to heise online.

Whether one should take the EU Commission's sudden advocacy for data protection seriously in view of the efforts to extend storage times for communication data within the EU is a completely different matter ...

GNU Modula-2 was unknown to me until now. It's nice that Modula-2 is also found in the GNU compiler family. Even though Modula-2 only holds historical interest for me - I much prefer dynamic languages like Python. But there were times when I used to program diligently in Modula-2.

MochiKit is a JavaScript library with a whole range of extensions for JavaScript. Above all, iterators, sensible functional concepts (filter, map, partial application), but also a whole range of new ideas, such as a very nice AJAX integration. Looks quite nice, I have to play around with it.

Top chefs are fighting for Linda (http://www.wdr.de/themen/wirtschaft/2/linda/index.jhtml?rubrikenstyle=wirtschaft) - and want to launch a petition. Hey, you have my signature for that. After all, I am extremely picky when it comes to potatoes. And no, it's not enough that the new potato variety has a name from Star Trek Voyager.

About unlucky people and favorites

Well, one thing was definitely clear today (aside from the fact that Armstrong still rides better than Ullrich, even in the time trial): Rasmussen is the unlucky one of the day. Two crashes, 5 breakdowns, and over 7 minutes caught up, that hurts. Fortunately, he arrived and he can console himself: today he will also be given a polka dot jersey.

The next Tour is already shaping up with the favorites: Ullrich and Basso right at the top of the list. Vinokourov certainly also a favorite, but definitely with a clear margin. It may be that at some point a new rider will come along - but these three are definitely in the running for the victory.

It will definitely be more interesting when Armstrong no longer starts - the favorites for next year are clearly much closer together and the battles should also become more exciting again. Ullrich and Vinokourov clearly the much stronger time trialists, Basso the much better climber - that will definitely be exciting.

Only one thing I still don't understand at the end of the Tour: why T-Mobile didn't take Zabel with them. Those idiots.

Django Again

Django - the upcoming web framework for Python - now has SQLite 3 support. This makes setting up a development environment for Django projects extremely simple: you just need Python 2.3 or Python 2.4, SQLite3, and PySQLite2. On a Mac, everything is already there except PySQLite2, which you can get from www.pysqlite.org and install using sudo python setup.py install. And you're ready to start with Django and work through the tutorials. No Apache needed, no PostgreSQL (though it's the nicest of all SQL databases, it's sometimes overkill for a development environment on a notebook), and especially not psycopg - whose installation unfortunately requires almost a full PostgreSQL source tree. So there's no excuse for Pythonistas not to get involved with Django.

Internet pillory of US law enforcement agencies

Shame on privacy rights, in the land of the brave and the free the US Department of Justice puts a sex offender database online. And who believes that wouldn't be so bad, it only affects rapists and pedophiles:

The online activities of US law enforcement are not only directed against convicted criminals. The police of Chicago recently launched a site where individuals are depicted and published by name who are suspected of "supporting prostitution". The site notes that the listed persons are considered innocent until a court has determined their guilt.

Great, isn't it? Let's just put your picture on display, the small side note the lynch mob will surely read before they unpack the rope and drag your ass to the nearest tree. And anyway, the term sex offender can't be broad enough ...