
DNA Analyses: Bavaria launches federal council initiative - who else if not Bavaria? Current incidents are seen as a welcome opportunity to quickly push through some changes. Never mind that these changes enable far more than fingerprints allow - and that there are many more possibilities for abuse (e.g. genetic analysis for assessing suitability).
The ruling of the Federal Constitutional Court that explicitly restricted DNA analyses to particularly serious crimes is being ignored as well. Who cares about the Federal Constitutional Court when populism works so well for stirring up sentiment...
Grafedia is something like links for hardware out there. You use a word as an email address under a domain and you receive a file in response. The concept is simple — it becomes interesting through the use of mobile phones to send the mail — and getting the result back on your mobile. And the fact that you can put the words somewhere out there on walls and similar places. Somehow crazy, somehow beautiful. (Found at Spreeblick)
LynuxWorks Introduces First User-Mode Linux Software for Apple PowerPC G5 Based on the Linux 2.6 Kernel - this now makes it possible to build logically separated virtual environments under Linux on PPC machines as well.
MathWorld News: The Mathematics of Tsunamis - interesting (I think) explanation of tsunami wave development. Found in zeitwissen:log.
Spent 5 minutes pondering which device had just sounded an alarm and what on earth it was supposed to remind me of. After checking several devices and programs, I realized it was an alarm that had actually been deleted long ago on my (not synced, therefore outdated) PDA. Insufficient alarm signal recognition capability due to alarm tone diversity overload...
Brian Mastbrook describes very nicely how Quicksilver combines the best of keyboard-driven interfaces and graphical interfaces. Unfortunately, QuickSilver only runs on 10.3 and later, which is why I'm still stuck with LaunchPad - which, however, in the latest versions (aside from the really extremely slow startup) can keep up quite well.
In general, I find this slowly developing idea of combining graphical and keyboard-driven interfaces very pleasant. Graphical interfaces are good at presenting complex structures (a directory structure becomes clear to me graphically faster than from the shell), but they are often quite cumbersome to use. Tools like QuickSilver and LaunchPad help tremendously. Apple's Universal Key Access would probably help me too - if I had 10.3...
Save Think Secret's Nicholas Ciarelli Petition is worth considering signing if you're an Apple user. In any case, this lawsuit from Apple is neither positive nor sensible - after all, the Apple world also lives in part from its rumors. Found at Spreeblick.
The Temboz RSS aggregator is a very nicely made aggregator in Python. It uses the Ultraliberal Feedparser for parsing and can import OPML. I find the interface nicely designed and the administration quite straightforward. And it has some nice features like the two-column layout and the fairly simple integrated filtering capability as well as quite useful feed list sorting options. I'm playing around with it a bit right now - even if that will probably reduce my motivation to write my own aggregator.
Working with Automator describes how the new automation tool in Mac OS X 10.4 works. Makes you curious ... (Found at Schockwellenreiter)
Shockwave Rider is having problems with his Zope server. Since I've been doing professional Zope hosting in my company for several years now and run quite a few massive portals (between 2000 and 3000 hits per minute are not uncommon - though distributed across many systems), here are some tips from me on scaling Zope.
- The most important step I would recommend to everyone is to streamline. Remove from Zope everything that doesn't need to be there - what can be created statically, what rarely changes, where no content management is needed: get rid of it. Put it in regular Apache directories. Use Apache's mod_rewrite to ensure the old URLs still work, but are served from Apache. This especially applies to all those little nuisances like layout graphics - they don't need to come from Zope, they're better served from Apache.
- Use Zope caching whenever possible. Whenever possible means: enough memory on the server so that even memory-hungry processes have some breathing room. Generally, Zope's built-in caching causes processes to get fatter and fatter - the cleanup in its own cache is quite useless. So implement process monitoring that shoots down and restarts a Zope process when it uses too much memory. Yes, that really is sensible and necessary.
- There are two good caching options in Zope: the RAMCacheManager and the HTTPCacheManager. The former stores results of Zope objects in main memory and can therefore cache individual page components - put the complex stuff in there. The second (HTTPCache) works together with Squid. Put a Squid in front of your Zope as an HTTP accelerator and configure the HTTP Cache Manager accordingly so that Zope generates the appropriate Expire headers. Then a large part of your traffic will be handled by Squid. It's faster than your Zope. Alternatively, you can configure an Apache as an HTTP accelerator with local cache - ideal for those who can't or don't want to install Squid, but do have options for further Apache configuration.
- Large Zope objects (and I mean really large in terms of KB) kill Zope. With caching they destroy your best cache strategy, and Zope itself becomes incredibly slow when objects get too large. The reason lies in Zope's architecture: all objects are first laboriously pieced together through multiple layers by various software layers. In memory - and therefore take up corresponding space in memory. Get rid of complex objects with huge KB numbers. Make them smaller. Create them statically via cron job. Serve them from Apache - there's nothing dumber than storing all your large PDFs in Zope in the ZODB, or even generating them dynamically there.
- Install ZEO. That thing rocks. Basically it's just the ZODB with a primitive server protocol. What's important: your Zope can be split into multiple process groups. You want this when you're using process monitoring to kill a rogue Zope process, but want the portal to appear as undamaged as possible from the outside - in that case just add mod_backhand to Apache, or another balancing technique between Apache and Zope. Additionally, ZEO also makes packing the ZODB (which should run daily) easier, since the pack runs in the background on the ZEO and the Zope servers themselves aren't greatly affected.
- If you have it, use an SMP server. Or buy one. Really - it brings a lot. The prerequisite is the aforementioned technique with multiple process groups - Python has a global interpreter lock, which means that even on a multiprocessor machine, never more than one Python thread runs at a time. Therefore you want multiple process groups.
- Performance is also gained by disabling layers. Unfortunately this often can only be realized with software changes, so it's more interesting for those who build it themselves. Move complex processes out of the Zope server and put them in Zope Products. Zope Products run natively without restrictions in the Python interpreter. Zope Python scripts and DTML documents, on the other hand, are dragged through many layers that ensure you respect Zope's access rights, don't do anything bad, and are generally well-behaved. And they make you slower. Products are worthwhile - but cost work and, unlike the other technical tips, aren't always feasible.
- Additionally, it has proven useful not to put too much data in the ZODB, especially nothing that expands it - the ZODB only gets bigger, it only gets smaller when packing. After some time you easily have a ZODB in the GB range and shouldn't be surprised by slow server starts...
- If more complex processes occur in the system, it can make sense to outsource them completely. I always use TooFPy for that. Simply convert all the more complex stuff into a tool and stick it in there - the code runs at full speed. Then simply access the tool server from Zope with a SOAP client or XMLRPC client and execute the functions there. Yes, the multiple XML conversion is actually less critical than running complex code in Zope - especially if that code demands considerable runtime. Zope then blocks one of its listeners - the number is static. And simply pushing it up doesn't help - thanks to the global interpreter lock, only more processes would wait for this lock to be released (e.g., for every C extension that's used). There's a good and fast C implementation for XMLRPC communication that can be integrated into Python, making the XML overhead problem irrelevant.
- If you use PostgreSQL as a database: use PsycoPG as the database driver. Session pooling really gets Zope going. Generally you should check whether the corresponding database driver supports some form of session pooling - if necessary via an external SQL proxy. Otherwise, Zope might hang the entire system during SQL queries because a heavy query waits for its result. Many have already fallen into this trap and learned that 16 Zope threads doesn't necessarily mean 16 parallel processed Zope accesses when SQL databases are involved.
Of course there's a lot more you can do, but the above are largely manageable on the fly and mainly depend on you having enough memory in the server (and possibly a multiprocessor machine - but it works without one too). Memory is important - the more the better. If you can, just put more memory in. You can't have too much memory...
What to do if even all that's not enough (yes, I've had that - sometimes only the really heavy-handed approach helps). Well, in that case there are variations of the above techniques. My favorite technique in this area is active caching. By this I mean that Zope is configured at one point for which documents should be actively cached. This then requires a script on the machine that fetches the pages from Zope and puts them in a directory. Apache rewrite rules then ensure that the static content is served from the outside. Basically you're ensuring that the pages most frequently visited and suitable for this technique (i.e., for example, containing no personalization data) simply go out as a static page, no matter what else happens - the normal caching techniques just aren't brutal enough, too much traffic still goes through to the server.
Another step is of course the use of additional machines - simply put more machines alongside and connect them using the ZEO technique.
Zope is fantastic software - especially the high integration of development environment, CMS, and server is often incredibly practical, and the easy integration of external data sources is also very nice. But Zope is a resource hog, you have to put it that simply.
Debian has a wonderful package system. And it has a whole range of very useful tools to make backports easier - for example, by using debootstrap to set up a chroot environment where you can safely gather the packages you need for the build and then create a corresponding package. I've used the whole thing several times, it's really great.
However, it can sometimes drive you crazy. I wanted to install the latest SQLite from Debian Testing. To do that, I first need the necessary tools to build the package. Since I had just set up a new chroot environment, not everything was there yet - for example, I was missing cdbs, a very powerful (and by now widely used) tool for easy creation of Debian packages. I had ported it once before, but I thought the opportunity was good to build a current version.
Or so I thought. It started off quite harmlessly - for the documentation it needs springgraph - a tool for formatting graphs. The tool itself actually has no build dependencies (except for the mandatory debhelpers). Fine. It also builds very quickly. When installing it, it complains about missing Perl modules for the GD2 integration. Okay, porting Perl modules is often tedious, but this one actually looked quite simple. A series of build dependencies, sure, but otherwise harmless. Except for the fact that it needs cdbs to build.
Aaaaarghl!!!!
Okay, I know what you have to do. Still. Sometimes I get the feeling that the Debian maintainers secretly get together to drive me crazy.