Content-type: matter-transport/sentient-life-form

bDS slowly usable

Now the new software is starting to be fun, everything you need for usage and creation is largely present. It still has a few minor flaws and there are one or two features I want, but I can already do everything I really need as a minimum. Publishing now works easily with a shell script:

#!/bin/sh

cd ~/Blogs/rfc1437.de/html

rsync -rav --delete \
	--exclude='thumbnails/' \
	--exclude='media/' \
	* git.rfc1437.de:git-server/html/

cd ..

rsync -rav --delete thumbnails/ git.rfc1437.de:git-server/html/thumbnails/
rsync -rav --exclude='*.meta' --delete media/ git.rfc1437.de:git-server/html/media/

Yes, that was just to test my source formatting. And?

End-to-End: Google will mit Javascript verschlüsseln - Golem.de. You can think what you want about Google, but there are still technicians with brains there who sometimes build something cool. Of course, we'll have to see how well the whole thing is implemented, but if someone like Google offers PGP integration for Gmail, that could finally be the impetus for the topic to gain further traction. The biggest problem with PGP and similar things is still the availability of communication partners who can actually do something with the corresponding technology.

Tails - About Tails. Hmm, maybe set up a USB stick with the system, for on the go and as needed.

reclaim hugo | Collected stuff from social networks - since the Reclaim Social project has now delivered the WordPress plugin in a first version, I have finally set up a site where I now collect my G+, Twitter, Facebook, Flickr, Github and YouTube activities. It already looks quite neat. By the way, I am using the Shprink One theme, which works very well with the formats and has a few nice features. And it is especially clean. And it works with different screen sizes, including mobile. Let's see how this thing behaves in continuous operation.

Welcome to RISC OS Pi in Documentation. Ok, quite cute - RISC OS now as a system for the Raspberry Pi. On the other hand, it probably has better performance than the old Acorn machines had back then.

Port 32764: Cisco confirms backdoor in routers. Ok, they confirm the existence - but where is the explanation, where it comes from? Why is it there and why hasn't Cisco removed it long ago? Does anyone seriously want to tell me that Cisco would never run a port scan on their own routers?

Ori File System. Hmm - not a real server, implemented as a real file system, sounds good. However, not available for Windows and therefore only conditionally suitable for my device mix. Might still be worth a look.

WordPress › WordPress 3.8 “Parker”. It's been a while since an update that actually appeals to me visually - and I like the new default theme so much that I'm seriously considering switching from my current one (which is still based on 2010, with minor adjustments). I found the 2011/2012/2013 themes rather meh. Especially 2013 was just plain irritating with its color scheme. 2014 will need a few patches, but that mainly refers to the design of gallery posts and asides - although I could almost live with the asides, maybe just tuck them into a sidebar or something. Hmm, let's see if I'll go through with it - the advantage would be that I could get rid of a lot of my own tinkered code and thus have less work with potential new versions that would require adjustments (although my adjustments have proven surprisingly stable, so far I haven't had to touch anything). What I do find really strange, though: the "Press This" bookmarklet has been almost unchanged in design for ages. Could also use some sprucing up!

Nice on the outside, nasty on the inside

It seems to be the motto of Koken, which I once praised quite a bit here. It is still one of the best-looking gallery systems with a really sleek admin interface. But all that sleekness cannot hide the fact that the code underneath is probably not as sleek as the layout after some problems that were almost not debuggable. To this day, the login form has problems with Chrome and Safari - and they haven't solved the problem. From the reactions, it's not even clear if they care at all. How stupid is it when a login form doesn't work because of some JavaScript hacks under Chrome? It's a simple form with username and password, what's so big about JavaScript there?

The crown was the Lightroom plugin again today. I used Koken because my old blog workflow - thanks to the stupid decision of the WP programmers to scrap the entire Atom publishing and let it rot instead of fixing it - went down the drain. I don't want anything complicated - just a simple way to upload a stack of pictures online with one click directly from Lightroom. It also works if the target is Flickr or Picasa or one of the other supported online services. But I want to control my pictures myself - and host them myself. Well, if a plugin just creates an empty album but doesn't upload a single picture (even though it fiddles around for hours and pretends to be active), then it's all just for the worse. If there are no logs or debugging possibilities or messages anywhere that help in the analysis, then it's all just for the trash.

So, for the time being, pictures will probably end up here again, currently with Dossier de Presse, a plugin for Lightroom that publishes via XMLRPC to WordPress. Which is not optimal, but the only thing currently available that works with current WordPress, current Lightroom, and at all. Since no metadata is transferred via XMLRPC in WordPress, such as image titles, I will probably have to remove this from my gallery layout somehow, otherwise only strange technical image names will appear there. Titling pictures is stupid anyway.

Computers could be so much more fun if software wasn't programmed by amateurs and blockheads 90% of the time ...

Roundcube - Free and Open Source Webmail Software. Thanks to NSA, one starts to think again whether one does not want to handle mail oneself. Ok, one really does not want to, because sorting spam is no fun, but well, if one does want to, this might be a somewhat more modern variant of webmail solutions. In addition, one can integrate it into Owncloud via an app. Although this is really just an iframe integration - so do not expect to be able to use your address book together or something like that. Personally, I will probably stick with pushing my mail through Google and letting the NSA sort my spam. But contacts and calendars could soon end up on my owncloud. Even the file sync has become really good in the latest versions.

Trusted Computing: Federal government warns against Windows 8. As long as Microsoft does not also remove the article from the network, like the same article in the Zeit. TLDR: TCP offers more and more control over computers to people who are not the owners - and increasingly restricts the rights of the owner to protect themselves from it. Which is why the BSI warns the government (the title is somewhat misleading) against using systems that use TCP in newer versions. Whereas the NSA is very enthusiastic about the new possibilities. Which gives the whole thing even more background in the current Prism discussion.

Here's what I found worth reading this week, collected and uncommented:

What I consider worth reading during the week and did not want to link separately, collected here and uncommented.

OX Documents: Online-Office-Suite als Open Source - Golem.de. Another candidate worth taking a look at - especially since the OX App Suite can almost completely replace Google Apps. However, I would then have to handle spam filtering myself (or simply route it through Google and have it sent directly to me, as there was an option for that). It somewhat conflicts with what Owncloud does, so I'll have to see which of the two services is actually more interesting for me. But maybe file syncing will also come to OX App Suite soon.

EverythingServerUbuntu - Ryzom - Ryzom Core Development Site. I should take a closer look at this - at the moment I'm playing a bit of Ryzom, just for relaxation on the weekend, and that has been open source for some time now - you can set up your own server with your own little world. It's not connected to the main system as far as I can see (which is a shame, it would be funny to jump with an avatar from there into my own little world), but still maybe something to play around with. Even if it's probably more complex than OpenSim to build something like Lilliput inside it.

Newsblur is one of the candidates to replace Google Reader that can also be self-hosted. However, it has somewhat broader requirements. Still worth a look? But first, let's wait and see what happens in the reader app market, because without an Android and OSX app that syncs with it, it's not very interesting for me.

Chumby tricks - ChumbyWiki. If you look around, it's quite interesting to see what all could be done with the Chumby. It was really an open platform after all. So perhaps it's not so bad that the manufacturer is gone and the infrastructure is slowly disappearing, because it shows whether an open system can really continue to live, even if the originally driving force is gone. And apart from that, it can still remain a fun tinkering project. I think I will really replace my old radio alarm clock and do more with this thing. So if I'm late for work in the near future and miss morning appointments, you know what it's all about - then my hacked alarm clock has crashed.

the_silver_searcher - Interesting for anyone who, like me, manages and searches through large source trees. Essentially something like ack - an automatic grep that runs through entire hierarchies and additionally filters files by various patterns, taking into account gitignore and hgignore. And all this not as a Perl script like ack, but in C with various native libraries and various optimizations for searching. It doesn't get faster than this unless you use pre-generated indexes (which come with their own problems).

If you want to use the Atom publishing protocol, you now need to install a plugin in WordPress 3.5 because it has been removed from the core. Of course, the WordPress developers didn't bother to include a corrected version of the Atom Publishing Protocol server in the plugin; it still contains the over 2-year-old bug with media uploads. Fortunately, my patch still works, but now it needs to be applied to a different file. Quite a mess, what they're doing there. And when I see how the bug in the WordPress core was ignored, my hope that someone will take the trouble to fix the plugin is pretty close to zero.

The SQLite RTree Module. And another extension for SQLite, this one a standard extension. R-Trees are tree structures optimized for range queries - that is, range queries such as "is this given rectangle contained in the list of rectangles".

The Gaia-SINS federated project home-page. Just quickly bookmarked in case I need it - spatial data (GIS data) can be efficiently indexed and queried in SQLite with an extension. Since I am a declared fan of SQLite, this is quite interesting. And it is implemented as a dynamically loadable extension (of course, this only works if the SQLite you are using is also enabled for extensions - unfortunately this is often not the case, installation might require a recompilation of SQLite, but it's not that terrible).

plan 9 was the system that took the ideas of Unix even further and, building on that, enabled a distributed system with distributed resources and seamless networking as early as the late 80s. Just think about where we would be today if it had become mainstream. Tablets that directly use network resources, that directly use complex applications on CPU servers in the network and that the developer can directly access for debugging from his workstation, without any hacks.

F-Droid. Quite interesting - an app store for open source applications for Android. So you don't have to constantly check yourself for news. Of course, some of the apps are also in the normal Google Play Store, but not every open source project wants to jump through Google's hoops.

git-annex. Definitely worth a look or two. In principle, it's something like a manually operated Dropbox - you can link folders with other folders and define sync relationships. But you can also define redundancies, ensuring that there are enough copies of files - if you delete a file, you get a warning if it was the last copy (and it is restored). Many commands for efficient management of various scenarios are added, and there are various backends for the data - for example, you can integrate Amazon S3 and include it as a backup repository with suitable means, or reference URLs from the web and make files always reconstructable (with this you can also integrate your own file server with an http interface). Or even use something like Google Mail as a backend and store your data in file attachments. Or use all the means of git to exclude temporary results of synchronizations, for example. Unlike Sparkleshare - which is also based on git - only the metadata is versioned in git here, not the files themselves. This of course has the disadvantage that file changes cannot be undone via it - for this you would need a versioning backend such as bup, which is then used as a data backup with versioning and definition of backup cycles. The advantage of the git-annex method, however, is that the data does not grow as gigantic as with Sparkleshare if you want to sync large files such as videos or digital images - only at the defined backup interface would the versions occur and you can explicitly determine which data goes there. Not for mouse pushers, but great for command line fetishists.

jq. A very cool tool for someone who has to deal with JSON a lot, especially in the Unix shell. The tool can process JSON data with functions that are inspired by sed, awk, and grep. And you can pipe jq in a Unix-typical way or even use internal pipes in transformation expressions à la awk. And it all comes as a single-executable without runtime libraries except for libc - so it should even work as a static binary and thus be trivially installable with scp.

ownCloud’s Latest Community Edition Adds Video Streaming, and Easy Mounting of Third-Party Storage. Sounds nice, especially the mounting of cloud storage. I wonder if it only resides in the cloud and is passed through, or if it is also additionally downloaded to the server. But I could maybe play around with my small installation. The improved sync also sounds good, as the previous one is really a bit strange with its regular timestamp file check.

Postgres-XC project Page. Multi-Master (Read and Write) Cluster for PostgreSQL. Supports replicated setups as well as partitioned setups (or mixed forms).

amoffat/sh. Cool little module that integrates external commands as if they were functions. You simply call a function git with a few named parameters and get the git output as a string. Makes shell scripts in Python much more compact and readable. Exactly the right thing for sysadmins.

Sitaramc/gitolite. Nice little tool for easily setting up repositories that others should also have access to - management is done through a dedicated admin repository.

Plop: Low-overhead profiling for Python. I need to take a closer look at this, it could be very interesting for the company's servers, especially the low profiling overhead of only 2% sounds exciting. And the visualization is definitely one of the better ones for Python profilers.

Lion: Mobile Backup Turn off local Time Machine | Jan-Kaspar Münnich. No idea if I already had this, too lazy to search, but if the local Time Machine backups are annoying (e.g. with smaller SSDs, this is anything but practical, especially if you occasionally perform larger file operations), you can turn them off with tmutil disablelocal and turn them on with tmutil enablelocal. For example, when performing larger cleanup tasks, it makes sense to turn them off, even if you activate them later to have the additional work safety. Or you simply archive everything on the notebook in git or mercurial, then you naturally don't need local backups either.

Ymacs -- An Emacs-like editor for the Web. As the title says. Emacs now also boots in the browser. However, there is still no usable editor for Emacs.

Fuse4X – The Easiest and Fastest Way to Create File Systems for Mac OS X. MacFuse is currently going the way of the dodo, so there's a need for an alternative if you want to load userspace filesystems (e.g., if you want to mount an SSH server). This version also looks more interesting than OSXFuse, because OSXFuse primarily aims for compatibility with the old MacFuse, while Fuse4X primarily aims for compatibility with the Linux reference and should make porting userspace filesystems easier.

Make runfcgi fail when database connection is open before fork. This is something I've been chasing for ages, most recently in a few pretty important projects. Flup works by first initializing the WSGI application and then making the forks for the workers with this initialized WSGI application. Unfortunately, we have database accesses during application initialization - as a result, the base process already has an open database connection, and each fork copies these data. But the socket of the connection doesn't go with it - the new process just thinks it's connected, but it's not. Accesses from the new processes then fail with an exception. In the linked patch, you can also replace the raise on the exception with connection.connection = None. This simply discards the connection that is already defective and always builds a new connection in new processes. With this, we have at least been able to fix this in a production environment (with psycopg2) and are optimistic that it will also help in the environment with pyodbc.

ronnix/fabtools. Looks interesting - a few tools for Fabric with which you can manage simple system packages and Python packages (also within virtual environments). Should take a look, could simplify a few things when initializing work environments. However, Vagrant currently only uses Chef and Puppet and not Fabric, if I remember correctly.

SET TRANSACTION ISOLATION LEVEL Transact-SQL. There you can find more information about the isolation level in MSSQL, especially what the snapshot feature means. In principle, this makes MSSQL behave similarly to PostgreSQL.

#18251 multithreading deadlock in django.models.loading.get_apps – Django. And another thing that might affect us - race conditions between Django threads during the initialization of Django applications. There's already a patch for this that fixes it in the Django internals.

Enabling Snapshot Isolation - SQLAlchemy 0.7 Documentation. Could this help us? MSSQL seems to have a rather unfavorable isolation level as default. Hmm, we'll probably try it out.

apenwarr/bup. A very interesting small tool that manages backups differentially - and that is based on the git data structures. In principle, one could consider it as an intelligent splitter for large files with minor changes in each version - for example, tar files from servers. For this purpose, an rsync-like algorithm is used to determine the differences and the result is stored in a git archive. The whole thing can also be combined with the previous git-annex and thus provide a space-saving backup solution.

git-annex. Interesting plugin for git with which you only manage the metadata of large files, but not their actual contents in git. The files themselves are managed as normal external files, but the directory structures and references are correctly versioned - and all this together with normally versioned files. This bypasses the problem that git has with large files. And thus something like a photo archive based on git becomes possible.

Mumble. It's somehow funny when you get your software tips from a political party, but never mind. Mumble is something like Teamspeak (or also a distant relative of Skype group chats or Google+ Hangouts), only open source and for self-hosting. Clients are also available for OSX, Windows, Linux, and iOS. By the way, there are also efforts to integrate mumble as an alternative to Vivox in OpenSim as a voice system.

Ubuntu 10.04: Why is ksmd eating CPU cycles? | Interphero Miscellany. Okay, just stumbled upon this on my server and blogging it for later reference. When using kvm, on newer Ubuntu versions, ksmd appears, which actually compares memory pages between virtual machines and only stores duplicate pages once - this saves RAM but consumes CPU. If you don't do over-commit on memory, you can safely turn it off.

Features | ownCloud.org. Hmm, now I just need iOS access to the OwnCloud and I think I have a longer installation session ahead of me. I probably won't be able to completely give up Dropbox or similar services (and I don't want to), but maybe it will be enough to switch back to the free version of Dropbox if I only manage the large amounts of data on my own server.

TeamPostgreSQL - PostgreSQL Web Admin GUI Tools. Looks good, a web interface for administering PostgreSQL databases that comes close to PGAdminIII and doesn't look as spartan as PHPPGAdmin. However, I have problems accessing databases from a project - it seems not everything is quite cleanly implemented, especially in the handling of sequences. Therefore, I can't say anything yet about whether it is performant. The whole thing comes with a Java-Tomcat server, so you can install and run it locally directly.

Matasano Security - Matasano Web Security Assessments for Enterprises. Analysis of cryptography in JavaScript. Summary: Cryptography in JavaScript is usually a bad idea, as the JavaScript is loaded from an untrusted source or untrusted network (if you trusted it, cryptography would hardly be necessary) and therefore a chicken-and-egg problem exists. Regarding the previous 0bin project: cryptography was not implemented to secure the user, but to secure the 0bin operator - it is therefore relatively irrelevant to the operator whether users are secure or not, it is only about "plausible deniability" for the operator. The situation is different, however, when a JavaScript encryption is implemented instead of using SSL.

sametmax/0bin. Interesting approach to circumvent the problems of pastebin hosting. With pastebin-like sites, the problem is that users post all kinds of content and the server operator can quickly be held liable. 0bin tries to shift this problem so that the operator cannot know what is in the pastebins, as they are stored encrypted and the encryption is done by the client via JavaScript. This works, of course, only if the judges also accept that the server operator cannot know what is going on - and not, for example, argue that he should then just install different pastebin software where he can know and still hold him responsible. Certainly an interesting approach, especially the idea of putting the key for the encryption in the hash of the URL (i.e., the part after the # in the address) and thus having a functioning URL, but still not providing the server with the key (since the hash of a URL is only used by the client and not communicated to the server).

R17 - flexible, scalable, relational data mining language. Looks quite interesting, basically something like a cross between AWK and SQL. The result isn't really pretty, but it seems practical - especially because you can easily use multiple processors, or even multiple machines (implicit parallelization), and thus also quite easily evaluate large amounts of data with ad-hoc queries. Because there is a simple format for passing data to further steps, it can also be easily adapted to new data sources without first running a lengthy export step there.

Plumbum: Shell Combinators and More — Plumbum: Shell Combinators. Looks interesting and much more thought out than some alternatives I've looked at (and much more expanded than shutil+glob).

ownCloud.org | Your Cloud, Your Data, Your Way!. I will definitely keep an eye on this, because once the OSX desktop client and the iOS client are available, this will be a clear alternative to Google Drive, Dropbox, or SkyDrive for me. After all, I already have my own server (classic Ehschonda solution), and I would only use Dropbox for integrating various iOS applications and then integrate their content into my own server and bring it to my desktops. Because no matter how good a cloud provider is (and so far Dropbox is one of the clearly better ones), my own server gives me more trust in the end.

pycounters. I need to check this out, it allows you to easily integrate counters into a project that provide data on things like function calls or similar - basically something like the Windows Performance Counters, but for Python projects.