I used to play around with grsecurity before, but the installation was a bit tricky - especially, you didn't know what to configure as a start and how to begin a reasonable rule-based security - the whole thing was more of a trial-and-error hopping than an understandable installation. However, for a security solution for an operating system, it is rather negative if you don't get the feeling of understanding what is happening there.
With the current versions of grsecurity, however, this has changed to a large extent. On the one hand, the patches run completely smoothly into the kernel, on the other hand there are two essential features that make the start easier: a Quick Guide and RBACK Full System Learning.
The Quick Guide provides a short and concise installation guide for grsecurity with a starting configuration for all the options that already offer a fairly good basis and excludes problematic options (which could exclude some system services). This way you get a grsecurity installation that offers a lot of protection but usually does not conflict with common system services. This is especially important for people with root servers - a wrong basic configuration could lock themselves out of the system and thus make the system unusable and a service case.
But the Full System Learning is really nice: here the RBAC engine is transformed into a logging system and it is logged which users execute what and what rights are needed for this. The whole thing is still controlled by corresponding basic configs that classify different system areas differently (e.g. ensure that the user can access everything in his home, but not necessarily everything in various system directories). You just let the system run for a few days (to also catch cron jobs) and then generate a starting configuration for RBAC from it. You can of course still fine-tune this (you should also do this later - but as a start it is already quite usable).
RBAC is basically a second security/rights layer above the classic user/group mechanisms of Linux. The root user does not automatically have all rights and access to all areas. Instead, a user must log in to the RBAC subsystem in parallel to his normal login (which happens implicitly through the system start for system services!). Rules are stored there that describe how different roles in the system have different access permissions.
The advantage: even automatically started system services are only allowed to access what is provided for in the RBAC configuration - even if they run under root rights. They only have limited capabilities in the system until they log in to the RBAC subsystem - but for this, a manual password entry is usually required for the higher roles. Attackers from the outside can indeed gain the user rights restricted by RBAC, but usually cannot get to the higher roles and therefore cannot interfere with the system as much as would be possible without RBAC.
The disadvantage (should not be concealed): RBAC is complex. And complicated. If you do something wrong, the system is locked - quite annoying for root servers that are somewhere out there in the network. You should always have fallback strategies so that you can still reach a blocked system. For example, after changes to the RBACs, comment out the automatic activation at system startup so that a reboot puts the system in a more open state in case of problems. Or have an emergency access through which you can still administer a blocked system to some extent. In general, as with all complex systems: Keep your hands off if you don't know what you're doing.
In addition to the very powerful RBAC, grsecurity offers a whole range of other mechanisms. The second major block is pax(important: here a current version must be used, in all older ones there is an evil security hole) - a subsystem that restricts buffer overflow attacks by removing the executability and/or writability from memory blocks. Especially important for the stack, as most buffer overflows start there. Pax ensures that writable areas are not executable at the same time.
A third larger block is the better protection of chroot jails. The classic possibilities for processes to break out of a chroot jail are no longer given, since many functions necessary for this are simply deactivated in a chroot jail. Especially for admins who run their services in chroot jails, grsecurity offers important tools, as these chroot jails were only very cumbersome to make really escape-proof.
The rest of grsecurity deals with a whole collection of smaller patches and changes in the system, many of which deal with better randomization of ports/sockets/pids and other system IDs. This makes attacks more difficult because the behavior of the system is less predictable - especially important for various local exploits, where, for example, the knowledge of the PID of a process is used to gain access to areas that are identified via the PID (memory areas, temporary files, etc.). The visibility of system processes is also restricted - normal users simply do not get access to the entire process list and are also restricted in the /proc file system - and can therefore not so easily attack running system processes.
A complete list of grsecurity features is online.
All in all, grsecurity offers a very sensible collection of security patches that should be recommended to every server operator - the possibility of remote exploits is drastically restricted and local system security is significantly enhanced by RBAC. There is no reason not to use the patch, for example, on root servers as a standard, given the rather simple implementation of the grsecurity patch in an existing system (simply patch the kernel and reinstall, boot, learn, activate - done). Actually, a security patch should be part of the system setup just like a backup strategy.
Now it would of course be even nicer if the actual documentation of the system was a bit larger than the man pages and a few whitepapers - and above all was up to date. This is still a real drawback, because the right feeling of understanding the system does not really set in without qualified documentation ...