WP Super Cache is highly recommended if you use several plugins like I do (and possibly access external services like the Tumblr plugin). Okay, there is a risk that a broken status from Tumblr might be cached in the sidebar, but this will resolve itself after a while. However, the runtime of a fully loaded WordPress is indeed a bit sluggish.