Let's say you started up and you initially didn't have the time or resources to host and run your own infrastructure. So you launched your service on AWS - the de-facto infrastructure-as-a-service provider. You got wildly successful, are now flush with cash, and believe that you could realize some savings (and have greater control?) by running your own infrastructre. But you're also smart enough to realize that no matter how hard you try, you possibly cannot achieve the elasticity or the scale of EC2 and S3. More significantly, it makes little sense to procure and provision hardware for peaks like the one animoto experienced in April '08 (slide 17 in Jeff Bezos' presentation), and then have your CPU utilization hover at 2% for most of the year. So what do you do? No sweat : you still have your private infrastructure provisioned to deal with regular workloads, but configure your load balancer (fronting your web-facing servers) to offload to EC2 when there are unexpected peaks. Sounds too far-fetched? Well, that's precisely what SmartSheet and Picnik did - use private infrastructure for the base load, with everything beyond offloaded to AWS. Perfect.
I found it a pleasant coincidence that this was posted on James Hamilton's blog just weeks after I hit a eureka moment arriving at exactly the same hybrid model (see? great minds think alike :-)) when thinking of how a present day startup might make the transition from running completely off the cloud to being run on its own infrastructure while being resilient to traffic spikes all along.