using S3 for software dependency management

I was fortunate enough this week to ignore the responsibilities of my normal software engineering job and focus all of my time on building a cool application using Amazon Web Services (AWS). A lot of the hype around AWS has to do with spiffy Web 2.0-style user interfaces and how you can combine data from multiple sources to generate web sites with consumer-level appeal, but I see things differently: new service businesses like AWS have the potential to change the way we build software.

After you spend a few years in a software engineering environment, you begin to notice that us software engineers spend way too much time doing things that really have nothing to do with writing software: we’re setting up revision control repositories, provisioning and managing a hardware fleet (in both production and test environments), and setting up software build and dependency management systems. In between all that, we still have to do the real work. Wouldn’t it be nice if we didn’t have to worry about that stuff?

I wrote some software this week to demonstrate how to use Amazon’s Simple Storage Service (S3) to manage software dependencies. More specifically, I wrote an Ivy resolver for S3 and a small dependency visualization tool to generate a directed graph of your software’s dependencies. Maintaining an artifact repository for your software dependencies is an absolute requirement for large, constantly evolving projects, but it’s one of those pesky tasks that just has nothing to do with actually writing software. Forget hardware to store your dependencies, forget scaling issues, forget the need for redundant hardware in another datacenter, and forget backups: just store your software artifacts in S3 and go for a beer.

Besides using S3 as a software artifact repository, I think there are other aspects of AWS that could greatly improve how we build software. Amazon’s Elastic Compute Cloud (EC2) service could be used as an on-demand build and test farm: you could spin up new virtual hosts in minutes that are provisioned with your own custom operating system image then build and test your software in these clean-room environments. This would be especially useful for organizations with software products that run on multiple different operating systems where maintaining a fleet of heterogeneous environments would be enormously expensive.