What do you think about having some kind of approvals based test suite? Basically, just generate the CF JSON for the stacks and compare them. If anything, it would be nice as a sanity check when you make a change that includes python logic.

Cool! This looks really nice to me. It'll be nice if we can rebase our private stack on the open source.

re: Dumping json & comparing: Yeah, that might be hard in this version - that said there's a pull request in process by @mhahn which will make writing tests like that a lot easier.

Mostly it'd just catch the 'oh, we thought we made a non-changing change but were wrong' stuff. It's hard to test this since it's so heavily tied into third party stuff.

Re: basing our internal stacks off of this - our stacks are sort of based off this (though it's the other way around) and we let ansible do more of the work, with less variables being managed in cloudformation, which is kind of nice. @bmarini was talking about that too - I'll see if there's a nice middle ground we can reach.

I just tried bringing this up from scratch and am running into the following issues after everything builds successfully:

on both minions (i only brought up 2 instead of 3):

$ docker logs `docker ps -l -q`
2015-07-24T07:49:47Z [INFO] Starting Agent: Amazon ECS Agent - v1.2.1 (5da1555)
2015-07-24T07:49:47Z [INFO] Loading configuration
2015-07-24T07:49:48Z [INFO] Checkpointing is enabled. Attempting to load state
2015-07-24T07:49:48Z [CRITICAL] Error creating state manager: stat '/data/': no such file or directory

If I remove the ECS_DATADIR and ECS_CHECKPOINT from the ecs_agent.env, I get the following:

# docker logs `docker ps -l -q`
2015-07-24T07:51:32Z [INFO] Starting Agent: Amazon ECS Agent - v1.2.1 (5da1555)
2015-07-24T07:51:32Z [INFO] Loading configuration
2015-07-24T07:51:33Z [INFO] Checkpointing not enabled; a new container instance will be created each time the agent is run
2015-07-24T07:51:33Z [INFO] Registering Instance with ECS
2015-07-24T07:51:33Z [ERROR] Could not register module="api client" err="Cluster not found."
2015-07-24T07:51:33Z [ERROR] Error registering: Cluster not found.

the ECS_CLUSTER variable is referencing an existing cluster.

on the controllers i get this:

$ docker logs `docker ps -l -q`
2015/07/24 07:47:12 [Driver '' not found.]

the only variables i didn't populate within example.env were:

empire_controller_cert_name: "''"

empire_controller_github_client_id:
empire_controller_github_client_secret:
empire_controller_github_organization:

but i don't think those would be causing these issues.

I think the new AMI should fix the issues that @mhahn had.

Ok - verified that the new AMI fixed things. @mhahn was able to get an environment up and running, and it sounds like it went smoothly once he tried the new AMI. I'm going to go ahead and merge this.