The idea and approach we would like to use with NOMAD development follow a bit what has been shown to be successful for other Big Data/cloud enterprises. We do not want to copy exactly the technologies, but some of them and the basic ideas behind them.
For this the following talk is instructive: Fast Delivery talk by Adrian Cockcroft (38 min)
Some basic ideas are also in this older article
Also Adrian Cockcroft Blog is quite interesting.
The idea is to let the developer free to experiment (freedom and responsibility), by making it simple to change and redeploy part or even the whole infrastructure and make it possible to have multiple version coexisting and talking to each other.
Trying to have micro-services can help having well defined and small units of development.
All that is automatizable should be automatized, in particular testing and integration testing. Create artificial failures to see how the system can cope with failures (Simian Army)
From our side we plan to use Docker
Ansible scripts written at CSC for our environment. They can provision typical computing and data processing cluster environments, including Hadoop.