Mesosphere's stack is in full production at major companies, including one of the largest financial services companies and one of the largest consumer electronics companies. General availability is next year, but paying customers are using it in production today--at very large scale.
Nice tools, but (and it feels funny to say this) they seem rather "sticks and stones" in comparison to the integration achieved with open source PaaS layers like deis atop CoreOS; Mesosphere's DCOS even more so.
They're general purpose devops automation frameworks... you can choose to take any of them and make this 'deep integrated' layer you speak of.
For example, Openstack deployments are commonly automated using Puppet scripts. (Really amazing stuff if you haven't seen it before...)
DCOS may choose to reinvent the wheel, but unless there's a core innovation in the way they're handling orchestration or automation, I think they're better off if they leverage an existing battle-tested toolchain. Adding another tool will just mean more work for the (underworked and lazy?) DevOps teams who will be responsible for managing DCOS.
Puppet and the like are state enforcement tools, not task schedulers.
As someone who used to run a 6000 core farm in 2007 (its not 25,000) I can tell you that puppet isn't going to help task placement. It can create machine that will run a certain app, but without some heavy programming it'll never balance or detect need and respond sensibly.
DevOps tools are rapidly evolving. Now that they've basically finished with machine provisioning, dependency management and orchestration, all major DevOps automation frameworks are going into managing reactive infrastructure. Enforcing a task schedule is just another form of state enforcement.
These are all pretty similar:
- configuration management:
ensure package oracle-java-8 is installed on machines A,B,C with this specific configuration.
- orchestration:
ensure my-awesome-java-app on machines A,B,C is running to databases on machine D,E,F
- deployment with constraints:
ensure that four instances of my-awesome-java-app are running on at least 2 physical machines with over 4TB free disk space.
- job runner:
ensure that script X runs on a cluster every __ minutes. when script X runs, send the output to script Y
I think that you'll see task placement and job scheduling primitives being integrated into DevOps tooling in the next 6-12 months.
Salt is terribly immature at the moment (I know because I use it professionally) I really wouldn't trust it for running tasks as well. Out of the box it starts to get horribly slow after around 500 nodes. (you need to spool up 600 tasks on 600 machines? that'll take 10 minutes guys.)
For distributed cron, we use jenkins. Which has the advantage of keeping "build history"
You do hit a limit when you go over 6000 "slots" (each slot accepts one task, and the main dispatcher is single threaded). Dispatching is simple and task building has simple syntax that easily grows to thousands of tasks in one job. monitoring is also simple, as each task ships logs and exit status back to the dispatcher. It also has mechanisms to cope with bad/slow/unhappy machines.
