Your application is not fair at all. It looks like your sleep mimics I/O boundedness, but in reality it does nothing but set an arbitrary request length. In a real world application I/O requests contend for resources with eachother. Sleep calls don't contend with each other at all.
This means that application servers can just keep stacking concurrent requests with perfect performance. If there was actually I/O being done that took 250ms, then you would find that it was the real bottleneck, and it would be the dominant factor in your benchmark, with the differences between app servers all but disappearing.
That benchmark is fair if you consider the context: I/O bound workoads. On those kinds of workoads, your app spends a large amount of time waiting for I/O. While waiting, it does not use any CPU. This is very well simulated using sleep calls.
It is not clear to me that my application fails at realistically simulating I/O (I agree with FooBarWidget below) (and I'm not sure where the word 'fair' comes from or what it means here, that's your word not mine), but I tried to provide as much code and information about exactly what I tested so you could decide for yourself, so fair enough.
(I agree that with real I/O, there wouldn't be _exactly_ (eg) 50ms of waiting in every request; it would differ from request to request, and might slow down under heavier load. That's true, but I don't think it matters for what I was trying to test -- I was not trying to test how well a given rdbms can handle load, for instance; I was trying to standardize that, to test the app's performance. My simulation assumes an (eg) 50ms _average_ iowait time, by just assigning every request 50ms of wait. Or instead of 50ms, whatever average iowait you wanted to posit and test, and I tested a few. It is indeed a simulation, but I think it captures the significant things for what we wanted to measure).
However! I would very much welcome seeing the results from a test done by you (or anyone else), with an application that simulates (or actually does) I/O in a way that you believe is more realistic and leads to more instructive results. It certainly would be interesting to see if it led to different results than my tests, or not. I suspect it would not (because I think my app is realistically simulating the parts that matter, naturally! but I've been wrong before!)
Feel free to fork my code to do so, if my code is a useful starting point for you. Please do share your results!
Of course, you could also run tests with some actual real world app as well, instead of an app simulating a real world load in a standardized way like I used, and see what those results look like. I'd certainly welcome seeing that too! There's certainly plenty of different sorts of tests that could be done. I'd love it if people started doing some of them and sharing their results. I tried to contribute my best effort to that.
Hmm, it seems though you're right, it seemed odd to me that the difference between puma would be so large at high concurrency, but when I look at it now the I see how the test does correctly mimic I/O for the purpose of the benchmark.
This means that application servers can just keep stacking concurrent requests with perfect performance. If there was actually I/O being done that took 250ms, then you would find that it was the real bottleneck, and it would be the dominant factor in your benchmark, with the differences between app servers all but disappearing.