IIS 7 Constraining Simultaneous Requests Limit?
I have been doing some performance testing for a company with a large server farm over the last couple of weeks. Although the farm had 20 or more servers, we started with just one sever to try and get some numbers we could use to extrapolate the overall tolerance of the larger system. The servers were all Windows 2008r2 64bit, IIS 7, running CF 9 enterprise with plenty of RAM. We were also running Fusion Reactor to help introspect ColdFusion.
As I slowly poured on more load I noticed something strange that I had never seen before. Although my "simultaneous requests" setting was set to 48. I could not get ColdFusion to handle more than 25 active connections. Under ordinary circumstances I could pound away at a server using my test framework and get enough requests active to overrun that simultaneous request setting and see the queue kick in. I was trying to max out the server but it was not behaving as expected. Active requests would "cap out" at 25 - as if my simultaneous request setting was set to 25 - but there were never any requests in the request queue. It was a head scratcher - but I kinda love those! Here's the skinny....
(NOTE: The comments on this post are important as well. And this Follow up post clears up some of the confusion.)
A quick look at Windows performance monitor "web service->current connections" showed it climbing well above 25 - 75, 80 and 90 connections. What I surmised is that instead of letting ColdFusion queue the requests in the JVM, somehow IIS was actually queuing them and passing a max of 25 concurrent threads over to ColdFusion.
The Fix
I'm not sure how or why, but I do know the what. The setting to change is the "Maximum Worker Processes" in your application pools. You find this setting by going into IIS manager, drilling down to the application pool you need to adjust and choosing "advanced settings". You'll see the setting on the properties dialogue. It will look like this:
In IIS 6 this used to be called "web farms" but I never remember it constraining so precisely like this. I set it to a multiple of 25 that got me above my simultaneous request setting. So if I was using say 64 I would set it to 3 (3 times 25 = 75). Sure enough when I put my extra load on the server perfmon "current connections" were reduced somewhat as CF began to handle more requests, and CF active requests now spiked above 25 and were able to queue as I expected. Take note that when you do this you are allowing IIS to spawn additional threads (and allocate memory and cpu cycles etc) so you can have a positive OR a negative impact depending on your resources and the load you are expecting.
Perhaps one of my ingenious readers can clue me in on why I saw this constraint. I certainly did not see anything specific about the number 25 but that was exactly the number I was held to until I changed this setting. Very curious.
15 Comments
-
Hey Mark,
while the setting you have changed does resolve your problem, it is not the correct setting for increasing the number of concurrent connections.
Having multiple worker processes is similar to have multiple load-balanced servers, and is useful for similar reasons: in case one worker fails, the others can pick up the load.
You don't need to have more than one worker process in order for IIS to be able to use all of your available CPUs. Multiple threads within a single worker will do that for you.
Take a look at this for changing your concurrent connections : http://blogs.x2line.com/al/archive/2010/01/04/3718... -
@Russ,
Oooh..... good tip. This setting looked to me like it had to do only with .NET managed code and not with the general application pool settings.
I understand the "Division of labor" with more than one worker process - and I just figured "multiples of 25" (For my particular setup). It is interesting that it's 12 per CPU and not 12 per core.
In any case - sounds like a good followup post :) Thanks as always Russ (I can always rely on a same day comment from you - ha). -
Actually guys, the real solution is in fixing a mistake on Adobe's part, which they've documented:
http://blogs.adobe.com/cfdoc/2009/12/iis_6_for_col...
There's a needed change to the maxworkerthreads value in either jrun_iis6_wildcard.in or jrun_iis7_wildcard.ini if you use IIS 7. The blog entry offers the details.
They say the problem only affects CF 9 or 9.01, not 8 or 8.01, and only IIS (6 or 7), and only on JRun it would seem.
FWIW, I pointed out that entry, with some other explanations for similar surprising limits one may hit in IIS, in an entry of my own:
http://www.carehart.org/blog/client/index.cfm/2011...
Hope that may be helpful to you other readers who find this. But I'll also review your two entries to see if there's new info from them I could add to that entry (or more accurately, that would have me point to your entries to find more info on them.)
Cheers. -
@charlie,
That's a great tip but in this case increasing maxconcurrentconnections (or multiple worker processes in the app pool) in IIS actually "fixed" the issue. I would have expected if the jrun limitation was the bug that this would not be the case - that I would still be "capped" at 25 after my IIS changes. no? -
Mark, remember that in IIS7 .net is tightly integrated, so things that seem like ASP.net only settings are probably still going to affect you.
ISAPI filters for example rely on the asp.net framework and if you don;t have the right version installed then they may not work, Helicon APE is one example.
Charlie, the cf connector settings seems related, but Marks change would have had no effect if it was being over ridden by the maxworkerthreads in the connector. Unless he is not running CF8/9 on JRUN. So it seems more likely that this setting only efefcts the number of concurrent requests that CF/Java will handle rather than IIS. -
Guys, I wouldn't be surprised if perhaps the limit of 25 (per maxworkerthreads in CF) might apply to each worker process/application pool. So, yes, increasing the number of them could lead to the corresponding bump on the CF side. More on that in a moment.
But Mark, you were asking in the entry what was magic about 25. This seems surely the answer (since it defaults to restricting CF 9 to running 25 requests from IIS, regardless of whether the CF request tuning settings are higher).
To prove this is it, have you tried changing it, to perhaps 50, to see another doubling still of how many requests get through? (Of course, you need to be able to pump enough requests from your stress tool to have that many really running "at once" in CF, though you could arrange them them to be slow in CF so that you could have that many running at once. I add this as much for others reading this. I realize you'd already know this, Mark.)
If it does turn out to be per worker process (whether per app pool, or per worker process within a pool) that would certainly be interesting info to know, and to pass on as a comment on the Adobe blog entry.
But there may be still more subtlety in observing all this. Note that the maxworkerthreads setting is per connector in CF (whether created with the CF web server config tool or from the command line) . Since one could define multiple connectors (even one per site), it would seem one could observe more than 25 requests at once (whether in the CF Server Monitor, FusionReactor, or SeeFusion) even without the maxworkerthreads change, since requests could be coming from different connectors.
Then add to this your observation that somehow it may be per worker process within the pool, and it really muddies the waters in observing the impact of these settings. I'll look forward to gaining clarity on this.
Finally, Mark, the feature you referred to previously as being "web farms" was actually "web gardens" (at least in IIS 6). Like this, it was a means in the properties of an app pool to increase the number of worker processes per pool. A good article explaining the distinction is http://www.codeproject.com/Articles/114910/What-is.... (Though note there is a new feature called Web Farm Framework for IIS 7, http://www.iis.net/download/WebFarmFramework .)
Hope that's helpful. -
@charlie,
Thanks for the input. Since I had a 1 site to 1 instance ratio for my test I can't tell for sure if it was IIS or CF (I'm still leaning toward IIS - sorry :). I'll set up a pure test and get a definitive answer for you. and thanks for the clarification on 'web gardens' vs. 'web farms'. -
Well, I meant that you could tell (even in that setup) if upping the maxworkerthreads would increase the number of requests you could run.
I appreciate that you've also "solved" it by adding 25 worker processes, but as you noted at the end of your entry, that comes at a fairly high price.
I get the sense that you've not entirely bought into what I pointed out as being the correct solution. I'm curious about that. It's not a theory. :-) I've seen the change have a dramatic impact on many servers. Did you check out the Adobe blog entry? It's a simple change (requiring a restart of IIS to take effect).
I know in an earlier comment you wondered if you'd 'still be "capped" at 25 after your IIS changes'. Well, like I said, it may be that the limit is per worker process (which would be interesting to confirm).
But upping that maxworkerthreads value (for your one site/one pool/one instance) while staying at one worker process would be far less overhead than running 25 of them, right? :-)
And to be clear, there would be no difference in overhead to CF, since they're just two different ways to get the same result in CF: allowing it to run at once whatever number of requests you choose to allow.
Just trying to make sure we're really understanding each other. This is indeed an important issue, and we three are among a relative few who care to dive deep into such issues. :-)
Going back to your entry, you asked what was magic about the cap of 25. It's precisely this maxworkerthreads value. I just don't sense you're buying that as the answer.I'll look forward to what your testing confirms. :-) -
Charlie,
Yeah... I'm not buying it without testing it. I can't see why the JRUN setting is related to max worker processes. Jrun doesn't "know" about IIS's worker threads in any real way does it? It just receives processing requests via the stack from the connector right? Through a TCP port? There's not 2 connectors - just one. So why would adding additional max worker processes suddenly "fix" (and did I detect a note of condescension :) the problem.
I get that Adobe logged a bug and a fix - but hey... they may not understand all the internals themselves (I certainly have better luck fixing this stuff than Adobe support engineers most of the time). So it wouldn't shock me if they found one problem and I've found another. Moreover the "magic 25" can be accounted for by the default cap of 12 concurrent requests per processor (I know I know that's 24 but it's very close).
Anyway - i'm going to schedule a time to get back on that server and reset it ... then we'll see :) -
Well, all I'm seeking your agreement on is that if changing the maxworkerthreads DID fix the problem, that it would be better than having 25 worker processes? One controls how many threads CF lets in from IIS, the other sets up 25 w3wp.exe's on the server.
My only point in pressing this is that you'd asked what was magic about 25, and this setting is exactly 25. It seemed you didn't know about the fix.
I meant no condescension (but people feel it anyway sometimes). You know I respect your work.
I only put "solved" in quotes because, as I then said, it seemed to come at a high price (so many w3wp.exe's).
As for why the additional worker processes helped things, like I said, that's curious for me, too, so I'll really look forward to hearing what you find.
Most important, I hope you'd simply test 1 worker process and 25 maxworkerthreads, to confirm that it does also fix the problem. And if so, I hope you'd agree it would seem the better solution (or if not, why not.) Again, this is all just about helping everyone else who may be following along, to have their CF servers running well. -
Charlie,
I'm not sure where you are getting your info - but I did not increase max worker processes to 25. I increased it to 3. And in any case doing so does NOT appear to result in additional w3wp.exe processes being spawned. I still see only 1 after boosting my setting to 3. Where do you see such an increase?
-Mark -
Charlie,
one correction... the max processes DO create an additional w3wp.exe process - but only up to the number you set (3 in this case).... -
Ah, my bad. I see now you said to increase it equal to the number of increments of 25 you wanted to see let through (so 3 for 75). Fair enough, and my apologies for missing that. I've been under the weather the past few days and certainly influenced by cold medicines! :-)
But still, the main point is that instead of adding 3 worker processes to "get around" the "constraint", you should be able to just change the constraint--in CF. Again, this is a known bug, with a clear connection to the limit of 25 you saw. That's all I've been trying to make most clear.
Your entry did ask if one of your "readers can clue me in on why I saw this constraint. I certainly did not see anything specific about the number 25 but that was exactly the number I was held to until I changed this setting."
This just seemed so clearly the answer. And I just know that this issue and its solution has gone sorely under-reported. I blogged it months ago, but that only reaches so far. When you brought it up, I just was delighted to have another chance to get the word spread, since people look to you also for troubleshooting solutions.
When you didn't respond with equal delight that it seemed to be "the answer", I'll admit I was a bit bummed. That's why I've wanted to just make sure there wasn't some misunderstanding. :-)
Finally, as for why when you increase the number of worker processes you do NOT see an increase in w3wp.exe's, I think you'll find that when you run your load test, you WILL see them.
IIS worker processes are set to quiesce themselves when not in use, so if you're looking now, without a test running, then you'll just likely see one per app pool.
I'll add that at least, this was things worked in IIS 6. I would be surprised if it was different in IIS 7. Do let us know what you find. And again, no offense intended at all. -
Darn, I was writing my note while replying to your earlier comment, so didn't see the next one until too late. Anyway, like I said, you may well see that sometimes you do only see 1 w3wp. But otherwise, the rest of the comment stands. :-)
-
Folks will want to catch that Mark did an update to this post, with more information and clarifications about some of the above. While it's listed now as a "related entry", in case someone may miss that, it seemed worthwhile to point out. Mark confirms that it was indeed the maxworkerthreads that was the limiter he was hitting, as well as adds some more observations about the pools discussion above.
http://www.coldfusionmuse.com/index.cfm/2011/11/5/...
Sites to Watch
Aggregators
RSS
Related Blog Entries