Configuring CF 7 Multiserver Instances - Including Custom JVM Settings
Multi-server Installation have some distinct advantages. Process isolation is the one that makes the most sense to me. Many, maybe most servers house more than 1 web site. Even Intranet servers seem to have more than 1 site configured. A "single site" server is pretty rare - at least outside of a cluster. Having a way to isolate 1 site from another on the server itself is an ideal way of keeping bad code or database connections on one site from dragging down another. It's not for everyone. It certainly adds another layer of complexity to your server. Still, it's an excellent solution in some cases. Consider this scenario.
You have a server with ten Coldfusion sites. Nine of the sites are lightly used and seem to experience no difficulty. The tenth site, however, is a pain in the Tukas. The traffic generates copious database calls and often drags all ten sites down. The only solution seems to be to restart Coldfusion as the threads start queuing up. What can you do about it? Create 2 instances of JRUN/Coldfusion. That in itself is actually pretty easy. Make sure that when you installed the Coldfusion server you chose "multiserver" as the type of install. If you did then you are ready for the following.
In the Coldfusion Administrator go to the "Enterprise Manager" and click on the "Instance Manager". Click "add new instance". Type in "badsite" and Coldfusion will copy files and configure stuff to create your new instance. Make sure and check the "Create Windows Service" option. Once Coldfusion is done you will see a new instance, "badsite" in the list. You can stop or start the service. It even has it's own admin where you can configure data sources and other settings -very cool. Use the "web server configuration tool" (in the macromedia program group) to map an instance to a web site. This process (of mapping site to instance) does little more than add a "wildcard mapping" to the site that points to JRun4\lib\wsconfig\1\jrun_iis6_wildcard.dll - where the "1" is the number of the instance. Your site should be ready to serve CFM files using your new instance.
If you take a look at your process viewer you might see something alarming depending on what your JVM.config file looks like. If, when you were serving all nine sites from the same instance, you configured your JVM to use 1.3 gigs of memory you will see your "jrun" instances are gloaming onto to 1.3 gigs virtual memory each (check the VM column). Unless you have 8 gigs on your server that could be a problem. In fact, when I was experimenting I added several instances and I began to see an error that said:
The JVM needed to allocate appropriate space using both physical and virtual memory. I increased the size of the page file and the error went away, but that didn't solve my problem. Obviously I want to keep Jrun running within physical memory as much as possible.Error occurred during initialization of VM
Could not reserve enough space for object heap
The Problem
It seems that when creating a new instance Jrun was using the settings from my JVM.config file for each instance. The total memory usage would equal the allocation in the JVM times the number of instances. If I allocated a 512 Megs heap and 128 Megs of MaxPermSize, and I had 3 instances I would end up with 640 Megs times 3 or 1.9 gigs. On a 2 gig server that's too much. To fix this easily I just scale back my settings in my JVM. In the scenario above I could scale back to 768 megs times 2 instances for 1.8 gigs. That is marginal, but probably doable. But there's a catch. There is an "admin" server that runs the Coldfusion administrator in the multi-server install. That's why you always see 1 more Jrun than you anticipated. I have to add that in. So I might end up with 512 megs X 3 instances for that 1.9 gig number.
But that is not optimal. Why should the little used "admin" server get a third of the resources? In my above scenario I might want to devote a full half or two thirds of memory to the heavily trafficked site. I will rarely want an even distribution. What I really need is to be able to install each instance with it's own settings - right?
The solution
The solution is involved, but it's doable.
- Create a JVM.config file for each of your instances. I used "jvm.config_small", "jvm.config_medium" and "jvm.config_large"; where small, medium and large represent how I was allocating the heap site.
- Open your services control panel and bring up the list of "Macromedia ...." services - these are your instances.
- From your list of instances in the services control panel, choose a service to configure for custom JVM. Let's say the name of your "badsite" instance is "Macromedia JRUN AS badsite". Open a command line and "CD" to the /Jrun4/bin directory. Type the following to remove the service. This does not remove any files - just "unregisters" the service.
c:\Jrun4\bin>jrunsvc -remove "Macromedia JRUN AS badsite"
-
Now "reregister" the same instance with the following command:
C:\JRun4\bin>jrunsvc -install badsite "Macromedia JRUN AS badsite" "Macromedia JRUN AS badsite" -config jvm.config_large
- Hit F5 on the services control panel and you should see your service. Try to start it. If you get an error go look in the jrun4/logs directory for the badsite-out.log and the badsite-err.log.
This is how I discovered to do it. As always I welcome input from folks who have a better mousetrap. Fire away and I'll give credit where credit is due. In fact, as I've been working through this (and I continue to learn all the ins and outs of configuring multiserver installs) Robi Sen has been a great help.
41 Comments
-
you are on the money MK. In fact, you can't use the JRun JMC to manage multiple JVM config files. You have to do what you outlined above.
I'll also thorow in that it is not so difficult to create a cluster of JRun instances running CF, in case that bad site is really bad.
The only drawback to all those instances is hooking them to IIS. If using IIS, you will need a seperate IIS site per instance, thus a seperate IP/URL for each. Yeah, I suppose host headers can be used, but not in conjuction with SSL, IIRC, IIS can't do SSL and host header stuff simultaneously on a site. I'll add it is possible to map a CF server to a virtual site/dir in IIS, but its a royal pain and not very maintainable.
DK -
Doug - thanks for the feedback. As for IIS and mapping a site/dir - do you mean that you could map the CF server isntance to a particular direcory instead of the virtual site itself? That would solve that SSL problem. Incidentally I've never even thought of using a "cluster of instances" for the same site - ingenous. -
yeah, you can map a CF server instance to a virtual site. Its tricky, ugly, and makes you feel all h4cky though :) Basicaly you add a dummy IIS site, use wsconfig to map your CF instance to it, note the jrunscripts mapping in IIS for the dummy site, add that info to your virtual site, then remove the dummy IIS site. I just don't feel comfortable maintaining a beast like that though.
As for clustering CF/JRun instances...I manage a couple prod servers that each host 10+ apps on a intranet. Each server is setup with a JRun/CF cluster that handles all the apps fine and dandy. The only thing I have had issues with are things like cached data in the app scope. If you need to update it, you have to ensure you are updating it on both instances, a royal PITA. -
Hi Mike,
I set up my project's CF server into about 7 different instances( like interactive, reporting, monitoring, support, etc.). What I wasn't prepared for was memory usage, which nicely offset the entire benefit from having different instances. So thanks for the tip - now I can skinny some of those instances up. Whats the best (ie quickest) way the gauge the requirements of each instance?
Saw you at Webdu (I think)... Is your book on CF optimisation out yet? -
Ben - hey, it's Mark not Mike, and do you have the name of that publisher... 'cause they haven't contacted me yet (ha). I don't believe we've met, but I'm quite sure it would be my pleasure. I'm glad the Post was helpful.
As for gauging memory, the quick and dirty way is probably just to set the minimum heap to something small like 256 and watch the process viewer to see where it maintains. Of course this might cause a lot of GC so you have to be careful.
Another way that is only slightly more difficult is to turn on Jrun metrics. The heap size is recorded there along with the thread counts. It should give a fairly detailed glimpse of each instance. I've been meaning to write a blog on enabling metrics (or maybe I already did... it's all running together :).
Mark -
Thanks Mark. Maybe you should write a book; There isn't much out there on this stuff from a CF perspective. If you like I'll write the chapter on things not to do :) -
Ben,
While there is not a specific book on tuning cf, frankly I have no idea why you would write a book just on cf since all the basic ideas of tuning j2ee app servers apply, a decent amount of this stuff is covered in the forta cf advanced book. actually there is alot of neat content in that book that gets ingnored including information on tuning the jvm, monitoring the jvm, figuring out your heap size.
as for marks comment on memory utilization there are a bunch of ways to see how much mem your app is using even on multi server instances. i like to use jrun metrics and have my log files outputed as csv's so i can do analytics in excel or mathmatica or what have you but you could on use perfmon or snmtp to monitor the jrun process memory utilization and the working set and you will get a nice view over time of how much of the heap is being used. -
This is good stuff Mark. Somehow I missed it when you first posted. Although it is a minor point in the grand scheme of your post, I never realized that you could define a custom jvm.config when installing an instance as a service. Because of that we have been running one of our instances in console mode and the rest as services... that is until today! -
Update: After a quick read of the EULA from Adobe, it appears that I was being a little naughty by bundling them together. Hence the quick removal/update of the original post (anyone who got here first was lucky). That doesn't mean that you still can't have a cracking CF dev environment. The zip I produced was files taken from 2 sources: Eclipse 3.3 and the ColdFusion Extensions for Eclipse and the CFEclipse download section of the site.
To download and install CFE follow the instructions here. To install the CF Extensions just:
1. Download http://www.cfreport.org/eclipse/ColdFusion_Extensi...... and unzip on your desktop
2. In Eclipse to o Help -> Software Updates -> Find and Install -> Search for new features to install
3. Click on New Local Site...
4. Choose the extracted folder on your desktop
5. Click Select
6. Click OK and then click Finish
7. You'll then be asked to select the features you want to install. Select ALL checkboxes and click next
8. "Obviously" read the Feature Licenses and the accept the terms
9. The installation page is an overview of the new features you've selected, all you need to do is click 'Finish' to run the update manager
10. The Adobe features will require you to install a signed feature, just click 'Install all'
11. Finally, restart
Thats it, you're now up and running. Your next two stops are the CFEclipse and Adobe's ColdFusion Extensions site. Also don't forget the reference section at the bottom of this post, espically Charlie Areharts list of over 70+ bloggers (part 1 and part 2)
References:
* CFEclipse.org
* trac.cfeclipse.org for documentation, bug tracking and logging, FAQ's etc
* CFEclipse Mailing lists
* ColdFusion Extensions for Eclipse
* Using the ColdFusion Extensions for Eclipse
* CFEclipse bloggers list, part 1 and part 2!
-
I'm getting ready to do this, as our "badsite" has been running out of threads/memory again, but having run across this phrase, I'm hesitant:
"You will want to do the steps above after adding instances in the CF Admin but before you use the web server configuration tool. "
Why is that? Is it going to break something? This isn't changing the CF instance, just the service that launches it, right?
Thanks again for pointing me here :) -
On a related note -- is there any way to tell which jrun.exe goes to which service? As it stands, I can only identify them by approximate guesses based on memory size, except for one instance that's running as its own user instead of service.
-
@JC:
To be honest I'm not sure why he said that (regarding re-creating the service before configuring the website). As far as I know, wsconfig is completely oblivious to the services configuration. Don't let this stop you, since even if your web server configuration was somehow affected wsconfig makes it easy to drop and re-create the configuration. In my experience I have not had a problem changing windows service details after web server configuration.
With respect to understanding which JRUN.exe is related to which service, a the process viewing tool can let you know, since the service (jrunsvc.exe) actually launches the JRUN instance and is registered against the service. I use Process Explorer from SysInternals (which was bought out by MS I believe). Process explorer lets you see all of the running processes in a tree view, so each jrun.exe is linked with its jrunsvc. By hovering over the jrunsvc for a particular jrunsvc.exe you can see the name of the service.
Hope that helps.
This is one blog post I keep coming back to refer to (mostly to eyeball the jrunsvc syntax). -
"Yeah, I suppose host headers can be used, but not in conjuction with SSL, IIRC, IIS can't do SSL and host header stuff simultaneously on a site."
Actually IIS can handle host-headers for SSL connections you just can't configure them through the IIS Manager (GUI). You have to use adsutil.vbs to configure the "SecureBindings" settings.
http://www.microsoft.com/technet/prodtechnol/Windo... -
Leah is definitely the one who should win. Science Fiction Conventions? What?! She deserves a win, indeed! -
I'm guessing the only way to do this in Javascript would be to use the onresize event, and then using the resizeTo method to attempt to keep the window at the size you want?
-
This is good stuff Mark. Somehow I missed it when you first posted. Although it is a minor point in the grand scheme of your post, I never realized that you could define a custom jvm. -
I'm guessing the only way to do this in Javascript would be to use the onresize event, and then using the resizeTo method to attempt to keep the window at the size you want? -
"You will want to do the steps above after adding instances in the CF Admin but before you use the web server configuration tool. " -
You missed a nice one! The appearance of local directory indexes has been improved greatly, with an attractive interface rather than the monospace font of yore.thanks -
To be honest I'm not sure why he said that (regarding re-creating the service before configuring the website). As far as I know, wsconfig is completely oblivious to the services configuration. Don't let this stop you, since even if your web server configuration was somehow affected wsconfig makes it easy to drop and re-create the configuration. In my experience I have not had a problem changing windows service details after web server configuration.
-
Update: After a quick read of the EULA from Adobe, it appears that I was being a little naughty by bundling them together. Hence the quick removal/update of the original post (anyone who got here first was lucky). That doesn't mean that you still can't have a cracking CF dev environment. The zip I produced was files taken from 2 sources: Eclipse 3.3 and the ColdFusion Extensions for Eclipse and the CFEclipse download section of the site. -
Actually IIS can handle host-headers for SSL connections you just can't configure them through the IIS Manager (GUI). You have to use adsutil.vbs to configure the "SecureBindings" settings.
-
You have to use adsutil.vbs to configure the "SecureBindings" settings.
-
To be honest I'm not sure why he said that (regarding re-creating the service before configuring the website). As far as I know, wsconfig is completely oblivious to the services configuration. Don't let this stop you, since even if your web server configuration was somehow affected wsconfig makes it easy to drop and re-create the configuration. In my experience I have not had a problem changing windows service details after web server configuration.
-
I use Process Explorer from SysInternals (which was bought out by MS I believe). Process explorer lets you see all of the running processes in a tree view, so each jrun.exe is linked with its jrunsvc. By hovering over the jrunsvc for a particular jrunsvc.exe you can see the name of the service. thanks
-
To be honest I'm not sure why he said that (regarding re-creating the service before configuring the website). As far as I know, wsconfig is completely oblivious to the services configuration.
-
With respect to understanding which JRUN.exe is related to which service, a the process viewing tool can let you know, since the service (jrunsvc.exe) actually launches -
Actually launches the JRUN instance and is registered against the service. I use Process Explorer from Sys Internals (which was bought out by MS I believe). Process explorer lets you see all of the running processes in a tree view, so each jrun.exe is linked with its jrunsvc. By hovering over the jrunsvc for a particular jrunsvc.exe you can see the name of the service....
http://www.roulette-software.net -
I'm getting ready to do this, as our "badsite" has been running out of threads/memory again, but having run across this phrase, I'm hesitant:
"You will want to do the steps above after adding instances in the CF Admin but before you use the web server configuration tool. "
Why is that? Is it going to break something? This isn't changing the CF instance, just the service that launches it, right? -
With respect to understanding which JRUN.exe is related to which service, a the process viewing tool can let you know, since the service (jrunsvc.exe) actually launches -
To be honest I'm not sure why he said that (regarding re-creating the service before configuring the website). As far as I know, wsconfig is completely oblivious to the services configuration. Don't let this stop you, since even if your web server configuration was somehow affected wsconfig makes it easy to drop and re-create the configuration. -
I use Process Explorer from SysInternals (which was bought out by MS I believe). Process explorer lets you see all of the running processes in a tree view, so each jrun.exe is linked with its jrunsvc. By hovering over the jrunsvc for a particular jrunsvc.exe you can see the name of the service. thanks
Sites to Watch
Aggregators
RSS
