ColdFusion Muse

Explained: Error Loading jvm.dll

Recently on CF-Talk I participated with super-gurus Dave Watts, Russ Michaels, Wil Genovese, Jochem Van Dieten and others trying to help Brook Davies overcome a ticklish obstacle. After updating here ColdFusion 8 server with the updater he was unable to start Coldfusion. Instead the /runtime/bin/cfusion-out.log presented him with this unhelpful error message.

"Error loading: C:/ColdFusion8/runtime/jre\bin\server\jvm.dll".
Brook tried different paths and different JVMs - all to no avail. Finally he fell to frantically uninstalling and reinstalling, updating and rolling back and (presumably) consuming large quantities alcohol. The list was helpful with a variety of suggestions. Suggestions ranged from a bit depth mismatch (a 32 bit OS trying to run a 64bit JVM for example), to paths, bad downloads, file I/O, wrong updaters - the whole gamut. But nothing seemed to work.

The Fix

Finally, user "Mack" (a newcomer to me) figured it out. The install in question was using a custom or incorrect version of the msvcr71.dll file. This is a core library on a windows machine. The jvm.dll cannot be instantiated without it - and in this case the version of that file was important as well. Mack copied the newest version into the /runtime/bin directory and the JVM was able to load. There are a couple things to note here. First, I don't now what the exact version should be. My information has Mack copying the dll from a running install into the bin directory and that did the trick. Secondly, you don't need to register the dll or overwrite it in the system32 directory. Apparently Jrun looks first in the /bin directory and then elsewhere - so as long as it's in the bin directory it's cool.

Finally, user Leigh (a princess I presume) noticed these 2 bugs in the Sun bug tracker for 1.6 which could be related.

Now perhaps an enterprising Muse reader will publish the exact version in the comments for us.

Muse Vs. .NET Integration - Part 3 a New Frontier

As we continue our saga against the evil empire our intrepid Jedi, Master Muse, has a new chapter of knowledge to share. So with this post I'm adding a few things you may need to know to work with .NET integration on ColdFusion - in this case ColdFusion 8.01 enterprise in "multi-server" mode. Before I continue it might behoove you to read my previous post on Muse vs. .NET integration Part I and Muse Vs. .NET Integration Part 2. These 2 previous posts provide a blow-by-blow description of issues faced and resolved to get this running. My father used to say "behoove" so I pulled it out of the attic to use here. It means "It might be in your interest" but as a child I always thought it had to do with goats or fauns or something - but I digress.

Here is the latest information we have uncovered in our never ending quest to get .NET integration working smoothly on ColdFusion 8 64bit. Many thanks to fellow guru and very large brained friend Guy Rish for his tireless efforts to uncover some of these items. I'd also like to thank Dennis N for his part, a brilliant engineer in his own right who has kept at this with us till we solve it or die trying. I'd mention his last name but I don't have his permission :) Meanwhile......

[More]

Muse Vs. .NET Integration - Part 2

When last we left our story the Muse had put in a 6-7 hour marathon coding session through the night to create a CFC proxy work-around for his .NET problem. This temporary solution allowed the customer to continue working the following day so we could go back to scratching our heads over why .NET integration on our 64bit Multi-server was experiencing a failure to communicate. If you want to catch up on the story read my previous Part 1 post.

To recap, no matter what we tried we could not get CF to communicate with .NET on our production server (CF 8 Enterprise 64bit Multi-instance Install). We had tried installing and de-installing the service a few times to no avail. The first task was to get a successful recompile in 64bit that would run in a test environment. Team member (and all around genius developer) Guy Rish took charge of that effort. In fact, he got the assembly to recompile and he was able to instantiate and access its methods from his local CF8 64bit developer install running on Win7. From this we knew it was possible to get this working. We only needed to tease out the environmental differences to figure out what was blocking us. That task fell to me...

[More]

Muse Vs. .NET Integration - Part 1

When Adobe (then Macromedia) came out with Coldfusion 8 one of the oft touted features was the .NET integration service. The idea was to provide the same easy-to-use accessibility that create object used to give to COM (although COM itself was unstable) and still gives to Java and web services. Just like ColdFusion gives you handy access to the universe of Java, the .NET integration service was designed to give you equally handy access to the world of .NET assemblies and managed code.

In practice however, I found that few developers chose to use it as a solution. Why? I think one reason is likely the emergence of Web Services and SOAP as a practical intermediate middle layer between various technologies - and especially between .NET and Java. When the integration service worked it was an effective painless solution. When it did not work however, it proved difficult to troubleshoot and configure. The fact that it was not a commonly adopted solution meant that fewer developers where asking questions about it or choosing it as solution to .NET-to-Java integration problems.

The Muse and his merry men (and women) were no exception in this regard. As an active ColdFusion shop doing over 1000 hours of ColdFusion consulting each month, we have worked with .NET integration only a handful of times in the years since its release. In each of those cases it was very simple implementation. The .NET service was usually chosen because of the use of client certificates or some other special requirement that included a bit of managed code that was not easy to duplicate in CF. In most cases, however, the web service implementation meant more "pure" CF code and better compatibility. For that reason the Muse never really delved into how this service was set up or how to troubleshoot or configure it. Until recently that is....

[More]

Web Service SSL Issue: Trouble with Client Based Certificates

I was recently brought in at the 11th hour to an issue where a web service worked fine on an older 32 bit install of ColdFusion, but was not working on the new 64bit install. All the keystore certs were up to date and the client was scratching their collective heads. The web service in question used SSL and presented a client certificate. Now in order to do this in CF you have to manually construct your SOAP request and pass it as XML using CFHTTP. This is because only CFHTTP has an attribute for clientcert and clientcertpassword (as of CF 8 I believe).

In any case, the code our customer was using was straightforward. It assembled a soap request and then issued a call to CFHTTP like so:

<cfhttp url="https://blahblah.com/?wsdl"
            charset="utf-8"
            port="443"
            method="post"
            result="response"
            clientcert="c:\somepath\somecert.p12"
            clientcertpassword="*some password*">

            
    <cfhttpparam type="xml" value="#soapPacketStuff#" />

</cfhttp>
On the "old" Coldfusion 8 "standard" server this worked fine. But on the brand spanking new, freshly patched and upgraded ColdFusion 8 enterprise server it was failing.

The Issue

The problem here is that a vulnerability in TSL (that's "SSL version 3.1" - see my previous post to learn more about the various SSL versions) was discovered in fall of 2009. This vulnerability allowed a "man in the middle" attacker to force a "renegotiation" of the handshake - thus allowing the insertion of arbitrary code into the request. Obviously this is a serious flaw. The fix was for vendors to simply disable the renegotiation feature in their products. So, for example, IIS 7 does not allow renegotiation by default.

So why is this issue not out there in the CF blogging world yet? First, I think that the use of client certs is a fairly small universe of ColdFusion applications. Second, Sun followed suit and fixed this issue with version 1.6.0_19 by disabling renegotiation. I think that many CF Servers are still using 1.6.0_18b or below - so this issue has yet to really rear its ugly head.

In any case, regular SSL requests continue to work as always with renegotiations disabled. A client certificate driven request is different however. It often requires renegotiation because of the complexity of the handshake (with 2 certificate pairs involved. That means client certificate driven CFHTTP SSL requests using the 1.6.0_19+ JVM will often fail to successfully negotiate a secure session. I want to say they will probably or certainly fail, but I'm not positive on that score.

The Fix - Such as it is

You have two ways of fixing this. You can roll back to 1.6.0_18. Seeing as how the current build is _20 you may not want to do this. Or, if you want to stay on 1.6.0_19+ you can add the following argument to your JVM.config file or files.

-Dsun.security.ssl.allowUnsafeRenegotiation=true
Obviously this allows for the server to renegotiate successfully. Your JVM will be vulnerable to the TLS exploit, but depending on your situation it may be a fairly low risk proposition. Still, you should take it into account. Consider running a separate JVM instance just for your web service. Some edge security devices can sniff out this issue as well.

Many thanks to my good friend and troubleshooting colleague Vlad Friedman from the Edge Web for figuring this out. I would also recommend that your read Chris Mahns blog entry on TLS Remediation.

Win2003 64 Bit Install Problems

This is a follow up to my post titled 64bit on Windows 2k3 Web Edition. In that post I did a play by play of an issue trying to install 64 bit ColdFusion 8 Enterprise on a Windows 2003 "Web Edition" platform. I have just finished a troubleshooting session with a nearly similar issue on Windows 2003 "Data Center" edition. If you have this problem you will know it because the install will silently fail after the files are extracted. Check in your temp directory and you will see a cryptic folder beginning with "I" containing two folders "windows" and "installdata". These 2 folders contain the extracted install files. Inside the "windows" folder you will also see an hotspot error log. It looks like "hs_err_pidXXX.log". You may have seen such files in your ColdFusion8/runtime/bin directory. They occur when the hot spot compiler has an unhandled exception.

Under the hood the ColdFusion install is firing off the "adobe_cf.exe" from that same windows folder. This exe file uses the adobe_cf.lax file as a launch settings file. Taking a closer look at the lax file and you will see some Java settings including two important settings. One is lax.nl.current.vm and the is lax.installer.win32.internal.property.0. Both of these settings point to a Java.exe file. By default the Java.exe file it points to is contained in that other temp folder called 'installerdata'. My guess is that the reason that the ColdFusion fails because the 64 bit JVM in the /installerdata folder is not compatible with something in the windows installation. In particular I think that there is a missing 64bit class or object.

The Fix

The fix is as follows.

  • Download and install the 32 bit 1.6 JDK.
  • Edit the adobe_cf.lax file to point to the java.exe file inside the jre folder of the newly installed 32 bit JDK.
  • Run the Adobe_cf.exe file.
The install should fire up at the point of the first splash screen and follow through to completion. To be clear - you are still installing the 64 bit version of ColdFusion. Only the "launch anywhere" process will be running 32 bit.

Troubleshooting and Optimizing Solr on ColdFusion 9

I had an interesting troubleshooting session recently with a customer. This customer had a very high powered server - SAS drives, 16 gigs of RAM, 64 bit throughout, Coldfusion 9 and an 8 gig Java Heap. The site had 70 or 80 search collections and they had switched to CF9 specifically to get beyond the limitations of Verity. Everything was performing well when suddenly the search service stopped responding to requests and simply started throwing "collection not found" errors. Coldfusion seemed fine and dandy. It continued to be responsive and it had no hanging threads. It was as if the search service had lost it's handles to the various collections. Restarting Solr solved the issue, but why was Solr locking up?

[More]

ColdFusion 8.01 64 bit on Windows 2k3 Web Edition - The Silent Killer

This post is a rundown of a recent troubleshooting adventure (If you don't know what a rundown is, ask Charles). I write these little post-mortems for a couple reasons. For one thing, I will need this solution (or some part of it) again at some point and my little pea brain is not going to retain all of this information. But what I usually do retain is the knowledge that somewhere on my blog I have a post about an issue with a Win2k3 64 bit CF install - or as Tevye would say, "Somewhere in the good book it must say something about a chicken!" The other reason is to benefit those of you in the ColdFusion universe who may be struggling with this issue. The Muse is all about community. If you are battling an install where the installer is failing silently after the splash screen, then this post will likely be a valuable read for you. If it doesn't solve your problem send me a note via Ask-A-Muse and I'll see if I can think of anything else you might try.

I was tasked with installing and optimizing ColdFusion 8 Enterprise 64 bit on a new server for a customer. I do this kind of task pretty often for companies who wish to have a clean and fully optimized install for migration. The new server was running Windows 2003 64 bit "Web Edition". Previously (in Windows 2000 I believe) the Web Edition would only support 2 Gigs of memory. But 2003 server supports up to 8 Gigs. I wanted to use 64 bit because the server was going to manage a large pool of modestly trafficked sites. It really didn't need the full fledged "multi-server" install. It just needed a nice big heap size. For that purpose the standalone Jrun install is perfect. In short order however, I found myself pitted against Win2k3 in a no holds barred install death match. Here's my story.

[More]

Address Resolution, Networking, and Cfdocument

Among the things that can befuddle even experienced developers, domain resolution ranks up at the top. Usually this is because we don't spend a lot of time worrying about resolution on our desktop or laptop or Iphone. DNS is an extremely mature technology and for the most part it just works with few issues. When it comes to a server however, there are several things that can trip up resolution. Without an understanding of exactly what is going on under the hood, you will find yourself destroying yet another keyboard with the ball of your fist as you shout "why won't you just work!"

Domain resolution comes into play on most ColdFusion applications, even if you don't think so. Among other things, resolution is important for:

  • Data Sources - how do you connect to an external server?
  • Ecommerce - how do you connect to a Gateway?
  • Web Services - how do you create your stub class?
So let's take a short journey down this path and see if we can uncover some of the general principles that will help us troubleshoot domain resolution issues.

[More]

Ask-a-Muse: How Do I Upgrade the JVM?

Muse Reader Rob Asks:
I have a silly question. How exactly do you upgrade the JVM on your ColdFusion server? My server is on Win2k3 x64 and the JVM version is 1.6.0_04. Do you specify it manually in the jvm.config file?

I'm glad you asked this question because it reminds me that I sometimes give advice without any follow through - which is the same problem I have with my 8 iron. Upgrading the JVM on a windows installation is pretty easy. Just remember that you will need the correct Java Runtime for your platform and ColdFusion version. Rob specified Win2k3 x64 so I assume he means he is running ColdFusion 8 enterprise 64 bit - in which case the target version is 1.6 update 14 (or 1.6.0_14). I usually start at the Sun Java download page. Once you have the right version in hand the rest is easy.

[More]

Certain JPGs Can Crash Your ColdFusion 8 Server

This issue was brought to my attention by Adrian Lynch on CF-Talk. It seems that if you use the new image functions in ColdFusion 8 against certain kinds of JPG images you can actually cause your JVM to crash. If you have code that uses the latest image functions to handle uploaded images you should definitely take note of this post. I cannot yet see how a user might take advantage of this bug to penetrate your server, but a malicious (or even non-malicious) user could easily perform a denial of service attack and cause your CF server to go up and down like Jack LaLanne doing jumping jacks. So if you fit into that category (handling uploaded images using CF 8 image functionality) here's the scoop.

[More]

Connector Problems - Win2k8, IIS7 and Multi-Server ColdFusion 8 x 64 Bit

I have not yet had this problem specifically, but it was pointed out by CFG Tom Forrest who spent some time wrangling with it. He was trying to use the connector widget to connect IIS 7 sites to ColdFusion instances (running in Multi-Server Mode). He reports as follows:

The connector refused to install anything into IIS. When I started it, the first window would appear. When I clicked "add" I would see something to the effect of, Installing required IIS7 components. It may take 2 to 5 minutes to complete. The window that allows you to set all the parameters would open, and you could select any of your running CF servers. However, you couldn't select any of the IIS sites that were created. Assuming you give up and click ok, allowing it to "install to all" you would get an error window stating error creating IIS application extensions ColdFusion.

According to Tom the fix is to install the IIS 6 Management Compatibility role service. This service allows an IIS 7 server to "act like" an IIS 6 server. Once installed the configuration tool began to work.

While I haven't had this specific problem, I have noticed that a number of other things are easier and more familiar with the IIS Management Compatibility installed. Thanks Tom.

Ask-a-Muse: Killing the Immortal Thread

Muse Reader Joe Asks:
How do I kill a request? Every other day or so there will be a runaway process that cannot be killed. Clicking on the red exclamation in the monitoring tool does not give an error but it does not kill the request either. My question is how to kill this process?

Ah the immortal thread - like a god coming down from Mt. Olympus and laughing with his (or her) hands on his mighty hips (see why I chose "his"? ... "her mighty hips" ... well, I just didn't want to go there). Such threads are mind bogglingly frustrating. In actual fact, there are some requests spawned by ColdFusion that may not be able to be terminated by ColdFusion. For the long version read on McDuff.

[More]

More Connection Magic

I outlined some issues in my last post on using CFHTTP and paying attention to connection issues. Muse reader Scott Pinkston posted a nice snippet of code that allows you to see what IP address is used to resolve a given domain name. Here it is complete with the function call (getHostAddress()) that gives you the IP address:

[More]

Checking the Size of the Spool Directory on Windows

If you ever send out a few tens of thousands of messages using CF you know the spool directory can get pretty crowded. If you are like me you sometimes want to keep an eye on it as those messages clear out to make sure there is nothing funky going on. If you use Windows Explorer this can be a maddening experience. Windows doesn't just retrieve a count of files. It retrieves the entire file list and meta data and it redraws the explorer window. When you have 50k messages in the spool folder it can take 10 to 30 seconds just for Windows to refresh the count so you can know how many were added or deleted to the folder.

Instead, I use a little tool called "t4edirsize" from tools 4 Ever. I have a "show spool" batch file on my servers that looks like this:

@echo

t4edirsize.exe d:\cfusion8\mail\spool
The output gives me all sorts of information including the number of files in the directory - but it usually only takes 50-200 milliseconds to run. Tools 4 ever command line tools along with Sys-Internals tools (now owned by Microsoft) like "pslist" and "pskill" are essential to your arsenal as a troubleshooter.

More Entries




Blog provided and hosted by CF Webtools. Blog Sofware by Ray Camden.