ColdFusion Muse - Coldfusion Troubleshooting

Always Check on the Last Thing You Changed

Mon, 13 May 2013 12:10:00 -0500

If you can sing this with a sort of smarmy accent like Eric Idle it makes it really pop to the tune of "Always Look on the Bright Side of Life".

Your server's feeling bad,
It can really Make you mad,
JRUN maxed can make you swear and Curse,
When your chewing CFGristle,
Don't Grumble, Give a Whistle
And this will help things turn out for the best

Always check on the last thing you changed
(whistle cheerfully here)
Always check on the last thing you changed.

If CF's being Rotten,
There's something you've forgotten
And that's to check the freaking SVN,
For anything that's newish
Roll it back, don't be bluish
Just pucker up and whistle, that's the thing

Always check on the last thing you changed
(whistle cheerfully here)
Always check on the last thing you changed.

...I'm not sure what was in that mimosa...

Debugging and a Return to Dodge City

Thu, 01 Nov 2012 10:49:00 -0500

One of the things the Muse likes best about ColdFusion is the excellent debug information provided during development. Of course you should never ever leave debugging enabled on a production server. Not only are you generating a great deal of additional data with each request (adding overhead), you are potentially exposing a mother lode of technical information that a nefarious hacker would salivate to see. But during development, the debug information is where you ought to live. Indeed, if you are not constantly checking the debug information start doing it now - make a habit of it! You will learn things about performance, iterations, database interactions, cookies, paths, and all sorts of goodies that will make you a better programmer.

I've had my head buried in the debug information since I started with ColdFusion. Back then (in the Wild West days of CF 4.01) we never heard of newfangled ideas like "cfqueryparam". We just stuffed our variables into queries willy nilly and trusted the good Lord to protect us. It feels like I have spent the last 7 or 8 years cleaning up after code written like that. But writing queries in the raw (unprotected I mean... I don't generally code naked, although I did experiment in college) had one main advantage. As you probably know a lot of debugging goes back to the database. The debug output pre-cfqueryparam was "well formed" query code that could be copied and pasted directly into a query tool like MSSQL studio or Navicat. This made debugging pretty easy. You could swipe a problem query out of the debug, run it and tweak it unit it gave you what you needed, then past it back into CF. But that changed when we all started using CFQUERYPARAM.

[More]

IIS 7 Max Worker Processes and ColdFusion Updated

Sat, 05 Nov 2011 23:12:00 -0500

In my previous post on this topic I indicated that IIS 7 seemed to be a constraining factor. That post lead to conversations with a couple of CF gurus (Charlie Arehart and Russ Michaels) who clued me in to a number of additional settings. If you are truly interested take the time to read the previous post and (especially) the comments before you read this post. What bothered me was that the issue I discovered (a cap on requests) can be affected by both IIS settings or JRUN settings (or both).

My conclusion is that the behavior I was trying to affect is actually the bug that Charlie pointed out to me on Adobe's site (found here). Charlie rightly indicates that this issue is under-recognized (I certainly had not run into yet). The behavior of this bug can be affected (fixed or mitigated) by adjusting IIS as described in my previous post as well as by using the Adobe-provided instructions. This lead to a bit of Muse head-scratching. How do these various processes really work together? This post hopes to clear that up (or at least add to our compendium of knowledge).

[More]

IIS 7 Constraining Simultaneous Requests Limit?

Fri, 04 Nov 2011 10:37:00 -0500

I have been doing some performance testing for a company with a large server farm over the last couple of weeks. Although the farm had 20 or more servers, we started with just one sever to try and get some numbers we could use to extrapolate the overall tolerance of the larger system. The servers were all Windows 2008r2 64bit, IIS 7, running CF 9 enterprise with plenty of RAM. We were also running Fusion Reactor to help introspect ColdFusion.

As I slowly poured on more load I noticed something strange that I had never seen before. Although my "simultaneous requests" setting was set to 48. I could not get ColdFusion to handle more than 25 active connections. Under ordinary circumstances I could pound away at a server using my test framework and get enough requests active to overrun that simultaneous request setting and see the queue kick in. I was trying to max out the server but it was not behaving as expected. Active requests would "cap out" at 25 - as if my simultaneous request setting was set to 25 - but there were never any requests in the request queue. It was a head scratcher - but I kinda love those! Here's the skinny....

(NOTE: The comments on this post are important as well. And this Follow up post clears up some of the confusion.)

[More]

ColdFusion Builder 2 Hotfix 1 Fails

Sat, 27 Aug 2011 13:11:00 -0500

If you have an issue where you are trying to install "hotfix 1" for CF Builder 2 on a Windows 64bit platform and it fails after decompressing the file, you might try the following. Look for the uncompressed files (usually in the temp directory), drill down and find the *.lax file. This is the file that install anywhere (a Java installer) uses to launch its own install procedure. In the file is a path to the JVM - look for the line that says something like lax.nl.current.vm=\\... Then set the path to a known good 64bit JRE SDK. You should have one if you are running eclipse. Once you make the change launch the install exe in the decompressed files folder (instead of the original file). This is a version of the fix found in my previous post on CF 8 64bit on Windows 2003 64bit Web edition. The previous post has all the details and then some. I'm told by CF aficionado Christian N. Abad that the fix works for CF Builder Hotfix 1 installs as well. I suspect as a general rule that it would work for any "install anywhere" process with this specific problem.

Jmeter Part 3: Script Recording and Crawling

Thu, 28 Jul 2011 12:39:00 -0500

This is my last post on the topic of Jmeter testing. I wanted to finish up by showing you 2 things. First, CF Guru Larry Lyons pointed me to a different test tool called "bad boy". This test tool has a good deal of capability in it's own right but the main reason I bring it to your attention is that you can record a test and export it to Jmeter in a few simple steps. Second, as promised in my previous posts (Jmeter testing and Jmeter Distributed testing) I'm going to share a Jmeter script that is able to crawl your site using the magic of regular expressions. First, let's explore Bad Boy.

[More]

Jmeter Part 2: Distributed Testing

Tue, 26 Jul 2011 12:58:00 -0500

In my last post I detailed using Jmeter to set up a simple Web Application test. Jmeter can bring a lot of users to the table with just a single workstation and you can affect some real pressure on your system. But there are times when the resources of the server simply exceed the amount of traffic you send its way. For example:

The site is heavily cached or pure HTML and can handle bucket loads of traffic.
The server is a huge horse with tons of RAM, fast disks and teamed NICs.
You are testing a cluster or load balancer - multiple servers working together for high capacity.

The idea behind distributed testing is to use multiple workstations with a single test plan. Jmeter makes this extremely easy (surprisingly easy considering how difficult it is with other testing products). Here are the details.

[More]

Site Testing With Jmeter

Mon, 25 Jul 2011 13:40:00 -0500

Recently I did a user group presentation on testing an application using Jmeter. Notice that I did not say "load" or "stress" or "performance" testing. There are many different types of testing you can do to your application. All applications can benefit from testing for performance or load - although there is a cost factor involved that can prevent some folks from taking the time to test smaller apps or apps with an expected predictable or lite load (like an internal application for example). Still, it is something that every developer should consider. Generally speaking there are 3 increasing levels of testing that rise "up the scale" in complexity and cost.

Performance Testing - To do a performance test you usually you put a reasonable or expected load on an application and examine the internals. By internals I mean memory usage, processor usage, database performance and specific pages that you suspect need tuning. There might be many other things you want to look at here - networking, third party services etc. The goal is to find bottlenecks and improve performance under the expected load. Almost everyone does some form of performance testing - even if it's just loading up a page and looking at the debug info to see what sort of time it takes to run a query or execute a routine. When finished you have a "tuned baseline" - a set of expected benchmarks you can reasonably expect your application to meet (until the database crashes or some data center guy trips over the power cord anyway).
Load Testing - Load testing is sort of generically thrown around to mean both performance and stress testing. In reality a true "load test" (at least by the book) means maximum load for a sustained period. In other words, pushing your application to the top of the curve and leaving it there for anywhere from an hour to a couple days to see if it degrades over time. So really it is more like "marathon" testing or "endurance" testing. Load testing teases out things you might not expect - bugs that occur over time in large data sets, crashes that have a threshold or duration attached to them, memory leaks, and timed problems that occur when events conflict (like backups or scans). Most folks do not do any load testing. Load testing is very expensive and time consuming and simply understanding the data set will require experience and patience. If you have invested in a high level application where perforamance and capacity are important long term factors then you should probably be load testing.
Stress Testing - The goal of a stress test is to see where your application will finally break, and how it recovers after crashing. In other words you add an increasing amount of requests to your app until it stops functioning, then you see if it recovers. Does it require a hard kill? Is any permanent damage done? Are there corrupt files? Orphaned DB records? Finding out where your system crashes gives you a baseline for capacity planning and allows you to fix issues related to potential crashes.

The following post details just the "minimum" or "basics" of setting up a test with a product called Jmeter (part of the awesome treasure trove of products found in the Apache Foundation vault). You can extrapolate from these instructions to do performance testing right away. With a little extra effort you can set up a stress test as well.

[More]

Preparing for When You Need Me Most

Mon, 24 Jan 2011 12:25:00 -0500

In the new order of things practically no one has a server "in-house" anymore. Unless you are a large company with a heavy IT presence, it is simply easier to rent space at a data center. This works very well for the most part, but one of the nuances of this system is remote access. Most data centers have a number of different methods for providing you with remote access. In some cases they use a VPN like Cisco AnyConnect (which I think is excellent). Once you are connected to the VPN you access the server or servers via an "internal" NAT address - an unroutable address usually beginning with 192 or 10. Then you use something like RDP (remote desktop protocol) to log into the server desktop and configure the server, deploy code or whatever. In other cases the NOC might simply provide open ports to your own static IP for the services you need (like RDP, MSSQL, MySQL etc). This approach opens the NOC up to the your security, but it's certainly better than open ports.

In addition, some hosts may allow these ports to be open and trust the next layer of security to keep the bad guys out. In fact, virtually all hosts offering shared hosting or web panels etc have to offer these open ports of necessity. The Muse hates this idea. Allowing open ports for RDP (3389), SQL (1433), and especially FTP is a bit like Goliath strutting up and down daring the Israelites to challenge him. Meanwhile, all rock throwing David could think is, "how could I possibly miss that guy?" But of course there's more to the story...

[More]

Accessing Both 64bit and 32bit Assemblies

Mon, 03 Jan 2011 11:07:00 -0500

Many readers will doubtless recall the war waged by the Muse against .NET on 64bit ColdFusion 8. If not, you can read all about it in this series of posts on Muse Vs. .NET Integration. It was clear from the outset that using 64bit CF against a 32bit assembly was not working for us. Naturally we went down the path of recompiling everything into 64bit and we had multiple obstacles to overcome. But the fact that we could not communicate with 32bit assemblies always puzzled me. The communication (like most things on a web server) happens through a socket to a listener provided by the integration service (just like Verity and Sequelink). I could not reasonably explain to my own satisfaction why it should be that a 32 bit assembly was inaccessible. After all, our tests using the assembly directly worked fine. I assumed in a vague sort of way that differences in how variables were stored and passed back and forth must be to blame.

A few weeks ago I got a tip from the always insightful Rick Root on CF-Talk - who figured this out with the help of the folks at Just CF (SupportObjective). His problem was different than mine. He had a mixed environment with both 32bit assemblies and 64bit assemblies. In his experimentation he discovered something that solves both our problems. When you call and assembly (any assembly) using .NET integration ColdFusion creates some jar files under the hood. You will find them in the /cfclasses/dotnetproxy folder. Here's a sample folder where the server is using 2 .NET assemblies:

Of course, one of the Jar files that ColdFusion creates is the actual assembly interface which "mirrors" the methods and properties of the .NET interface. It's the one with the cryptic name that looks like "-23480983_3023903.jar". The other file however is special. It's the one that is always named dotNetCoreProxy.jar. It is compiled only once (unless you delete it) on the first time you instantiate a .NET assembly. It provides (presumably) the interface to the proxy listener.

The Fix

Now here's the kicker. If the first assembly you call is a 32bit assembly - in other words, if you call the 32bit framework first - then this little jar file is compiled to access the 32 bit framework and cannot access the 64bit framework. In fact, there's some good evidence that it can't reliably access 32bit DLLs either (it seems error prone). However, if you access a 64bit assembly first then the jar file is compiled in such a way as to be able to access both the 64bit and 32bit frameworks.

Conclusion

To avoid having to run this gauntlet I would suggest accessing a .NET Assembly on the 64 bit framework first and backing up the subsequent dotNetCoreProxy.jar file to avoid future confusion. I'm sure you can find a nice "Hello World" assembly to compile to 64bit. If you have legacy .NET integration code and you are moving from 32bit to 64bit then it will likely not work out of the box until you have done this (or it may work once or twice and then give you inscrutable errors). You will need to get the correct dotNetCoreProxy.jar file compiled (the 64bit version) before you can access your 32 bit assembly dlls successfully and reliably. The good news is that you don't necessarily need to recompile all your assemblies to use the 64bit CLR - although there may be a performance or compatibility reason to do so.

Explained: Error Loading jvm.dll

Sat, 04 Dec 2010 10:51:00 -0500

Recently on CF-Talk I participated with super-gurus Dave Watts, Russ Michaels, Wil Genovese, Jochem Van Dieten and others trying to help Brook Davies overcome a ticklish obstacle. After updating here ColdFusion 8 server with the updater he was unable to start Coldfusion. Instead the /runtime/bin/cfusion-out.log presented him with this unhelpful error message.

"Error loading: C:/ColdFusion8/runtime/jre\bin\server\jvm.dll".

Brook tried different paths and different JVMs - all to no avail. Finally he fell to frantically uninstalling and reinstalling, updating and rolling back and (presumably) consuming large quantities alcohol. The list was helpful with a variety of suggestions. Suggestions ranged from a bit depth mismatch (a 32 bit OS trying to run a 64bit JVM for example), to paths, bad downloads, file I/O, wrong updaters - the whole gamut. But nothing seemed to work.

The Fix

Finally, user "Mack" (a newcomer to me) figured it out. The install in question was using a custom or incorrect version of the msvcr71.dll file. This is a core library on a windows machine. The jvm.dll cannot be instantiated without it - and in this case the version of that file was important as well. Mack copied the newest version into the /runtime/bin directory and the JVM was able to load. There are a couple things to note here. First, I don't now what the exact version should be. My information has Mack copying the dll from a running install into the bin directory and that did the trick. Secondly, you don't need to register the dll or overwrite it in the system32 directory. Apparently Jrun looks first in the /bin directory and then elsewhere - so as long as it's in the bin directory it's cool.

Finally, user Leigh (a princess I presume) noticed these 2 bugs in the Sun bug tracker for 1.6 which could be related.

Now perhaps an enterprising Muse reader will publish the exact version in the comments for us.

Muse Vs. .NET Integration - Part 3 a New Frontier

Tue, 09 Nov 2010 23:47:00 -0500

As we continue our saga against the evil empire our intrepid Jedi, Master Muse, has a new chapter of knowledge to share. So with this post I'm adding a few things you may need to know to work with .NET integration on ColdFusion - in this case ColdFusion 8.01 enterprise in "multi-server" mode. Before I continue it might behoove you to read my previous post on Muse vs. .NET integration Part I and Muse Vs. .NET Integration Part 2. These 2 previous posts provide a blow-by-blow description of issues faced and resolved to get this running. My father used to say "behoove" so I pulled it out of the attic to use here. It means "It might be in your interest" but as a child I always thought it had to do with goats or fauns or something - but I digress.

Here is the latest information we have uncovered in our never ending quest to get .NET integration working smoothly on ColdFusion 8 64bit. Many thanks to fellow guru and very large brained friend Guy Rish for his tireless efforts to uncover some of these items. I'd also like to thank Dennis N for his part, a brilliant engineer in his own right who has kept at this with us till we solve it or die trying. I'd mention his last name but I don't have his permission :) Meanwhile......

[More]

Muse Vs. .NET Integration - Part 2

Tue, 19 Oct 2010 11:20:00 -0500

When last we left our story the Muse had put in a 6-7 hour marathon coding session through the night to create a CFC proxy work-around for his .NET problem. This temporary solution allowed the customer to continue working the following day so we could go back to scratching our heads over why .NET integration on our 64bit Multi-server was experiencing a failure to communicate. If you want to catch up on the story read my previous Part 1 post.

To recap, no matter what we tried we could not get CF to communicate with .NET on our production server (CF 8 Enterprise 64bit Multi-instance Install). We had tried installing and de-installing the service a few times to no avail. The first task was to get a successful recompile in 64bit that would run in a test environment. Team member (and all around genius developer) Guy Rish took charge of that effort. In fact, he got the assembly to recompile and he was able to instantiate and access its methods from his local CF8 64bit developer install running on Win7. From this we knew it was possible to get this working. We only needed to tease out the environmental differences to figure out what was blocking us. That task fell to me...

[More]

Muse Vs. .NET Integration - Part 1

Tue, 19 Oct 2010 00:23:00 -0500

When Adobe (then Macromedia) came out with Coldfusion 8 one of the oft touted features was the .NET integration service. The idea was to provide the same easy-to-use accessibility that create object used to give to COM (although COM itself was unstable) and still gives to Java and web services. Just like ColdFusion gives you handy access to the universe of Java, the .NET integration service was designed to give you equally handy access to the world of .NET assemblies and managed code.

In practice however, I found that few developers chose to use it as a solution. Why? I think one reason is likely the emergence of Web Services and SOAP as a practical intermediate middle layer between various technologies - and especially between .NET and Java. When the integration service worked it was an effective painless solution. When it did not work however, it proved difficult to troubleshoot and configure. The fact that it was not a commonly adopted solution meant that fewer developers where asking questions about it or choosing it as solution to .NET-to-Java integration problems.

The Muse and his merry men (and women) were no exception in this regard. As an active ColdFusion shop doing over 1000 hours of ColdFusion consulting each month, we have worked with .NET integration only a handful of times in the years since its release. In each of those cases it was very simple implementation. The .NET service was usually chosen because of the use of client certificates or some other special requirement that included a bit of managed code that was not easy to duplicate in CF. In most cases, however, the web service implementation meant more "pure" CF code and better compatibility. For that reason the Muse never really delved into how this service was set up or how to troubleshoot or configure it. Until recently that is....

[More]

Web Service SSL Issue: Trouble with Client Based Certificates

Fri, 27 Aug 2010 14:28:00 -0500

I was recently brought in at the 11th hour to an issue where a web service worked fine on an older 32 bit install of ColdFusion, but was not working on the new 64bit install. All the keystore certs were up to date and the client was scratching their collective heads. The web service in question used SSL and presented a client certificate. Now in order to do this in CF you have to manually construct your SOAP request and pass it as XML using CFHTTP. This is because only CFHTTP has an attribute for clientcert and clientcertpassword (as of CF 8 I believe).

In any case, the code our customer was using was straightforward. It assembled a soap request and then issued a call to CFHTTP like so: On the "old" Coldfusion 8 "standard" server this worked fine. But on the brand spanking new, freshly patched and upgraded ColdFusion 8 enterprise server it was failing.

The Issue

The problem here is that a vulnerability in TSL (that's "SSL version 3.1" - see my previous post to learn more about the various SSL versions) was discovered in fall of 2009. This vulnerability allowed a "man in the middle" attacker to force a "renegotiation" of the handshake - thus allowing the insertion of arbitrary code into the request. Obviously this is a serious flaw. The fix was for vendors to simply disable the renegotiation feature in their products. So, for example, IIS 7 does not allow renegotiation by default.

So why is this issue not out there in the CF blogging world yet? First, I think that the use of client certs is a fairly small universe of ColdFusion applications. Second, Sun followed suit and fixed this issue with version 1.6.0_19 by disabling renegotiation. I think that many CF Servers are still using 1.6.0_18b or below - so this issue has yet to really rear its ugly head.

In any case, regular SSL requests continue to work as always with renegotiations disabled. A client certificate driven request is different however. It often requires renegotiation because of the complexity of the handshake (with 2 certificate pairs involved. That means client certificate driven CFHTTP SSL requests using the 1.6.0_19+ JVM will often fail to successfully negotiate a secure session. I want to say they will probably or certainly fail, but I'm not positive on that score.

The Fix - Such as it is

You have two ways of fixing this. You can roll back to 1.6.0_18. Seeing as how the current build is _20 you may not want to do this. Or, if you want to stay on 1.6.0_19+ you can add the following argument to your JVM.config file or files. -Dsun.security.ssl.allowUnsafeRenegotiation=true Obviously this allows for the server to renegotiate successfully. Your JVM will be vulnerable to the TLS exploit, but depending on your situation it may be a fairly low risk proposition. Still, you should take it into account. Consider running a separate JVM instance just for your web service. Some edge security devices can sniff out this issue as well.

Many thanks to my good friend and troubleshooting colleague Vlad Friedman from the Edge Web for figuring this out. I would also recommend that your read Chris Mahns blog entry on TLS Remediation.