ColdFusion Muse

ColdFusion Debugging on Production

Today's short note is brought to you by "Don't Do That On Production!" At CF Webtools often times we get called in to help troubleshoot servers that are failing to perform well. We often hear the same sort of symptoms that goes like this. The server has been running fine for months then suddenly for no reason it's slow, CPU usage is high, and it hangs or crashes multiple times per day. This always prompts us to ask the same question. "What was changed just before these symptoms started?" And the answer is usually "Nothing was changed (as far as they knew)". In all reality the person we're talking to may not the be only person with access to make changes to the server. Or they may not in fact have access at all and they are relying on information provided to them by an IT team member. We take notes, assume nothing, and question everything (on the server).

We had this scenario play out a few times in the past few weeks with three servers from three different companies. The reason I'm writing this note is the same problem occurred on each server. The short answer is someone enabled ColdFusion Debugging on the production server. ColdFusion is a very powerful rapid development platform, but it has a few gotchas if you are not careful. Such as enabling debugging on a production server. Debugging output provides a massive amount of information and for obvious security reasons we never want this enabled on a production server. Yes, I know you can restrict debugging output to a certain IP address, but that does not prevent the debugging output from being generated. It's just not displayed. The generation of debugging output takes more CPU power and at times more JVM memory. On a low load web application you may not notice a difference. However, on a high load, high traffic production web application the extra resources needed to generate the debugging output may in fact cause all those symptoms described above.

In each of the cases we saw these past few weeks, we were reviewing the servers settings, looking at the results of Fusion Reactor, and reviewing ColdFusion settings. On the first server we almost missed the fact that debugging was enabled. By the time we were troubleshooting the third server with similar symptoms we were checking to see if debugging was enabled before we did anything else. Disabling debugging resolved the bulk of the performance issues. We then used this time to review each server and offered additional performance tuning recommendations based on each servers resources and application needs.

This falls into the category of "Don't Do That On Production!" Please leave debugging to your development and staging servers.

CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @

ColdFusion, SSL, SNI, SAN and Wildcards - Stuff You Need to Know

The Muse welcomes back his friend and colleague (and super genius guru) Wil Genovese with an timely post on SSL and Certificate types. If you have had your head in the ground (or perhaps you have been guest staring on "Naked and Afraid" or "Survivor") you may have missed the hubbub surrounding TLS, SSL and changes and support. There is a lot going on and it is more important than ever that you get your hands around the issue to keep your users safe. Wil has done Yeomen's work identifying the types of certs, the versions of ColdFusion and Java that support them, and work arounds and caveats for those of you who need them. You will likely want to bookmark this one. Take it away Wil.


Surviving Poodle - ColdFusion and SSL 3

There's been a great deal of buzz about poodle. Poodle is an SSL exploit capable of highjacking a session using a browser's ability to "negotiate downward" the level of SSL it uses. It's recent prolifieration has put some urgency into the efforts to force existing applications and platforms to deny the use of any standard of SSL less than version 3.0. Super guru Wil Genovese ( recently did some troubleshooting on a ColdFusion server with an issue related to this necessary configuration step. Wil writes:

We ran into an issue when a company contacted us at CF Webtools because ColdFusion was suddenly no longer able to connect to their email providers mail servers. One day ColdFusion was sending emails to their clients just fine and the next day it was failing. As you know these issues are usually best resolved by asking "What changed?" As far as the client knew, nothing had changed - but we knew enough not to stop digging.


Side-by-side Configuration Error installing CF 11 on Win2008r2

One of my colleagues, Chris Tierney, was installing a pristine copy of ColdFusion 11 on a Windows 2008 server. He followed our standard protocol which is to install the server using the "built in" Web server, then create instances (we typically use multi-server mode) then use wsconfig.exe to connect the instances to IIS. It all went as planned until he tried to run wsconfig.exe (FYI - you must always run this as administrator). He got an error as follows: Cannot run program "C:\Users\ADMINI~1.CFW\AppData\Local\Tem
p\2\\ExecuteAppCmd\ExecuteAppCmd.exe": CreateProcess error=14001, The applicatio
n has failed to start because its side-by-side configuration is incorrect.

Followed by an odd stack trace. After experimenting with permissions and googling he stumbled on Bug 3761543 in the ColdFusion bugbase. The issue is not very well documented. Apparently the MS C++ package installed on 08 is 32 bit. I'm not clear if we installed it or it shipped with 08, but remember, you need the MS C++ 64bit SP1 Redistributable. Here's the download link from Microsoft so you don't have to hunt it down.

The Fix

Microsoft Visual C++ 2008 SP1 Redistributable Package (x64)

One more time: This download fixes the "side-by-side configuration" when installing ColdFusion 11 on Windows 2008r2.

email connection crossover workarounds

As a follow up to yesterday's post (regarding sending mail and having it end up in someone else's "sent" folder) I thought I might put some flesh on the workaround suggested both in the bug report and on CF-Talk. The suggestion is to:

Create a CNAME to point to the SMTP server address so that both websites were looking at different domain names.
This idea is workable up to a point so I thought I would explore it for my readers.


CFHTTP, IIS 8 and Server Name Indication (SNI)

Guest Post by Wil Genovese

(Muse Introduction)
Most readers know that the Muse is deeply indebted to a large and talented group of developers working here at CF Webtools. These folks solve problems and undertake Herculean programming tasks on a daily basis. They are constantly making me look good and I would not be able to play golf or spend the day wise-cracking in IM and tormenting my assistant Melissa without them on my side. Among these folks is one of my favorite characters, CF guru Wil Genovese. Wil has worked with us for a few years now and he writes an excellent blog at If you have not already done so, you should add it to your list of must read blogs.

Meanwhile, a few days ago Wil was trying to troubleshoot a head scratching issue with CFHTTTP and SSL. Now such issues almost always come down to getting the certificates properly installed in the keystore, using the correct URL (correct in all respects for the certificate), name resolution and SSL protocol levels (as in "do you need to lower Java's draconian SSL defaults to allow for less secure protocol"). After beating his head against the wall repeatedly Wil finally decided the issue was on the other end - the certificate on the server was somehow wrong, misconfigured or behaving unexpectedly. I thought this was dubious at best, but as is so often is the case the Muse was wrong and Wil found out (with apologies to Monty Python) something completely different. It turns out a new feature in IIS 8 (Windows Server 2012) was the culprit. Since this setting affects all Java versions prior to 1.7 and even affects CF 10 on Java 1.7, you should probably pay attention. My guess is that you will run into this issue eventually - given the ubiquity of IIS and the coming upgrades to Windows server 2012.

Anyway, I invited Wil to write the following entry detailing his findings. If you want to know more read on:


Always Check on the Last Thing You Changed

If you can sing this with a sort of smarmy accent like Eric Idle it makes it really pop to the tune of "Always Look on the Bright Side of Life".

Your server's feeling bad,
It can really Make you mad,
JRUN maxed can make you swear and Curse,
When your chewing CFGristle,
Don't Grumble, Give a Whistle
And this will help things turn out for the best

Always check on the last thing you changed
(whistle cheerfully here)
Always check on the last thing you changed.

If CF's being Rotten,
There's something you've forgotten
And that's to check the freaking SVN,
For anything that's newish
Roll it back, don't be bluish
Just pucker up and whistle, that's the thing

Always check on the last thing you changed
(whistle cheerfully here)
Always check on the last thing you changed.

...I'm not sure what was in that mimosa...

Debugging and a Return to Dodge City

One of the things the Muse likes best about ColdFusion is the excellent debug information provided during development. Of course you should never ever leave debugging enabled on a production server. Not only are you generating a great deal of additional data with each request (adding overhead), you are potentially exposing a mother lode of technical information that a nefarious hacker would salivate to see. But during development, the debug information is where you ought to live. Indeed, if you are not constantly checking the debug information start doing it now - make a habit of it! You will learn things about performance, iterations, database interactions, cookies, paths, and all sorts of goodies that will make you a better programmer.

I've had my head buried in the debug information since I started with ColdFusion. Back then (in the Wild West days of CF 4.01) we never heard of newfangled ideas like "cfqueryparam". We just stuffed our variables into queries willy nilly and trusted the good Lord to protect us. It feels like I have spent the last 7 or 8 years cleaning up after code written like that. But writing queries in the raw (unprotected I mean... I don't generally code naked, although I did experiment in college) had one main advantage. As you probably know a lot of debugging goes back to the database. The debug output pre-cfqueryparam was "well formed" query code that could be copied and pasted directly into a query tool like MSSQL studio or Navicat. This made debugging pretty easy. You could swipe a problem query out of the debug, run it and tweak it unit it gave you what you needed, then past it back into CF. But that changed when we all started using CFQUERYPARAM.


IIS 7 Max Worker Processes and ColdFusion Updated

In my previous post on this topic I indicated that IIS 7 seemed to be a constraining factor. That post lead to conversations with a couple of CF gurus (Charlie Arehart and Russ Michaels) who clued me in to a number of additional settings. If you are truly interested take the time to read the previous post and (especially) the comments before you read this post. What bothered me was that the issue I discovered (a cap on requests) can be affected by both IIS settings or JRUN settings (or both).

My conclusion is that the behavior I was trying to affect is actually the bug that Charlie pointed out to me on Adobe's site (found here). Charlie rightly indicates that this issue is under-recognized (I certainly had not run into yet). The behavior of this bug can be affected (fixed or mitigated) by adjusting IIS as described in my previous post as well as by using the Adobe-provided instructions. This lead to a bit of Muse head-scratching. How do these various processes really work together? This post hopes to clear that up (or at least add to our compendium of knowledge).


IIS 7 Constraining Simultaneous Requests Limit?

I have been doing some performance testing for a company with a large server farm over the last couple of weeks. Although the farm had 20 or more servers, we started with just one sever to try and get some numbers we could use to extrapolate the overall tolerance of the larger system. The servers were all Windows 2008r2 64bit, IIS 7, running CF 9 enterprise with plenty of RAM. We were also running Fusion Reactor to help introspect ColdFusion.

As I slowly poured on more load I noticed something strange that I had never seen before. Although my "simultaneous requests" setting was set to 48. I could not get ColdFusion to handle more than 25 active connections. Under ordinary circumstances I could pound away at a server using my test framework and get enough requests active to overrun that simultaneous request setting and see the queue kick in. I was trying to max out the server but it was not behaving as expected. Active requests would "cap out" at 25 - as if my simultaneous request setting was set to 25 - but there were never any requests in the request queue. It was a head scratcher - but I kinda love those! Here's the skinny....

(NOTE: The comments on this post are important as well. And this Follow up post clears up some of the confusion.)


ColdFusion Builder 2 Hotfix 1 Fails

If you have an issue where you are trying to install "hotfix 1" for CF Builder 2 on a Windows 64bit platform and it fails after decompressing the file, you might try the following. Look for the uncompressed files (usually in the temp directory), drill down and find the *.lax file. This is the file that install anywhere (a Java installer) uses to launch its own install procedure. In the file is a path to the JVM - look for the line that says something like\\... Then set the path to a known good 64bit JRE SDK. You should have one if you are running eclipse. Once you make the change launch the install exe in the decompressed files folder (instead of the original file). This is a version of the fix found in my previous post on CF 8 64bit on Windows 2003 64bit Web edition. The previous post has all the details and then some. I'm told by CF aficionado Christian N. Abad that the fix works for CF Builder Hotfix 1 installs as well. I suspect as a general rule that it would work for any "install anywhere" process with this specific problem.

Jmeter Part 3: Script Recording and Crawling

This is my last post on the topic of Jmeter testing. I wanted to finish up by showing you 2 things. First, CF Guru Larry Lyons pointed me to a different test tool called "bad boy". This test tool has a good deal of capability in it's own right but the main reason I bring it to your attention is that you can record a test and export it to Jmeter in a few simple steps. Second, as promised in my previous posts (Jmeter testing and Jmeter Distributed testing) I'm going to share a Jmeter script that is able to crawl your site using the magic of regular expressions. First, let's explore Bad Boy.


Jmeter Part 2: Distributed Testing

In my last post I detailed using Jmeter to set up a simple Web Application test. Jmeter can bring a lot of users to the table with just a single workstation and you can affect some real pressure on your system. But there are times when the resources of the server simply exceed the amount of traffic you send its way. For example:

  • The site is heavily cached or pure HTML and can handle bucket loads of traffic.
  • The server is a huge horse with tons of RAM, fast disks and teamed NICs.
  • You are testing a cluster or load balancer - multiple servers working together for high capacity.
The idea behind distributed testing is to use multiple workstations with a single test plan. Jmeter makes this extremely easy (surprisingly easy considering how difficult it is with other testing products). Here are the details.


Site Testing With Jmeter

Recently I did a user group presentation on testing an application using Jmeter. Notice that I did not say "load" or "stress" or "performance" testing. There are many different types of testing you can do to your application. All applications can benefit from testing for performance or load - although there is a cost factor involved that can prevent some folks from taking the time to test smaller apps or apps with an expected predictable or lite load (like an internal application for example). Still, it is something that every developer should consider. Generally speaking there are 3 increasing levels of testing that rise "up the scale" in complexity and cost.

  • Performance Testing - To do a performance test you usually you put a reasonable or expected load on an application and examine the internals. By internals I mean memory usage, processor usage, database performance and specific pages that you suspect need tuning. There might be many other things you want to look at here - networking, third party services etc. The goal is to find bottlenecks and improve performance under the expected load. Almost everyone does some form of performance testing - even if it's just loading up a page and looking at the debug info to see what sort of time it takes to run a query or execute a routine. When finished you have a "tuned baseline" - a set of expected benchmarks you can reasonably expect your application to meet (until the database crashes or some data center guy trips over the power cord anyway).
  • Load Testing - Load testing is sort of generically thrown around to mean both performance and stress testing. In reality a true "load test" (at least by the book) means maximum load for a sustained period. In other words, pushing your application to the top of the curve and leaving it there for anywhere from an hour to a couple days to see if it degrades over time. So really it is more like "marathon" testing or "endurance" testing. Load testing teases out things you might not expect - bugs that occur over time in large data sets, crashes that have a threshold or duration attached to them, memory leaks, and timed problems that occur when events conflict (like backups or scans). Most folks do not do any load testing. Load testing is very expensive and time consuming and simply understanding the data set will require experience and patience. If you have invested in a high level application where perforamance and capacity are important long term factors then you should probably be load testing.
  • Stress Testing - The goal of a stress test is to see where your application will finally break, and how it recovers after crashing. In other words you add an increasing amount of requests to your app until it stops functioning, then you see if it recovers. Does it require a hard kill? Is any permanent damage done? Are there corrupt files? Orphaned DB records? Finding out where your system crashes gives you a baseline for capacity planning and allows you to fix issues related to potential crashes.
The following post details just the "minimum" or "basics" of setting up a test with a product called Jmeter (part of the awesome treasure trove of products found in the Apache Foundation vault). You can extrapolate from these instructions to do performance testing right away. With a little extra effort you can set up a stress test as well.


Preparing for When You Need Me Most

In the new order of things practically no one has a server "in-house" anymore. Unless you are a large company with a heavy IT presence, it is simply easier to rent space at a data center. This works very well for the most part, but one of the nuances of this system is remote access. Most data centers have a number of different methods for providing you with remote access. In some cases they use a VPN like Cisco AnyConnect (which I think is excellent). Once you are connected to the VPN you access the server or servers via an "internal" NAT address - an unroutable address usually beginning with 192 or 10. Then you use something like RDP (remote desktop protocol) to log into the server desktop and configure the server, deploy code or whatever. In other cases the NOC might simply provide open ports to your own static IP for the services you need (like RDP, MSSQL, MySQL etc). This approach opens the NOC up to the your security, but it's certainly better than open ports.

In addition, some hosts may allow these ports to be open and trust the next layer of security to keep the bad guys out. In fact, virtually all hosts offering shared hosting or web panels etc have to offer these open ports of necessity. The Muse hates this idea. Allowing open ports for RDP (3389), SQL (1433), and especially FTP is a bit like Goliath strutting up and down daring the Israelites to challenge him. Meanwhile, all rock throwing David could think is, "how could I possibly miss that guy?" But of course there's more to the story...


More Entries

Blog provided and hosted by CF Webtools. Blog Sofware by Ray Camden.