ColdFusion Muse

ColdFusion Debugging on Production

Today's short note is brought to you by "Don't Do That On Production!" At CF Webtools often times we get called in to help troubleshoot servers that are failing to perform well. We often hear the same sort of symptoms that goes like this. The server has been running fine for months then suddenly for no reason it's slow, CPU usage is high, and it hangs or crashes multiple times per day. This always prompts us to ask the same question. "What was changed just before these symptoms started?" And the answer is usually "Nothing was changed (as far as they knew)". In all reality the person we're talking to may not the be only person with access to make changes to the server. Or they may not in fact have access at all and they are relying on information provided to them by an IT team member. We take notes, assume nothing, and question everything (on the server).

We had this scenario play out a few times in the past few weeks with three servers from three different companies. The reason I'm writing this note is the same problem occurred on each server. The short answer is someone enabled ColdFusion Debugging on the production server. ColdFusion is a very powerful rapid development platform, but it has a few gotchas if you are not careful. Such as enabling debugging on a production server. Debugging output provides a massive amount of information and for obvious security reasons we never want this enabled on a production server. Yes, I know you can restrict debugging output to a certain IP address, but that does not prevent the debugging output from being generated. It's just not displayed. The generation of debugging output takes more CPU power and at times more JVM memory. On a low load web application you may not notice a difference. However, on a high load, high traffic production web application the extra resources needed to generate the debugging output may in fact cause all those symptoms described above.

In each of the cases we saw these past few weeks, we were reviewing the servers settings, looking at the results of Fusion Reactor, and reviewing ColdFusion settings. On the first server we almost missed the fact that debugging was enabled. By the time we were troubleshooting the third server with similar symptoms we were checking to see if debugging was enabled before we did anything else. Disabling debugging resolved the bulk of the performance issues. We then used this time to review each server and offered additional performance tuning recommendations based on each servers resources and application needs.

This falls into the category of "Don't Do That On Production!" Please leave debugging to your development and staging servers.

CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @

A Whole Heaping Helping of Normal

When I'm called into a data center or large application to look at a ColdFusion performance problem I often find myself in a "war room" with highly skilled technicians and admins who may or may not know much about the JVM runtime. One of the things that require experience is knowing what normal memory patterns look like for a typical production heap. Windows admins will often go directly to the task manager or resource monitor to examine memory usage. If they are uninitiated the first thing they say is "Wow... Jrun is using a lot of memory." This often doesn't sit right to them, particularly if the site is idle. So this post is to help those windows folks (and Linux as well) to an understanding of what normal heap activity looks like.


Dynamic Compression on ColdFusion 9 and IIS7

Maybe your already know that web servers can compress outgoing content. Compressed content arrives at the browser which decompresses it and is able to render it. This is all generally seamless to the user and results in a more effective use of bandwidth. Now, compressing static files (like .html files) is a no brainer for web servers. They simply pre-compress the files and store them in a file cache somewhere. When the original file is called for the web server serves up the compressed file instead.

Dynamic files are more problematic. There's no correlation between the file name and the buffered output of a ColdFusion page for example. Consider search results. One user might receive 10 results and another user might receive 10 completely different results. Still another user might receive 100 results. How is the web server supposed to compress that data? Like your app server it does it "on the fly". It waits for ColdFusion to return the response buffer, compresses the file in memory (as I understand it) and then outputs the buffer to the browser. At least that's the way it works in theory. In practice you might find that ColdFusion 9 and IIS 7 don't quite have this figured out yet.

Before I give you the blow-by-blow (and thankfully a solution) I want to make it clear that this problem and solution come to me by way of my good friend and colleague Vlad Friedman of Edgeweb Hosting. EdgeWeb consistently receives the highest possible reviews from its customers and Vlad is one of the brightest folks I know in our corner of the IT world. Now let's talk about our little problem shall we?


Query Caching Run Amuck: Know Your FIFO Buffer

Query caching is one of those underutilized features of ColdFusion that can exponentially speed up your application. It is also one of those misunderstood features that, when used incorrectly can be very disappointing. Let me say at the start that the Muse believes you should use query caching. If you don't believe I'm a fan then check out my post titled, Good Developers Practice Safe Query Caching. It's not a panacea, but it definitely has it's uses. Almost every application has some queries that can be cached - and saving round trips to the database is the holy grail of application tuning. But in this post we want to talk about naughty developers who cache irresponsibly... developers who do not understand the nature of the FIFO buffer.


Webmaniacs Live - Mike Brunt and CF Server Tuning

CF Optimization Guru Mike Brunt gave an awesome presentation on tuning ColdFusion. In a short 50 minutes he covered such topics as JVM configuration, multi-server install and load testing. It was an excellent overview. One recommendation (that he made in is lyric British access) was to install See Fusion or Fusion Reactor instead relying on the built in monitor. His take was that, because the monitor is running under ColdFusion, it is too easily become affect itself by performance - or perhaps could become part of the problem instead of being part of the solution.

I had thought, based on a presentation by Adam Lehman on CF 8 a year ago, that the monitor ran in a different JVM space than CF. One of the things I have noticed with the monitor always shows a Flex gateway thread in the request monitoring. That thread is the monitor itself. So I'm thinking Mike might be right. Perhaps the server monitor does exist "inside" of ColdFusion. I like the new server monitor and I've found it useful. But I really like SeeFusion's default view which shows requests and visuals of the heap on one page. I think it is a better aggregate of the things I want to know immediately when I'm looking at a performance problem.

Verbose Garbage Collection

Another excellent tip is a "how to" on enabling verbose garbage collection. This is easy to do. You simply add the following arguments to the JVM.config file and restart ColdFusion.

    -XX:+PrintHeapAtGC -verbose:gc

The information is going to be stored in a log file in the default logs directory (like /runtime logs). How would you use this? One of the best ways to use it is to figure out when new generation collection is going on, when tenure (old "stop the world") collection is going on, and how much data is going from new to old. You can get a good picture of what is going on by examining this log information and using your imagination. Check out Mike's blog on GC tuning for a good outline of the process.

Scheduled Task Performance

So you have a template that runs fine when you are loading it into a browser but runs like a water buffalo on Percodan when it's run as a Coldfusion scheduled task? Here's a tip I saw on CF-Talk that serves as a good reminder. Take a look at the debug settings. By default, debugging includes the address (often called the "loopback" or "local" address). Coincidentally this is the very IP address that the Coldfusion Scheduler will report as the calling address (cgi.remote_addr) - meaning the template in question will generate all of that lovely debug information. This can result in an otherwise acceptable level of performance turning painfully slow. Muse rule of thumb - disable debugging in production environments. Ok... so sometimes I enable it briefly to get some information when I'm troubleshooting a specific process, but it should always be off during normal operations.

Let me add as an anti-thesis, turn on debugging in a development environment. It's surprising how many developers work without it. Personally, I spend more time looking at the debug info than the actual output. One of the first things you should do when writing a new query for example, is check it out in the debug. How many records returned? How long did it take? Can I make it better? One of Coldfusion's big advantages (in my view) is the copious amount of debug information provided for you with each request. Use it!

Coldfusion Optimization and the Windows Legacy

Coldfusion has been navigating the Internet waters in the good ship Java for some time now (6 or 7 years I think). If you are old enough in Internet years to remember Coldfusion 5 you probably know that Coldfusion was originally written for windows in C++ and ported over to Linux, Solaris and UNIX. These ports were not reputed to be particularly good and Coldfusion was largely considered to be a windows server application (and probably justifiably so). In 2002 with "CFMX" Coldfusion offered the Web world an application that was not just a windows application with a port to other platforms. Instead it was a truly cross-platform effort. Since that time Coldfusion has found it's way onto Linux in increasing numbers. In fact we are seeing more Coldfusion on Linux than ever before - particularly users who need Coldfusion Enterprise.

And why not? Coldfusion runs splendidly on Linux. We use CentOS at CF Webtools and it sings along happily with very few issues. In fact, it is possible to allocate about 50 percent more memory to the heap on a Linux machine (using the 32 bit JVM). That is a significant advantage that CF on Linux has over CF on windows. Here's a blog post by CF Webtools own Linux guru Ryan Stille on that topic. Meanwhile, here's a take on the server optimization from the good old days to today:


Making String Comparisons Faster Using Javacast

In my Previous Post on the subject of variable comparison I tried to use JavaCast() to influece the comparison operator being used. I was attempting to keep Coldfusion from having to ask "Is it a float" or "Is it a string" or "why doesn't Ray Camden ever win People's 'sexiest man alive'". I thought that if I could pre-cast variables into Java objects, Coldfusion could use that information to cut to the chase and know exactly which operator to use without testing. Alas, that proved to be false hope. If you read the post you will already know that the difference between comparing typless and typed data in Coldfusion is not worth the effort.

Well as usual, smarter people than I (than me?) read this blog. The amazing Mark Mazelin (who's payment gateway API looks very interesting) suggested that I try to use the comparison operator directly, as in "object.compareTo()". I sometimes forget about all the nifty methods and properties attached to these objects when they are created. I set about testing his idea.


Multiple Garbage Collectors: Can Two Sanitation Engineers be Better Than One?

I ran across this JVM configuration on a server recently. I should note that the server in question was having some issues, so this is not an endorsement of this approach. I simply had not seen this sort of configuration on a Coldfusion server before. Here are the JVM arguments:

java.args=-server -Xms512m -Xmx768m -XX:MaxPermSize=192m -XX:PermSize=64m -XX:NewSize=48m -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC ...
What seemed unusual to me is that this particular set of arguments allows for 2 garbage collectors to be specified. Both the "UseParNewGC" switch and the "UseConcMarkSweepGC" switch are set. I did not know that multiple GCs could be specified. I set about finding out how this was possible.


Iterations, Comparisons and JavaCast (Oh My)

Testing Gurus like Dave Watts and Robi Sen will tell you that iterative tests are only interesting in a passing academic sort of way. Indeed if you are testing a real world application don't bother setting up a fancy-pants loop with 50,000 iterations to see if you should use "IS" or "EQ". If you are going to test then take the time to test real world operations in your application. But while we are on the subject of iterative tests I got to pondering what goes on under the hood. For example, what happens when you do "var1 IS var2" in a CFIF.


Ask-A-Muse: How Do I Use Client Variables?

Muse Reader Asks:
We have a client variables database that is currently storing over 1.5 million records. In the Coldfusion Administrator, client variables are set to purge every 90 days. I was thinking of purging every 7 to 30 days to reduce the number of database entries and hopefully improve performance. What are my options?

It's a good question. Performance depends on many thing including hardware, traffic, tuning, and capacity. Still, 1.5 million rows is a large table and as you might imagine it will perform less ably than a table with just a few hundred thousand rows. Let’s talk about these variables and how they are used.


Looking for Bottlenecks? Watch Out for Cfdirectory

I have a client with a file intensive application. It allows users to upload images and manage galleries. It's very slick and uses the flash uploader to accomplish multiple file uploads. He was having performance problems with the uploader. The flash uploader is a nifty way to upload a wheel-barrow full of files in a single operation. You can even check for things like file size and type in advance instead of waiting for the whole file to arrive on the server.

What we began to notice is that some requests took longer than others, a lot longer. I, being the expert troubleshooter that I am, naturally thought it was file sizes. I assumed that requests for a 2 meg file upload naturally took longer than requests that handled files of 200k. When we looked closer, however, it turns out that was not the case. A much more sinister culprit was lurking.


Variables Scope Memory Leak (say it ain't so)

I just read this fabulous post from Mike Schierberl's blog on Variables Scope Leaking and I thought I would pass it on as a good read. To summarize, it is a common practice to create a component into the application scope and include an "init()" type function in it that returns an instance of itself. For example you might have an "employee" component who's "init()" function takes an employee id as an argument and returns an instance of "employee" populated with user data. That's pretty standard. If you return this new populated instance into the variables scope you would expect the variable to be discarded at the end of the request. Mike's post shows that this does not happen as expected. Because the variables returned are referenced (as apposed to "by value") they persist beyond the request termination.

As a fix you can add structClear(variables) to the end of your request - in the onRequestEnd() function for example. I can't explain it better than Mike. He includes a sample and a flash movie of how he came to his conclusions. It's very thorough.

New Information From Robi Sen

Robi Sen ran his own tests against the SUN JVM. He contests the assertion that this is a Coldfusion issue and he's focused on the JVM. His results may be found Comments (2) | Print | | Digg It!

Superfluous Data Binding Can be Good for You

If you have read my previous posts on "Execution Plans" and Data Binding you know I am a big believer in using Cfqueryparam for performance as well as security. Today I picked up a tip on this topic from (a blog I had never read before). I had never considered it before, but a query with no "WHERE" clause does not using binding because it lacks the opportunity. Therefore, a statement like:

      SELECT * FROM Users   
Would not include a prepare statement and not hit the execution plan. Hmmm.... The solution (originally detailed by Barny) is to add a superfluous binding merely for the purpose of kicking off the "prepare statement" method. Like so:
      SELECT * FROM Users   
      WHERE 1 = <cfqueryparam cfsqltype="CF_SQL_INTEGER" value="1">
That get's the job done nicely.

Innovative Multi-Server Configuration Uses

Ben Forta posted this item on CF-Talk while discussing the nuances of moving to a multi-server instance. I thought it was a great illustration of some of the uses of CF Multi-server.

Ben Forta wrote:
It sounds like you are considering dedicated specialized instances, and I am a big fan of this. There is one customer I work with who has several sites in a single instance, like you do, but they have also created specific instances for specific needs. They have one that just does mailing, it receives requests from other instances, and process them. They have another which executes report requests on scheduled intervals. And another that builds Verity indexes when needed. These specialized instances require fewer threads, fewer data sources, no RDS and debugging (which you'd never want enabled on production servers anyway), and so on. That works very well. Of course, you may also want separate instances for specific apps, but that is a separate discussion.

I especially love the idea of a specialized instance for verity indexes. Re-indexing Verity collections is a huge resource hog, and larger collections can cause problems. I don't know why I never considered this approach before - very ingenious.

More Entries

Blog provided and hosted by CF Webtools. Blog Sofware by Ray Camden.