The Muse is a slow moving tools user. I just switched to CF Builder about a year ago in fact. Before that I was using CF Eclipse. I still use the venerable Homesite for quick troubleshooting on production servers or to review code. I'm using it to write this blog in fact. I have written my blogs in hand coded HTML for many years - which I'm sure, explains the copious amounts of in-line CSS throughout. But I'm a full convert to CF Builder now and have been for some time. All my "Code-for-production" work is done using CFB with integrated SVN through a local environment. Recently on one of the email lists I follow I heard a tip from mad-genius Kevin Miller (Websolete.com).

Like the Muse, Kevin runs CF Builder 2 on Windows 7 64bit as a plugin against a heavily customized version of eclipse 64bit. Also like the Muse he finds it to be an underachiever. CFB (or perhaps it's just 64bit Eclipse and we are throwing CFB in there unnecessarily) tends to use way more processor than seems necessary and at times seem to lag and catch up like a fat man at a marathon. On rare occasions when I'm editing large files or (in particular) browsing files on the network it simply peters out and needs to be defibrillated - also like the fat man I suppose. Of course I have edited the eclipse.ini file and I run a large heap. I've experimented with different GC's and other params. In the end I just kind of assumed that.... well that it was a dog like many Java desktop apps (don't judge me). And at age 46 I've learned to settle for a lot of things, so I was reasonably content to simply live with it. I know - it sounds horrible when I write it down like that (doh!).

But Kevin suggested switching from javaw.exe to the jvm.dll by adding the "-vm" switch. I had assumed that "javaw.exe" was necessary because it builds a Windows GUI. But that "GUI" is not the Eclipse gui (at least I don't think so). More likely it is that annoying system tray thingy that keeps begging me to let it spout all the wonderful things Oracle is doing while it upgrades my SDK. Anyway, with a little trial and error I found that Eclipse runs splendidly using the jvm.dll and I'm having fewer lags and problems with. Kevin's post is here if you want the full story. Let me add to his comments that I had a bit of trouble finding the right "jvm.dll". The first one I tried was apparently 32 bit. The one I needed was in the jre6 folder - which eventually I found installed elswhere on my machine. Thinking back I think the SDK install does ask for a specific location for those files - so that makes sense.

Once I had right path I opened my eclipse short cut and added..

eclipse.exe -vm "C:\Program Files\Java\jre6\bin\server\jvm.dll"

...to the target area so that I'm using it each time I open eclipse. Note the necessary quotes around the path. They are needed because of the space in "program files". As always I count on my readers to add to the conversation - just be nice. Remember I grew up on Andy Griffith, not South Park. So let's leave Kenny alone on this post. On the other hand Aunt Bee is fair game.

The Muse is on the lookout for any and all ColdFusion resources in the Boston/Providence area. One of our Tier 1 clients is looking to build a new team in that area and is asking for our assistance. So if you are a senior developer who thinks you can get through my grueling test and interview process and you want to work for a terrific consulting company that has doubled in size in the last year - send your resume to jobs@cfwebtools.com and let's have a chat.

On a related note I am looking for some local Omaha talent as well. Specifically I need a design resource that specializes in User Interface Design - creating user lexicons, visibility analysis, consistency, error and feedback standards, recognition vs. recall, minimalism etc. - and you live in or around the Omaha area (and by around I mean Sioux City, Lincoln, Council Bluffs and the Omaha Metro area), then feel free to contact me. We might have a great opportunity for you to make a significant contribution to an amazing suite of applications for a thriving business. Again - you would have to live in and around Omaha, or at least be willing to brook the subject of moving to Omaha within a reasonable time frame.

For those of you waiting for me to call you about the resume you already sent me in the last few weeks (there are about 15 or 20 of you) hang in there. We are in a 4 week cycle right now so you can bet I'll be contacting you very soon.

When planning for scalability one of the things that is sometimes left out is the impact of indexing bots on your site. If you have a news or ecommerce site that is constantly changing, you definitely want bots to be indexing your site. How else are the latest and greatest products or stories going to show up in organic searches after all? But you also want bots to be well behaved. It would great if you could greet the bots at the door and say "Hey... it's 2:00am, not much going on so index to your heart's content." Or, "Whoa there fella - do you have a reservation? This is Cyber Monday and I'm afraid all our seats are full for paying customers. Can you come back in 12 hours?" But that sort of smart interaction is sadly not in the cards. Some bots have defined rules, some do not. Some honor things you put in the robots.txt file others do not. So here are some tips that might save you some time.

[More]

Here's a problem that comes under the heading of the right hand being unaware of what the left hand is doing. Seriously the right hand is like sitting back, relaxing, maybe pushing the remote and the left hand playing the drums, waving at passers-by and tap-dancing wearing a little tuxedo. The right hand says, "Hey.. what in the ham sandwich is going on over there?" And the left hand says... Eh... I'd say that is as far as I want to go with the whole left-hand-right-hand thing.

The Problem

From time to time I end up troubleshooting an FTP connection. Like most system admins I hate FTP with a capital ick. Insecure, clunky, fault intolerant... It's like arriving at the Oscars in a VW Bus. It gets you there but there has to be a better transport than this! Where we need to support FTP (and always through a VPN please - do the rest of us a favor!), we use a product called FileZilla FTP Server. It's a great product and you can be up and running in about 5 minutes on just about any windows platform. Simply add users, folders, IP restrictions etc. and you are off and running.

The only problem is that occasionally people will call and simply can't get connected. The server is able to recognize them and verify their credentials but when it issues its first server side command (list I think) it times out and drops the connection. After a while I realized that the folks who were experiencing this problem were people running the FileZilla client. That's right - the FileZilla client has trouble connecting to the FileZilla server. If I asked them to use a different client the problem went away.

The Solution

A few weeks ago I finally figured out the solution based on a comment by super-genius-guru Wil Genovese of Trunkful.com who (I'm thrilled to say) works for the Muse and does miraculous things almost every day for our company and staff. He mentioned the problem with FileZilla and UTF-8 encoding. Some experimentation helped me determine an actual fix. Apparently the FileZilla client is passing the string UTF-8 for encoding (which looks ok to me) and the FileZilla server is expecting "UTF8". Fortunately the site manager has a way of specifying a "custom encoding" string. Click on a site then click on the "charset" properties tab. One of the choices is "custom character set". Choose it and enter UTF8 (without the dash). You should be able to connect fine after that.



Note: this problem doesn't exist with every version of client and server combination.

This is a post about solving a particular problem with SMTP relay that involves mass emails. Whenever I write a post on this topic there are 2 things that my savvy readers always feel compelled to tell me:

• "Hey Muse, make sure you are not sending SPAM" - Thanks for the advice. The Muse takes a dim view of SPAM. Like everyone else I'm tired of being told I've won the lottery, have friends in Nigeria, and need to be more concerned about my size. The emails in this case are not spam - but if you are tempted to make that comment I appreciate that you have the Muse' back.
• "Hey Muse, you are crazy to do this yourself - Thanks for that as well. And please don't hesitate to tell me (again) about the various services that are out there - all of which are better equipped technically, mentally, physically, and ecumenically to handle my email so that I need not be an expert on the topic. I always appreciate that input. The only thing I like better than leaving money on the table is transferring it to another vendor. In fact it's an axiom of business to never do anything for money that you can pay someone else to do for you. Next week I'm writing about server troubleshooting and that will give you an additional opportunity to tell me (again) about how that no one needs to worry about that anymore either because the cloud fairies do it all magically.

[More]

Warning: The following post is not related to ColdFusion, SQL, Troubleshooting or anything else of a technical nature. For this post active ingredients include faith, sentimentality, syrupy sweetness, hyperbole, statements of love, and over-all good-feeledness. Side effects may include smiles, light hearted chuckles, blushing, desire to hug your friends and loved ones and an internal urge to answer that pressing question "does my life matter". You have been notified. Seriously folks, if blogs that include statements related to faith bother you - my apologies. I have no desire to offend anyone, but Thanksgiving seems like an appropriate holiday for the following.

On this Thanksgiving Day I hope that you all have something to be thankful for. No matter where you are in life it pays to stop and consider the little things. This is not mere Muse sentimentality - although I do tend to cry at the end of "Armageddon" and every time I hear the song "I'm a little teapot" (it's a long story). No, what I mean is that living a life in tune with gratitude is a recipe for being content. You can make an amazing income, have power and influence and be an outstanding talent - but if you haven't learned how to be content then you may have the "good life" but you have yet to learn how to "live well".

Now I don't know everything about being content. I certainly have a ways to go. I am sometimes frustrated by others. I often feel an excess of pride and ambition. I'm not always as grateful as I could or should be. Still, the older I get, the more I am convinced that I am fortunate and blessed - and the more I want to share that blessing with others. More often than not I am content - and that is a powerful gift. Gratitude is a huge ingredient in the contentment recipe. Gratitude brings perspective and perspective allows me to rest and enjoy what I have in my life.

So what am I thankful for?

[More]

Recently one of our systems started misbehaving. In this system we had 2 ColdFusion 8 servers connecting to a single MSSQL 2005 server. All the hardware was quite good - plenty of RAM, Fast disks, moderate traffic etc. The system had been in place for some time. But (and isn't this often the case) we moved a new design in place with some changes to the query code and suddenly our well-behaved system started acting like a sugar-laden toddler in the cereal aisle.

Watching the "running request" counter for ColdFusion I noticed that it was slowly accumulating requests. When that happens (threads slowing building up over time) you usually have to prepare for some frustrating troubleshooting ahead. When a server is "crashing" you can often pinpoint the error. Crashing servers tend to suddenly fill up running requests and the request queue and the log files will generally have some clues occurring right around that time. But this was different. In this case the request count climbed slowly and was seemingly random. And these threads did not show up in the list of "active requests" in the CF monitor either. Aha! I thought. This is my old networking issue! You might not remember this but a few years ago we discovered that auto-synching ports can sometimes cause phantom connections to hang on a DB intensive application (see this post).

But a quick checkup of network settings showed that this was not the case. Network connectivity was excellent and both DB and the 2 servers were connected through the same Cisco switch. So it was on to the database. Why the DB? Why not scour through JVM settings and fiddle with CF request settings? For one thing, 80% of the time it's not CF or the DB but some combination of the 2 (bad query writing, resource constrained DB, drivers etc). In this case the 2 common denominators were the database and the new code - but I believed the DB was our lowest hanging fruit.

Processor Usage

Sure enough a check of the database showed processor usage that did not look normal. Wait a minute Muse... don't you have any baseline numbers for that assumption? Nope, not at this point. I'm letting my experience guide me. When you have 4 cores and 2 of them are at a flat lined 50% you generally know something is wrong. In fact a quick check of the accumulating requests on CF showed a 20-25% per thread correlation. In other words, each of my hanging threads was using 20% or so of one core on the SQL server. Once it was hung that thread continued to use 20% of one SQL server core in perpetuity until CF was restarted.

The funny thing was that under regular load the DB processors was extremely underutilized until one of these threads was produced. The DB processors would stay at between 1 and 5 percent most of the time - practically idle. When one of these "special threads" came along, one proc out of the 4 of would "jump up" to 20 or 25 percent but the rest would idle along as before.

Finding the Problem

We tried a great many things. We patched and hot fixed, shrunk and optimized files, added and removed indexing etc - all of which was helpful and necessary, but none of which permanently "fixed" our problem). Finally, I was looking at the "activity monitor" in MSSQL05. The activity monitor "process info" view shows a list of connections along with some extra data, process ID, database, status etc. If you double click on an item in the row you will see the currently running query or task. You have to sort of "get lucky" to see it since most of them fly by pretty fast.

In any case I was watching this view (refreshing every 10 seconds) while there were no hanging threads and suddenly I saw something that made me scratch my head. A process ID was duplicated about 3 or 4 times. Each of the duplications had a "wait time" and a "wait type" of CXPACKET. So this process ID was spawning multiple threads under a single ID. And the wait time made me think that this might be our offending process. Looking at the processor utilization I noticed that sure enough, I had a 20% utilization on one core. Going back to my CF servers my suspicions were confirmed. We had a hanging thread on one of the servers - so this CXPACKET thing required some more investigation.

First however, I thought I might try to mitigate the problem from within the activity monitor by killing off this process ID. If I was successful I would have a new mitigation technique that would not involve any potential user disruption, with the exception of whoever was running the query that was locking up these threads (and they were probably tapping their fingers on the desk waiting anyway). So I tried the "kill process" button from the activity monitor, but I had to kill all of them individually and I couldn't catch them all before they re-spawned - or maybe I'm just too old. Turning to SQL Studio I ran the query KILL 55 (where 55 was the process ID in question). That did the trick and it was indeed a magic bullet. As soon as I "killed" that process ID - all the sub-processes were terminated as well. My CF server dropped the hanging thread and SQL Processor usage went back down to normal.

The Fix

Ok so now what? I could hire a temp to sit in front of the activity monitor all day and kill off any process ID with a CXPACKET Wait type that correlated to a CF hanging thread. I could probably write a complicated SQL script to find these threads and terminate them (I kind of liked that idea actually). In the end I chose to do a little research into CXPACKET wait types. I was fortunate to stumble onto this post by Pinal Dave. It turns out that a CXPACKET wait is related to parallelism. Now parallelism is how MS SQL chops up the work load of a query and makes full uses of your processors to get the work done. Much like cfthread splits work out and then joins it back together, SQL splits the work out and then an "organizing" thread "waits" for all the individual threads to complete. Once they have all completed it assembles the data for return to the client. Make sure and read the full article as well as the comment by Jonathan Kehayias at the bottom - excellent stuff!

In any case my SQL server was suffering from not being able to reassemble threads from this division of labor. I'm not sure why that might be (I have some ideas) but the long and short of it is that attempts at parallelism for query execution were causing hanging Jrun threads on my CF server. Following Pinal's guide (and a couple of MS resource pages) I tried setting the max degree to 2 and the threshold to 20, 25, 30... looking for a "sweet spot" where most of my queries would execute without parallelism, reserving it for the report or aggregation queries in the admin section of this site. Unfortunately that didn't work. The issue here was likely a specific query with some new joins in it that was always going to trigger parallelism and ofen fail to complete - causing our hanging thread issue.

Finally, I set the "max degree" to 1. Doing this meant that there would be a 1 to 1 relationship between threads, process IDs and queries. In other words, a given query would never use more than one core execution thread. Now you might think this is problematic because it doesn't make full use of SQL's tuning engine. Technically you are right. I would only say that in a typical web application the query traffic generally consists of dozens of very short queries where parallelism would actually add additional time to the process. So in a typical web application you lose very little by minimizing the degree of parallelism. And indeed that appeared to be the case in our web application. Our CXPACKET waits, hanging threads and egregious processor usage all went away and things have been functioning smoothly since then.

The Aftermath

The Muse knows his readers well. Some of you want to hammer me about not fixing the real problem - that specific query in the code. Not to worry. Using SQL's performance dashboard we teased out the worst offenders and set our ColdFusion developers to analyzing the code. But I suspect the version of SQL or something about the hardware, hyperthreading or NUMA to be a more likely culprit. I have never seen SQL's execution planner cause a problem when it turns to parallel execution before. Still - it's always a good idea to fine tune that query code.

In my previous post on this topic I indicated that IIS 7 seemed to be a constraining factor. That post lead to conversations with a couple of CF gurus (Charlie Arehart and Russ Michaels) who clued me in to a number of additional settings. If you are truly interested take the time to read the previous post and (especially) the comments before you read this post. What bothered me was that the issue I discovered (a cap on requests) can be affected by both IIS settings or JRUN settings (or both).

My conclusion is that the behavior I was trying to affect is actually the bug that Charlie pointed out to me on Adobe's site (found here). Charlie rightly indicates that this issue is under-recognized (I certainly had not run into yet). The behavior of this bug can be affected (fixed or mitigated) by adjusting IIS as described in my previous post as well as by using the Adobe-provided instructions. This lead to a bit of Muse head-scratching. How do these various processes really work together? This post hopes to clear that up (or at least add to our compendium of knowledge).

[More]

One of my terrific, fantastic, stupendous, extraordinary, (doesn't that make you want to come and work for me?) developers named Denny has recently been working on consuming an API from a popular service who shall remain anonymous. I'll only say that when I hear their name I think of lederhosen and that cough drop commercial with the fancy pants guys and big horns. Anyway, Denny apparently uses art to vent his frustrations. Here's what he came up with.

Ok so he's not Picasso yet - but it is definitely refrigerator worthy don't you think?

I have been doing some performance testing for a company with a large server farm over the last couple of weeks. Although the farm had 20 or more servers, we started with just one sever to try and get some numbers we could use to extrapolate the overall tolerance of the larger system. The servers were all Windows 2008r2 64bit, IIS 7, running CF 9 enterprise with plenty of RAM. We were also running Fusion Reactor to help introspect ColdFusion.

As I slowly poured on more load I noticed something strange that I had never seen before. Although my "simultaneous requests" setting was set to 48. I could not get ColdFusion to handle more than 25 active connections. Under ordinary circumstances I could pound away at a server using my test framework and get enough requests active to overrun that simultaneous request setting and see the queue kick in. I was trying to max out the server but it was not behaving as expected. Active requests would "cap out" at 25 - as if my simultaneous request setting was set to 25 - but there were never any requests in the request queue. It was a head scratcher - but I kinda love those! Here's the skinny....

(NOTE: The comments on this post are important as well. And this Follow up post clears up some of the confusion.)

[More]

Fair warning - this is a pretty "101" post so it might be a tad ho-hum for some of you. It's surprising to me how many developers I meet who stare at me blankly when I suggest that the use their HOSTS file for one purpose or another. The HOSTS file has been around since the first networks - although it's gone by a few names over the years. In its current version HOSTS has been around largely unchanged since ARPANET and is in fact the predecessor to DNS (which has reached a venerable age in its own right). As a web developer, learning some easy fundamentals about the use of this file is a practical and fundamental skill - so let's explore it a bit shall we?

[More]

This falls along the lines of one of those tips that matters only to sys admins, firewall managers or network engineers. So if you aren't a networking geek (or don't aspire to become one) you can skip this tip. Here's the skinny.

When setting up a windows server I like to use an "internal only" IP address - one that is not statically NAT'ted to anything - as the source IP address. In most cases this means the IP address presented when making outgoing requests is the external address of the firewall instead of the "real static" ip. If you don't know what I mean by source IP, remote into your server and use a brower to go to what is my ip. Whatever it gives you back is your "source" IP address - the IP presented by outgoing requests. In fact if you check your source IP from your desktop in a typical corporate office and then go to a neighbor's computer and check it there it there you will likely see the same IP address. This is because most desktops sit behind a firewall and the firewall has an assigned IP address that it presents as the "source" IP for most traffic. And that "external firewall address" is also the one I often choose to use for outgoing traffic from a server.

Ok, so why is that a problem? Well a server is a little different. In most cases it will have one or more IPs that are "statically mapped" to its own live internet ips. For example, let's say the DNS record for www.example.com is pointed to 72.10.20.10. "Inside" the network the server actually has an IP of 192.168.10.10. When web traffic hits the firewall for the "external" address (72.10.20.10) it looks at its translation table and knows that the "inside" address that "equals" 72.10.20.10 is actually 192.168.10.10 - and then it checks to see if the traffic is allowed (that's the "firewall" function of a firewall) and forwards the traffic to port 80 on 192.168.10.10. That's "network address translation" to a "statically mapped ip address" (whew!!). Ok, take a drink of water - the dizziness will pass momentarily.

Now for a variety of reasons I often don't want the IP address that the server presents when making outgoing connections to be "statically mapped". Instead I often prefer it to present the outside IP address of the firewall (as a sort of a generic proxy for my whole network). That used to be pretty easy. In Windows server 2000 and 2003 that was easy. I would just make sure that the first IP address I added (the one that you actually "see" in the little network IP properties window before you click on "advanced" - was a non-statically mapped IP. All outgoing traffic would "choose" this first IP by default and Voila! I have the results I was looking for. Then I could just add my other "statically mapped" IPs in the advanced tab and move on.

With Windows 2008r2 however this source IP address can switch to one of the other IPs in the pool. So even though I added my non-static IP first eventually my server might switch to using the statically mapped IP. This is probably only an annoyance for me. But if you are one of the tiny minority of people who geek out over such things here is the solution.

The Fix

In order to get the behavior you want you start out the same. Add your non-static IP as the first IP address per usual. Then instead of adding additional IPs using the "advanced" tab, open a command line and use netsh to add them with the netsh command and the "skipassource=true" flag. It's that "skipassource" flag that does the magic. Here's the syntax for you.

netsh int ipv4 add address network_1 192.168.10.10 255.255.255.0 skipassource=true

One of the things that separate advanced developers from intermediate (at least around here) is the use of Java. Most advanced CF developers know that if ColdFusion doesn't provide precisely the functionality you are looking for you can usually find something in Java that will do the job. Now I am not talking about petulant PHP or Java developers who are being "forced" to write in ColdFusion. Such developers tend to write rather awful code that jumps through hoops in order to make ColdFusion do something the PHP way or the Java way. These folks never figure out how to take advantage of ColdFusion strengths and they are often left with code that must be refactored. Still, ColdFusion and Java are blood brothers. It's axiomatic that if CF can't do what you want, Java can usually come to the rescue. In this post we will discuss a method to treat your Java development just like your ColdFusion development - compiling it automatically at at application refresh for easy development. But first, let's talk about why working with Java can be a bit tricky for ColdFusion folks.

[More]

If you have an issue where you are trying to install "hotfix 1" for CF Builder 2 on a Windows 64bit platform and it fails after decompressing the file, you might try the following. Look for the uncompressed files (usually in the temp directory), drill down and find the *.lax file. This is the file that install anywhere (a Java installer) uses to launch its own install procedure. In the file is a path to the JVM - look for the line that says something like lax.nl.current.vm=\\... Then set the path to a known good 64bit JRE SDK. You should have one if you are running eclipse. Once you make the change launch the install exe in the decompressed files folder (instead of the original file). This is a version of the fix found in my previous post on CF 8 64bit on Windows 2003 64bit Web edition. The previous post has all the details and then some. I'm told by CF aficionado Christian N. Abad that the fix works for CF Builder Hotfix 1 installs as well. I suspect as a general rule that it would work for any "install anywhere" process with this specific problem.