ColdFusion Muse

Troubleshooting CFHTTP - 3 Tips

Hang around ColdFusion long enough and you will eventually find yourself making use of CFHTTP. The tag allows you to make an HTTP request from your Coldfusion server to an HTTP resource. Why would you want to do this? HTTP is a great low level protocol for data exchange. There's a reason why most web services run over HTTP - it is easy to understand and implement. As for CFHTTP, there are a great many examples, but here are a few.

  • RSS Feeds - You might want to consume several RSS feeds and create a single feed from them. This is essentially what is going on at Full As a Goog and other sites like it.
  • Ecommerce - Many Ecommerce gateways receive HTTP requests over SSL from web servers in order to process Credit card transactions.
  • Data Files - You might use it to pick up text files for import or export.

Here's another example from the simple and practical side of my personal life. I listen to XM radio and I'm a baseball fan. The list of games and channels is online in the XM Radio program guide, but I wanted to access it from my mobile phone. I wanted to figure out my desired channel in the car without needing to scroll through dozens of channels. Using CFHTTP I unpacked the XM program guide for Major League Baseball, found the "print" version for the games, and figured out how to pass it today’s date so that it would show the channels for today’s game at the top. I created a server side script that adjusted the date and used CFHTTP to retrieve the file starting with today. The result can be viewed here. It loads fast on my Moto Q and now I can find the Cubs at a glance. Here are the 3 or 4 lines of code I needed to create the page.

<Cfset dt = dateformat(now(),'mm/dd/yyyy')/>

<cfset u = "http://www.xmradio.com/schedule/sport/get_mlb_schedule.jsp?print=yes&startDate=#dt#"/>

<cfhttp url="#u#"/>

<Cfoutput>
    #cfhttp.filecontent#
</CFOUTPUT>

This simple example only scratches the surface of what is possible with CFHTTP. There are, however, some nuances to using it. Troubleshooting CFHTTP is challenging at times. Without the benefit of experience you can find yourself going bald in a hurry. So, here are 3 tips from the muse to help you troubleshoot CFHTTP.

DNS Issues and Connection Issues

The most common problem when using CFHTTP is simply that the server cannot resolve the URL or is blocked from access. When the problem is DNS you will get a short "connection failure" message in place of the cfhttp.filecontent variable. If the problem is that the URL is being resolved (DNS is working) but the content is being blocked from retrieval you might get a hanging request and a timeout (or you might get a response from the blocking entity).

To troubleshoot this issue you need access to the server - remote access is fine, but you need to be able to issue HTTP requests from the server. It does you little good to load the URL into the browser of your workstation and say "see... it resolves." That doesn't tell you wether the server can resolve it or not. Remember, servers live in a very a different environment from your laptop or desktop. They are usually surrounded by devices, routing and rules intended to protect them. They also sometimes use a different DNS server (or at least configured differently) on the "inside" network where they reside.

So you will need to verify that the server can resolve your URL. That's step 1. If you determine that that the server cannot resolve the URL then you have a few choices. You can attempt to get it to resolve by configuring DNS, changing the network config etc. Or you can add a hosts file entry to circumvent DNS (this will not solve a problem where the request is blocked but it will get you past DNS resolution issues). Either way, you have to enable the server to "get at" the content.

SSL Issues

Using HTTP over SSL requires that the public and private keys work together to "handshake" and allow the request to succeed. The public keys have a hierarchy of "trust" that is based on the entities that issued the certs (like Verisign for example). The browser handles all of this internally and generally trusts about everyone (or at least asks if you want to trust a cert). But remember, you are not using a browser. You are using the JVM to make your request. Just like the browser, the JVM has a list of entities that it trusts as well. It's call the "trusted keystore". If your using CFHTTP to make SSL requests and having trouble, part of the reason could be because the cert is not yet "trusted" by the keystore. The fix is a cryptic procedure using a command line tool that ships with Java. The instructions may be found here. Note, troubleshooting this issue will require a bit more patience than just a connection issue, and if your cert is a "self-signed" cert you are going to need some Advil.

Compression

Actually, I just learned about this issue from my good friend and CF/SQL guru Mike Klostermeyer (who if he ever decided to blog could put me to shame). It turns out that if the request is made against an IIS server and the IIS server has compression enabled for serving web pages, the CFHTTP process might have trouble unpacking the results into something readable. The fix, as it turns out, is to add some headers as cfhttpparams - like so:

<cfset u = 'http://www.myexample.com'/>
    
    <cfhttp url="#u#" method="get">
        <cfhttpparam type="Header" name="Accept-Encoding" value="deflate;q=0">
        <cfhttpparam type="Header" name="TE" value="deflate;q=0">
    </cfhttp>
As I understand it, these headers instruct IIS to send the content uncompressed. If you ever find yourself tied in knots over this particular issue, this will keep you sane (or at least keep you from jumping out of a 4 story window).

If you are a muse reader and you have tips on troubleshooting CFHTTP, feel free to add to the discussion.

Comments
jc's Gravatar If all else fails, check the redirect setting. Change it to whatever it wasn't.

This especially applies when you're using CFHTTP to build a bot/spider sort of tool. I've had to build a "pretend user" to log into an administrative site and perform a variety of tasks on several occasions.

Carrying cookies between CFHTTP sessions can be a pain, too. I'll post a snippet of code I wrote (there's probably a better way to do it though).

(after a successful CFHTTP)
<cfset myCookies=MyCFHTTPVarName.responseheader['Set-Cookie']>
<cfloop from="1" to="#structCount(mycookies)#" index="i">
<cfset cookieStruct[#listfirst(listfirst(myCookies[i],';'),'=')#] = #listlast(listfirst(myCookies[i],';'),'=')#>
</cfloop>

~~~

(inside the following CFHTTP tag)
<cfloop list="#structkeylist(cookieStruct)#" index="i">
<cfhttpparam type="Cookie" name="#i#" value="#cookieStruct[i]#">
</cfloop>
# Posted By jc | 9/9/08 10:33 AM
Brad Wood's Gravatar Heh-- yeah, that HTTP compression can be a bear to figure out.

There are several versions of peoples' fixes to it, and I think it kind of depends on the server you are connecting to. I will confirm, it is not just IIS that will screw with you. Apache will do it too.

Also, I've blogged some behavior of CFHTTP's URL resolving on redirection that I think is a bit buggy:
http://www.codersrevolution.com/index.cfm/2008/6/2...
# Posted By Brad Wood | 9/9/08 11:35 AM
Shimon's Gravatar I've opened this article from RSS to read just half an hour before I was told we've ran into compression issue in my company!

Also I'm not sure if this should be a tip or just a remark, Steven Erat has a little bit different version of this code that helps him to deal with IIS compression (note the asterisk):

<cfhttpparam type="header" name="Accept-Encoding" value="*" />
<cfhttpparam type="Header" name="TE" value="deflate;q=0">

http://www.talkingtree.com/blog/index.cfm/2004/7/2...
# Posted By Shimon | 9/9/08 2:32 PM
jc's Gravatar Shimon - The asterisk bit fails in some circumstances. I forget what specifically, but I remember having to switch from that to the version Mark had in the OP.
# Posted By jc | 9/9/08 2:39 PM
Shimon's Gravatar @jc - thanks, I'll keep that in mind
# Posted By Shimon | 9/9/08 3:30 PM
Geoff's Gravatar Also, bear in mind the JVM maintains its own never-expiring dns cache. ColdFusion *ignores* the TTL of the domain you're trying to retrieve with cfhttp and caches the dns lookup forever. So if the ip address ever changes, you're stuck until a CF restart.

You'd need to fiddle with the JVM settings to turn off this feature...
# Posted By Geoff | 9/9/08 3:55 PM
Steven Erat's Gravatar Here's a reference for configuring DNS caching TTL:
http://www.talkingtree.com/blog/index.cfm/2006/5/2...

I've been using a network packet analyzer called Ethereal for 6 years or so. Ethereal is now marketed as Wireshark. Its incredibly useful for everything from watching the TCP SYN/ACK handshake, to viewing the entire HTTP request/response conversation. (It can't read the contents of SSL encrypted connections, but it can at least report the handshake and certificate exchange)

Under ColdFusion in the runtime/bin directory there's also sniffer.exe, a modification of Apache's TCP Monitor. It acts as a proxy and dumps the request/response in a little window as its happening. Tell it to listen some high random port on localhost, say 8181 and to proxy for the remote host/port say company.com:80, then make your cfhttp connect to localhost:8181 and that request details will be displayed as its proxied to company.com.
# Posted By Steven Erat | 10/22/08 2:35 PM
Eric Hoffman's Gravatar I was in fact about to jump out a two story window to determine if that could supersede my cfhttp pain. Then remembered from the past about IIS Compression vs. CFHTTP.

Thanks for the snippet...brought back all the memories from the last time I hit that wall, fixed it, and got back to work!
# Posted By Eric Hoffman | 1/19/09 3:07 PM
Asha's Gravatar To resolve the compression issue in ColdFusion9 a new attribute is added we can set compression="none" on cfhttp tag.We dont need to add headers in cfhttpparam.
# Posted By Asha | 1/5/10 6:23 AM
James Moberg's Gravatar The new CF9 compression="none" parameter does't work when performing a GET on a compressed text file and using the "name" parameter to create a query object. I had to add the extra cfhttpparam options in order for it to work.
# Posted By James Moberg | 6/9/11 4:29 PM
Tom Loiacono's Gravatar I echo James' post. Compression=none worked fine until I hit a GET that is grabbing a text file and putting it in a name var. However, adding the extra cfhttparams isn't working. Jury's still out; working on it...
# Posted By Tom Loiacono | 12/5/11 5:27 PM
Tim Brown's Gravatar Any one come across a solution for this issue? This question has the exact issue I am encountering ( http://stackoverflow.com/questions/2218615/cfhttp-... ). I have tried every possible suggested compression header and attribute setting to get this to work on my local host. I have messed with the JVM settings to ensure that the http cache is not an issue.

I have one site on my local host http://mysite.local that calls another local host site http://api.local/. When I update my api.local host file entry to point to a coworker's IP that as a local copy of the site the cfhttp call works instantly. Any thoughts would be much appreciated.
# Posted By Tim Brown | 2/4/12 9:19 PM



Blog provided and hosted by CF Webtools. Blog Sofware by Ray Camden.