ColdFusion Muse

Cryptocurrency Miners Hacking Servers

Are your free CPU cycles making others rich? There's a chance they are and it's at your expense. A recent article at Vice.com states that "At Least 1.65 Million Computers Are Mining Cryptocurrency for Hackers So Far This Year". If this is to be believed then it's possible a server you are running has been compromised and is actually mining cyrptocurrency for the hackers.

Cyrptocurrency is an anonymous, digital currency that is supposed to be untraceable. It's used on the internet to purchase more and more products and services. One of the most common forms of cryptocurrency is Bitcoin. This is from the Wikipedia entry on Bitcoin.

Bitcoin is a worldwide cryptocurrency and digital payment system called the first decentralized digital currency, since the system works without a central repository or single administrator. It was invented by an unknown programmer, or a group of programmers, under the name Satoshi Nakamoto and released as open-source software in 2009. The system is peer-to-peer, and transactions take place between users directly, without an intermediary. These transactions are verified by network nodes and recorded in a public distributed ledger called a blockchain. Besides being created as a reward for mining, bitcoin can be exchanged for other currencies, products, and services. As of February 2015, over 100,000 merchants and vendors accepted bitcoin as payment. Bitcoin can also be held as an investment. According to research produced by Cambridge University in 2017, there are 2.9 to 5.8 million unique users using a cryptocurrency wallet, most of them using bitcoin. ...

Bitcoin Mining is a record-keeping service that runs on peoples computers, servers, or specialized Mining Devices, that are setup by individuals to help process Bitcoin transactions. As a reward for doing this you are given newly created bitcoins and transaction fees. ie. You can make money by mining for Bitcoin.

This reward is enough that hackers have taken it to the next level and started hacking servers around the world so they can install mining software and use YOUR computers and servers to make money for themselves. Just this week it was discovered that some of Showtime's web servers were mining cryptocurrency. This isn't a new thing either. Back in 2014 Iowa State University servers were also hacked for the purpose on mining Bitcoins. These are not isolated occurrences. They are happening regularly. This practice is free to the hackers an costly to the owners of the servers. Here's why.

Case Study

CF Webtools has seen this type of hack in the real world. We recently had a company come to us seeking our services for both Server Administration and ColdFusion programming. Part of taking this new company on as a client we performed a security review on all of their servers. They also had existing issues that we needed to look at in particular. One of their web servers was rebooting multiple times per day at what seemed like "random" intervals.

Upon review we found the web server was always running at 100% CPU usage with no services claiming to be using that much CPU power. Certainly not ColdFusion or IIS. After completing additional research we decided to install a malware removal tool and scanned for malware. It didn't take long to find that indeed there was malware running on the server. What we found surprised us only because we had not seen this in action before. It was a cryptocurrency miner and it was so intensive that it would crash the server. All attempts to remove the malware failed. It would end up back on the server in a short period of time. The fact is this server was compromised. To resolve the issue we sent one of our decommissioned, but powerful servers, preinstalled with a clean OS to their data center. Then our Operations Manager went on the road to install the new server as well as a physical firewall. We essentially rearchitected their entire server setup. Meanwhile the malware removal tool did it's best to keep the malware at bay while I recreated their web server on the new server. It was a busy week (or more), but we were able to clean the code on the clients server and put that on the new server. We also had to research and rebuild all the dependancies from scratch. When it was all said and done we replaced the compromised server with the new one and put all their servers behind a Cisco ASA.

This case of Hacking for Bitcoins proved costly in the end to the company who's systems were compromised all while providing a free profit to the hacker(s).

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

ColdFusion Security Patches ColdFusion 11 Update 13 and ColdFusion 2016 Update 5

Adobe just released security updates for ColdFusion 11 and ColdFusion 2016. This is a critical security update and you should be updating your ColdFusion servers. The information below is from the CF Webtools operations group. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com. Meanwhile, this info below will help IT staff and DIY types get started.

With ColdFusion 11 Update 13 and ColdFusion 2016 Update 5 there are additional manual updates that are required to complete the security patch. The additional requirements are the same for both ColdFusion 11 and ColdFusion 2016 and the remaining information pertains to both versions. Both updates require that ColdFusion be running on Java version 1.8.0_121 or higher. For reference, ColdFusion 11 comes with Java version 1.8.0_25 and ColdFusion 2016 comes with Java version 1.8.0_72. The Java that needs to be installed is different from the "Windows User" Java client that may already be installed. The installer is available from Oracle. Once the new Java version installed, the jvm.config file for each ColdFusion instance needs to be updated to point to the new Java version installation path. If you're running the Enterprise version of ColdFusion, there's a likely chance there is more than one ColdFusion instance running.

Part of the instructions from Adobe says that if your ColdFusion server is installed as J2EE server then there is an addiction manual configuration that you ned to do. However, every installation of ColdFusion since the release of ColdFusion 10 is a J2EE or JEE installation. What Adobe really meant was that if you are using a third party JEE server and not the built-in Tomcat JEE server.

If your ColdFusion server is running on a third party JEE server such as WebLogic, Wildly, custom Apache Tomcat, etc (Not the built in Tomcat that comes with ColdFusion), then the following step needs to be completed.

Set the following JVM flag, "-Djdk.serialFilter=!org.mozilla.** ", in the respective startup file depending on the type of Application Server being used.

For example,

  • On Apache Tomcat Application Server, edit JAVA_OPTS in the 'Catalina.bat/sh' file
  • On WebLogic Application Server, edit JAVA_OPTIONS in the 'startWeblogic.cmd' file
  • On a WildFly/EAP Application Server, edit JAVA_OPTS in the 'standalone.conf' file

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them or if they are not familiar or knowledgeable with ColdFusion contact us at CF Webtools to patch your servers.

As always, if you need help migrating to the next version, scanning your ColdFusion server for security vectors or installing this patch and new Java version, contact your project or account manager directly, or send an email to support@cfwebtools.com. You can also simply respond to this email (or call 403-408-3733).

*Note: ColdFusion11 when it was first released came with a version of Java 1.7.0_nn. Adobe later re-released ColdFusion 11 with Java 1.8.0_25. If you have ColdFusion 11 still running on Java 1.7 I highly recommend that Java be upgraded to Java 1.8. Oracle is no longer supporting Java 1.7 and 1.7 is long past it's end of life. Even though the Adobe instructions for this current security update states that you can run Java 1.7.0_131, I highly recommend upgrading to Java 1.8.

Mary Jo Sminkey: Robust Error Handling Part 3 - Ajax

Part 3 in the super-guru's Mary Jo Sminkey's series on robust error handling. Enjoy!


So in my last post on error handling, I promised in this next one to cover logging of Ajax requests. As more and more of our site involves Ajax requests, this has become an important part of my error logging and debug work. If you want to know how to do this read on!

[More]

Building a Robust Error Handler Part 2

The Muse is fortunate to have a number of very talented and smart developers work for him. Mary Jo Sminkey has been with CF Webtools for several years now and we simply love having her around. She's a fantastic developer and a fantastic resource. You might catch her at cf.objective() where she will be speaking next week.

Meanwhile, you might remember part 1 of this series – Building a Robust Error Handler. Below is part 2 in that 3 part series. Enjoy!


Well it's taken a couple years to get around to it, but at long last here is part 2 of my error handling post! Of course, in that time I've continued to work on improvements to my error handler, so many changes that I'm going to need to break THIS post up into multiple parts as well. I'll try not to take too long to get the final part out to you!

So in part one we looked at things like using the error handler for both global errors as well as in cfcatch blocks, dumping all the available scopes while scrubbing them for any sensitive data, keeping track of already reported errors to prevent receiving tons of duplicate emails for the same error, etc.

In this post (part 2) we will expand the functionality of the error handler and talk about strategies for use throughout the application. Let's start with how to handle errors "in depth".

[More]

ColdFusion, SSL, SNI, SAN and Wildcards - Stuff You Need to Know

The Muse welcomes back his friend and colleague (and super genius guru) Wil Genovese with an timely post on SSL and Certificate types. If you have had your head in the ground (or perhaps you have been guest staring on "Naked and Afraid" or "Survivor") you may have missed the hubbub surrounding TLS, SSL and changes and support. There is a lot going on and it is more important than ever that you get your hands around the issue to keep your users safe. Wil has done Yeomen's work identifying the types of certs, the versions of ColdFusion and Java that support them, and work arounds and caveats for those of you who need them. You will likely want to bookmark this one. Take it away Wil.

[More]

ColdFusion and JVM Versions and SSLv3-TLS Security Magic

This is the second entry by Wil Genovese (Trunkful.com) in our effort to provide a complete picture of how CF, Various versions of JVMs and various versions of SSL all work together. Wil's previous article on Surviving Poodle detailed a blow by blow description of how to troubleshoot a system broken due to the upgrading of SSL. This article includes some detailed technical information as well as the results of some painstaking tests. It is our hope that it will serve as a guide. It represents yet another reason to insure that you are upgrading to the latest JVM and CF version. Take it away Wil:

[More]

Surviving Poodle - ColdFusion and SSL 3

There's been a great deal of buzz about poodle. Poodle is an SSL exploit capable of highjacking a session using a browser's ability to "negotiate downward" the level of SSL it uses. It's recent prolifieration has put some urgency into the efforts to force existing applications and platforms to deny the use of any standard of SSL less than version 3.0. Super guru Wil Genovese (Trunkful.com) recently did some troubleshooting on a ColdFusion server with an issue related to this necessary configuration step. Wil writes:

We ran into an issue when a company contacted us at CF Webtools because ColdFusion was suddenly no longer able to connect to their email providers mail servers. One day ColdFusion was sending emails to their clients just fine and the next day it was failing. As you know these issues are usually best resolved by asking "What changed?" As far as the client knew, nothing had changed - but we knew enough not to stop digging.

[More]

Side-by-side Configuration Error installing CF 11 on Win2008r2

One of my colleagues, Chris Tierney, was installing a pristine copy of ColdFusion 11 on a Windows 2008 server. He followed our standard protocol which is to install the server using the "built in" Web server, then create instances (we typically use multi-server mode) then use wsconfig.exe to connect the instances to IIS. It all went as planned until he tried to run wsconfig.exe (FYI - you must always run this as administrator). He got an error as follows:

java.io.IOException: Cannot run program "C:\Users\ADMINI~1.CFW\AppData\Local\Tem
p\2\\ExecuteAppCmd\ExecuteAppCmd.exe": CreateProcess error=14001, The applicatio
n has failed to start because its side-by-side configuration is incorrect.


Followed by an odd stack trace. After experimenting with permissions and googling he stumbled on Bug 3761543 in the ColdFusion bugbase. The issue is not very well documented. Apparently the MS C++ package installed on 08 is 32 bit. I'm not clear if we installed it or it shipped with 08, but remember, you need the MS C++ 64bit SP1 Redistributable. Here's the download link from Microsoft so you don't have to hunt it down.

The Fix

Microsoft Visual C++ 2008 SP1 Redistributable Package (x64)

One more time: This download fixes the "side-by-side configuration" when installing ColdFusion 11 on Windows 2008r2.

Business Referral - don't forget

A week ago, in my post, The Muse Has Cash... I started a new program to reward community members for leads. Thank you for all the input and for the many leads generated already. We really appreciate it and we'll make you proud! After a quarter or so I will report on the success of the program.

Meanwhile, a few have pointed out that the previous post is lengthy so I wanted to put up the "express checkout" version. It's simple, if you refer a lead to us (email cfleads@cfwebtools.com) that results in new business, we will pay back to you 8% of the gross revenue from that customer in the first 10 months of working with the customer. If the customer spends 10k with us, you make $800.00. Simple and easy. So hook the Muse up. We are looking for another record year!

Note: In the past the Muse has offered bonuses for referring developers to us. This program is for new business, not developers - although as always if you are looking for work send me your resume. We typically hire several times a year.

Running ColdFusion 10 in Distributed Mode

A recent discussion on CF-Talk had a member trying to find instructions for setting up ColdFusion in "Distributed mode" on a CF 11 server. If you have never heard of "distributed mode" you are not alone among CF developers. It's not a common setup in my experience. Here's a quick 101. Many processes and daemons on a given server communicate using the TCP stack. TCP provides a predictable, high capacity, mature protocol for piping requests and data in and out of server processes. In this case, IIS or Apache communicate with CF through the local loopback (localhost or 127.0.0.1) IP address and a port - usually 8012 (or 8013 or 8014) chosen at installation. Naturally you can alter the IP address to which you are connecting, changing it from the loopback to... well any IP that's listening on 8012. That means you can set up your ColdFusion servers separately from your web servers. After all it's just IP networking. Why would you do that? The Muse will let his guest handle that question.

Meanwhile, to preempt (or perhaps spur) discussion, the Muse will note that this process is similar to something called "Reverse proxy" which functions in much the same way. The difference being with CF distributed, IIS on the front end handles all the "http stuff" and passes the request to CF just like it was a local engine, whereas with reverse proxy the HTTP request is simply redirected to the alternate server. That means the alternate server needs to be a full webserver plus application server. While that increases the overhead a bit, it has some advantages - but that's a topic for a different post.

Back to our CF-Talk question, it quickly became apparent that not many folks actually knew how to accomplish this task on a CF 10 (or 11) server due to the underlying platform switch to Tomcat. After some back and forth Byron Mann chimed in with some very specific instructions on how to get this done. Byron is a lead engineer at HostMySite - which makes the Muse feel better about their ColdFusion support. :) Here's his tutorial.

[More]

IIS Vulnerability Steals Payment Information (By Wil Genovese - CFG)

Super guru Wil Genovese (Trunkful.com) is back to describe an IIS vulnerability that was inserted using a long-known (and patched) CF vulnerability. The Muse will make 2 points. First, if you are hit with this one call us! We will gladly put our shoulder to the wheel and help you dig out. Second, don't forget to patch your servers and keep up on the latest security news. No matter what your chosen platform you need to be vigilant and attentive. Take it away Wil.

First let me point out that the vulnerability that was found has a patch that has been available since January of 2013. So as the Muse said, patch your servers! I first read about this attack in a PC World article titled, PCWorld - Attackers exploited ColdFusion vulnerability to install Microsoft IIS malware. I spent hours reading all the linked websites and blog posts by the security researcher that discovered the IIS Malware (see this Trustwave post) trying in vain to learn the name of said DLL that gets installed, where it gets installed and how to detect the file(s). The few details I found were not completely useful. While I learned the behavior of the malware I never learned how to find the offending DLL or even the file name. I did discover that no existing anti-malware or anti-virus software would detect this rogue DLL. I repeated my futile search every few weeks to see if anything new was being reported.

Since knowing how to locate and expunge such things is part of my job I needed a way to find it, but how? I could search any of the servers at CF Webtools until the cows come home, but if none of them have been hit with this malware I will never find it. What I needed was a server that had been exploited to examine. Over the past year with the slightly larger than usual number of security holes discovered in ColdFusion we've had a few new clients come to us for help in patching and repairing servers. None of the IIS modules on those servers stood out to me as 'unusual', but I wasn't looking directly for this. Finally we had a company come to us for help with a breach.

[More]

Building a Robust Error Handler

If you have been around the ColdFusion world as long as the Muse you have heard of Mary Jo Sminkey. Mary Jo built a popular ColdFusion ecommerce platform called CFWebstore. She has vast experience in ColdFusion and a seemingly boundless fountain of energy. Her eclectic interests range from technology to baking to dog training. As far as CF Webtools and the Muse can tell, Mary Jo excels at everything she does. We frankly suspect she is actually twins or triplets pretending to be only one person :) The following article is by Mary Jo and details her approach to application specific error handling. She has a detailed and thorough knowledge of the topic. Using this approach she has been able to reduce the number of errors on a very high traffic E-commerce site to practically nil. In the first of 2 articles MJ (as we call her with great affection) details the structure and usage of the handler.

Building a Robust Error Handler (by Mary Jo Sminkey)

Let's face it, sometimes we put less effort into the error handler than into the rest of our code. We might put something in place that throws up a "user friendly" page, and maybe email a dump of the catch or error structure, but when the site goes live, and we are deluged with errors due to search bots, hack attempts and poorly coded pages we turn it off or send all those emails to a seldom-visited mailbox. Sometimes we implement error handling as cftry/cfcatch blocks that do little more than preventing errors from being thrown, instead of helping us track down the issue.

I look at the error handler as a way to help make a site as bug-free as possible. By having it email me as much information as possible about errors, I troubleshoot, fix and patch, and get to a point where errors are the exception rather than the rule. In this article, we'll look at building a single-page, comprehensive error handler. In a future article, we'll look at integrating that error handler with the open source bug tracker BugLogHQ. Before we begin with our error handler let's talk about our error handling strategy.

[More]

email connection crossover workarounds

As a follow up to yesterday's post (regarding sending mail and having it end up in someone else's "sent" folder) I thought I might put some flesh on the workaround suggested both in the bug report and on CF-Talk. The suggestion is to:

Create a CNAME to point to the SMTP server address so that both websites were looking at different domain names.
This idea is workable up to a point so I thought I would explore it for my readers.

[More]

ColdFusion email security Bug: Your mail in the wrong sent folder?

A recent conversation on CF-Talk piqued my interest. It turns out there is a tricky bug with regard to sending authenticated mail. Here at CF Webtools we have internal relays (protected, internal only IPs, listed in SPF and handling domain keys) whose sole purpose is to relay mail from our web servers - so we do not have "authenticated" email per se. But in the case of this bug (you can see the report here) it's possible for email from one user to wind up in the "Sent" folder of email from an entirely different user. Needless to say this is a security concern for those of you on shared servers especially.

Here are the conditions that need to be met for this to occur (as I understand it).

[More]

A Frank Discussion About Protection

I know it's an uncomfortable topic. I understand that you would like to keep your validation private. You would probably rather learn about this from your friends at the coffee shop, Jeremy who is two cubes down from you, or some guy on a forum (shudder). Still, the Muse has an assignment in life to point these things out and make sure you are well informed and prepared when temptation strikes. Oh I know what you say now. I know what I'm doing. The risk factor is slight. I'm too small... I mean... my application is too small to need it. But take it from me - you will need to understand how to use protection or bad things will happen. So let's talk about it.

[More]

More Entries




Blog provided and hosted by CF Webtools. Blog Sofware by Ray Camden.