ColdFusion Muse

ColdFusion MailSpoolService Performance

In my last article about the Adobe ColdFusion MailSpoolService I mentioned that I was going to try to get specifics on expected performance in the Standard Edition vs Enterprise edition of the MailSpoolService. Adobe has not respond to my requests with actual data. While attending the ColdFusion Summit 2017 I tried to get a clear answer from any of the Adobe ColdFusion engineering team members that were at the conference. They didn't know the answer. Because I didn't get the response I wanted from Adobe I decided to start testing.

My first test was to setup a Windows VM with ColdFusion 11 installed with a standard license. I also created a simple CFML page that uses CFMAIL to send an email with a CFLOOP to send that same email a lot of times. To make this a more realistic test I made up a new disposable email address on our mail server at CF Webtools and sent the emails from my email server on AWS. This means that the ColdFusion MailSpoolService has to actually communicate with a mail server. SMTP connections can at times take time. The emails I generated have several paragraphs of Lorem Ipsum text to simulate actual email sizes. My first test was to verify one email did indeed get sent. It did. The next test was to send 1000 emails while timing with my iPhone's stop watch. We also have ColdFusion 11 Enterprise which meant I was able to test the performance against the Enterprise Edition. Lastly, I was asked to test on the Developer Edition because it is often stated that the Developer Edition is essentially Enterprise Edition with a two connection limit. I ran this test a couple times each from ColdFusion 11 Standard, ColdFusion 11 Developer, and ColdFusion 11 Enterprise servers.

Standard Edition
It took approximately 23 minutes to process 1000 emails in the mail spool. This comes down to about 44/45 emails per minute. Which works out to about 11/12 emails per 15 second pooling interval or 2600 email an hour. Which is a little more that 60,000 emails per day processing 24 hours straight without any connection issues. That's not too shabby for being the single threaded version of the MailSpoolService.

Developer Edition
After running the same tests a couple times in Developer Edition I got the exact same results as I did for Standard Edition.

Enterprise Edition
This is where you can say "You get what you pay for!". Before I go into the numbers let me also remind everyone that the Enterprise Edition of the MailSpoolService is multi-threaded and you can specify the number of threads. I think the default is 10 threads. This setting is in the Mail section of the ColdFusion Administrator Enterprise Edition ONLY in the sub section "Mail Spool Settings". There is nothing that indicates that there is a maximum number of threads. My tests are with 10 threads.

I had to run this test several more times just to make sure I saw what I saw. All 1000 emails were sent in a single polling of the mailSpoolService. That's 1000 emails sent in under 15 seconds. I ramped it up a bit and sent 5000 emails. This time it took two polling intervals and sent 5000 emails in about 30 seconds. To get absurd I increased the test to 10,000 emails and the Enterprise Edition cleared those out in less than 60 seconds. This means it took 4 polling intervals to process 10,000 emails which comes out to 2,500 every 15 seconds with 10 MailSpoolThreads. I wanted to verify this exactly so I decreased the polling interval from 15 seconds to 30 seconds. I wanted to fill the mail spool completely beforehand and see how many emails were processed on each polling interval. What I saw is that I'm not nearly at the limit of what the Enterprise Edition MailSpoolService can handle. By slowing down the polling interval my CFML script was able to put all 10,000 emails into the mail spool folder before the MailSpoolService started processing. Then it happened, all 10,000 emails were process in one single polling interval of less than 15 seconds time. I'm not sure were the limit is, but it's fairly clear that the Enterprise edition can send more emails than most of us will ever need. Even if you're running a bulk mail service.

Summary
My results are not scientific. However, I do believe they are closer to what real people will see on real servers based on my experience with hundreds of servers. It would be really nice if Adobe would respond with some real numbers so we could help clients decide if this feature is worth buying Enterprise Edition instead of Standard Edition. However, based on my testing, if sending emails is your high priority and the amount of emails is going to be over 50,000 emails per day then you might want to weigh the cost of an Enterprise license.

Note:
The reason I was testing on ColdFusion 11 is this is the version that several different clients have that are having issues with the MailSpoolService. I think I know that for one client they really are trying to send near or over 50,000 emails per day and this is why they thought there was an issue with the MailSpoolService.

Comments
Russ Michaels's Gravatar If you need to send large quantities of mail fast, then I recommend taking a look at mailgun or SendGrid. You can use their API to send out emails instead of SMTP, thus bypassing the cfmailspool and SMTP process entirely. This is significantly quicker if you are sending out the same email to all address as I believe that will be done with a single API call. Plus you have the advantage of their bounce processing, reporting and other useful tools that you don't get with cfmail.
# Posted By Russ Michaels | 11/27/17 5:43 PM
James Moberg's Gravatar You should compare the perform of using a local SMTP service on the same server as ColdFusion. (We installed Microsoft IIS SMTP Service and sent email without having to use authentication when generated messages on the same box.) We've found this process to be a lot faster than repeatedly connecting to a third-party SMTP server using a single ColdFusion thread.

In the past, we also had issues with losing occasional messages w/both Standard & Enterprise. We would log the data, send and then... nothing. Messages wouldn't even end up in the CF SMTP log.

In the end, we migrated our sending to go through SparkPost (similar to SendGrid that Russ is recommending). It's a lot easier to generate one message and provide custom variables to be "mail merged" on their end than it is to personalize and generate them all using CF. SparkPost offers both SMTP & Rest API, but if you choose SMTP, you'll need to restart ColdFusion alot because their IPs change and ColdFusion tends to cache the IP forever.
# Posted By James Moberg | 11/27/17 5:58 PM
James Moberg's Gravatar I believe Enterprise is capable of keeping the SMTP connections open between "sends". This would make it faster. I haven't monitored it, but I'm guessing Standard has to open a new connection, authenticate, send, close and repeat while sending via SMTP.
# Posted By James Moberg | 11/27/17 6:02 PM
James Moberg's Gravatar Sorry... I should have stated that we use Microsoft IIS SMTP Service to "relay" email to our dedicated mail server.
# Posted By James Moberg | 11/27/17 6:04 PM
Wil Genovese's Gravatar James, Actually for the server that prompted me to run these tests, it is using Microsoft IIS SMTP to relay mail and it is NOT getting any faster results sending mail from the Standard Edition. Also, I did review the SMTP logs on the server that was having 'issues' and found that a new connection was being made for each mail sent from ColdFusion 11 Standard Edition. It does appear that the mailSpoolService in Standard Edition is very restricted. My whole point of this has been to try to get actual facts from Adobe as to the mail spool performance levels in Standard vs Enterprise. That data has been elusive so I did basic testing to see what I could come up with.

Regards,
Wil
# Posted By Wil Genovese | 11/28/17 6:53 PM
Wil Genovese's Gravatar Russ, I'm well aware of other resources for sending email outside of ColdFusion. My whole point of this has been to try to get actual facts from Adobe as to the mail spool performance levels in Standard vs Enterprise. That data has been elusive so I did basic testing to see what I could come up with.

Regards,
Wil
# Posted By Wil Genovese | 11/28/17 6:56 PM
James Moberg's Gravatar It does seem like CFStandard is intentionally "hobbled". As long as CF has been out, you'd think a feature like SMTP connection caching would have become standard and not something that requires additional licensing. Modular pricing would be nice. I'd pay a little extra to enable non-hobbled SMTP & PDF features, but don't need many of the other "enterprise" features (API Manager, etc).

I recall that back in the CF5/6 days (~2002) that some developer blog about generating similar CFMail files and used a different program to monitor and perform the sending process. I did manage to find this article from 2002 that recommended writing text files directly to the mail server's outbound queue folder, but this isn't what I recall reading before. (Many CF-related blogs are no longer available.)
http://coldfusion.sys-con.com/node/41634
# Posted By James Moberg | 11/28/17 8:03 PM



Blog provided and hosted by CF Webtools. Blog Sofware by Ray Camden.