We'll delve into a Spotflow story about a delay in the delivery of verification emails during the registration process for our product. Spotflow provides a developer-friendly, self-service IoT platform that collects and processes data, simplifying IoT device management and solution development.
Self-service of the platform is a very important aspect of our product from the very beginning when a potential customer can try the platform with a free trial. The free trial allows them first-hand experience with the platform and influences their decision to choose the product.
However, we experienced delays in delivering verification emails during the registration process, an integral part of the trial. To address this, we evaluated our existing email delivery solution, took steps to rectify the issue, and considered whether transitioning to a new email service provider was necessary.
The journey started with a pressing issue. We were experiencing around a 4-minute delay with our verification emails. We noticed this delay, specifically with emails sent to Gmail. However, it was unclear whether this delay also applied to emails sent to other mailbox providers such as Yahoo, Hotmail, etc. This was more than just an inconvenience. It was a critical problem that was preventing users from completing their registration to our platform, leading to a loss of potential customers.
Our initial suspicion fell on our current service provider, SendGrid, and the reputation of the dedicated IP address we were using. Our dissatisfaction with SendGrid had been simmering for a while, and this issue seemed like the final straw. Consequently, we have opted to conduct a thorough examination of the issue and rigorously test our solution to ensure that our decisions are grounded in empirical data.
Our testing criteria were straightforward. We aimed to distribute the verification email to a vast array of distinct email addresses and determine whether and when the emails were delivered. Numerous independent tools for testing email deliverability are available on the market (this analysis is beyond the scope of this blog post). However, GlockApps proved to be the most compatible with our needs. GlockApps provides Spam Test packages, eliminating the need for us to purchase any subscriptions. We selected a package that included 10 Spam tests, which we anticipated would suffice for our comprehensive testing.
It’s worth noting that the GlockApps portal is quite user-friendly, and the test is relatively simple. The spam test is conducted using a seed list of GlockApps’s test email addresses set up at different mailbox providers and regions. The next step is to send the email to the seed email addresses. The email to be tested must contain a unique GlockApps identifier, which can be placed anywhere within the email content. This ID allows the tool to discern which email is associated with the concrete spam test. For the tests themselves, we created a C# script that sends our verification email to all the seed emails via the email delivery services we selected.
Before discussing our experience testing our solution with SendGrid, it's important to understand Spotflow's history with this tool. We initially selected SendGrid, a well-known Email Service Provider (ESP), believing it to be a reliable choice. However, we encountered several issues during setup, resulting from underdeveloped features.
Our first concern arose during the setup process. We required separation between our development/testing and production environments. SendGrid uses 'Subuser accounts' derived from a 'Parent account' to organize and manage different email streams. The parent account isn't intended to send emails, which are left to the subuser accounts. This division is suggested for separating transactional and marketing emails as well. However, we found this architecture clumsy and the naming convention inappropriate for concepts related to the separation of environments or transactional/marketing email streams.
We also struggled to understand SendGrid's approach to two-factor authentication (2FA). It offers 2FA only via SMS or Authy, an app owned by SendGrid's parent company, Twilio. We were not satisfied with either approach. Using SMS (or voice over GSM) for 2FA is widely considered insecure. Authy, although a popular 2FA application, needs to be associated with a phone number, which opens unnecessary attack vectors. Security keys, passkeys, or at least generic TOTP are not supported. But security is not the only issue associated with SendGrid's strange 2FA strategy. In combination with the fact that there might only be a single administrator account, there is no way to provide redundant administrator access to SendGrid because the administrators would be forced to share a phone number, which is highly impractical. Fortunately, it is possible to add additional user accounts with an almost identical set of permissions.
Another unexpected limitation was that SendGrid only supports custom Single Sign-On (SSO) via SAML for identity platforms such as Okta and Azure AD. It does not offer SSO/social login through common social accounts like Google, Microsoft, or GitHub.
Despite our dissatisfaction with these aspects of SendGrid, we chose to continue using it as it fulfilled its primary function. However, we stopped tolerating SendGrid when we encountered email delivery problems. Let's examine SendGrid's spam test results for our verification email template.
Spam Test Results
Before we delve into the spam test results, it's important to mention that we use SendGrid with a dedicated IP address. Moreover, our DNS records for DKIM, SPF, DMARC, and reverse DNS on the IP address are verified.
After conducting the spam test for the verification email template (see picture above), we obtained the following results on SendGrid.
To clarify, when referring to 'tabs', this means the tabs other than the Primary Inbox, such as Gmail's Promotions tab.
Surprisingly, about half of the emails ended up in the spam folder. We didn't anticipate such poor results with a dedicated IP, which we believed would bolster our reputation.
Focusing on delivery times (for emails that didn't land in spam), we encountered an initial issue of roughly a 4-minute delay in two Gmail addresses. We initially thought this was the worst possible delivery time, but we were mistaken. Approximately 20 emails (to Yahoo and AOL) took about an hour to deliver, arguably worse than landing in spam.
Given these disappointing results, we decided to try other email delivery services, including Mailgun, Mailchimp, AWS SES, Resend, and Postmark. The story of our journey towards better email deliverability had just begun.
Due to our limited time capacity, we were unable to test all services in detail and had to conduct the testing in two rounds. In the first round, we carried out a high-level analysis of the email delivery service. This included assessing its compatibility with our solution and toolset, features, integrations, SDKs, CLIs, user-friendliness, and configuration setup, alongside the current experience of developers. Those services that passed the first round were then subjected to a spam test in the second round, using the same procedure as for SendGrid.
After the high-level analysis, we disqualified any service that did not meet our requirements, needs, and expectations.
The first email delivery service we eliminated is Mailgun. We decided not to test it due to poor user experiences during configuration and portal usage. We found the Mailgun portal to be user-unfriendly, and it occasionally displayed incomplete error messages. There was a situation when we could not log in because the Mailgun authorization service was not functioning. Among all the email delivery services we tested, Mailgun had the worst domain and DNS records configuration experience. Perhaps this is nitpicking, but the Mailgun portal doesn't implement the sending of DNS records via email to a colleague who holds the authority to set DNS records. The support for Mailgun's SDK appears to be lacking and immature. From our perspective, it notably lacks support for languages such as C#, Python, and Node.js. For these reasons, we did not proceed with further testing. Even if Mailgun passes the spam test, these reasons would still prevent us from integrating it into our product.
In our high-level analysis, we decided to cease further testing of Mailchimp. Mailchimp primarily focuses on marketing emails, with transactional emailing only offered as an add-on feature accessible with their Premium or Standard plans (starting at $19 for 500 contacts per verified domain). This add-on requires the purchase of email blocks, starting at $20/month for 25,000 emails, with the cost per block decreasing as volume increases. However, unused email quotas do not roll over to the next month. Furthermore, Mailchimp lacks a C# SDK, and we found its interface to be complex and difficult to navigate. However, it might be acceptable for existing Mailchimp customers. For our transactional email needs, Mailchimp seems like an overcomplicated solution.
While AWS SES is an affordable solution (around $30 with a dedicated IP), its lower cost may be reflected in its limited out-of-the-box features such as tracking, monitoring, deliverability metrics, and handling of spam complaints and bounced emails. This suggests that the price does mirror the feature set offered. It's important to note that while AWS SES does offer monitoring for sending activity and sender reputation, these are not out-of-the-box features and require effort to utilize. Furthermore, an AWS support plan would be necessary for customer service.
We prefer an email delivery service that includes robust out-of-the-box features and support. While this might be more expensive upfront, it could save time and effort in service setup and functionality in the long run.
Postmark has emerged as a promising alternative to SendGrid, impressing us with its commitment to maintaining a reputable service. When you sign up for Postmark, you can initially only send emails to the domain associated with your registration email. Furthermore, you cannot send emails to any address outside the added and verified domains until your account is approved. Postmark Admins manually review each new account to prevent their platform from being used to send spam or other reputation-damaging emails.
Our initial impressions of the portal and documentation were highly favorable. The Postmark environment is incredibly intuitive, enabling quick service setup, such as domain verification.
Postmark's pricing model is interesting, offering equal access to all features across different plans, with pricing dependent solely on the number of emails sent per month. This model doesn't differentiate between transactional or marketing emails like most of the others email delivery services. The trial version offers 100 emails per month, and the first paid version, priced at $15, provides 1000 emails per month. Additionally, a dedicated IP address can be purchased as an add-on (50$).
We were particularly impressed with the quality of customer support. When we needed to submit a larger monthly limit for our GlockApps spam test, we reached out to support. Postmark responded swiftly and positively, increasing the trial version limit to 500 emails without any issues.
Postmark sets itself apart from its competitors with its default data retention policy. It retains email content, metadata, and events (delivery, click, open) for 45 days, significantly longer than other services, which typically offer only a few days of retention.
Moreover, Postmark offers an official C# SDK and CLI. An interesting feature is the option to separate into 'servers', which can be used to segregate projects and provide specific statistics.
Our experience with Postmark was so satisfying that we decided to explore its service in greater depth.
In comparison to SendGrid, we conduct a spam test for verification email template using the trial version of Postmark without a dedicated IP address. Overall, Postmark achieved 15.3% improvement compared to SendGrid.
There has been a significant improvement in email delivery times. SendGrid delivered 36 emails in more than 5 seconds, while Postmark only took that long for 11 emails. However, we recorded four occurrences where Postmark delivered emails just over 10 minutes.
Postmark demonstrates a notable improvement over SendGrid, both in terms of overall rating and email delivery times.
Resend, a relatively new player in the field of email delivery services, debuted as an open-source platform in 2022 and joined Y Combinator's winter batch in 2023. Built on AWS SES, Resend stands out with its robust base, common email delivery functionalities, and competitive pricing model.
Its pricing model, at $20 per month for 50,000 emails, is comparable to other providers. A dedicated IP can be added for an extra $30 per month. Considering the potential costs of configuring and building email delivery functionalities on top of AWS SES, Resend may be a cost-effective choice. For Spotflow, which uses the email delivery provider mainly for transactional emails, the quality, manageability, and reputation of transactional email delivery are more valuable than the monthly cost, which varies by only a few dollars.
Resend's website and documentation left a positive impression on us, providing a well-made, user-friendly environment. It is evident that Resend is focused on providing a high-quality developer experience. However, Resend does not match its competitors in terms of features. Despite its active development, it lacks a templating tool.
Resend offers custom SSO capabilities for enterprise users, with support for popular identity providers such as Okta, and compatibility with both SAML and OIDC (OpenID Connect) protocols. Users on lower plans can use SSO/social login as well.
In terms of customer support, Resend is slower than Postmark. We experienced a slow process when requesting an increase in the email sending limit due to spam tests. Resend also falls short on data retention, offering only 3 days compared to Postmark's 45, and lacks an official C# SDK. However, we found the community C# SDK, which Resend recommends, to work well.
Resend also has fewer integration options than other services, but we found a well-written and functional SMTP integration guides for the common services.
This concludes our initial insights into Resend. Let's now explore how it performed in our test.
Similar to Postmark, we conducted a spam test for the verification email template using the trial version, without a dedicated IP address. Resend achieved an overall improvement of 26.3% compared to SendGrid and 11% compared to Postmark.
We've observed a notable improvement in email deliverability, with all emails being delivered within one minute. Crucially, the issue previously encountered with Gmail, where emails took up to four minutes to arrive, was not present.
Resend significantly outperformed SendGrid and even surpassed Postmark, which also reached good results.
The GlockApps Spam test offers a view of several spam filters, including Google Spam Filter, Barracuda, and Spam Assassin. However, the Google Spam Filter results weren't particularly useful for us as they don't provide a numerical value, just a status that remains the same across all tested ESPs. The results for Barracuda and Spam Assassin are interpreted as follows:
SendGrid reached the worst result with the Barracuda spam filter, where it was most likely to be considered spam. Postmark and Resend performed well on this filter, with Resend nearly scoring zero.
For Spam Assassin filter, Postmark had the best results. Resend and SendGrid had similar, relatively moderate scores.
After six months of using SendGrid, our frustrating experiences and poor results from spam tests led us to stop using their service. We found SendGrid underperformed significantly compared to Postmark and Resend, with an average delivery time of approximately 11.5 minutes.
Postmark and Resend were close competitors. Both left a positive technical impression on us and were user-friendly. From a developer's perspective, Resend was more appealing. In terms of integrations, SDKs, and features, Postmark was slightly ahead. While Postmark's support was excellent, Resend could improve in this area. Resend does not offer a templating tool. For pricing, Resend offers 50,000 emails for $20 per month, whereas Postmark provides 1,000 emails for $15 per month, making it the more expensive option for us.
We considered both Resend and Postmark. However, the deciding factor was the spam test results.
Resend outperformed in all aspects of the verification email template spam test. The median total email delivery time, unaffected by outliers or extreme values, was a strong indicator of its superiority. Based on our analysis and tests, we've decided to replace SendGrid with Resend in Spotflow.
As we close this chapter of our journey towards better email deliverability, our journey doesn't end here. At Spotflow, we've come across a new challenge related to the content of the emails themselves, which we're eager to explore in our upcoming post, Spotflow's Journey Towards Better Email Deliverability - Part 2. Stay tuned as we delve into our experience with Resend, including the technical challenges we faced during setup, our approach to delivery issues related to email content, and the solutions we devised.
