verification email delivery illustration

Spotflow’s journey towards better email deliverability - part 1

Facing delays in email verification delivery can lead to a significant loss of potential customers. In this article, we will share our journey of troubleshooting these delays and evaluating the performance of well-known email service providers like SendGrid, Postmark, and Resend. Discover how our email deliverability improved as we transitioned to a new provider.
Michael Mikus, Software Engineer at Spotflow
Michael Mikus
Software Engineer
Published on:
July 10, 2024
Tech
Join the waitlist!
Be the first to experience next-gen monitoring platform for embedded devices.
You're on the waitlist!
Oops! Something went wrong while submitting the form.
Join the waitlist!
Be the first to experience next-gen monitoring platform for embedded devices.
You're on the waitlist!
Oops! Something went wrong while submitting the form.

In This Article

We'll delve into a Spotflow story about a delay in the delivery of verification emails during the registration process for our product. Spotflow provides a developer-friendly, self-service IoT platform that collects and processes data, simplifying IoT device management and solution development.

Self-service of the platform is a very important aspect of our product from the very beginning when a potential customer can try the platform with a free trial. The free trial allows them first-hand experience with the platform and influences their decision to choose the product.

However, we experienced delays in delivering verification emails during the registration process, an integral part of the trial. To address this, we evaluated our existing email delivery solution, took steps to rectify the issue, and considered whether transitioning to a new email service provider was necessary.

Email Deliverability Issue

The journey started with a pressing issue. We were experiencing around a 4-minute delay with our verification emails. We noticed this delay, specifically with emails sent to Gmail. However, it was unclear whether this delay also applied to emails sent to other mailbox providers such as Yahoo, Hotmail, etc. This was more than just an inconvenience. It was a critical problem that was preventing users from completing their registration to our platform, leading to a loss of potential customers.

Our initial suspicion fell on our current service provider, SendGrid, and the reputation of the dedicated IP address we were using. Our dissatisfaction with SendGrid had been simmering for a while, and this issue seemed like the final straw. Consequently, we have opted to conduct a thorough examination of the issue and rigorously test our solution to ensure that our decisions are grounded in empirical data.

The Original Spotflow Verification Email Template

Delivery Testing Tool

Our testing criteria were straightforward. We aimed to distribute the verification email to a vast array of distinct email addresses and determine whether and when the emails were delivered. Numerous independent tools for testing email deliverability are available on the market (this analysis is beyond the scope of this blog post). However, GlockApps proved to be the most compatible with our needs. GlockApps provides Spam Test packages, eliminating the need for us to purchase any subscriptions. We selected a package that included 10 Spam tests, which we anticipated would suffice for our comprehensive testing.

It’s worth noting that the GlockApps portal is quite user-friendly, and the test is relatively simple. The spam test is conducted using a seed list of GlockApps’s test email addresses set up at different mailbox providers and regions. The next step is to send the email to the seed email addresses. The email to be tested must contain a unique GlockApps identifier, which can be placed anywhere within the email content. This ID allows the tool to discern which email is associated with the concrete spam test.  For the tests themselves, we created a C# script that sends our verification email to all the seed emails via the email delivery services we selected.

Evaluating SendGrid

Before discussing our experience testing our solution with SendGrid, it's important to understand Spotflow's history with this tool. We initially selected SendGrid, a well-known Email Service Provider (ESP), believing it to be a reliable choice. However, we encountered several issues during setup, resulting from underdeveloped features.

Our first concern arose during the setup process. We required separation between our development/testing and production environments. SendGrid uses 'Subuser accounts' derived from a 'Parent account' to organize and manage different email streams. The parent account isn't intended to send emails, which are left to the subuser accounts. This division is suggested for separating transactional and marketing emails as well. However, we found this architecture clumsy and the naming convention inappropriate for concepts related to the separation of environments or transactional/marketing email streams.

We also struggled to understand SendGrid's approach to two-factor authentication (2FA). It offers 2FA only via SMS or Authy, an app owned by SendGrid's parent company, Twilio. We were not satisfied with either approach. Using SMS (or voice over GSM) for 2FA is widely considered insecure. Authy, although a popular 2FA application, needs to be associated with a phone number, which opens unnecessary attack vectors. Security keys, passkeys, or at least generic TOTP are not supported.  But security is not the only issue associated with SendGrid's strange 2FA strategy. In combination with the fact that there might only be a single administrator account, there is no way to provide redundant administrator access to SendGrid because the administrators would be forced to share a phone number, which is highly impractical. Fortunately, it is possible to add additional user accounts with an almost identical set of permissions.

Another unexpected limitation was that SendGrid only supports custom Single Sign-On (SSO) via SAML for identity platforms such as Okta and Azure AD. It does not offer SSO/social login through common social accounts like Google, Microsoft, or GitHub.

Despite our dissatisfaction with these aspects of SendGrid, we chose to continue using it as it fulfilled its primary function. However, we stopped tolerating SendGrid when we encountered email delivery problems. Let's examine SendGrid's spam test results for our verification email template.

Spam Test Results

Before we delve into the spam test results, it's important to mention that we use SendGrid with a dedicated IP address. Moreover, our DNS records for DKIM, SPF, DMARC, and reverse DNS on the IP address are verified.

After conducting the spam test for the verification email template (see picture above), we obtained the following results on SendGrid.

SendGrid Spam Test Overview Results
Inbox + tabs Inbox Tab Spam Missing
50.8% 59 1 57 1

To clarify, when referring to 'tabs', this means the tabs other than the Primary Inbox, such as Gmail's Promotions tab.

Surprisingly, about half of the emails ended up in the spam folder. We didn't anticipate such poor results with a dedicated IP, which we believed would bolster our reputation.

Focusing on delivery times (for emails that didn't land in spam), we encountered an initial issue of roughly a 4-minute delay in two Gmail addresses. We initially thought this was the worst possible delivery time, but we were mistaken. Approximately 20 emails (to Yahoo and AOL) took about an hour to deliver, arguably worse than landing in spam.

SendGrid Spam Test Delivery Time Results
Delivery Time 0s - 5s 6s - 59s 1 min - 4:59 min 5 min <
# 78 14 2 20

Given these disappointing results, we decided to try other email delivery services, including Mailgun, Mailchimp, AWS SES, Resend, and Postmark. The story of our journey towards better email deliverability had just begun.

Evaluation of Email Delivery Services

Due to our limited time capacity, we were unable to test all services in detail and had to conduct the testing in two rounds. In the first round, we carried out a high-level analysis of the email delivery service. This included assessing its compatibility with our solution and toolset, features, integrations, SDKs, CLIs, user-friendliness, and configuration setup, alongside the current experience of developers. Those services that passed the first round were then subjected to a spam test in the second round, using the same procedure as for SendGrid.

Eliminated Email Delivery Services

After the high-level analysis, we disqualified any service that did not meet our requirements, needs, and expectations.

Mailgun

The first email delivery service we eliminated is Mailgun. We decided not to test it due to poor user experiences during configuration and portal usage. We found the Mailgun portal to be user-unfriendly, and it occasionally displayed incomplete error messages. There was a situation when we could not log in because the Mailgun authorization service was not functioning. Among all the email delivery services we tested, Mailgun had the worst domain and DNS records configuration experience. Perhaps this is nitpicking, but the Mailgun portal doesn't implement the sending of DNS records via email to a colleague who holds the authority to set DNS records. The support for Mailgun's SDK appears to be lacking and immature. From our perspective, it notably lacks support for languages such as C#, Python, and Node.js. For these reasons, we did not proceed with further testing. Even if Mailgun passes the spam test, these reasons would still prevent us from integrating it into our product.

Mailchimp

In our high-level analysis, we decided to cease further testing of Mailchimp. Mailchimp primarily focuses on marketing emails, with transactional emailing only offered as an add-on feature accessible with their Premium or Standard plans (starting at $19 for 500 contacts per verified domain). This add-on requires the purchase of email blocks, starting at $20/month for 25,000 emails, with the cost per block decreasing as volume increases. However, unused email quotas do not roll over to the next month. Furthermore, Mailchimp lacks a C# SDK, and we found its interface to be complex and difficult to navigate. However, it might be acceptable for existing Mailchimp customers. For our transactional email needs, Mailchimp seems like an overcomplicated solution.

AWS SES

While AWS SES is an affordable solution (around $30 with a dedicated IP), its lower cost may be reflected in its limited out-of-the-box features such as tracking, monitoring, deliverability metrics, and handling of spam complaints and bounced emails. This suggests that the price does mirror the feature set offered. It's important to note that while AWS SES does offer monitoring for sending activity and sender reputation, these are not out-of-the-box features and require effort to utilize. Furthermore, an AWS support plan would be necessary for customer service.

We prefer an email delivery service that includes robust out-of-the-box features and support. While this might be more expensive upfront, it could save time and effort in service setup and functionality in the long run.

Evaluating Postmark

Postmark has emerged as a promising alternative to SendGrid, impressing us with its commitment to maintaining a reputable service. When you sign up for Postmark, you can initially only send emails to the domain associated with your registration email. Furthermore, you cannot send emails to any address outside the added and verified domains until your account is approved. Postmark Admins manually review each new account to prevent their platform from being used to send spam or other reputation-damaging emails.

Our initial impressions of the portal and documentation were highly favorable. The Postmark environment is incredibly intuitive, enabling quick service setup, such as domain verification.

Postmark's pricing model is interesting, offering equal access to all features across different plans, with pricing dependent solely on the number of emails sent per month. This model doesn't differentiate between transactional or marketing emails like most of the others email delivery services. The trial version offers 100 emails per month, and the first paid version, priced at $15, provides 1000 emails per month. Additionally, a dedicated IP address can be purchased as an add-on (50$).

We were particularly impressed with the quality of customer support. When we needed to submit a larger monthly limit for our GlockApps spam test, we reached out to support. Postmark responded swiftly and positively, increasing the trial version limit to 500 emails without any issues.

Postmark sets itself apart from its competitors with its default data retention policy. It retains email content, metadata, and events (delivery, click, open) for 45 days, significantly longer than other services, which typically offer only a few days of retention.

Moreover, Postmark offers an official C# SDK and CLI. An interesting feature is the option to separate into 'servers', which can be used to segregate projects and provide specific statistics.

Our experience with Postmark was so satisfying that we decided to explore its service in greater depth.

Spam Test Results

In comparison to SendGrid, we conduct a spam test for verification email template using the trial version of Postmark without a dedicated IP address. Overall, Postmark achieved 15.3% improvement compared to SendGrid.

Postmark Spam Test Overview Results
Inbox + tabs Inbox Tab Spam Missing
66.1% 77 1 39 1

There has been a significant improvement in email delivery times. SendGrid delivered 36 emails in more than 5 seconds, while Postmark only took that long for 11 emails. However, we recorded four occurrences where Postmark delivered emails just over 10 minutes.

Postmark Spam Test Delivery Time Results
Delivery Time 0s - 5s 6s - 59s 1 min - 4:59 min 5 min <
# 103 7 0 4

Postmark demonstrates a notable improvement over SendGrid, both in terms of overall rating and email delivery times.

Evaluating Resend

Resend, a relatively new player in the field of email delivery services, debuted as an open-source platform in 2022 and joined Y Combinator's winter batch in 2023. Built on AWS SES, Resend stands out with its robust base, common email delivery functionalities, and competitive pricing model.

Its pricing model, at $20 per month for 50,000 emails, is comparable to other providers. A dedicated IP can be added for an extra $30 per month. Considering the potential costs of configuring and building email delivery functionalities on top of AWS SES, Resend may be a cost-effective choice. For Spotflow, which uses the email delivery provider mainly for transactional emails, the quality, manageability, and reputation of transactional email delivery are more valuable than the monthly cost, which varies by only a few dollars.

Resend's website and documentation left a positive impression on us, providing a well-made, user-friendly environment. It is evident that Resend is focused on providing a high-quality developer experience. However, Resend does not match its competitors in terms of features. Despite its active development, it lacks a templating tool.

Resend offers custom SSO capabilities for enterprise users, with support for popular identity providers such as Okta, and compatibility with both SAML and OIDC (OpenID Connect) protocols. Users on lower plans can use SSO/social login as well.

Resend portal

In terms of customer support, Resend is slower than Postmark. We experienced a slow process when requesting an increase in the email sending limit due to spam tests. Resend also falls short on data retention, offering only 3 days compared to Postmark's 45, and lacks an official C# SDK. However, we found the community C# SDK, which Resend recommends, to work well.

Resend also has fewer integration options than other services, but we found a well-written and functional SMTP integration guides for the common services.

This concludes our initial insights into Resend. Let's now explore how it performed in our test.

Spam Test Results

Similar to Postmark, we conducted a spam test for the verification email template using the trial version, without a dedicated IP address. Resend achieved an overall improvement of 26.3% compared to SendGrid and 11% compared to Postmark.

Resend Spam Test Overview Results
Inbox + tabs Inbox Tab Spam Missing
77.1% 90 1 24 3

We've observed a notable improvement in email deliverability, with all emails being delivered within one minute. Crucially, the issue previously encountered with Gmail, where emails took up to four minutes to arrive, was not present.

Resend Spam Test Delivery Time Results
Delivery Time 0s - 5s 6s - 59s 1 min - 4:59 min 5 min <
# 108 4 0 0

Resend significantly outperformed SendGrid and even surpassed Postmark, which also reached good results.

Brief Summary of Spam Filter Results

The GlockApps Spam test offers a view of several spam filters, including Google Spam Filter, Barracuda, and Spam Assassin. However, the Google Spam Filter results weren't particularly useful for us as they don't provide a numerical value, just a status that remains the same across all tested ESPs. The results for Barracuda and Spam Assassin are interpreted as follows:

  • Barracuda score: This score ranges from 0 (definitely not spam) to 10 or higher (definitely spam).
  • Spam Assassin score: Spam Assassin scores typically vary from -10 to 10, with lower scores indicating a lower likelihood of spam and higher scores indicating a higher likelihood.
Spam Filters Results
SendGrid Postmark Resend
Google Spam Filter Not Spam/Phishy Not Spam/Phishy Not Spam/Phishy
Barracuda 8.00 1.12 0.20
Spam Assasin -0.5 -3.5 -0.5

SendGrid reached the worst result with the Barracuda spam filter, where it was most likely to be considered spam. Postmark and Resend performed well on this filter, with Resend nearly scoring zero.

For Spam Assassin filter, Postmark had the best results. Resend and SendGrid had similar, relatively moderate scores.

Final Decision

After six months of using SendGrid, our frustrating experiences and poor results from spam tests led us to stop using their service. We found SendGrid underperformed significantly compared to Postmark and Resend, with an average delivery time of approximately 11.5 minutes.

Postmark and Resend were close competitors. Both left a positive technical impression on us and were user-friendly. From a developer's perspective, Resend was more appealing. In terms of integrations, SDKs, and features, Postmark was slightly ahead. While Postmark's support was excellent, Resend could improve in this area. Resend does not offer a templating tool. For pricing, Resend offers 50,000 emails for $20 per month, whereas Postmark provides 1,000 emails for $15 per month, making it the more expensive option for us.

We considered both Resend and Postmark. However, the deciding factor was the spam test results.

Spam Test Results Overview
SendGrid Postmark Resend
Inbox + tabs 50.8% 66.1% 77.1%
Average delivery time (s) 689.65 s (~11m:30s) 22.09 s 1.83 s
Median delivery time (s) 3.0 s 2.0 s 1.0 s
Max delivery time (s) 6897 s (1h:54m:57s) 607 s 11 s

Resend outperformed in all aspects of the verification email template spam test. The median total email delivery time, unaffected by outliers or extreme values, was a strong indicator of its superiority. Based on our analysis and tests, we've decided to replace SendGrid with Resend in Spotflow.

As we close this chapter of our journey towards better email deliverability, our journey doesn't end here. At Spotflow, we've come across a new challenge related to the content of the emails themselves, which we're eager to explore in our upcoming post, Spotflow's Journey Towards Better Email Deliverability - Part 2. Stay tuned as we delve into our experience with Resend, including the technical challenges we faced during setup, our approach to delivery issues related to email content, and the solutions we devised.

Join the waitlist!
Be the first to experience next-gen monitoring platform for embedded devices.
You're on the waitlist!
Oops! Something went wrong while submitting the form.
Join the waitlist!
Be the first to experience next-gen monitoring platform for embedded devices.
You're on the waitlist!
Oops! Something went wrong while submitting the form.

About Author

Michael Mikus, Software Engineer at Spotflow

Michael Mikus

Software Engineer
With several years of industry experience, Michael has been instrumental in developing data processing solutions for agricultural robots and other IoT devices using cloud technologies. This has laid a solid foundation for his expertise in co-creating an IoT platform. Michael's professional journey spans various facets of software engineering, specializing in back-end technologies. He thrives on tackling technological challenges and is enthusiastic about sharing the valuable insights he acquires. His contributions to this blog offer a comprehensive exploration of diverse subjects, primarily emphasizing software development.
right arrow icon
Back to blog post list
Tech

Read our Other Latest Blog Posts

Get real-time insight into your fleet today!

Start Free Trial
Observe your devices with Spotflow for 14 days. No strings attached.