In our previous article, we evaluated several email delivery providers/services (ESPs) in response to the deliverability issues we experienced with SendGrid. Our assessment was twofold. Initially, we conducted a high-level analysis and firsthand user experiences with various providers, including Mailgun, Mailchimp, AWS SES, Resend, and Postmark. Subsequently, based on the results from the first stage, we selected Resend and Postmark for spam testing. The primary concern that prompted this evaluation was the delayed delivery of verification emails.
Resend emerged as the most effective solution. It significantly improved email delivery and reduced delivery times compared to the other services. Despite some limitations, Resend's performance and user-friendly environment convinced us to replace SendGrid.
We’ll continue with our story about the delay of our verification email for the Spotflow IoT Platform. First, we'll share the first steps with Resend and the challenges we faced during the setup. Next, we'll dive into our issues with email deliverability related to the email content and how we proceeded to solve the problem.
Our initial steps with Resend involved creating an official Spotflow account and selecting a subscription plan that fits our needs. Our current intention is to use Resend solely for transactional emails. The Free plan isn't suitable for us, as it only offers one domain within a single region and doesn't provide the option to purchase a dedicated IP address. But do we really need a dedicated IP address? Our experience with SendGrid showed worse spam test results despite having a dedicated IP compared to other email delivery services with shared IP addresses.
Given our requirement to use two domains in Europe (production and test/development), we must choose the Pro plan, priced at $20 per month. This plan accommodates up to 50,000 emails per month, offers unlimited domain verifications, and supports multi-region email sending capabilities. However, we still need to research whether or not we should use a dedicated IP address.
Our research revealed that dedicated IP addresses, despite seeming exclusive, can transfer the responsibility for maintaining a good email reputation from Email Service Providers (ESPs) to customers. This means that if customers do not manage their email-sending practices carefully, they could face issues like high bounce rates, spam complaints, and blacklisting. Problems may arise if customers send too many emails too quickly, target only a limited number of mailbox providers (Gmail, Hotmail, etc.), or fail to follow best practices for email content.
Furthermore, new dedicated IPs can be problematic as they have no established reputation, which is vital for mailbox providers to trust email traffic. Mailbox providers and blacklists also monitor entire IP ranges and domains, meaning issues with one IP can impact the entire subnet or domain. Additionally, mailbox providers are now focusing more on domain reputation rather than IP reputation, making domain authentication standards like DKIM increasingly important.
Dedicated IP addresses are mainly beneficial for extremely high-volume senders (generally hundreds of thousands of emails per month). They can help avoid throttling at mailbox providers but require a large volume of emails to all major mailbox providers to maintain a good reputation.
For low-volume senders, shared IP addresses typically offer a more reliable option for maintaining good email deliverability rates. Consequently, selecting a reputable Email Service Provider (ESP) that meticulously monitors its shared IP addresses is vital for ensuring optimal performance and consistent deliverability.
Based on these findings and Resend spam test results with shared IPs, we decided to opt for shared IP addresses for our transactional emails.
After successfully upgrading Resend to the production level, we decided to reassess its quality by repeating the spam test from the previous article. In the overall results, Resend showed a slight improvement of 0.9% compared to the previous test.
However, we encountered an unpleasant surprise when examining the email delivery time. The problem we had previously observed reappeared. During the test, an email was delivered to Gmail after 4 minutes, an issue not seen in the previous test.
Examining the basic statistics, we see that the average delivery time has increased slightly. However, the median delivery time has remained the same. The maximum delay observed was the previously mentioned 4 minutes.
Despite improvements after switching to Resend, we are facing a delay of over 4 minutes in email delivery to Gmail. This issue existed with SendGrid as well but Resend generally does not exhibit further email delivery delays. This latency affects Spotflow's free trial and user registration, potentially discouraging new clients. Therefore, it's crucial that we address this problem.
As a first step, we decided to analyze email message headers via Google Admin Toolbox Message Analyzer. This email header analyzer inspects email header fields and displays important information about the email message, including details related to the source, destination, and forwarding email servers. From the headers analyzes, it's evident that Google routes our email through its servers multiple times.
The last server, for some reason, processes our email for four minutes. That server seems to have a special purpose. Emails that are delivered without delay pass through only three servers.
After conducting brief internet research, we identified a probable cause for our issue (see details here). Gmail has introduced a security feature for early detection of phishing attempts. This is done using an algorithm that flags and delays potentially suspicious messages, allowing for additional checks on the content. Emails can be delayed by up to 4 minutes.
The issue appears to lie in the content of the verification email. One potential cause could be the email's subject. We use Auth0 to send the verification email, where we have used the default subject "Verify your email" for the Auth0 Verification email template. Other companies might also use this approach, and as a result, we could be flagged by Gmail/Google as a suspicious sender.
Another potential issue could be the Auth0 symbol used for email verification. It's a checkmark inside a white circle on a black background, obtained from the following URL via HTTP: http://cdn.auth0.com/website/emails/product/top-verify.png.
We attempted to manually send emails with various modifications (e.g., different subjects, with or without the checkmark link) to the problematic Gmail accounts. However, the results were inconsistent. Sometimes, even with the problematic email components, the emails didn't end up in the phishing quarantine.
Google's criteria for selecting emails for phishing quarantine are unclear and seem to depend on unpublic factors. We suspect that our original email verification template is already flagged by a particular algorithm as potentially phishing, and minor changes may not be effective.
Therefore, we decided to overhaul the template entirely, modify the email subject, and exclude non-https links. The original template contained about 570 lines, while the new template has only around 130. It's important to note that we're unsure if the template code's length impacts the algorithm in question.
We conducted a spam test on the new email verification template, just as we did with the original one. This resulted in a decrease in the number of missing emails (the percentage of messages not delivered to the seed email addresses).
The most important aspect of this test is email delivery times. With the new template, we have not observed any emails delivered later than one minute.
The maximum delivery time is 7 seconds, while the average time is around one second.
We also manually tested the new template on problematic Gmail accounts and didn't experience any issues with email delivery. The implemented changes to the email verification template appear to have resolved the email delivery issue. While we can't make a definitive conclusion yet, we haven't observed any problems with Gmail email delivery over the past month.
We decided to revisit SendGrid and evaluate its performance using the GlockApps spam test for the new verification email template, comparing it to Resend. We used brand new GlockApps seed email addresses for this test, so the results cannot be directly compared with the previous spam tests.
Overall, Resend showed an 8.4% improvement over SendGrid. Just to remind you, when referring to ‘tabs,' this means the tabs other than the Primary Inbox, such as Gmail's Promotions tab.
When comparing the delivery times of emails for both ESPs, Resend also performed better. None of Resend's emails were delivered later than one minute, while SendGrid took over five minutes to deliver seven emails.
Resend also outperformed SendGrid in the last comparison, achieving superior results in average, median, and maximum email delivery times.
Resend surpassed SendGrid in every aspect during the recent verification email template spam test. This outcome further supports our decision to replace SendGrid with Resend.
It's important to note that we didn't conduct the spam test for Postmark, despite its potential for a relevant comparison. However, we had already ruled out Postmark based on prior spam tests. In this spam test, our primary aim was to compare our past email delivery provider with the current one.
Our transition from SendGrid to Resend for email delivery services has been successful. Resend has shown significant improvements in email delivery times and spam test results over SendGrid. We've also been able to address a persistent issue with Gmail delivery times by revising our verification email template. With these changes, we've seen a substantial increase in delivery efficiency, which is crucial for our user registration process.
Spotflow learned that selecting an email service provider (ESP) based solely on popularity is not ideal. It is crucial to engage in high-level analysis and testing, as these efforts pay off in the long run. Even less established ESPs can show good results and may leverage more modern technologies.
Additionally, we discovered that having a dedicated IP address for the email stream does not necessarily enhance email deliverability. Deliverability is influenced not only by the choice of ESP but also by factors such as email content. We hope our experience provides valuable insights for others facing similar email delivery challenges.
