The spammer that was able to send 4 million emails
What happened?
Last weekend someone was able to sign-up for a Prezly trial and send out over 4 million emails unnoticed. I mean, we did notice it after the weekend but then it was too late.
This will increase our SendGrid bill but that's not the worst. It also affects our sending reputation which is slowly recovering after the hotfix on Monday.
TLDR: The 'send a test email' functionality allowed comma-separated multi-send and was hijacked to send out spam (with [TEST] prefix).
How could this happen?
We have a number of limits in place for trial accounts: Any account that has not entered their credit card details can not interact with DNS (newsroom domain, DKIM/SPF sender domain verification) and there are time-based volume limits in place with regard to email sending.
The truth is that we didn't build these limits for trial accounts but rather for some high-usage customers and found out later that the same validation should be applied to trials.
This weekend on Saturday night we had a ton of incoming trial requests with similar email addresses that looked sketchy. As we've become more confident in the automatic enforcement of account limits we don't manually check logs/verify account activity (anymore).
These trials hijacked the 'send a test email' functionality in the email composer that allows customers to enter a comma-separated list of addresses to send a few people a sample email campaign. The email subject is prefixed with [TEST] - though.
The spammers abused this feature to enter a very long string of email addresses (up to 10k email addresses per submission) and do that over and over.
After doing more research about what exactly went down I believe later some automation was used where we were seeing multiple accounts exploit the same vulnerability at the same time.
4 million emails. Sigh...
Patching the hole
After Jesse noticed something weird happened over the weekend it took a few lines of code to add additional validation to prevent this from happening. But as we didn't want to remove that functionality we decided that trial users can only send test emails to their own email addresses.
We're aware this is not the perfect solution and some of the higher-quality trial accounts might be annoyed by this limitation.
In the long-term, I think we'll implement some kind of scoring system where those limitations are applied based on some (automated or manual) qualification/risk score.
What else could go wrong?
Taking a step back from this specific case I took some time to think about other potential ways a trial account could harm our system or significantly increase our cost for running Prezly.
Here is what I came up with:
- Use our newsroom/CDN to host files
- Overload the system by importing huge/corrupt contact files
- Use the API to create a huge amount of content increasing our Algolia charges
- Trigger a shit-ton of `identify()' calls to increase volume-based MTU billing in for example segment.com
Although I am aware that only scenario 1. will provide value to the attacker with more incentives than the other more malicious scenarios, I think this is something to be aware of.
In Scenario 1) we do have some kind of measures where we clean up old/sleeping accounts and thus there would be no long-term benefits of using Prezly as trustworthy file storage.
Going Forward
This case has reminded me again that we should probably do even more to protect our customers and reputation. And while I want to make sure that good leads/PR agencies and brands that sign up for a Prezly trial have an awesome first experience. That experience is impacted by some of the limitations we put in place to protect ourselves from bad actors.
Some ideas and things I will be exploring in the next few weeks. For some of the larger ideas, I will create a Shape-Up/Scoping document and pitch them in the next betting session.
Use our Trial Qualification
We are already manually qualifying all incoming trials in the Prezly app. This allows us to run other automation and playbooks based on the quality of the lead.
As our sales team is manually going through those leads, connecting with them on LinkedIn, and figuring out if this is a real company vs a bot, I think this is a good indicator of how much we can trust those accounts.
We could keep all the trial limitations (and the extra limitation we built in for this spammer) in place for low/unqualified accounts where we get out of the way for trials that were manually marked as having higher potential.
Improve monitoring & Alerts
We only send automated alerts (SMS/slack messages/phone calls) if there are reliability issues with our systems. An increase in error rates, failure to connect with backstream systems, and issues with load or pods crashing.
But we don't proactively monitor and alert customer behaviour. This means that creating 100k stories or importing 5 million contacts would probably go unnoticed unless it would trigger any reliability alerts due to a slower database or the application being unable to keep up.
But because we have some components that auto-scale the damage might already be done before we even notice it.
Going forward I think we need to proactively monitor total system usage with things like:
- Number of emails sent
- Number of stories published
- Number of newsrooms created
- Amount of domains verifications failed
- Total amount/size of uploads on CDN
- Contact import records being processed
After we have some baseline metrics we could set up certain thresholds and alert the right people if the number is exceeding 'normal' app usage. It would probably have helped a lot in noticing this issue a little earlier.
Isolate the email reputation for Trial Accounts
Almost all customers (outside of some Enterprise customers) share the same IPs and email-sending infrastructure. We do a lot to protect our reputation:
- Warning customers of increased spam rates
- Forcing unsubscribes. No way to revert a recipient's unsubscribe
- Dropping soft bounces for future campaigns
- Discouraging 'email all' behaviour
- Suspending accounts that smell like spam
This enabled us to keep our sending reputation high as landing in the inbox is one of the key concerns for new customers trying our product.
But every software has bugs or oversights. And I think it's safe to say there will be similar situations in the future.
So wouldn't it be good to isolate the sending behaviour of trial accounts, and maybe for newly joined customers on lower plans, to their own sending IPs and email reputation?
Ask for Credit Card
Some of my favourite SaaS companies ask for a credit card during the sign-up process and talked about it (Nathan Barry/Convertkit). I am pretty confident requiring a CC would do what you'd expect:
- Lower conversion rate
- Increase usage for trials
- More efficient funnels for trials
And although this would probably discourage bad actors from trying to exploit Prezly it comes with a price. Because now you have fewer trials together with less information on what people do when they are first activating their account.
So I'm not sure about credit cards yet. And this discussion on Indiehackers proves that I'm not the only one with that concern.
In an ideal world, I would not have to spend time on this and focus on building an awesome product and company. Yesterday I learned about Heroku shutting down their free plans stating 'abuse' as the main reason. So I'm sure we're not the only ones struggling with this.