Skip to main content

Spam form sporm fire

Fire description below.

How are we now blocking SPORM?​

Sporm is spam forms...

We have several approaches as there are several different types:

Our main thing is a system of monitoring all urls entered in saved powrmails. If a url is used in 7 or more powrmails, it notifies the #formspam channel of slack and a human can verify if it is legit or not. If it is not, we mark it as sporm in the admin dashboard https://www.powr.io/admin/powrmail_url_status and our system blocks all apps owned by a user who saved a powrmail with that url.

Apps blocked in this way are NOT notified. The malicious users continue to make more (which are automatically blocked if they use same url) and when they submit, their browser gets a 200 response, but no app form response is created or email sent.

This new system is put in place after we identify several hundred apps linking to a malicious site. we've only found 4 and we are AUTO blocking these apps right away before it even gets to the sporm system.

These urls are any made with the gg.gg, u.to, and v.ht link shorteners, or any one that uses the string "datingg" (note the 2 g's)... these 4 strings are responsible for several thousand spam apps, half sending pornographic photos and invitations to people, the others are a mix of financial scams, and more traditional phishing emails.

Fire Description​

Friday around 4pm, US team identified some users posting responses to a spam form generating some porn/finance scam emails. This came up in the existing jerky spammer notification on formspam channel. We followed the standard procedure to dephish user and block sparkpost account.

While debugging we noticed that there were several other apps that were not caught in this. These were sending several app form responses to various apps from a same ip from various ips.

To fix this, we decided to "block" users from submitting multiple responses within 30 seconds from same ip, ping the formspam channel and if identified as bad form, add to the list of BLOCKED_APP_IDS in heroku which will also block the response from creation. We send a 200 back to the spammer is not aware of the block. https://gitlab.com/powr/powr/-/merge_requests/4234 https://gitlab.com/powr/powr/-/merge_requests/4237

On Friday, Saturday and Sunday, Brent and Praneeta added identified spams to the heroku env. There were several other forms which were getting flagged which were legit.

On Sunday US night time, which is KZ Monday morning, the formspam channel was hit again.

Logs from there: Somebody started spamming powr (sparkpost bounce rate 9k).

5.15pm Aidana noticed spam attack at #formspamChannel 5.25pm Sergey come up with idea of blocking all submissions which could be potential_spam? => blocks all new free users who sends more than 50 form submissions, that are received by slack 5.34pm Sergey created branch https://gitlab.com/powr/powr/-/merge_requests/4240 to stop spams code stops sending emails but keeps posting in slack (since it happens before pushing to sparkpost server) 6pm branch is deployed and bounce rate goes down 6.14 Yerassyl finds better way of filtering spammers 6.28pm new branch by Sergey: https://gitlab.com/powr/powr/-/merge_requests/4242 => spam SPAM_FORM_LIMIT is based on heroku - so that we don’t push changes on numbers every time -> number of responses decreased from 50 to 30 (heroku vars SPAM_FORM_LIMIT) 6.29pm Form spammers stopped, sparkpost bounce rate 40

Monday morning: We reverted https://gitlab.com/powr/powr/-/merge_requests/4240 - because we don't want to block potential legit forms. We noticed we are not really blocking apps from the heroku env var. We pushed another change to fix that and block only submissions from same ip if the user has powrmail, and only has the email field. https://gitlab.com/powr/powr/-/merge_requests/4244 (30.seconds, was the next fix)

afternoon: We pushed a change to move the BLOCKED IPS to db - https://gitlab.com/powr/powr/-/merge_requests/4245 Now we can block apps through tsh too, will block

evening - we noticed that all the apps had pattern - datingg. and gg.gg. We ran a check and found 647 apps. 3:55pm We got hit again and we had around 1018 apps that were spam.

Meanwhile, heroku went down so we could not use rails c, so Ivan manually added these to the db.

Also, heroku went down so POWr was down.

We also pushed a rake task which will run every 10 mins to find apps which have spam url patter in powrmail and add them to block list. https://gitlab.com/powr/powr/-/merge_requests/4247

Current plan​

  • if there are individual spam instances, mark them as blocklisted through TSH
  • if a pattern of urls - add to the rake task from https://gitlab.com/powr/powr/-/merge_requests/4247
  • do not block or delete the app / user - we should send 200 so the spammer doesn't know we are blocking them.

Future plan, create a list of all links in powrmail and have a running list of spam urls. will block emails which has the urls and if urls appear in multiple place, we have someone verify if it is legit. Any urls in the spam emails should also be verified.

This is being spec'd in this doc- https://docs.google.com/document/d/1CmRG3Vu0VsQoqkCSKUTzLX_5VOwGjPm9gkozW1u1sM4/edit#heading=h.vlqv3xdrhq8z

We identified this list of issues we have - https://docs.google.com/document/d/14L940mZV7nxJII62Nl6hKbZwrGuWcAEqLC2KVSLuQP0/edit

List of apps https://docs.google.com/spreadsheets/d/14SfVTGmUXmn-Xxvzqje059nv_uy-xj9R9AZc1rE0Ywo/edit#gid=0

Feb 7 Spammers hit again- updated existing apps, not create - https://gitlab.com/powr/powr/-/merge_requests/4274 - was pushed as fix Also hardcoded block "datingg and gg.gg" on save https://gitlab.com/powr/powr/-/merge_requests/4275

Puru created endpoint for wix to get list of blocklisted wix instance ids - https://gitlab.com/powr/powr/-/merge_requests/4276

https://gitlab.com/powr/powr/-/merge_requests/4262 - was pushed to record recurring urls, alert form spam channel. If a url is marked spam all the associated users are "Blocklisted"

Feb 8 Spammers are trying different urls and are spoofing IKEA There are 2 new emails they have created - One is spoofing IKEA(http://archive.sendpulse.com/u/NzMxMDQyNA==/12tc3/) and the other one is the lady in the fields (https://meyerweb.com/eric/tools/dencoder/)