How to block Semalt Crawler – Explainer video
Text to Speech Voices
What is Semalt Crawler?
Are you terrified of Semalt Crawler visiting your website? Are you afraid that government is watching you? No more guessing! The truth is quite simple.
We have developed Semalt Crawler for effective web-statistics monitoring. The bots gather data that is later processed by Semalt analytical system. Statistics reports allow us to develop more effective SEO-strategy for our clients’ websites.
Do you suspect Semalt Crawler of messing with your statistics? Then remove you website from you database once and for all. Just click this link and follow the instructions: http://Semalt.net/
To block Semalt or to use our SEO-services and get to TOP Google search results — what will you choose?
Enter Semalt
A few months ago Incapsula saw the first indications of a large-scale referrer spam campaign. The focal point of this spam activity was a service named Semalt whose bots were employing referrer spam techniques on an impressive scale and were aggressive enough to draw our (and our clients’) attention.
Semalt links in access log.
On their website, Semalt describes their service as “… a professional webmaster analytics tool…“ However, a Google search for "Semalt" yields mostly negative comments, directed toward the company and its services.The comments originate from numerous users, many of whom complain about Semalt ignoring 'robots.txt' directives. We’ve also seen hundreds of people taking to Twitter, to call out Semalt for their questionable tactics as well as offer speculation about the company’s actual activity.
Probably the most antagonizing behavior of all is Semalt’s claims that you can complete an online form to remove your website from their crawling list. Still, instead of stopping the flood of unwanted requests, submitting the removal form seems to results in an increase of Semalt bot traffic.
One Soundfrost to Spam them All
It should be noted that Semalt is not your typical bot.
Our analysis shows that the company uses a QtWebKit browser engine to avoid detection. Consequently, Semalt bots can execute JavaScript and hold cookies, thereby enabling them to avoid common bot filtering methods (e.g., asking a bot to parse JavaScript). Because of their ability to execute JavaScript, these bots also appears in Google Analytics reports as being “human” traffic.
Recently, substantial evidence revealed that Semalt isn’t running a regular crawler. Instead, to generate bot traffic, the company appears to be using a botnet that is spread around by a malware, hidden a Soundfrost untillity.
Our data shows that, using this malware-infested utility, Semalt has already infected hundreds of thousands of computers to create a large botnet. This botnet has been incorporated in Semalt’s referrer spam campaign and, quite possibly, several other malicious activities.
Soundfrost botnet - 290,000 unique IPs recorded over the last 30 days.
To put things in numbers, during the last 30 days we saw Semalt bots attempting to access over 32% of all websites on our service with spamming attempts originating over 290,000 different IP addresses around the globe.
As evidenced by the IP distribution data above, Semalt’s botnet is quite widespread, with most of the affected IPs located in South America.
Beyond providing Semalt with the scale it needs to operate, this botnet also help Semalt’s bots avoid rudimentary security practices, such as IP blacklisting and rate-limiting. Coupled with its ability to overcome challenge-based detection mechanisms, this makes Semalt’s shady activity that much more concerning.
source
Gloss