What Is Semalt – a Crawler
Semalt’s crawler is a bot generated by a website analytics company based out of Ukraine. There is no way to explain this without making this Semalt company look like… terds. Spend 5 minutes on Twitter or searching Google to see that Semalt has angered a LOT of people. One company was so enraged they wrote a Cease and Desist Notice! Here’s a sneak peek. Click the image to read it:
WHOA she’s not messin’ around! Not sure this is the best way to attract new clients, Semalt!
The Semalt referral traffic you see in your analytics is NOT real traffic. Semalt’s crawler hits your site while gathering data for their reporting tools. While they are not the only company to collect web data, they stand out as the only recent company that I know of to mess up millions (millions is a guesstimate) of users’ stats in the process.
Traffic From Semalt Crawlers
How Semalt Is Messing Up Your Google Analytics Data
Which of Your Website Stats Are Messed Up by Semalt’s Crawler
Semalt’s bot falsely inflates your total visitor count and negatively impacts your website engagement and conversion stats. Here are a few ways:
- Total Visits – Each time the bot hits your site counts as a visit
- Bounce Rate – Each semalt fake visit counts as a bounce, falsely inflating your bounce rate
- Pages Per Visit – Each semalt crawler visits only 1 page, a LOW and bad number that drives down your pages per visit average.
- Conversion Rate – If you have goals or ecom conversions, Semalt visits never convert. They drive down your website’s conversion rate.
Compare website stats below to see the impact of Semalt’s corrupt data:
Genuine data with semalt EXCLUDED, last 30 days:
Now look what happens to the numbers with the Semalt traffic data included:
The conversion rate drops 0.22%, the average time on site is off by 28 seconds, and total visits is falsely inflated by 58 visits.
How to Remove Semalt from Your Google Analytics
You can either create an advanced segment or create a custom view to get rid of the corrupt Semalt crawler data in your website stats.
1. Advanced Segment
- An advanced segment can backdate through your historic stats.
- An advanced segment can be turned off and on anytime easily with no permanent impact on your stats.
- Can not be used in conjunction with other Advanced segments.
To install my Google Analytics advanced segment, log in to your Google Analytics account THEN click (or copy and paste to your address bar) this Google Analytics Custom Segment Template link: https://www.google.com/analytics/web/template?uid=tk1n27A8RseamzFuD4FIfg.
You can also import my segment from the Google Analytics Template Gallery. It is called “Remove All Semalt Bot Traffic”.
Attention Lawton Digital Marketing Clients: you have this segment automatically accessible in your Google Analytics accounts! Find it in Google Analytics custom segments here:
If LDM also provides your real-time performance and web stats, you can access the segment from your RavenTools reporting dashboard here:
2. Filtered View
- Filtered view does NOT fix your historic corrupted stats. It starts collecting data on the date created.
- Removes the irritating Semalt data before you ever load your stats, so you never have semalt stats in your data again.
Here is what the filtered view looks like:
A filtered view is a few steps to setup. If you want Lawton Digital to set up a new view for you beyond the screen shot above, contact us to help.
If you try it on your own, it is IMPORTANT to create a NEW view rather than altering your existing one! Creating a filter on an existing view permanently alters the data in your Google Analytics view. Although you can remove the filter later, you can not bring back the data.
The Bad News
There is no current easy solution. Hundreds of new blog posts go up every day with creative solutions to block or remove it from your analytics. The Semalt company created a form on their website where we could input URLs to request removal from their “seed list.” After personally testing it 3 times on 6+ different sites, I can tell you for sure that it DOES NOT WORK. In fact, on one of my sites the number of hits INCREASED after I submitted my website for removal! Check out this stat showing Semalt traffic to my website more than doubling after submission to request removal:
The Good News
Complaints against Semalt are skyrocketing all across the web. The bad press is quickly giving Semalt a terrible reputation as a digital analytics company. Hopefully, they will eventually be forced to change their crawler so that it no longer adds false numbers to our Google Analytics web stats.
What Lawton Digital Marketing is Doing for Clients
We are and have been experimenting with solutions for a few weeks. We have not filtered out the semalt traffic on purpose from your web stats yet because we want to make sure that we stop the traffic from hitting your site in the first place. The Google Analytics filters above are a great bandaid for analytics accuracy, but they do not stop the actual crawler.
We are testing .htaccess file modifications on lawton-digital.com. In less techy lingo this means we add code telling Semalt crawlers to buzz off. The first test got rid of the crawler that had historically hit the site. Semalt crawler then started hitting from a different domain. This means they did not play nicely, making people even angrier. Here is what happened to my stats after blocking the first crawler:
Check out this Tweeter’s reaction to continued issues:
THEN Semalt made like a Chameleon so that it could get to the site from a different domain. SNEAKY! Now we implemented a new test with a broader code snippet that hopefully will solve the problem once and for all.
Google Analytics also recently launched an option to “Exclude traffic from known bots and spiders.” This feature just came out, and we are testing it too! We do not have enough results yet to determine whether this new feature includes Semalt. Google has not directly addressed the question.
Stay Up on This Topic
If you are interested in staying up on the trend, feel free to check out Twitter feed for mentions of #semalt or @semaltcom. Be warned that you may see me get a bit heated in my language used if you choose to check it out! You see a strong response because I am passionate about data-driven marketing, and Semalt’s crawler is messing up your stats. Semalt has the power to fix this issue, and they have chosen not to.
Let’s hope that Semalt learns their lesson, fixes their crawler and is able to turn over a new leaf.