Current state of bot filtering
-
We currently filter common bot user-agents from our data
-
These user-agents will flow through your funnel as usual and tracking functions normally. However, we drop the data in our pipeline before it reaches the database, making it not appear in reporting
-
We also filter certain visits as situations arise (e.g. odd but prevalent requests)
-
We also disregard incoming cost data if the value passed is too high
-
Right now, there is no user-level filter to add your own bot-exclusion or blocking rules
Overall bot-filtering feature plans
We plan to add the ability to create rules that are either global, or apply to specific funnels only, or to specific traffic sources only. Our plans are as below:
- Provide filtering rules as a new section, where you define global (account-level) rules based on:
- User-agent
- IP address/ranges
- Referrer domain/path
- Countries
- ISP name
- Funnel
- Traffic Source
- You will then be able to apply several actions:
- Mark as filtered but let visitor continue as normal down funnel path
- Do not log the data, but let visitor continue as normal down funnel path
- Bounce the traffic to a specific URL and do not log the data at all
So these actions are basically soft, soft-delete, and hard-bounce.
That's for the future though! On to dealing with the now...
Dealing with bots using data resets
Right now, we have added some basic filtering on our side to get rid of known bots.
If you have other traffic you want to get rid of, the only thing you can do about annoying data is delete it entirely.
You can do this via the updates section > reset stats. The easiest way will be to know the IPs of the traffic that you want to delete.
You can then pick funnel groups, funnels and traffic sources (or just select all), then toggle on the IP filtering and enter IPs/ranges, like so:
Click Estimate and it will find matching data, which you can then delete.
How to find these IPs? For that, you can use our Raw Events section and just scan through the results. Typically you will find specific ISPs, referrers or user-agents are responsible for the junk you want to remove.
You can manually find and copy the related IPs, or export to Excel and filter to get what you want there.
For now, I would just keep a text file with the IPs/ranges so that you can copy/paste into the above form every time you want to do this, and so you can grow your filtering list over time -- an IP blacklist basically.
We do plan to add user-agent and ISP name to the above filtering options in the future, which may make these even easier, prior to our filtering module.
Sorry for the inconvenience!
Also, please note reset stat operations are slow -- once submitted they go to our database which itself manages these operations, its out of our control. So it might take 15 seconds, it might take 10 minutes -- you just need to wait and let it proceed.
If after 1 hour your data has not been removed, you can try again and see if the estimate has changed, or reach out to us for help.