What is omgili ?
Omgili is a web crawler associated with web data extraction from forums, discussion boards, and Q&A sites. The crawler’s name stands for “Oh My God I Love It” and has been observed scraping user-generated content for purposes related to sentiment analysis, data mining, and commercial intelligence. It is often linked to web monitoring and competitive intelligence platforms. The user-agent string generally includes “omgili” and has been known to access a broad range of forum-like platforms.
Who is operating omgili ?
Omgili was originally developed by a company called Omgili Ltd. However, in recent years, the crawler appears to be operated by webz.io, a commercial data-as-a-service company specializing in web-scale structured content extraction. Webz.io provides feeds of online discussions, reviews, and dark web content to clients in cybersecurity, finance, and market research. More information is available at webz.io.
Why you should be interested in omgili ?
If your site contains forums, user reviews, or structured discussions, omgili is likely to target it. This crawler extracts high volumes of conversational data, which can create server strain and potentially expose user-generated content to commercial reuse. It has also been flagged by several bot detection services for aggressive behavior. Website owners concerned with privacy, data governance, or unauthorized use of content should monitor and control its access.
How to block omgili ?
1. robots.txt File:
Add the following rule to your robots.txt file
# block omgili User-agent: omgili Disallow: /
2. Server-Side Filtering:
Block requests containing “omgili” in the user-agent string using Apache/Nginx configuration.
3. Traffic Analysis:
Monitor unusual spikes in requests to user-generated content endpoints. Omgili often behaves like a bulk data miner.
About the bot
Owner: webz.io (formerly Omgili Ltd.)
Owner URL: webz.io
Bot URL: webz.io
Bot User Agent: omgili/0.1 +http://omgili.com
Respects robots.txt: No
