Product Release: Crawl Requests
Researchers can now direct our crawlers at the keywords, profiles, or Telegram channels that they'd like to track.
TLDR
Open Measures is launching a new tool, Crawl Requests, to allow researchers to point crawlers at the information they care about most.
Partnered users can now add any keyword, profile, or Telegram channel they’d like to continuously monitor.
Background
The research community needs scraping for unique keywords, user or group profiles, and Telegram channels for project-specific investigations. Historically, we met that need with request-based manual configuration and our Crawl Requests API. Today, we’re happy to share the launch of the Crawl Request dashboard, where users can take control of Open Measures’ scope of coverage themselves.
Here’s a high level overview of how this new tooling works:
With Crawl Requests, users will have better control over Open Measures’ crawlers and the ability to target thorough crawls of subsets comprised of sources we collect. Depending on the dataset, we will either enumerate through a list of targeted keywords, user profiles, or channels.
The *
in the above diagram represents the other crawling processes that Open Measures maintains to collect data from our sources. Crawl Requests are a standalone collection stack that runs in parallel to these default data collection systems.
Keywords and Profiles
Most sites that Open Measures collects from have native search bars. When a crawl request is made for a keyword, Open Measures’ crawlers run a search for that keyword using the dataset's search bar, enumerate all the results, and save the collected data.
Crawl Requests allows Open Measures users to request keyword crawling on a per source basis.
Telegram Channels
Crawl Requests also allows partners to submit Telegram channels. When Open Measures receives a request to crawl Telegram data, we back-crawl the channel all the way to the first message before monitoring all data coming in live going forward.
Bottom Line
Open Measures’ Crawl Requests UI is now available to partnered users and organizations. This innovative new application allows users to take control of our crawlers, pointing them at unique keywords, profiles, and channels that their researchers care about most.
Would you like access to Crawl Requests? Reach out to us below: