Product Release: Crawl Requests

Researchers can now direct our crawlers at the keywords, profiles, or Telegram channels that they'd like to track.

Apr 11, 2024

Open Measures is launching a new tool, Crawl Requests, to allow researchers to point crawlers at the information they care about most.
Authenticated users can now add keywords, profiles, or Telegram channels they’d like to back-crawl and continuously monitor.

Background

In the Crawl Requests UI, users can view all crawl requests, manage request statuses, add new requests, and monitor trends.

The research community needs comprehensive coverage of unique keywords, user or group profiles, and Telegram channels for project-specific investigations. Historically, we met that need with request-based manual configuration and our API. Today, we’re happy to share the launch of the Crawl Request App, where users can take control of Open Measures’ scope of coverage themselves.

Here’s a high level overview of how this new tooling works:

Crawl Requests works in five steps: Partners provide targets for the crawl (keywords, profiles, or channels) (1). Open Measures’ crawlers pick up jobs (2) and run collection for data using a source’s native search interface (3). That data is then collected and stored (4). Partners can then access the collected data on the Open Measures platform via API, Research Dashboard, etc (5).

With this new feature, users will have better control over Open Measures’ collections and the ability to target thorough crawls of subsets comprised of sources we collect. Depending on the dataset, we will either enumerate through a list of targeted keywords, user profiles, or channels.

The * in the above diagram represents the other crawling processes that Open Measures maintains to collect data from our sources. Crawl Requests are a standalone collection stack that runs in parallel to these default data collection systems.

Keywords and Profiles

*Requests for profiles or keywords are made on a source-by-source basis*

Most social media sites have native a search bar. When a crawl request is made for a keyword, Open Measures’ crawlers run a search for that keyword using the dataset's search bar, enumerate all the results, and index the data.

*Example search interface on Gab. In this example, the keyword request is for "trump 2024"*.

Crawl Requests allow Open Measures users to request keyword crawling on a per source basis.

Telegram Channels

Crawl Requests also allow partners to submit Telegram channels. When Open Measures receives a request to crawl Telegram data, we back-crawl the channel to the first message and then continuously monitor for new data moving forward.

Bottom Line

Open Measures’ Crawl Requests App and API are now available to partnered users and organizations. This innovative new application allows users to take control of our crawlers, pointing them at unique keywords, profiles, and channels that their researchers care about most.

Would you like access to Crawl Requests? Reach out to us below:

Open Measures Newsletter