Product Release: Crawl Requests

With Crawl Requests, researchers can direct our crawlers at the keywords, profiles, or Telegram channels they’d like to track.

TLDR

• Open Measures is launching a new tool, Crawl Requests, to allow researchers to point crawlers at the information they care about most.

• Authenticated users can now add keywords, profiles, or Telegram channels they’d like to back-crawl and continuously monitor.

Background

In the Crawl Requests UI, users can view all crawl requests, manage request statuses, add new requests, and monitor trends.

The research community needs comprehensive coverage of unique keywords, user or group profiles, and Telegram channels for project-specific investigations. Historically, we’ve met that need with request-based manual configuration and our API.

Today, we’re happy to share the launch of the Crawl Request App, where users can take control of Open Measures’ scope of coverage themselves. Below is a high level overview of how this new tooling works:

Diagram illustrating the functionality of our new feature, Crawl Requests (in five steps, explained in more detail below).

Crawl Requests works in five steps:

1. Partners provide targets for the crawl (keywords, profiles, or channels)

2. Open Measures’ crawlers pick up those jobs.

3. The crawlers run collection for requested data using a source’s native search interface.

4. The data is collected and stored.

5. Partners can then access the collected data on the Open Measures platform via API or Research Dashboard.

With this new feature, users will have better control over Open Measures’ collections and will gain the ability to target thorough crawls of subsets comprised of sources we collect. Depending on the dataset, we will either enumerate through a list of targeted keywords, user profiles, or channels.

The * in the diagram above diagram represents the other crawling processes that Open Measures maintains to collect data from our sources. Crawl Requests are a standalone collection stack that runs in parallel to these default data collection systems.

Keywords and Profiles

Screenshot showing that requests for profiles or keywords are made on a source-by-source basis

Most social media sites have native a search bar. When a crawl request is made for a keyword, Open Measures’ crawlers run a search for that keyword using the dataset's search bar, enumerate all the results, and index the data.

Example search interface on Gab. In this example, the keyword request is for "trump 2024".

Crawl Requests allows Open Measures users to request keyword crawling on a per source basis.

Telegram Channels

The addition of new Telegram channels, now available at the touch of a button.

Crawl Requests also enables users to add Telegram channels. When Open Measures receives a request to crawl Telegram data, we back-crawl the channel to the first message and then continuously monitor for new data moving forward.

Conclusion

Open Measures’ Crawl Requests App and API are now available to partnered users and organizations. This innovative new application allows users to take control of our crawlers and point them at unique keywords, profiles, and channels that their researchers care about most. For access to Crawl Requests, reach out to us at info@openmeasures.io.

---

Identify online harms with the Open Measures platform.

Organizations use Open Measures every day to track trends related to networks of influence, coordinated harassment campaigns, and state-backed info ops. Click here to book a demo.