Search Engine Scraper

Harvest URL’s from Search Engines

Search Engine Harvester

Included with ScrapeBox
Fastest Scraper Built
Multi-Threaded Connections
+30 Search Engines
Proxy Support
Refresh Proxies While Harvesting
100% Trainable
Harvest Any Queries
Custom Useragents
Harvester Stats
Keyword Stats

With the release of ScrapeBox v2.0 we have created the fastest, most power Search Engine Scraper ever built. It’s the first desktop SERP Scraper we have ever seen surpass scraping speeds of over 1 Million URL’s per Minute!

ScrapeBox has a custom search engine scraper which can be trained to harvest URL’s from virtually any website that has a search feature. It may be a simple WordPress blog with a search feature that you want to harvest all the URL’s from on a particular keyword or number of keywords, or a major search engine like Google, Bing or Yahoo.

The custom scraper comes with approximately 30 search engines already trained, so to get started you simply need to plug in your keywords and start it running or use the included Keyword Scraper. Besides the major players, some of the included engines are Lycos, Ask.com, Rambler, AltaVista, Mojeek, Blekko, Excite, HotBot, IXQuick, DogPile, Blingo as well as ISP specific search engines like Charter, Verizon, Comcast and Orange.co.uk. There’s even an engine for YouTube to harvest YouTube video URL’s and Alexa Topsites to harvest domains with the highest traffic rankings.

It’s also multi-threaded with adjustable connections so you could run 3000 connections at once and harvest thousands or even millions of URL’s per second from all engines at once or conservatively run one connection for PC’s with slower internet speeds. You can also configure options on proxy retries, removing dead proxies while harvesting, refresh proxies while harvesting as can be seen here.

Trainable Harvester

Trainable harvester with over 30 search engines and the ability to easily add your own search engines to harvest from virtually any site.

You can add country based search engines, or even create a custom engine for a WordPress site with a search box to harvest all the post URL’s from the website.

Training new engines is pretty easy, many people are able to train new engines just by looking at how the 30 included search engines are setup. We have a Tutorial Video or our support staff can help you train specific engines you need. You can even export engine files to share with friends or work colleges who own ScrapeBox too.

For power users, there’s even more advanced options. For each engine you can customize all the header data ScrapeBox sends with each request, you can change the useragent to use low bandwidth mobile search engines, you can set custom cookies, clear cookies before each request, follow redirects and even append the domain to harvested URL’s of search engines with relative links.

Harvester Stats

Detailed statistics can be obtained when harvesting, we realize not everyone wants to scrape millions of URL’s. Some people want granular results too.

So we provide harvester statistics so you can log how many results were obtained for every keyword in every search engine.

The harvester can also save the keyword with every harvested URL so you can easily identify what keywords produced what results.

Search Engine Harvester Tutorial

View our video tutorial showing the Search Engine Scraper in action. This feature is included with ScrapeBox, and is also compatible with our Automator Plugin.