Skills

Utility for crawling websites

Spiderman makes it trivial work to write a crawler. Just define some components, use segues to connet them, finally write some script to specify data that needs to be transfer from one component to the next, that's all!

CLI for crawling & analysing repositories

Endless Crawler Cached Data

An easy-to-use Node web crawler storing cookies, following redirects, traversing pages and submitting forms.

Auto-generated API client for Path of Exile Forum Crawler

Server-Side Rendering Proxy

BlogSearch index building tool for generic static webpages (crawler)

Expired domains finder for crawler.ninja

The error crawler that powers http://plucky.io/

the crawler of NBA score

A crawler to fetch last news from Iranian(Persian) news agencies.

## install

TRA Receipt Crawler

Hal Client for JavaScript

Node module for web-scraping on torrent video websites

A weibo crawler

Manga downloader using starkana.com.

* Install the package globally.

A DOM crawler to extract results from https://www.mytischtennis.de/clicktt/home/

Use static-crawler to generate static html version of a single page application. No matter of the framework you are using, React, VueJs, Angular and others, this tool will work with most of them, since it is using puppeteer to render the pages.

a simple node crawler

Generate sitemap just throw any link.

crawling thirdparty licenses from package manager files (packages.json, pom.xml, ...)