🕷️ Scrapex at one of my workplaces, I've developed a multi-threaded crawling/scraping tool developed and written in the Dart Lang. FEATURES: ----------------- * Do crawling with caching support, so it doesn't hurt the target. using a strategy like SWR (stale-while-revalidate), incremental invalidating/revalidating cache. * Perform parsing & extractions of information. * Ensuring every parameter is customizable via command line args, parameters such as concurrency, cpuThreads, delayBetweenRequests, urlsCacheHours, htmlCacheDays, useJsonl (supports for OUTPUT JSONL), offline (indicate if it should work in offline mode), verbose, etc. * self-contained binary deployment in EC2 environment. * Rotating IPs using free proxies to maintain IP reputation. * Using a multithreaded approach to reduce execution times. * Can be tuned for performance, network conditions based on hosting server's specs. Technologies: Dart, Dart Isolate, Squadron, Pool, Riverpod, Dart HTML (DOM Parsing), Freezed, Dio, Interact, Build CLI, Self-Contained Binary.