Local handler debugging — Sneakpeek 0.2.2 documentation

# Local handler debugging

You can easily test handler without running full-featured server. Here’s how you can do that for the DemoScraper that we have developed in the tutorial.

Add import in the beginning of the file:

    from sneakpeek.scraper.runner import ScraperRunner

And add the following lines to the end of the file:

    async def main():
        result = await ScraperRunner.debug_handler(
            DemoScraper(),
            config=ScraperConfig(
                params=DemoScraperParams(
                    start_url="https://www.ycombinator.com/",
                    max_pages=20,
                ).dict(),
            ),
            middlewares=[
                RequestsLoggingMiddleware(),
            ],
        )
        logging.info(f"Finished scraper with result: {result}")

    if __name__ == "__main__":
        asyncio.run(main())

For the argument ScraperRunner.debug_handler takes:

1. An instance of your scraper handler

2. Scraper config

3. [Optional] Middleware that will be used in the handler (see full list of the middleware here)

Now you can run you handler as an ordinary Python script. Given it’s in demo_scraper.py file you can use:

    python3 demo_scraper.py