Python Web Scraping Expert

About NewsCore

NewsCore is an
AI-native market intelligence platform
that helps leading organizations monitor, collect, and analyze strategic information from the web in real time. We enable Fortune-500 corporations, major industrial players, and public institutions to detect emerging risks, competitive moves, regulatory signals, and market trends automatically. We're building
the next generation of automated monitoring systems
, powered by advanced AI, large-scale crawling, and information retrieval technologies.

Your Responsibilities

We are looking for
a scraping expert
to strengthen our ingestion and crawling stack.

You will design resilient pipelines capable of retrieving
millions of documents per day
across:

Websites, Social media, Institutional portals, Company websites, Industry sources

You'll work directly with our CTO and tech teams.

What you will work on

Large-Scale Scraping Architecture

Build and maintain high-volume scraping pipelines capable of ingesting 5M+ article / day

Ensure resiliency, retries, concurrency, proxy rotation, throttling, and anti-bot bypassing

Maintain robust source coverage and maximize recall rate

News & Media Scraping

Scrape news sites, press portals, industry pages, and structured business data

Extract HTML content cleanly (publication date, article content, title, authors, metadata, …)

Social Media Scraping

Extend and improve our stack for: X, Telegram, WhatsApp, Instagram, Tiktok

Manage sessions, cookies, mobile scraping, and API constraints

Backend Integration

Integrate crawling systems with: Django/Fast API backend, Celery workers, Background tasks, Redis caching, Internal ingestion APIs

Persist structured content into our database with indexing, versioning, and enrichment layers

Quality & Monitoring

Detect scraping failures, parsing errors, and source mismatches

Continuously improve content quality, coverage, freshness, and extraction accuracy

Build internal metrics to measure scraping performance at scale

Required Hard Skills

Strong experience in scraping (2+ years minimum)

Proven record in web scraping at scale (1M+ docs/day)

Experience with scraping frameworks: lxml , Selenium / Playwright, Requests, BeautifulSoup

Large-scale scraping best practices: concurrency, task scheduling, proxy rotation, caching strategies, error handling & bypass rules

Experience with social media scraping: (X, Telegram, WhatsApp, etc.)

Strong knowledge of: background jobs (Celery or equivalents), Django integration, Python parsing & normalization, HTML content extraction, Database ingestion

Comfortable with: CI/CD, Git, Quality-oriented code (tests, docs)

Bonus Skills (Nice to Have)

Scraping anti-bot techniques (captcha, session hijack, mobile / device emulation, …)

Performance testing & benchmarking

Knowledge of NLP, embeddings, search pipelines, information retrieval

Soft Skills

Strong ownership mindset

Autonomous and proactive

Fast learner, adaptable to fast-changing environments

Fluent English (written and spoken)

Why Join Us?

Cutting-edge AI startup
building the future of market intelligence used by global leaders.

High-impact work
on core systems powering large-scale data collection and strategic insights.

Ownership opportunities
, including potential equity and long-term incentives based on profile.

Elite engineering culture
, working with top-tier engineers, AI and data experts.

Recruitment Process

We propose 3 rounds over two weeks:

30-minute call
with Ludovic to understand your goals and situation.

2-hour test interview
with François to assess your hard skills.

45-minutes- tech interview
- deeper dive into your technical expertise.