Scrapper

STATUS [ACTIVE]

UPDATE 06/20/2024

At this time Intermarche,SystemeU and Leclerc use Datadome protection

• Intermarche -> Impossible for me to bypass the new version of Datadome -> Target waiting
• SystemeU -> Same to Intermarche , bypass with proxy and IP Rotating is possible
• Leclerc -> Same to Intermarche

PRESHOT 2024 TARGET EVOLUTION

• SystemeU -> Update the version of the DataDome Solution
• Auchan and Carrefour add DataDome Solution
• Monoprix no protection
• Leclerc need to rebuild the pathing of the website to use correctly the DataDome solution

PRESHOT 2024 TOOL EVOLUTION

• php-webdriver -> Maybe Deprecated soon for WebScraping
• puppeteer -> need more update for hide the headless mode (waiting)
• playwright -> microsoft tool (Ubuntu 20.* or newer)
• selenium -> next test for scrapping target (Famous tool)

Disclaimer

• This tool is not for collect personal information
• Please respect the RGPDs rules

What is a scrapper

A tool to collect any information of website pages :For example javascript,html,css sources It's possible to look the content of a website pages with the browser with this tips :

• firefox view-source:https://www.mozilla.org/fr/ or CTRL^U
• CRTL+MAJ+I for web inspector -> Console, possibility to change the display conten

Or with special library and framework like :

• Selenium (Python,...)
• Goutte (Symfony)
• Scrapy (Python)

Or API :

• ScrapFly
• ScraperAPI

How

• With PHP and the docXpath
• With php-webdriver
• With puppeteer
• With puppeteer-extra

Why

• For my project PriceComparator
• Developpement of your own tools is important to understand and learn many things.

Paths

Paths

dev
└── JSON_updates.php
project
├── infos_programs.php
└── project.php
src
├── control_google_.js
├── DatadomeBreaker/
├── libJSON/
├── scrape.js
├── scrape_su.js
├── scrapper_auchan.php
├── scrapper_carrefour.php
├── scrapper_intermarche.php
├── scrapper_leclerc.php
├── scrapper_monoprix.php
├── scrapper.php
├── scrapper_systemeu.php
├── test_extra_puppeteer.js
└── test_rq_submod.js
your_project
├── process_p.php
├── proofs/
├── README.md
└── usage.php
composer.json
package.json
README.md

Usage

LIKE A PACKAGE :

• curl -sS https://getcomposer.org/installer | php7.2 OR #2:
  • php7.2 composer.phar update
• #2 -> composer install :
  • composer update
• In move your_project folder in the root of your project for test the functions

LIKE A PROJECT :

• composer require php-webdriver/php-webdriver
• project.php for known how the different tools works
• scrapper*.php the differents files for scraping mission
• vendor add lib for php-webdriver
• node_modules(hide with .gitignore) for node.js module
• *.json/*.txt for different test to build program to efficient scraping

php project/project.php --info it's a good start

Version

V1.5.1

• Basic version of scrapper :

  • http, https
  • html content generate by JS -> puppeteer
  • cloudflare security
  • text in tag with another tag $\color{green}\textsf{(V2.0\ scrapper.php)}$

• Specific version for specific website :

  • The french supermarket compagny :
    • Leclerc [BLOCKED]:
      • parse specific JS -> json
      • usage of https of basic version :
      • NoBot Solutions DataDome Solution
      • Try Bypass NoBot Solutions with knownledge of all stores (libJSON/leclercs.json) (works before Datadome Solution buy)
    • Carrefour :
      • parse specific JS -> json
      • usage of php-webdriver
      • NoBot Solutions -> Cloudflare
    • Auchan :
      • parse text in html tag
      • usage of php-webdriver
      • NoBot Solutions
    • Monoprix :
      • parse specific JS -> json
      • usage of puppeteer or php-webdriver is possible
      • products for all stores in the target country
      • NoBot Solutions
    • Intermaché [BLOCKED] :
      • parse specific JS -> json
      • usage of php-webdriver
      • NoBot Solutions -> DataDome Solution
    • SystemeU [BLOCKED]:
      • parse specific JS -> json (products only on the display page)
      • usage of puppeteer or php-webdriver IMPOSSIBLE
      • NoBot Solutions -> DataDome Solution
      • Necessary to use puppeteer-extra-plugin-stealth -> not enough
      • Try Bypass with src/libJSON/* (scrape2() in scrape_su.js) but blocked again

Features