ShopFlixBot - Crawler Documentation & Seller Integration Guide

Overview

Shopflix uses a web crawler (ShopFlixBot) to download and process XML product feeds from seller websites. This document covers the crawler's behavior, network identity, and the steps sellers (or their technical teams) need to take to ensure uninterrupted access.

User-Agent

ShopFlixBot identifies itself using the HTTP header:

User-Agent: ShopFlixBot

In some cases, it may also use a generic browser-style header:

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/37.0.2062.94 Chrome/37.0.2062.94 Safari/537.36

Crawled URLs

ShopFlixBot only accesses the following resources:

• XML product feed - the feed URL as registered in the Merchant Portal.
• Image URLs - image links referenced within the XML, required for product creation.
• Product page URLs - product links referenced within the XML, used for occasional validation.

IP Addresses

All requests originate from the following IPs:

Request Behavior

• ShopFlixBot maintains a reasonable request rate at all times.
• ShopFlixBot does not perform any form of authentication (no cookies, tokens, or login flows).
• ShopFlixBot requires access to seller XML URLs without rate limits.

Seller-Side Configuration

To ensure uninterrupted feed access, sellers must whitelist ShopFlixBot's IPs and User-Agent in their server and security infrastructure. The instructions below cover the most common scenarios.

General (Any Firewall / WAF / Rate Limiter)

1. Whitelist the IPs listed above in your firewall and rate-limiting tools.
2. Allow the User-Agent ShopFlixBot in any ACL or bot-filtering mechanism.
3. Exclude the IPs from rate limiting - ShopFlixBot needs unrestricted access to the XML feed endpoint.

Cloudflare-Specific Setup

Follow the steps below to allow access for ShopFlixBot and our platform's IPs.

Step 1 - Create a "Skip Security" Rule (CRITICAL)

Navigate to Security → WAF → Custom Rules and create a new rule:

• Rule name: Allow ShopFlix Feed Fetcher
• Expression:

(ip.src in {18.193.166.252 3.125.27.163}) and (http.user_agent contains "ShopFlixBot")

• Action: Skip
• Skip options: Select "All WAF rules", "All rate limiting rules", "All managed rules" and "Bot Fight Mode".

Important note for Free Plans: If you have Cloudflare's Free Plan, "Bot Fight Mode" is often overly strict and may not comply with the rule above. If the problem persists: Go to Security → Bots and fully disable Bot Fight Mode (set it to Off).

Step 2 - Restrict to Feed URLs (Optional)

If you want the exception to apply only to feeds, use:

(http.request.uri.path contains "/feed") and (ip.src in {18.193.166.252 3.125.27.163} or http.user_agent contains "ShopFlixBot")

Step 3 - Technical Verification (Curl Test)

Please verify the connection by running the following command from a terminal:

curl -A "ShopFlixBot" -L https://www.yourdomain.gr/your-feed.xml

Expected result:

• Returns Raw XML data.
• No CAPTCHA pages or Cloudflare Challenges.
• No 403 Forbidden errors.

Connection Details:

• IPs: 18.193.166.252, 3.125.27.163
• User-Agent: ShopFlixBot

Testing & Troubleshooting

To simulate a ShopFlixBot request from the command line:

curl -v -A "ShopFlixBot" -L https://www.example.com/feed.xml

If the feed is inaccessible, verify that both the IP whitelist and User-Agent skip rule are in place, and check your server/CDN access logs for blocked or challenged requests.