Scraping from a Wordpress custom plugin

Joined
Dec 31, 2016
Messages
596
Likes
420
Degree
2
I've been writing my scrape code in Python (mostly Scrapy), which is cool, but I feel like its becoming sort of a pain, having it off site, with stuff like IP adress issues, having to FTP the data to my server etc.

I also have some other ideas for integration with Wordpress for a product, sort of like Google Site Kit.

My question is:

Is there any reason not to build a plugin in Wordpress that scrapes external data and stores it in a separate Wordpress table?
 

Ryuzaki

女性以上のお金
Staff member
BuSo Pro
Digital Strategist
Joined
Sep 3, 2014
Messages
3,725
Likes
7,066
Degree
8
The first thing that comes to mind for me is resource demand and what this would do to your site speed and user experience.

The second thing would be burning your IP address in some fashion and ending up on browser's "be careful visiting this site" list.
 
Joined
Dec 31, 2016
Messages
596
Likes
420
Degree
2
The first thing that comes to mind for me is resource demand and what this would do to your site speed and user experience.

The second thing would be burning your IP address in some fashion and ending up on browser's "be careful visiting this site" list.
That's a very good point.

No easy shortcuts eh.
 
Joined
Apr 27, 2019
Messages
7
Likes
6
Degree
0
Couldn't you just create your own API to access your scraped data when you need it? Or have the data in a Firebase database that you can access in realtime?