URL Scrapping 3rd party service?

[miketpowell]

We are going to build at app in 2017 that monitors a number of attributes of content URLs on sites as they are published.

One major element of this app would be getting the new URLs of content in real time from a ton of different websites.

Since this is a pretty common "need" are there any good 3rd party services that already do something like this?

 

[eliquid]

depends, do these sites have rss feeds/site maps of their new content as its pushed out?

 

[miketpowell]

Most will but cannot be sure everyone will. But it will mostly be content/news style sites that want the content found easily. We'd want to be finding the new URL very quickly, minutes after published ideally and not like once a day. Even every hour would not be ideal.

 

[eliquid]

If even every hour is not good enough, I have 1 last ditch method I know of. Im not going to share it on the board though. Secret sauce type stuff...

 

[Jared]

I had this exact same question a few months back.

RSS will work for most, obviously.

Assuming you're on Windows, I'd recommended RSS Owl, which is a traditional desktop RSS client. You might also check out Feed Notifier, which sits in your system tray, and will notify of you new posts via popup notification/slider. You can customize how often either one checks each feed, allowing you to get updated as often as is needed.

For those sites with erratic RSS feeds, or without them (like Facebook pages), there's a Chrome plugin called Distill Web Monitor.

You can visit a page, and select certain elements on that page. The plugin will search the page, however often you set it to, looking for any changes to those elements. If it finds changes, it'll notify you with a desktop pop up and/or sound effect. If you set it up properly, you can use it to monitor a site's homepage, blog page -- wherever new content gets posted -- and be notified when a new article is added.