Roll Call: Who's Grinding Late on a Saturday Night? (Aussies, You Can't Fool Me)

The hunger to rip apart your competitors.

 
gJ4s25K.gif
 
It's 00:57 am here. Been grinding for more than 9 hours now.

Just finished restructuring my new website I started last month.

It's a niche I'm very excited about and has ALOT of potentials + fairly OK competition , so I'm going all out on this.

Hopefully , everything works as planned.
 
Still in the office.. recording the last two videos for content. 5 total today, one of every day of the up coming work week.
 
Grindin away. I can usually tell if it's a good day by the first hour in the morning - if it is I spend the entire day writing, reaching out, planning and executing. I love this stuff.
 
Working on the main project this eve/morn/moment. Roughing out a multi-tenancy API so I can allow bands/venues the ability to do band-or-venue-name.music-project-domain.tld. The way I have things, Nginx just proxies everything through to the Phoenix App with an environment variable containing the subdomain set via fastcgi_param. Then I take care of the rest with routes, scopes and the authentication system. A good incentive for the next level when I implement a tiered subscription system. Not really worried about the negatives of using subdomains because SEO really isn't a concern of mine for this project other than the "SEO 101" stuff.
 
w7t23yU.gif
 
This is me on a Saturday night, editing long-form content I outsourced, and going bananas when I realize it takes just as long as if I'd written it myself since the editing happens as a byproduct of that:

giphy.gif

However, I don't know the topic so there's no avoiding it. And it's GLORIOUS content.
 
Messing with scraping sites in Elixir and getting it all to work with Chrome in "headless" mode (https://developers.google.com/web/updates/2017/04/headless-chrome). The end goal is to have a fleet of scrapers that can all be controlled by either a web based GUI or by API calls. I've got something similar already that I've coded in Ruby, but there is nothing that even begins to come close to the level of fault-tolerance I can achieve with Elixir.
 
Grinding Saturday night, grinding Sunday morning . . . grinding damn near 24 hours a day.
 
Wakey-wakey . . .
 
xSLdODO.gif

Trying to get some API thingy going...
 
Been all day trying to find content to have rewritten.

This is reminding me how much garbage is out there on the internet and how easy it should be for most people fly past 95% of competitors. Unfortunately for them, 95% of those people fall in that "95% of competitors."
 
I wrote here a couple weeks back about messing with headless Chrome in Elixir. Since then I've created several modules for my own use and a couple of Internal projects for Clients. I'm adding the finishing touches to the following module I designed to get up to 2 pages of results (configurable via a "page_depth" parameter) from Google. This is how it looks when I run it on my dev box in IEx, which is Elixir's interactive shell:

Code: 
iex(20)> results = BigG.get_results_for_keyword("best coffee maker 2017", 1, false, proxy_address, user_agent)
[debug] Starting Session
[debug] Ending any old sessions
[debug] Starting Headless Chrome Session
[debug] Navigating to Google Homepage
[debug] Page Title is Google
[debug] On Google Homepage
[debug] CAPTCHA Not Detected
[debug] Searching for: best coffee maker 2017
[debug] Filling In Form
[debug] Sending Enter Key Event
[debug] Waiting
[debug] CAPTCHA Not Detected
[debug] There are 10 organic results
[debug] Ending Session
{:found_results,
 %{page_one_results: [%{description: "Jul 7, 2017 - The coffee machine you use at home is a very personal decision and choosing the right model can be tricky. Let us help!",
      position: 1, title: "Best Coffee Makers of 2017 - CNET",
      url: "https://www.cnet.com/topics/coffee-makers/best-coffee-makers/"},
    %{description: "Ninja Coffee Bar Brewer, Thermal Carafe (CF087) Cuisinart DCC-3200. BUNN NHS Velocity Brew 10-Cup Home Coffee Brewer. Zojirushi EC-YGC120. Mr. Coffee 10-Cup Thermal coffee maker System PSTX91. Hamilton Beach 49980A Mr. Coffee 4-Cup Programmable coffee maker, DRX5.",
      position: 2,
      title: "Top 10: Best Automatic Drip Coffee Makers of 2017 - Little Coffee Place",
      url: "https://www.littlecoffeeplace.com/coffee-makers"},
    %{description: "Cuisinart Perfect Temp 14-Cup Programmable Coffeemaker. Zojirushi Fresh Brew (Plus Thermal Carafe) Coffee Maker. The Cuisinart DGB-700BC Grind-and-Brew 12-Cup Automatic Coffeemaker. The Bonavita 8-Cup Stainless Steel Coffee Brewer. Technivorm Moccamaster 10-Cup Coffee Brewer with Thermal Carafe.",
      position: 3,
      title: "The 10 Best Drip Coffee Makers of 2017 – Choose Wisely - Home ...",
      url: "https://www.homegrounds.co/best-drip-coffee-maker-reviews/"},
    %{description: "Looking for the perfect drip coffee maker for your home? We have reviewed the best drip coffee makers out there to help you make the buying decision.",
      position: 4,
      title: "Best Drip Coffee Makers Reviews [Tested] - Top Picks 2017",
      url: "https://freshpresso.net/best-drip-coffee-makers/"},
    %{description: "We've reviewed the latest and best coffee makers of 2016 at the Good Housekeeping Institute, so you don't have to. Here are the ones you want!",
      position: 5,
      title: "50+ Best Coffee Makers & Coffee Machine Reviews 2017",
      url: "http://www.goodhousekeeping.com/appliances/coffee-maker-reviews/"},
    %{description: "Drip coffee machines can be fickle. Sometimes, the coffee tastes watered-down. Other times, it tastes like you're sipping on a cup of charred coal. But with the ...",
      position: 6,
      title: "The Best Coffee Maker List (2017) - BrownsCoffee.com",
      url: "http://www.brownscoffee.com/best-coffee-maker-list/"},
    %{description: "May 22, 2017 - Looking for a budget-friendly coffeemaker? One that's single-serve? Or better yet, one that doubles as a cappuccino maker? We have an option ...",
      position: 7,
      title: "20 Best Coffee Makers of 2017 - Reviews of Coffee Machines & Maker ...",
      url: "http://www.bestproducts.com/appliances/small/g216/best-coffee-makers-machines/"},
    %{description: "The humble coffee maker is any coffee-lovers best friend, so it's important to find just the right model and style for the type of brew you love. Whether you like a ...",
      position: 8,
      title: "The 7 Best Coffee Makers to Buy in 2017 - The Spruce",
      url: "https://www.thespruce.com/best-coffee-makers-to-buy-4118535"},
    %{description: "... venturing downtown to the nearest cafe? Then check out some of our favorite coffee makers on the market, from Bunn to Chemex. ... Home > Home > The best coffee maker you can buy ... By Dan Evon — Updated May 12, 2017 10:02 am. 7.",
      position: 9, title: "The Best Coffee Makers You Can Buy | Digital Trends",
      url: "https://www.digitaltrends.com/home/the-best-coffee-makers/"},
    %{description: "Mar 17, 2017 - Take a look at our picks of the top rated automatic drip coffee makers for 2017. These have received consistently high ratings from a lot of the ...",
      position: 10, title: "10 Best Drip Coffee Maker - Reviews 2016 | CMPicks",
      url: "http://coffeemakerpicks.com/best-drip-coffee-makers/"}],
   page_two_results: [], total_results: 46800000}}

If results are found, they get returned as a tuple containing the result state as well as a "list of maps" of the results which includes the Title, Position and Meta Desc of each one. The module can also detect CAPTCHA, rotate proxies/user agent each request and detect error states such as timeouts. I don't have any plans to make the module available to the public but I can at least share that all anyone needs to do this in Elixir is to have the latest/greatest Chrome installed, the latest version of Chromedriver (Can skip Chrome and use PhantomJS), "Hound" (https://github.com/HashNuke/hound) and around 250-300 lines of code to make it do the thing.
 
Add me to the list. My head is only half in it so it's not grinding, but it's still production.
 
SmokeTree said:
I wrote here a couple weeks back about messing with headless Chrome in Elixir. Since then I've created several modules for my own use and a couple of Internal projects for Clients. I'm adding the finishing touches to the following module I designed to get up to 2 pages of results (configurable via a "page_depth" parameter) from Google. This is how it looks when I run it on my dev box in IEx, which is Elixir's interactive shell:

Code: 
iex(20)> results = BigG.get_results_for_keyword("best coffee maker 2017", 1, false, proxy_address, user_agent)
[debug] Starting Session
[debug] Ending any old sessions
[debug] Starting Headless Chrome Session
[debug] Navigating to Google Homepage
[debug] Page Title is Google
[debug] On Google Homepage
[debug] CAPTCHA Not Detected
[debug] Searching for: best coffee maker 2017
[debug] Filling In Form
[debug] Sending Enter Key Event
[debug] Waiting
[debug] CAPTCHA Not Detected
[debug] There are 10 organic results
[debug] Ending Session
{:found_results,
 %{page_one_results: [%{description: "Jul 7, 2017 - The coffee machine you use at home is a very personal decision and choosing the right model can be tricky. Let us help!",
      position: 1, title: "Best Coffee Makers of 2017 - CNET",
      url: "https://www.cnet.com/topics/coffee-makers/best-coffee-makers/"},
    %{description: "Ninja Coffee Bar Brewer, Thermal Carafe (CF087) Cuisinart DCC-3200. BUNN NHS Velocity Brew 10-Cup Home Coffee Brewer. Zojirushi EC-YGC120. Mr. Coffee 10-Cup Thermal coffee maker System PSTX91. Hamilton Beach 49980A Mr. Coffee 4-Cup Programmable coffee maker, DRX5.",
      position: 2,
      title: "Top 10: Best Automatic Drip Coffee Makers of 2017 - Little Coffee Place",
      url: "https://www.littlecoffeeplace.com/coffee-makers"},
    %{description: "Cuisinart Perfect Temp 14-Cup Programmable Coffeemaker. Zojirushi Fresh Brew (Plus Thermal Carafe) Coffee Maker. The Cuisinart DGB-700BC Grind-and-Brew 12-Cup Automatic Coffeemaker. The Bonavita 8-Cup Stainless Steel Coffee Brewer. Technivorm Moccamaster 10-Cup Coffee Brewer with Thermal Carafe.",
      position: 3,
      title: "The 10 Best Drip Coffee Makers of 2017 – Choose Wisely - Home ...",
      url: "https://www.homegrounds.co/best-drip-coffee-maker-reviews/"},
    %{description: "Looking for the perfect drip coffee maker for your home? We have reviewed the best drip coffee makers out there to help you make the buying decision.",
      position: 4,
      title: "Best Drip Coffee Makers Reviews [Tested] - Top Picks 2017",
      url: "https://freshpresso.net/best-drip-coffee-makers/"},
    %{description: "We've reviewed the latest and best coffee makers of 2016 at the Good Housekeeping Institute, so you don't have to. Here are the ones you want!",
      position: 5,
      title: "50+ Best Coffee Makers & Coffee Machine Reviews 2017",
      url: "http://www.goodhousekeeping.com/appliances/coffee-maker-reviews/"},
    %{description: "Drip coffee machines can be fickle. Sometimes, the coffee tastes watered-down. Other times, it tastes like you're sipping on a cup of charred coal. But with the ...",
      position: 6,
      title: "The Best Coffee Maker List (2017) - BrownsCoffee.com",
      url: "http://www.brownscoffee.com/best-coffee-maker-list/"},
    %{description: "May 22, 2017 - Looking for a budget-friendly coffeemaker? One that's single-serve? Or better yet, one that doubles as a cappuccino maker? We have an option ...",
      position: 7,
      title: "20 Best Coffee Makers of 2017 - Reviews of Coffee Machines & Maker ...",
      url: "http://www.bestproducts.com/appliances/small/g216/best-coffee-makers-machines/"},
    %{description: "The humble coffee maker is any coffee-lovers best friend, so it's important to find just the right model and style for the type of brew you love. Whether you like a ...",
      position: 8,
      title: "The 7 Best Coffee Makers to Buy in 2017 - The Spruce",
      url: "https://www.thespruce.com/best-coffee-makers-to-buy-4118535"},
    %{description: "... venturing downtown to the nearest cafe? Then check out some of our favorite coffee makers on the market, from Bunn to Chemex. ... Home > Home > The best coffee maker you can buy ... By Dan Evon — Updated May 12, 2017 10:02 am. 7.",
      position: 9, title: "The Best Coffee Makers You Can Buy | Digital Trends",
      url: "https://www.digitaltrends.com/home/the-best-coffee-makers/"},
    %{description: "Mar 17, 2017 - Take a look at our picks of the top rated automatic drip coffee makers for 2017. These have received consistently high ratings from a lot of the ...",
      position: 10, title: "10 Best Drip Coffee Maker - Reviews 2016 | CMPicks",
      url: "http://coffeemakerpicks.com/best-drip-coffee-makers/"}],
   page_two_results: [], total_results: 46800000}}

If results are found, they get returned as a tuple containing the result state as well as a "list of maps" of the results which includes the Title, Position and Meta Desc of each one. The module can also detect CAPTCHA, rotate proxies/user agent each request and detect error states such as timeouts. I don't have any plans to make the module available to the public but I can at least share that all anyone needs to do this in Elixir is to have the latest/greatest Chrome installed, the latest version of Chromedriver (Can skip Chrome and use PhantomJS), "Hound" (https://github.com/HashNuke/hound) and around 250-300 lines of code to make it do the thing.
Click to expand...

This looks awesome. What are some of your intended uses for this, automating rank tracking, keyword research? Could something like this be modified to track Amazon prices and buybox ownership?
 
ryandiscord said:
What are some of your intended uses for this, automating rank tracking, keyword research?
Click to expand...

One of the basic uses would be checking a URL to make sure it's indexed, which can be used as one of the facets of a web asset monitoring app, part of one of my current projects. The end game for me is to have my own collection of private APIs that I can use (or a partner in a JV) to get data from and interact with Google, Bing, Twitter, Facebook, Pinterest, etc. Building modules/services/APIs like this is pretty much what I do on weekends and for fun in general.

If you're interested in building things like this in general, Python might be a better language to start with as Elixir syntax and functional programming in general is a bit different from PHP, Perl, Ruby, C#. @turbin3 wrote a guide on scraping with Python and Scrapy that I think is one of the best written resources on general scraping I've seen and is definitely worth the read.
 
You must log in or register to reply here.
Back