Roll Call: Who's Grinding Late on a Saturday Night? (Aussies, You Can't Fool Me)

Discussion in 'Water Cooler' started by Ryuzaki, Feb 25, 2017.

  1. turtle

    turtle

    Joined:
    Jun 10, 2015
    Messages:
    89
    Likes:
    58
    Degree:
    0
    The hunger to rip apart your competitors.

     
    SmokeTree and CCarter like this.
  2. CCarter

    •  
    •  
    •  
    CCarter If they cease to believe in u, do u even exist? Staff Member

    •  
    •  
    Joined:
    Sep 15, 2014
    Messages:
    1,735
    Likes:
    3,679
    Degree:
    5
  3. Encrypted

    Encrypted

    Joined:
    Nov 18, 2016
    Messages:
    15
    Likes:
    9
    Degree:
    0
    It's 00:57 am here. Been grinding for more than 9 hours now.

    Just finished restructuring my new website I started last month.

    It's a niche I'm very excited about and has ALOT of potentials + fairly OK competition , so I'm going all out on this.

    Hopefully , everything works as planned.
     
    SmokeTree likes this.
  4. Steve Brownlie

    •  
    •  
    Steve Brownlie Building Links

    Joined:
    Nov 16, 2015
    Messages:
    248
    Likes:
    435
    Degree:
    1
    2 days in a row up at 4am. If I can get to sleep a bit more consistently in the late evening so I get 5-6 hours in I might just stick at this. Love the quiet.
     
    built and turtle like this.
  5. BobTheBuilder

    •  
    BobTheBuilder

    Joined:
    May 12, 2017
    Messages:
    9
    Likes:
    3
    Degree:
    0
    I thought I was the only one

    [​IMG]
     
  6. SSG

    SSG Dōmo arigatō, Mr. Roboto

    Joined:
    Nov 20, 2016
    Messages:
    33
    Likes:
    33
    Degree:
    0
    Still in the office.. recording the last two videos for content. 5 total today, one of every day of the up coming work week.
     
  7. Inhibition

    Inhibition

    Joined:
    Dec 20, 2014
    Messages:
    45
    Likes:
    42
    Degree:
    0
    Grindin away. I can usually tell if it's a good day by the first hour in the morning - if it is I spend the entire day writing, reaching out, planning and executing. I love this stuff.
     
  8. SmokeTree

    •  
    •  
    SmokeTree Developer/Linux Consultant

    Joined:
    Sep 23, 2014
    Messages:
    163
    Likes:
    245
    Degree:
    1
    Working on the main project this eve/morn/moment. Roughing out a multi-tenancy API so I can allow bands/venues the ability to do band-or-venue-name.music-project-domain.tld. The way I have things, Nginx just proxies everything through to the Phoenix App with an environment variable containing the subdomain set via fastcgi_param. Then I take care of the rest with routes, scopes and the authentication system. A good incentive for the next level when I implement a tiered subscription system. Not really worried about the negatives of using subdomains because SEO really isn't a concern of mine for this project other than the "SEO 101" stuff.
     
  9. Prentzz

    •  
    Prentzz

    Joined:
    Mar 19, 2015
    Messages:
    176
    Likes:
    249
    Degree:
    1
    [​IMG]
     
    CCarter likes this.
  10. CCarter

    •  
    •  
    •  
    CCarter If they cease to believe in u, do u even exist? Staff Member

    •  
    •  
    Joined:
    Sep 15, 2014
    Messages:
    1,735
    Likes:
    3,679
    Degree:
    5
  11. Ryuzaki

    •  
    •  
    Ryuzaki 女性以上のお金 Staff Member

    Joined:
    Sep 3, 2014
    Messages:
    2,223
    Likes:
    3,924
    Degree:
    6
    This is me on a Saturday night, editing long-form content I outsourced, and going bananas when I realize it takes just as long as if I'd written it myself since the editing happens as a byproduct of that:

    [​IMG]

    However, I don't know the topic so there's no avoiding it. And it's GLORIOUS content.
     
  12. contract

    contract We're all gunna mine it brah.

    Joined:
    Jun 2, 2015
    Messages:
    260
    Likes:
    323
    Degree:
    1
  13. Concept

    Concept

    Joined:
    May 17, 2015
    Messages:
    230
    Likes:
    307
    Degree:
    1
    :evil: Love being back in the game
     
  14. CCarter

    •  
    •  
    •  
    CCarter If they cease to believe in u, do u even exist? Staff Member

    •  
    •  
    Joined:
    Sep 15, 2014
    Messages:
    1,735
    Likes:
    3,679
    Degree:
    5
    [​IMG]
     
    Concept likes this.
  15. Concept

    Concept

    Joined:
    May 17, 2015
    Messages:
    230
    Likes:
    307
    Degree:
    1
    [​IMG]
    :evil:
     
    Bastion likes this.
  16. SmokeTree

    •  
    •  
    SmokeTree Developer/Linux Consultant

    Joined:
    Sep 23, 2014
    Messages:
    163
    Likes:
    245
    Degree:
    1
    Messing with scraping sites in Elixir and getting it all to work with Chrome in "headless" mode (https://developers.google.com/web/updates/2017/04/headless-chrome). The end goal is to have a fleet of scrapers that can all be controlled by either a web based GUI or by API calls. I've got something similar already that I've coded in Ruby, but there is nothing that even begins to come close to the level of fault-tolerance I can achieve with Elixir.
     
    Calamari likes this.
  17. Jared

    Jared Breaking the Shackles of a Lifetime of Bummery

    Joined:
    Apr 27, 2015
    Messages:
    222
    Likes:
    211
    Degree:
    1
    Grinding Saturday night, grinding Sunday morning . . . grinding damn near 24 hours a day.
     
    real likes this.
  18. Jared

    Jared Breaking the Shackles of a Lifetime of Bummery

    Joined:
    Apr 27, 2015
    Messages:
    222
    Likes:
    211
    Degree:
    1
    Wakey-wakey . . .
     
  19. CCarter

    •  
    •  
    •  
    CCarter If they cease to believe in u, do u even exist? Staff Member

    •  
    •  
    Joined:
    Sep 15, 2014
    Messages:
    1,735
    Likes:
    3,679
    Degree:
    5
    [​IMG]

    Trying to get some API thingy going...
     
    SmokeTree likes this.
  20. Ryuzaki

    •  
    •  
    Ryuzaki 女性以上のお金 Staff Member

    Joined:
    Sep 3, 2014
    Messages:
    2,223
    Likes:
    3,924
    Degree:
    6
    Been all day trying to find content to have rewritten.

    This is reminding me how much garbage is out there on the internet and how easy it should be for most people fly past 95% of competitors. Unfortunately for them, 95% of those people fall in that "95% of competitors."
     
    SmokeTree likes this.
  21. Trankuility

    •  
    Trankuility Breathe the body deep

    Joined:
    Oct 28, 2015
    Messages:
    538
    Likes:
    532
    Degree:
    2
    Workin and twerkin.

    [​IMG]

    I do it for the children.
     
    Jared, animalstyle and SmokeTree like this.
  22. SmokeTree

    •  
    •  
    SmokeTree Developer/Linux Consultant

    Joined:
    Sep 23, 2014
    Messages:
    163
    Likes:
    245
    Degree:
    1
    I wrote here a couple weeks back about messing with headless Chrome in Elixir. Since then I've created several modules for my own use and a couple of Internal projects for Clients. I'm adding the finishing touches to the following module I designed to get up to 2 pages of results (configurable via a "page_depth" parameter) from Google. This is how it looks when I run it on my dev box in IEx, which is Elixir's interactive shell:

    Code:
    iex(20)> results = BigG.get_results_for_keyword("best coffee maker 2017", 1, false, proxy_address, user_agent)
    [debug] Starting Session
    [debug] Ending any old sessions
    [debug] Starting Headless Chrome Session
    [debug] Navigating to Google Homepage
    [debug] Page Title is Google
    [debug] On Google Homepage
    [debug] CAPTCHA Not Detected
    [debug] Searching for: best coffee maker 2017
    [debug] Filling In Form
    [debug] Sending Enter Key Event
    [debug] Waiting
    [debug] CAPTCHA Not Detected
    [debug] There are 10 organic results
    [debug] Ending Session
    {:found_results,
     %{page_one_results: [%{description: "Jul 7, 2017 - The coffee machine you use at home is a very personal decision and choosing the right model can be tricky. Let us help!",
          position: 1, title: "Best Coffee Makers of 2017 - CNET",
          url: "https://www.cnet.com/topics/coffee-makers/best-coffee-makers/"},
        %{description: "Ninja Coffee Bar Brewer, Thermal Carafe (CF087) Cuisinart DCC-3200. BUNN NHS Velocity Brew 10-Cup Home Coffee Brewer. Zojirushi EC-YGC120. Mr. Coffee 10-Cup Thermal coffee maker System PSTX91. Hamilton Beach 49980A Mr. Coffee 4-Cup Programmable coffee maker, DRX5.",
          position: 2,
          title: "Top 10: Best Automatic Drip Coffee Makers of 2017 - Little Coffee Place",
          url: "https://www.littlecoffeeplace.com/coffee-makers"},
        %{description: "Cuisinart Perfect Temp 14-Cup Programmable Coffeemaker. Zojirushi Fresh Brew (Plus Thermal Carafe) Coffee Maker. The Cuisinart DGB-700BC Grind-and-Brew 12-Cup Automatic Coffeemaker. The Bonavita 8-Cup Stainless Steel Coffee Brewer. Technivorm Moccamaster 10-Cup Coffee Brewer with Thermal Carafe.",
          position: 3,
          title: "The 10 Best Drip Coffee Makers of 2017 – Choose Wisely - Home ...",
          url: "https://www.homegrounds.co/best-drip-coffee-maker-reviews/"},
        %{description: "Looking for the perfect drip coffee maker for your home? We have reviewed the best drip coffee makers out there to help you make the buying decision.",
          position: 4,
          title: "Best Drip Coffee Makers Reviews [Tested] - Top Picks 2017",
          url: "https://freshpresso.net/best-drip-coffee-makers/"},
        %{description: "We've reviewed the latest and best coffee makers of 2016 at the Good Housekeeping Institute, so you don't have to. Here are the ones you want!",
          position: 5,
          title: "50+ Best Coffee Makers & Coffee Machine Reviews 2017",
          url: "http://www.goodhousekeeping.com/appliances/coffee-maker-reviews/"},
        %{description: "Drip coffee machines can be fickle. Sometimes, the coffee tastes watered-down. Other times, it tastes like you're sipping on a cup of charred coal. But with the ...",
          position: 6,
          title: "The Best Coffee Maker List (2017) - BrownsCoffee.com",
          url: "http://www.brownscoffee.com/best-coffee-maker-list/"},
        %{description: "May 22, 2017 - Looking for a budget-friendly coffeemaker? One that's single-serve? Or better yet, one that doubles as a cappuccino maker? We have an option ...",
          position: 7,
          title: "20 Best Coffee Makers of 2017 - Reviews of Coffee Machines & Maker ...",
          url: "http://www.bestproducts.com/appliances/small/g216/best-coffee-makers-machines/"},
        %{description: "The humble coffee maker is any coffee-lovers best friend, so it's important to find just the right model and style for the type of brew you love. Whether you like a ...",
          position: 8,
          title: "The 7 Best Coffee Makers to Buy in 2017 - The Spruce",
          url: "https://www.thespruce.com/best-coffee-makers-to-buy-4118535"},
        %{description: "... venturing downtown to the nearest cafe? Then check out some of our favorite coffee makers on the market, from Bunn to Chemex. ... Home > Home > The best coffee maker you can buy ... By Dan Evon — Updated May 12, 2017 10:02 am. 7.",
          position: 9, title: "The Best Coffee Makers You Can Buy | Digital Trends",
          url: "https://www.digitaltrends.com/home/the-best-coffee-makers/"},
        %{description: "Mar 17, 2017 - Take a look at our picks of the top rated automatic drip coffee makers for 2017. These have received consistently high ratings from a lot of the ...",
          position: 10, title: "10 Best Drip Coffee Maker - Reviews 2016 | CMPicks",
          url: "http://coffeemakerpicks.com/best-drip-coffee-makers/"}],
       page_two_results: [], total_results: 46800000}}
    
    If results are found, they get returned as a tuple containing the result state as well as a "list of maps" of the results which includes the Title, Position and Meta Desc of each one. The module can also detect CAPTCHA, rotate proxies/user agent each request and detect error states such as timeouts. I don't have any plans to make the module available to the public but I can at least share that all anyone needs to do this in Elixir is to have the latest/greatest Chrome installed, the latest version of Chromedriver (Can skip Chrome and use PhantomJS), "Hound" (https://github.com/HashNuke/hound) and around 250-300 lines of code to make it do the thing.
     
    ryandiscord likes this.
  23. Calamari

    •  
    •  
    Calamari

    Joined:
    Oct 6, 2014
    Messages:
    623
    Likes:
    743
    Degree:
    3
    Add me to the list. My head is only half in it so it's not grinding, but it's still production.
     
    SmokeTree likes this.
  24. ryandiscord

    ryandiscord

    Joined:
    Apr 7, 2016
    Messages:
    187
    Likes:
    114
    Degree:
    1
    This looks awesome. What are some of your intended uses for this, automating rank tracking, keyword research? Could something like this be modified to track Amazon prices and buybox ownership?
     
  25. SmokeTree

    •  
    •  
    SmokeTree Developer/Linux Consultant

    Joined:
    Sep 23, 2014
    Messages:
    163
    Likes:
    245
    Degree:
    1
    One of the basic uses would be checking a URL to make sure it's indexed, which can be used as one of the facets of a web asset monitoring app, part of one of my current projects. The end game for me is to have my own collection of private APIs that I can use (or a partner in a JV) to get data from and interact with Google, Bing, Twitter, Facebook, Pinterest, etc. Building modules/services/APIs like this is pretty much what I do on weekends and for fun in general.

    If you're interested in building things like this in general, Python might be a better language to start with as Elixir syntax and functional programming in general is a bit different from PHP, Perl, Ruby, C#. @turbin3 wrote a guide on scraping with Python and Scrapy that I think is one of the best written resources on general scraping I've seen and is definitely worth the read.