Recurring tasks for your web app?
Use cron + curl.

Work on a web app long enough, and you’ll come across some sort of task that needs to be accomplished by your app at regular intervals. Some examples might include pruning inactive user accounts, sending reminder emails, or running custom analytics.

My first introduction to these sorts of tasks was at a previous job. A Rails app needed to run a task at regular intervals, and used a gem called Clockwork. At first glance, this seems to be a decent solution to the problem – a lightweight Ruby daemon which triggers your code at regular intervals, using a nice DSL. However, after using it for a while, I decided it perhaps wasn’t the perfect tool. First of all, it adds a dependency to your app, and you need to take the time to learn the API and how to use it. You also need to monitor the Clockwork process to ensure that it restarts after a crash. You also need to ensure that the Clockwork process runs on a single app server; otherwise your job will be run multiple times. Plus, it turns out that timezones are hard.

When a recent requirement came up that could be solved by a periodic task, I ruled Clockwork out, and decided to use some other common tools to accomplish the same goal. As you may have surmised, I ended up using cron to invoke curl, and hit a private URL in my app. While I’m sure many folks can point out downsides to such an approach, it has a lot of appeal to me:

  1. Uses very basic, well-understood software that ships with most (all?) Linux systems.
  2. Hits your webapp, so you don’t have to deal with setting up a DB connection pool, logging, etc.

Cron syntax inscrutable to you? A quick Google search should clear it up nicely. Don’t want to expose a sensitive or resource-intensive task via a public URL? Use basic auth to secure it, returning a 404 if the auth is missing. As a bonus, cron will use your systems time, so you don’t have to worry about the scheduled job missing a beat during DST (yes, this did in fact happen to me).

Comments 0 comments · Posted by Posted by Nathan at 14:03 · Tags cron curl nerdery linux


Signup form automation with Selenium

For a recent project, I was tasked with creating a large number of user accounts on various 3rd party websites. Specifically, Spotify, Last.fm, and Basis. I started off by doing the ol’ copypasta, but after about the third Spotify account, I realized that I was going to go brain dead before finishing them all. Then I remembered Selenium. I had my list of proposed usernames, email addresses, and passwords – why couldn’t I just feed those into an automated browser instance?

Well, the answer is, of course, that I could – with varying degrees of success. For most account-based websites, the general steps you’ll take to create a bunch of accounts is as follows:

  1. Go to signup page
  2. Enter user data
  3. Submit form
  4. Log out

Spotify was the easiest of the three to automate. Their signup form is very basic, with (surprisingly) no CAPTCHA or email validation. The only problem is that when you successfully submit the form, the resulting page automatically prompts a desktop client download. I wasn’t able to quickly find a way to dismiss that, so simply paused my script so that I could manually click “cancel.” Much easier than filling out the form.


require "selenium-webdriver"
driver = Selenium::WebDriver.for :firefox

accounts = [
  # account info redacted...
]

accounts.each do |account|
  driver.navigate.to "https://www.spotify.com/us/signup/"

  driver.find_element(:name, 'username').send_keys(account[:username])
  driver.find_element(:name, 'password').send_keys(account[:password])
  driver.find_element(:name, 'email').send_keys(account[:email])
  driver.find_element(:name, 'confirm_email').send_keys(account[:email])

  driver.find_element(:name, 'dob_month').send_keys(months[(0..11).to_a.sample])
  driver.find_element(:name, 'dob_day').send_keys((1..28).to_a.sample)
  driver.find_element(:name, 'dob_year').send_keys((1950..2000).to_a.sample)
  driver.find_element(:id, 'register-male').click

  driver.find_element(:id, 'register-button-email-submit').click

  # Enough time to manually dismiss client download prompt
  sleep 5

  driver.navigate.to "https://www.spotify.com/us/logout"
end

Last.fm was a bit tricker, as they include a reCAPTCHA widget in their form. While modern versions of reCAPTCHA look like just a simple checkbox, there is obviously some complex logic going on behind the scenes. As a human, you click their “I am not a robot” checkbox and you’re done. When a script fills out the form and clicks the checkbox element, the checkbox is checked, but an image recognition CAPTCHA is presented. Rather than try to figure out the logic behind triggering the additional level of protection, I simply stopped my script after loading the challenge, and manually clicked through. It was kind of fun, actually, to see the different queries and how fast I could complete them. I suspect the CAPTCHA is triggered by the absense of human-like activities (i.e. normal JavaScript events that would be triggered by a human, like mouseover), but didn’t feel like figuring it out, because I’m lazy.


require "selenium-webdriver"
require "pry"
driver = Selenium::WebDriver.for :firefox

accounts = [
  # account info redacted...
]

accounts.each do |account|
  driver.navigate.to "https://secure.last.fm/join"

  driver.find_element(:name, 'userName').send_keys(account[:username])
  driver.find_element(:name, 'email').send_keys(account[:email])
  driver.find_element(:name, 'password').send_keys(account[:password])
  driver.find_element(:name, 'passwordConf').send_keys(account[:password])
  driver.find_element(:id, 'id_terms').click

  captcha_iframe = driver.find_element(:css, 'iframe[title="recaptcha widget"]')
  driver.switch_to.frame(captcha_iframe)
  captcha_checkbox = driver.find_element(:id, 'recaptcha-anchor')
  driver.action.move_to(captcha_checkbox).click(captcha_checkbox).perform

  # switch back to the main document
  driver.switch_to.default_content

  # Wait until Control+D from user to continue
  binding.pry
end

Basis was the most annoying form to fill out. It’s some sort of terrible SPA, so you have to wait until the DOM content loads before filling in any data. It also has some sort of weird logic that requires each input to be tabbed into, or else it thinks the POST fails when the form is submitted (it’s not, actually). You can also tell it’s some horrible JavaScript form because some input elements don’t actually have name attributes, which also makes them terribly annoying to find/fill out. I ended up enumerating over all the input elements, then trying to differentiate based on a data-bound-to property. The problem was that multiple elements had the same data-bound-to value, so I had to ignore any subsequent elements after the first. This form also asks for a lot more data (name, height, weight, etc.) that I had to generate. That’s why I love Ruby’s Array#sample method.


require "selenium-webdriver"
require "pry"
driver = Selenium::WebDriver.for :firefox

accounts = [
  # account info redacted...
]

accounts.each do |account|
  driver.navigate.to "https://app.mybasis.com/#register"

  # It's a stupid Javascript app, so wait until the DOM elements load
  wait = Selenium::WebDriver::Wait.new(:timeout => 10) # seconds
  wait.until { driver.find_element(:name, 'email') }

  # Email
  driver.find_element(:name, 'email').send_keys(account[:email])
  driver.find_element(:name, 'confirm_email').send_keys(account[:email])

  # Password
  driver.find_element(:name, 'password').send_keys(account[:password])
  driver.find_element(:name, 'confirm_password').send_keys(account[:password])

  # Birthday
  birthday = "#{(1..12).to_a.sample.to_s.rjust(2, "0")}/#{(1..28).to_a.sample.to_s.rjust(2, "0")}/#{(1970..2000).to_a.sample}"
  driver.find_element(:name, 'anatomy.dob').send_keys(birthday)

  # Gender
  driver.find_element(:name, 'gender').click

  # Timezone
  driver.find_element(:name, 'settings.timezone').send_keys("(GMT-04:00) New York")

  # Find elements that don't have any sort of queryable attribute
  height_counter = 0
  weight_counter = 0
  elements = driver.find_elements(:css, 'input')
  elements.each do |elem|
    case elem.attribute('data-bound-to')
    when 'n:profile.first_name'
      elem.send_keys(%w(James John Robert Michael William Mary Patricia Linda Barbera Elizabeth).sample)
    when 'n:profile.last_name'
      elem.send_keys(%w(Smith Doe Brown Jones Johnson Miller Davis Wilson Anderson Taylor).sample)
    when 'n:anatomy.height'
      if height_counter == 2
        next
      elsif height_counter == 1
        elem.send_keys((0..11).to_a.sample)
      else
        elem.send_keys((4..6).to_a.sample)
      end
      height_counter += 1
    when 'n:anatomy.weight'
      if weight_counter == 1
        next
      else
        elem.send_keys((110..180).to_a.sample)
      end
      weight_counter += 1
    end
  end

  # Manually submit form, then log out
  binding.pry
end

I’ll probably never have to do anything similar in the near future, but hopefully these examples will help you out if you have a similar (mind-numbing) task in your future.

Comments 0 comments · Posted by Posted by Nathan at 11:03 · Tags ruby web selenium


Review: Nexus 5X

One of the problems with being a technophile is the rapid pace at which new technology is released. It seems that every other day I see something drool-worthy that would make my life immeasurably better, if only I could cram it into my house alongside my other troves of junk. I try to resist these base urges, but don’t always succeed. A few weeks ago I was tempted by Google’s Black Friday sale, and ended up purchasing a new 32GB Nexus 5X for $350 (the same price I paid for a 16GB Nexus 4, three years ago), and figured I’d provide my thoughts regarding one of Google’s newest reference devices.

When reading other reviews before my purchase, I was worried about the “feel” of the phone. It’s made out of plastic, and one reviewer at The Verge mentioned that it felt cheap and “hollow.” Personally, I didn’t experience the same reaction – the phone feels fine to me. It is made out of plastic, which I was worried about. The phones that I’ve used the most in previous years (iPhone 4/5) have been primarily constructed of glass/metal, which are heavy and feel nice. The Nexus 5X is surprisingly light (as you might expect) for being a 5”+ phone, but the matte plastic finish on the back is actually pleasing. To be honest, I think that plastic is a decent tradeoff for a phone these days, even though “premium” phones have metal bodies. The problems with the metal enclosure are weight, as well as the fact that metal interferes with wireless reception. What Apple (and other manufacturers) have done is to make cutouts in the back of the phone. The iPhone 5 had a decent-looking solution for this, which was to use glass panels at the top and bottom of the phone’s back. The glass panels obviously didn’t match at all with the aluminum enclosure, but it looked pretty good. The iPhone 6 family eschews the glass cutouts for plastic lines around the back of the phone, and they look terrible. To be honest, I prefer the solid plastic back of the Nexus – at least it’s a uniform.

The other area where I prefer the design of the Nexus 5X is its camera bump. I guess we’ve reached a point these days where the demands of high-performance camera lenses and thin enclosures mean that camera lenses protrude from the back of our phones. The iPhone 6 has a little nub, which asymmetrically juts out from the upper right side of the phone. The Nexus 5X puts the camera in the center of the body, and the plastic back smoothly rises up around it to create a pleasing beveled effect. Seriously, if you’re going to have the camera extend from the phone’s back, at least embrace it, as opposed to making it look like an embarrassing afterthought.

My last phone was an iPhone 5, so I’ve never used Touch ID. For that reason, I can’t really compare it to Google’s version, which they call Nexus Imprint. The position of the fingerprint reader on the Nexus 5X is OK – yes, you have to pick the phone up in order to use it, but one of the things I like about Android phones is the software buttons, so the sensor had to go somewhere else. The sensor will both turn on and unlock the phone, as opposed to the power button (which shows the lock screen), so it’s nice to have that choice (I use the lock screen for media controls). I will say that once you have a phone with biometric security, it would be hard to go back – it’s obviously super convenient. It’s pretty egregious that Apple has had a lock on this tech for the last two years, but I guess that’s what you get when you buy the company.

USB Type-C. I like it. It’s the future for sure. Reversible connectors are so much better it’s not even funny. But for now, it’s kind of a pain – you’ll have to buy some new cables or adapters. I wanted to do some remote debugging, and was annoyed to remember that I needed to physically connect the phone, and didn’t have a Type-A to Type-C cable. One good thing about the iPhone’s ubiquity is that everyone has a Lightening adapter. You can forget your charger at home and it’s not too big of a deal. If you don’t bring a USB-C cable for your Nexus, you’re outta luck.

As far as the rest of the package, the software is Google. I find that I use Google services a lot more these days, and they have the added advantage of a web-based presence. For example, I can upload a book to Play Books from my computer, start reading, then download the book/pick up where I left off on my phone. I really like the Google Now launcher, which to be honest is still probably the best part of Android phones. There are a few games that don’t have Android ports (Kero Blaster, for example), but I can live without those. I like how Android/Chrome makes web apps more like first-class citizens. In general, the software Just Works.

I have until the end of January to decide whether or not I want to keep the Nexus, but as of right now I’m thinking I’ll stick with it. It does enough of the things I care about well enough for me to be satisfied. At least until I see the next iPhone revision.

Update

I ended up returning the phone. What an anti-climax, eh? I really liked the hardware, and to be honest I kinda like Android’s UI more than iOS. It really came down to two things: slow interface, and bad camera. I kept on experiencing random delays when trying to launch apps, or otherwise interact with the phone. Not terrible, but enough to be on par with my 3 year old iPhone 5. What’s the point of buying a new phone if it performs the same as the old one? The camera was also a disappointment. It took decent shots, but I had two separate instances where I pulled the phone out to take a picture, and couldn’t. The first time, it gave me an error message like “Camera is disconnected.” Um, what? I had to reboot. Second time, the camera app just never started. I had done the ol’ “double-tap power button to wake camera” trick, and it just sat there with a black screen. Both times I missed the shot I wanted; even a slow camera is better than none at all.

I read the other day that Google is publishing an update which supposedly fixes some of the sluggish performance of the 5X. It is tempting, but I’m not sure I want to pay to be a beta tester. Maybe I’ll take the leap again if I can independently verify the fix is, in fact, a fix.

Comments 0 comments · Posted by Posted by Nathan at 16:12 · Tags google android nexus review


Bare-bones SSH Tunneling

This year, I wanted to work remotely for a day or two in the week leading up to Christmas. I went through the trouble of getting an RSA dongle and setting up the terrible VPN software on my machine, only to discover that I couldn’t access some resources that were within my company’s DMZ. When I asked about this, the “official” response was that it was expected for users to connect to an intermediary machine first, then connect to the desired (restricted access) resource. Whiskey Tango Foxtrot! The problem here is that I don’t have an intermediary machine.

This situation can be (mostly) solved by using SSH tunneling, which can take network traffic directed at a local port, and send it to a remote port via a man-in-the-middle server. In my case, I have access to an internal-only webserver, which itself has access to the DMZ. The two tasks that I needed to do on restricted server were HTTPS access (port 443) and SFTP access (port 22).

HTTPS


ssh -L 3000:<inaccessible_server>:443 <your_username>@<accessible_server> -N

The -L option means we want to do local port forwarding, and -N specifies that no command should be executed on the remote server. You could also add -f if you want to put the ssh process in the background. Run this command, and load localhost:3000 in your browser. Unfortunately, you’ll see a “certificate mismatch” error message, because the SSL certificate doesn’t match the URL. A hacky way to fix this is to modify your /etc/hosts file, and add an alias (on a new line):


127.0.0.1 <inaccessible_server_url>

Then you can hit <inaccessible_server_url> as (almost) normal, the only difference being that you have to specify port 3000 instead of the normal HTTPS port 443.

SFTP

Basically do the same thing as with the HTTPS setup, except change the remote port to be 22:


ssh -L 3000:<inaccessible_server>:22 <your_username>@<accessible_server> -N

Add your hosts alias, then modify your SFTP connection to hit port 3000, and connect as normal.

(Note that adding the DNS alias only matters if you’re accessing the server via URL; if you’re connecting via IP, you’ll have to change the server to localhost.)

References

Comments 0 comments · Posted by Posted by Nathan at 15:12 · Tags SSL VPN


PM2: The Swiss Army knife
for your Node.js app

I recently discovered PM2. It is great. Tell your friends. I can’t believe that one app does so many (necessary) things:

(Re)starts your app

A naive approach to running a Node application would be to start the process and put it in the background. But what happens when it (inevitably) crashes? Give PM2 your application configuration, and it will start the app and ensure it respawns after a crash.

Monitor memory/CPU usage

App got a memory leak? Churning CPU? Watch your usage levels in real time.

Run app in fork/cluster mode

Normally you’d have to write your app specifically to take advantage of clustering, but with PM2, you can just specify whether you want a single process or multiple processes. By default, it’ll spin up as many workers as your CPU has cores.

Log output from stdin/stderr

While there are a number of sophisticated logging solutions for Node, at this point in my usage of it I really only care about getting console.log statements and stack traces written out to external files. By default, Node outputs these messages to stdout/stderr, which PM2 will watch and write off to log files. You can easily get a stream of logs using pm2 logs, or limit to one application using pm2 logs <appname>.

Easily set up log rotation

As with any web application that generates log files, after a while the files get too large and unwieldy. PM2 has a command that will automatically generate a logrotate config file that targets the default PM2 log location.

Auto-restart app during development

When developing, you have to restart your Node server in order to see changes. Frameworks like Rails have a “development” mode they can use to re-load code before each request, but with Node you’ll probably rely on something like supervisor, which watches the filesystem of your project for changes, then kills/respawns your Node process. Or you can just use PM2, which does the same thing.

Start app after server reboot

This is a necessary evil that I’d wager many developers don’t think about, similar to log rotation. Unless you create a custom init script, if your server ever reboots, you will have to log in and restart the PM2 daemon. PM2 can generate a startup script for you!

Deployment

I haven’t used this aspect of PM2, but am hoping to soon. Instead of manually SSH’ing to your server in order to do deploys, just use the functionality built in to PM2, which will check out code from your VCS and (re)start the Node process. It supports multiple server targets, as well as rollbacks.

Comments 0 comments · Posted by Posted by Nathan at 12:12 · Tags node.js sysadmin pm2