For a recent project, I was tasked with creating a large number of user accounts
on various 3rd party websites. Specifically, Spotify, Last.fm, and Basis. I started
off by doing the ol’ copypasta, but after about the third Spotify account, I realized
that I was going to go brain dead before finishing them all. Then I remembered
Selenium. I had my list of proposed usernames, email
addresses, and passwords – why couldn’t I just feed those into an automated browser
instance?
Well, the answer is, of course, that I could – with varying degrees of success.
For most account-based websites, the general steps you’ll take to create a bunch
of accounts is as follows:
Go to signup page
Enter user data
Submit form
Log out
Spotify was the easiest of the three to automate. Their signup form is very basic,
with (surprisingly) no CAPTCHA or email validation. The only problem is that
when you successfully submit the form, the resulting page automatically prompts
a desktop client download. I wasn’t able to quickly find a way to dismiss that,
so simply paused my script so that I could manually click “cancel.” Much easier
than filling out the form.
Last.fm was a bit tricker, as they include a reCAPTCHA widget in their form.
While modern versions of reCAPTCHA look like just a simple checkbox, there is
obviously some complex logic going on behind the scenes. As a human, you click
their “I am not a robot” checkbox and you’re done. When a script fills out
the form and clicks the checkbox element, the checkbox is checked, but an
image recognition CAPTCHA is presented. Rather than try to figure out the logic
behind triggering the additional level of protection, I simply stopped my script
after loading the challenge, and manually clicked through. It was kind of fun,
actually, to see the different queries and how fast I could complete them.
I suspect the CAPTCHA is triggered by the absense of human-like activities
(i.e. normal JavaScript events that would be triggered by a human, like mouseover),
but didn’t feel like figuring it out, because I’m lazy.
require "selenium-webdriver"
require "pry"
driver = Selenium::WebDriver.for :firefox
accounts = [
# account info redacted...
]
accounts.each do |account|
driver.navigate.to "https://secure.last.fm/join"
driver.find_element(:name, 'userName').send_keys(account[:username])
driver.find_element(:name, 'email').send_keys(account[:email])
driver.find_element(:name, 'password').send_keys(account[:password])
driver.find_element(:name, 'passwordConf').send_keys(account[:password])
driver.find_element(:id, 'id_terms').click
captcha_iframe = driver.find_element(:css, 'iframe[title="recaptcha widget"]')
driver.switch_to.frame(captcha_iframe)
captcha_checkbox = driver.find_element(:id, 'recaptcha-anchor')
driver.action.move_to(captcha_checkbox).click(captcha_checkbox).perform
# switch back to the main document
driver.switch_to.default_content
# Wait until Control+D from user to continue
binding.pry
end
Basis was the most annoying form to fill out. It’s some sort of terrible SPA,
so you have to wait until the DOM content loads before filling in any data.
It also has some sort of weird logic that requires each input to be tabbed into,
or else it thinks the POST fails when the form is submitted (it’s not, actually).
You can also tell it’s some horrible JavaScript form because some input elements
don’t actually have name attributes, which also makes them terribly annoying
to find/fill out. I ended up enumerating over all the input elements, then trying
to differentiate based on a data-bound-to property. The problem was that multiple
elements had the same data-bound-to value, so I had to ignore any subsequent
elements after the first. This form also asks for a lot more data (name, height, weight, etc.)
that I had to generate. That’s why I love Ruby’s Array#sample method.
require "selenium-webdriver"
require "pry"
driver = Selenium::WebDriver.for :firefox
accounts = [
# account info redacted...
]
accounts.each do |account|
driver.navigate.to "https://app.mybasis.com/#register"
# It's a stupid Javascript app, so wait until the DOM elements load
wait = Selenium::WebDriver::Wait.new(:timeout => 10) # seconds
wait.until { driver.find_element(:name, 'email') }
# Email
driver.find_element(:name, 'email').send_keys(account[:email])
driver.find_element(:name, 'confirm_email').send_keys(account[:email])
# Password
driver.find_element(:name, 'password').send_keys(account[:password])
driver.find_element(:name, 'confirm_password').send_keys(account[:password])
# Birthday
birthday = "#{(1..12).to_a.sample.to_s.rjust(2, "0")}/#{(1..28).to_a.sample.to_s.rjust(2, "0")}/#{(1970..2000).to_a.sample}"
driver.find_element(:name, 'anatomy.dob').send_keys(birthday)
# Gender
driver.find_element(:name, 'gender').click
# Timezone
driver.find_element(:name, 'settings.timezone').send_keys("(GMT-04:00) New York")
# Find elements that don't have any sort of queryable attribute
height_counter = 0
weight_counter = 0
elements = driver.find_elements(:css, 'input')
elements.each do |elem|
case elem.attribute('data-bound-to')
when 'n:profile.first_name'
elem.send_keys(%w(James John Robert Michael William Mary Patricia Linda Barbera Elizabeth).sample)
when 'n:profile.last_name'
elem.send_keys(%w(Smith Doe Brown Jones Johnson Miller Davis Wilson Anderson Taylor).sample)
when 'n:anatomy.height'
if height_counter == 2
next
elsif height_counter == 1
elem.send_keys((0..11).to_a.sample)
else
elem.send_keys((4..6).to_a.sample)
end
height_counter += 1
when 'n:anatomy.weight'
if weight_counter == 1
next
else
elem.send_keys((110..180).to_a.sample)
end
weight_counter += 1
end
end
# Manually submit form, then log out
binding.pry
end
I’ll probably never have to do anything similar in the near future, but hopefully
these examples will help you out if you have a similar (mind-numbing) task in your future.
One of the problems with being a technophile is the rapid pace at which new technology is
released. It seems that every other day I see something drool-worthy that would
make my life immeasurably better, if only I could cram it into my house alongside
my other troves of junk. I try to resist these base urges, but don’t always
succeed. A few weeks ago I was tempted by Google’s Black Friday sale, and ended
up purchasing a new 32GB Nexus 5X for $350 (the same price I paid for a 16GB Nexus 4,
three years ago), and figured I’d provide my thoughts regarding one of Google’s newest
reference devices.
When reading other reviews before my purchase, I was worried about the “feel” of
the phone. It’s made out of plastic, and one reviewer at The Verge mentioned that
it felt cheap and “hollow.” Personally, I didn’t experience the same reaction –
the phone feels fine to me. It is made out of plastic, which I was worried
about. The phones that I’ve used the most in previous years (iPhone 4/5) have
been primarily constructed of glass/metal, which are heavy and feel nice. The
Nexus 5X is surprisingly light (as you might expect) for being a 5”+ phone,
but the matte plastic finish on the back is actually pleasing. To be honest, I
think that plastic is a decent tradeoff for a phone these days, even though
“premium” phones have metal bodies. The problems with the metal enclosure are
weight, as well as the fact that metal interferes with wireless reception. What
Apple (and other manufacturers) have done is to make cutouts in the back of the
phone. The iPhone 5 had a decent-looking solution for this, which was to use
glass panels at the top and bottom of the phone’s back. The glass panels obviously
didn’t match at all with the aluminum enclosure, but it looked pretty good. The
iPhone 6 family eschews the glass cutouts for plastic lines around the back
of the phone, and they look terrible. To be honest, I prefer the solid plastic
back of the Nexus – at least it’s a uniform.
The other area where I prefer the design of the Nexus 5X is its camera bump.
I guess we’ve reached a point these days where the demands of high-performance
camera lenses and thin enclosures mean that camera lenses protrude from the
back of our phones. The iPhone 6 has a little nub, which asymmetrically juts
out from the upper right side of the phone. The Nexus 5X puts the camera in
the center of the body, and the plastic back smoothly rises up around it to create a
pleasing beveled effect. Seriously, if you’re going to have the camera extend
from the phone’s back, at least embrace it,
as opposed to making it look like an embarrassing afterthought.
My last phone was an iPhone 5, so I’ve never used Touch ID. For that reason,
I can’t really compare it to Google’s version, which they call Nexus Imprint.
The position of the fingerprint reader on the Nexus 5X is OK – yes, you have
to pick the phone up in order to use it, but one of the things I like about
Android phones is the software buttons, so the sensor had to go somewhere else.
The sensor will both turn on and unlock the phone, as opposed to the power
button (which shows the lock screen), so it’s nice to have that choice (I use
the lock screen for media controls). I will say that once you have a phone with
biometric security, it would be hard to go back – it’s obviously super convenient.
It’s pretty egregious that Apple has had a lock on this tech for the last two
years, but I guess that’s what you get when you buy the company.
USB Type-C. I like it. It’s the future for sure. Reversible connectors are so much
better it’s not even funny. But for now, it’s kind of a pain – you’ll have to
buy some new cables or adapters. I wanted to do some remote debugging, and was annoyed to
remember that I needed to physically connect the phone, and didn’t have a
Type-A to Type-C cable. One good thing about the iPhone’s ubiquity is
that everyone has a Lightening adapter. You can forget your charger at home
and it’s not too big of a deal. If you don’t bring a USB-C cable for your Nexus,
you’re outta luck.
As far as the rest of the package, the software is Google. I find that I use
Google services a lot more these days, and they have the added advantage of
a web-based presence. For example, I can upload a book to Play Books from my
computer, start reading, then download the book/pick up where I left off on
my phone. I really like the Google Now launcher, which to be honest is still
probably the best part of Android phones. There are a few games that don’t have
Android ports (Kero Blaster, for example), but I can live without those. I like
how Android/Chrome makes web apps more like first-class citizens. In general,
the software Just Works.
I have until the end of January to decide whether or not I want to keep the
Nexus, but as of right now I’m thinking I’ll stick with it. It does enough of
the things I care about well enough for me to be satisfied. At least until I
see the next iPhone revision.
Update
I ended up returning the phone. What an anti-climax, eh? I really liked the hardware,
and to be honest I kinda like Android’s UI more than iOS. It really came down
to two things: slow interface, and bad camera. I kept on experiencing random
delays when trying to launch apps, or otherwise interact with the phone. Not terrible,
but enough to be on par with my 3 year old iPhone 5. What’s the point of buying a
new phone if it performs the same as the old one? The camera was also a disappointment.
It took decent shots, but I had two separate instances where I pulled the phone
out to take a picture, and couldn’t. The first time, it gave me an error message
like “Camera is disconnected.” Um, what? I had to reboot. Second time, the camera app just never
started. I had done the ol’ “double-tap power button to wake camera” trick, and
it just sat there with a black screen. Both times I missed the shot I wanted; even a slow camera
is better than none at all.
I read the other day that Google is publishing an update which supposedly fixes
some of the sluggish performance of the 5X. It is tempting, but I’m not sure
I want to pay to be a beta tester. Maybe I’ll take the leap again if I can
independently verify the fix is, in fact, a fix.
This year, I wanted to work remotely for a day or two in the week leading up
to Christmas. I went through the trouble of getting an RSA dongle and setting
up the terrible VPN software on my machine, only to discover that I couldn’t
access some resources that were within my company’s DMZ.
When I asked about this, the “official” response was that it was expected for
users to connect to an intermediary machine first, then connect to the desired
(restricted access) resource. Whiskey Tango Foxtrot! The problem here is that
I don’t have an intermediary machine.
This situation can be (mostly) solved by using SSH tunneling, which can take
network traffic directed at a local port, and send it to a remote port via
a man-in-the-middle server. In my case, I have access to an internal-only
webserver, which itself has access to the DMZ. The two tasks that I needed
to do on restricted server were HTTPS access (port 443) and SFTP access (port
22).
The -L option means we want to do local port forwarding, and -N specifies
that no command should be executed on the remote server. You could also add
-f if you want to put the ssh process in the background.
Run this command, and load localhost:3000 in your browser. Unfortunately, you’ll
see a “certificate mismatch” error message, because the SSL certificate doesn’t
match the URL. A hacky way to fix this is to modify your /etc/hosts file, and
add an alias (on a new line):
127.0.0.1 <inaccessible_server_url>
Then you can hit <inaccessible_server_url> as (almost) normal, the only difference
being that you have to specify port 3000 instead of the normal HTTPS port 443.
SFTP
Basically do the same thing as with the HTTPS setup, except change the remote
port to be 22:
Add your hosts alias, then modify your SFTP connection to hit port 3000, and
connect as normal.
(Note that adding the DNS alias only matters if you’re accessing the server via
URL; if you’re connecting via IP, you’ll have to change the server to localhost.)
A naive approach to running a Node application would be to start the process and
put it in the background. But what happens when it (inevitably) crashes? Give
PM2 your application configuration, and it will start the app and ensure it
respawns after a crash.
Normally you’d have to write your app specifically to take advantage of clustering,
but with PM2, you can just specify whether you want a single process or multiple processes. By default, it’ll spin up as many
workers as your CPU has cores.
While there are a number of sophisticated logging solutions for Node, at this point
in my usage of it I really only care about getting console.log statements and
stack traces written out to external files. By default, Node outputs these messages
to stdout/stderr, which PM2 will watch and write off to log files. You can easily
get a stream of logs using pm2 logs, or limit to one application using pm2 logs <appname>.
As with any web application that generates log files, after a while the files get too large and
unwieldy. PM2 has a command that will automatically generate a logrotate config
file that targets the default PM2 log location.
When developing, you have to restart your Node server in order to see changes.
Frameworks like Rails have a “development” mode they can use to re-load code
before each request, but with Node you’ll probably rely on something like supervisor,
which watches the filesystem of your project for changes, then kills/respawns
your Node process. Or you can just use PM2, which does the same thing.
This is a necessary evil that I’d wager many developers don’t think about,
similar to log rotation. Unless you create a custom init script, if your server
ever reboots, you will have to log in and restart the PM2 daemon. PM2 can generate
a startup script for you!
I haven’t used this aspect of PM2, but am hoping to soon. Instead of manually
SSH’ing to your server in order to do deploys, just use the functionality
built in to PM2, which will check out code from your VCS and (re)start the Node
process. It supports multiple server targets, as well as rollbacks.
When learning the code base of a new project, there are always places where
you look at what’s been written, and have a “WTF” moment. Normally, it’s best
to leave written code in place – if it ain’t broke, don’t fix it. However,
I let myself get a bit carried away with a recent project, and rewrote the
bulk of the server-side component. I thought I’d pull out a small example of
how you can take some verbose code, and condense it down to something more
readable.
In this example, we have an array of email addresses, some of which may be
prepended with the string "coordinator-". Why the original developer didn’t
just create two database columns, user_emails and coordinator_emails, I
can’t imagine. In any case, we want to send a message to the “non-coordinator”
email addresses.
Here was the original code:
var to = "";
for(var i=0; i<emails.length; i++) {
var email = emails[i];
if(email.indexOf('coordinator') == -1) {
to += email;
if(i != emails.length-1) to += ",";
}
}
to = to.slice(0, -1);
Straightforward enough, right? Create an empty string, and concatenate the
appropriate email addresses (include a separating comma), then remove the final
comma. So, if the emails array looks like ['bob@gmail.com', 'larry@gmail.com', 'coordinator-joe@gmail.com'],
the final output is the string "bob@gmail.com, larry@gmail.com".
But there’s an easier way to do this, and it involves being knowledgeable about
common Array manipulation methods. The first thing we want to do is remove any
email addresses that contain "coordinator-". Filter them out, if you will.
By Jove, there just happens to be an Array method named filter that we can use!
Next, we want to join the resulting email addresses together, with a comma between
them. What do you know? join is an Array method as well!
The whole thing boils down to one (long) line (pardon the ES6 syntax):