nathandemick.com

That time when I went to an all-girls junior high jazz band concert

I was reminiscing to a co-worker about my time in Japan, and told him a story that I’m not sure I’ve ever told anyone else. So, why not regale the internet with pointless personal anecdotes?

For those who don’t know, I taught ESL in Japan from the summer of 2004-2005. Somehow I had ended up connecting with the owner of a private school that was located in Utsunomiya. Definitely something of a backwater locale, even though it’s only 60 miles away from Tokyo. Most of my teaching was in the main office that was close to the center of the city, but I also had to regularly commute to a satellite office that was in nearby Tochigi (bizarrely, on a rail line operated by a department store). The school I worked for, being a small family-run business, had all ages of students: I sometimes “taught” preschool kids, and my oldest student was a guy in his 80s (whenever he missed a class I worried that he had died).

One of my students at this satellite office was a junior high student named Moe (pronounced moh-ay). She was actually one of my better students; since she was more fluent in English, our lessons were more high-level, and therefore less boring for me. Towards the end of the 2005 school year, my Japanese co-worker and I had finished a lesson with Moe, when she invited us both to a concert that her school band was giving. While my co-worker demurred, I thought, “Why not?” and told her I’d go (toward the end of my year-long contract, I’d determined that I was not going to stay in Japan, so attempted to have a “try anything” attitude during my remaining time).

The concert was on a Sunday, and the day before I was working in the Tochigi office as usual (yes, I had to work on Saturdays, and got Sunday/Monday off). Imagine my bemusement/concern when, at the end of the day, my co-worker suddenly pulls out a giant bouquet of flowers. “Here, you can give these to Moe.” she blathered. I worried about what people might think of a 20-something foreigner giving flowers to a young female student, but felt obligated at the same time. Cursing her under my breath, I took the bouquet, trying not to think about what I was going to do with it.

The next day, for the first time ever, I took the train to Tochigi for non-work-related purposes, and walked to the school. It felt pretty weird to be commuting on my day off. I ended up stuffing those damn flowers into my backpack, and figured I’d worry about them later. As I walked into the school auditorium’s foyer, I could almost hear people’s necks snapping as they turned to look at me. Of course, I was the only non-Japanese person there. And I don’t know why I didn’t think of it before, but of course it was an all girls school. The awkwardness of my presence had just increased exponentially. I brazened it out and found an inconspicuous seat near the back.

From what I remember, the concert itself was actually enjoyable. I didn’t go in expecting too much (remembering my own school-age band concerts), but as you might imagine, the Japanese always seem to take things to the next level. Not only was the music played impeccably, they also threw in some choreographed movement with their instruments.

After the thing ended, the students dispersed into the audience to receive congratulations from their various family members, and I had to deliver that albatross-like bouquet. Fortunately, I was able to hand it over pretty inconspicuously, thanks to the crowd.

After the meet ‘n greet, the students were called back up to the stage for a group photo. Imagine my chagrin when the photographer, seeing a white guy hanging around, yelled, “Who’s he with?” When it became known that I was Moe’s English teacher, he told me to get up on the stage for a photo, a suggestion that was enthusiastically received by the girls themselves. “Jeez, you’re tall!” I remember one remarking.

So, somewhere in Tochigi, there’s a photo of an all-girls junior high jazz band with a nerdy guy standing in the middle of the group, trying desperately not to look as awkward as he feels. Good times 😅.

· 0 comments

Review: Sony Xperia Z3 Compact

I seem to continually have dalliances with the Android world. Last year I bought (then sold) a Sony Xperia Z3, then later purchased/returned an LG Nexus 5X. I always seem to end up going back to my trusty iPhone 5, though. There are just enough dealbreakers with any Android phone that make them tough to live with as a daily driver.

Earlier this spring, I was revamping a few of my iOS apps, and figured that since they’re written using JavaScript and published as “apps” in a Cordova wrapper, I might as well try publishing on Google Play as well. I’d returned my Nexus 5X long ago, so started trolling Swappa for a used Android phone. I’d had my eye on the Xperia Z3 compact for a while: its stylish design, coupled with a large screen inside a small(-ish) chassis had me itching to try one out. I grabbed one for a decent price, released my apps to absolutely zero downloads, and then tried using it as my main phone for a bit.

The thing that I love the best about the Z3 Compact is its screen-to-body ratio. The whole device is just slightly larger than my iPhone 5, but the screen is about 33% larger. Watching videos and reading books is much more enjoyable. Plus, since the overall phone size is manageable, you can use it without resorting to the “smartphone claw grip of death.” The body of the phone looks pretty classy as well – I got the black version, which is like the prototypical black slab that all other smartphones descend from. Unfortunately, Sony never evolved its industrial design past the iPhone 4 era: the phone is all glass, save for the plastic sides. I always treated it fairly gingerly, as I was never sure how hard I could set it down without cracking the back.

Unfortunately, the software front is where most Android phones fall down for me. While most of Google’s software is on point (Chrome, Gmail), the Android messaging scene is pretty fragmented. What I want is a Google-endorsed messenger that piggybacks on top of SMS, similar to Apple’s iMessage. The app that might have fit that category (Hangouts) is deprecated, and users are advised to use a standalone SMS app. Rather than double down on Hangouts, Google instead is promoting two new messengers, Allo and Duo, that you’ll have to get all your friends to download in order for them to be useful at all. Blah. At that point I’d rather use Facebook Messenger, which at least most people have installed.

Sony also tries to include a lot of “value-added” software, which is mostly worthless. They have all sorts of extra media apps, but without a compelling syncing solution for my Mac, I didn’t load any music or videos onto the phone’s internal storage. A few bloatware-type apps (AR Fun, wtf) were immediately disabled (can’t delete these apps, of course). I could sign up for a “My Xperia” accout, but it’s not immediately apparent what benefit it would give me, aside from yet another set of account details to remember (cursory internet research tells me it’s similar to Apple’s Find My iPhone).

Dispite these annoyances, the phone was a pleasure to use, once I disabled/ uninstalled offending Sony software. Since I mostly rely on very basic smartphone apps (Maps, Email, Browser, Books), I don’t mind missing out on the hottest new iOS microtransaction bandit, errrr, game.

However, I ended up getting rid of the phone. Since my wife has an iPhone, the lack of iMessage is a real killer. She’d try to send me a video of the kids, and it would be sent as a hyper-compressed MMS. Static images would fare no better. Rather a minor thing, but messaging is the core of my smartphone use.

The other unfortunate thing is that the Z3 Compact has no upgrade path. It shipped with Android 4.4, and has been (slowly) upgraded through Android 6, but it’s anyone’s guess how much longer it will receive updates. While my iPhone 5 is nearing the end of its update life (4 years!), I know that I can buy a new phone with updated internals in the exact same form factor, and have it be supported for another 3+ years. The Z3 Compact has no obvious successor. Sony released the Z5 Compact last year, but it uses the problematic Snapdragon 810 processor, and the US version has its fingerprint sensor disabled(!). The Z5 Compact’s design is also a bit more stodgy. And the Z5 series is the last of the Z’s – Sony’s recent crop of X-series phones are about as forgettable as they come. I don’t want to get too attached to a dying phone.

I guess I’ll appreciate the Z3 Compact for what it was at the time, and hope that one day Apple will release an iPhone 5-sized device with an iPhone 6-sized screen.

· 0 comments

Self-Hosted Comments for a Static Blog

Last year, when I ditched my managed hosting, I converted my blog to use Jekyll, a static site generator. At the time I was pretty overwhelmed with all the tasks necessary to move my digital life, and didn’t want to throw “configure PHP/MySQL” onto the pile. Also, much to my chagrin, when I looked over my Wordpress installation prior to exporting its content, I noticed a bunch of suspicious-looking files that could only have been created by script kiddies exploiting Wordpress vulnerabilities. I thought I had been pretty concientious in keeping my instance up to date, but apparently not. So a static site it was.

One of the downsides to a static site is that… it’s static. No comments or any other form of interactivity. At first I tried hooking up Disqus, which I had actually also used in my Wordpress blog, rather than the native comments. Problem is, I dislike Disqus. As with any free service, if you aren’t paying, then you are the product. Managing a Disqus account is annoying, and it’s yet another 3rd party service that gets to track you around the web. So, during a bit of downtime, I created my own basic blog comments app.

It’s a Ruby app written with Sinatra that has two routes: GET and POST. Comments are stored in a SQLite database (I’m not anticipating heavy traffic). Include the client-side script wherever you want your comments to appear in a post. It uses reCAPTCHA for abuse prevention.

I learned a few things doing this project, including how to use nginx as a reverse proxy (only used Apache before), create an Upstart init script (sigh), and deploy a basic Ruby app on a VPS. It’s amazing how much of this stuff you don’t have to do when working at a company with sysadmin folks XD.

It should be fairly easy for anyone to take this code, made a few minor modifications, and run it for their own blog. I would also be happy to answer any questions about getting it up and running… just leave me a comment!

· 0 comments

Recurring tasks for your web app? Use cron + curl.

Work on a web app long enough, and you’ll come across some sort of task that needs to be accomplished by your app at regular intervals. Some examples might include pruning inactive user accounts, sending reminder emails, or running custom analytics.

My first introduction to these sorts of tasks was at a previous job. A Rails app needed to run a task at regular intervals, and used a gem called Clockwork. At first glance, this seems to be a decent solution to the problem – a lightweight Ruby daemon which triggers your code at regular intervals, using a nice DSL. However, after using it for a while, I decided it perhaps wasn’t the perfect tool. First of all, it adds a dependency to your app, and you need to take the time to learn the API and how to use it. You also need to monitor the Clockwork process to ensure that it restarts after a crash. You also need to ensure that the Clockwork process runs on a single app server; otherwise your job will be run multiple times. Plus, it turns out that timezones are hard.

When a recent requirement came up that could be solved by a periodic task, I ruled Clockwork out, and decided to use some other common tools to accomplish the same goal. As you may have surmised, I ended up using cron to invoke curl, and hit a private URL in my app. While I’m sure many folks can point out downsides to such an approach, it has a lot of appeal to me:

  1. Uses very basic, well-understood software that ships with most (all?) Linux systems.
  2. Hits your webapp, so you don’t have to deal with setting up a DB connection pool, logging, etc.

Cron syntax inscrutable to you? A quick Google search should clear it up nicely. Don’t want to expose a sensitive or resource-intensive task via a public URL? Use basic auth to secure it, returning a 404 if the auth is missing. As a bonus, cron will use your systems time, so you don’t have to worry about the scheduled job missing a beat during DST (yes, this did in fact happen to me).

· 0 comments

Signup form automation with Selenium

For a recent project, I was tasked with creating a large number of user accounts on various 3rd party websites. Specifically, Spotify, Last.fm, and Basis. I started off by doing the ol’ copypasta, but after about the third Spotify account, I realized that I was going to go brain dead before finishing them all. Then I remembered Selenium. I had my list of proposed usernames, email addresses, and passwords – why couldn’t I just feed those into an automated browser instance?

Well, the answer is, of course, that I could – with varying degrees of success. For most account-based websites, the general steps you’ll take to create a bunch of accounts is as follows:

  1. Go to signup page
  2. Enter user data
  3. Submit form
  4. Log out

Spotify was the easiest of the three to automate. Their signup form is very basic, with (surprisingly) no CAPTCHA or email validation. The only problem is that when you successfully submit the form, the resulting page automatically prompts a desktop client download. I wasn’t able to quickly find a way to dismiss that, so simply paused my script so that I could manually click “cancel.” Much easier than filling out the form.


require "selenium-webdriver"
driver = Selenium::WebDriver.for :firefox

accounts = [
  # account info redacted...
]

accounts.each do |account|
  driver.navigate.to "https://www.spotify.com/us/signup/"

  driver.find_element(:name, 'username').send_keys(account[:username])
  driver.find_element(:name, 'password').send_keys(account[:password])
  driver.find_element(:name, 'email').send_keys(account[:email])
  driver.find_element(:name, 'confirm_email').send_keys(account[:email])

  driver.find_element(:name, 'dob_month').send_keys(months[(0..11).to_a.sample])
  driver.find_element(:name, 'dob_day').send_keys((1..28).to_a.sample)
  driver.find_element(:name, 'dob_year').send_keys((1950..2000).to_a.sample)
  driver.find_element(:id, 'register-male').click

  driver.find_element(:id, 'register-button-email-submit').click

  # Enough time to manually dismiss client download prompt
  sleep 5

  driver.navigate.to "https://www.spotify.com/us/logout"
end

Last.fm was a bit tricker, as they include a reCAPTCHA widget in their form. While modern versions of reCAPTCHA look like just a simple checkbox, there is obviously some complex logic going on behind the scenes. As a human, you click their “I am not a robot” checkbox and you’re done. When a script fills out the form and clicks the checkbox element, the checkbox is checked, but an image recognition CAPTCHA is presented. Rather than try to figure out the logic behind triggering the additional level of protection, I simply stopped my script after loading the challenge, and manually clicked through. It was kind of fun, actually, to see the different queries and how fast I could complete them. I suspect the CAPTCHA is triggered by the absense of human-like activities (i.e. normal JavaScript events that would be triggered by a human, like mouseover), but didn’t feel like figuring it out, because I’m lazy.


require "selenium-webdriver"
require "pry"
driver = Selenium::WebDriver.for :firefox

accounts = [
  # account info redacted...
]

accounts.each do |account|
  driver.navigate.to "https://secure.last.fm/join"

  driver.find_element(:name, 'userName').send_keys(account[:username])
  driver.find_element(:name, 'email').send_keys(account[:email])
  driver.find_element(:name, 'password').send_keys(account[:password])
  driver.find_element(:name, 'passwordConf').send_keys(account[:password])
  driver.find_element(:id, 'id_terms').click

  captcha_iframe = driver.find_element(:css, 'iframe[title="recaptcha widget"]')
  driver.switch_to.frame(captcha_iframe)
  captcha_checkbox = driver.find_element(:id, 'recaptcha-anchor')
  driver.action.move_to(captcha_checkbox).click(captcha_checkbox).perform

  # switch back to the main document
  driver.switch_to.default_content

  # Wait until Control+D from user to continue
  binding.pry
end

Basis was the most annoying form to fill out. It’s some sort of terrible SPA, so you have to wait until the DOM content loads before filling in any data. It also has some sort of weird logic that requires each input to be tabbed into, or else it thinks the POST fails when the form is submitted (it’s not, actually). You can also tell it’s some horrible JavaScript form because some input elements don’t actually have name attributes, which also makes them terribly annoying to find/fill out. I ended up enumerating over all the input elements, then trying to differentiate based on a data-bound-to property. The problem was that multiple elements had the same data-bound-to value, so I had to ignore any subsequent elements after the first. This form also asks for a lot more data (name, height, weight, etc.) that I had to generate. That’s why I love Ruby’s Array#sample method.


require "selenium-webdriver"
require "pry"
driver = Selenium::WebDriver.for :firefox

accounts = [
  # account info redacted...
]

accounts.each do |account|
  driver.navigate.to "https://app.mybasis.com/#register"

  # It's a stupid Javascript app, so wait until the DOM elements load
  wait = Selenium::WebDriver::Wait.new(:timeout => 10) # seconds
  wait.until { driver.find_element(:name, 'email') }

  # Email
  driver.find_element(:name, 'email').send_keys(account[:email])
  driver.find_element(:name, 'confirm_email').send_keys(account[:email])

  # Password
  driver.find_element(:name, 'password').send_keys(account[:password])
  driver.find_element(:name, 'confirm_password').send_keys(account[:password])

  # Birthday
  birthday = "#{(1..12).to_a.sample.to_s.rjust(2, "0")}/#{(1..28).to_a.sample.to_s.rjust(2, "0")}/#{(1970..2000).to_a.sample}"
  driver.find_element(:name, 'anatomy.dob').send_keys(birthday)

  # Gender
  driver.find_element(:name, 'gender').click

  # Timezone
  driver.find_element(:name, 'settings.timezone').send_keys("(GMT-04:00) New York")

  # Find elements that don't have any sort of queryable attribute
  height_counter = 0
  weight_counter = 0
  elements = driver.find_elements(:css, 'input')
  elements.each do |elem|
    case elem.attribute('data-bound-to')
    when 'n:profile.first_name'
      elem.send_keys(%w(James John Robert Michael William Mary Patricia Linda Barbera Elizabeth).sample)
    when 'n:profile.last_name'
      elem.send_keys(%w(Smith Doe Brown Jones Johnson Miller Davis Wilson Anderson Taylor).sample)
    when 'n:anatomy.height'
      if height_counter == 2
        next
      elsif height_counter == 1
        elem.send_keys((0..11).to_a.sample)
      else
        elem.send_keys((4..6).to_a.sample)
      end
      height_counter += 1
    when 'n:anatomy.weight'
      if weight_counter == 1
        next
      else
        elem.send_keys((110..180).to_a.sample)
      end
      weight_counter += 1
    end
  end

  # Manually submit form, then log out
  binding.pry
end

I’ll probably never have to do anything similar in the near future, but hopefully these examples will help you out if you have a similar (mind-numbing) task in your future.

· 0 comments