Side-stepping the Secretary Problem, unwittingly.

"How many applicants do you need before you find the right one? One? Five? The entire community, every time? :laughcry:"

— A fellow slacker in the Clojurians Slack.

The asker isn't joking. Any proposal to use unfamiliar technology, especially programming languages like Clojure inevitably triggers managerial hand-wringing about the hiring pool.

And what could cause more Managerial Hand Wringing than than trying to hire in a niche of a niche… You see if you can go back to 2014 and find QA people, in Pune/India, willing to learn to read and write Clojure code, to help backend engineers test a rather large SaaS, written in Clojure?

I got to know of the so-called "Secretary Problem" ¹ in the ensuing discussion. The problem statement is used to study Optimal Stopping Theory. It sets up a recruiting scenario, and explores how to maximize the odds of selecting the best applicant. It has been a fun rabbit hole to explore.

A key constraint of the secretary problem is Once rejected, an applicant cannot be recalled.

Once upon a time, my colleague, Mayank, and I violated this unknown-to-us constraint, in our otherwise-conventional tech hiring loop by:

• Not only explicitly keeping the door open for re-applications, after a cooling-off period.
• But also offering to help people learn stuff we were interested in hiring for, on an opt-in basis. We would send people curriculum and make ourselves available for office hours on a fixed weekly schedule, to those who showed interest in learning.

Over about late 2013 through 2014, him and I were hiring for programmers for our QA team (which was him and I :) at helpshift.com, a Clojure-happy startup. We wanted curiosity-driven people with enough technical proficiency that we could teach and train, because we were writing test tools and suites in Clojure. Not to mention all sorts of impractical to automate manual testing of our systems, which demanded some baseline capacity to wade through reams of technical manuals and documentation, and the desire and ability to learn and use our cli tools, scripts, and configurations, if not craft them.

We knew the tester-programmer combo is a tough ask in the first place, especially in India, where the hiring pool is almost entirely filled with manual-only testers ². So we had braced ourselves for a noisy pipeline.

In this context, I figured it would be crazy to lose anyone with potential. In my previous career, as a business manager, I had seen companies lose candidates with exactly the right life experiences, but the wrong degree or pedigree or test score. So I spitballed the idea to offer reapplication + opt-in support. My colleague loved the idea, and so it was.

Luckily, the company culture was already about "nontraditional" hiring, looking for diamonds in the rough, so to speak ³. And since we didn't have an HR function, the two of us were left to our own devices.

In hindsight, having to go through HR, however supportive, would have been horrible because we would have had to compromise by saying something like "Oh so sorry this time, but you are part of our candidate pool and we will be in touch, pinky promise. Later. You can also try to reapply after six months".

Having known such rejection emails, and the following perpetual silences as well as black holes of re-application, I read those types of rejection emails as garden-variety weasel-wordy ass-coverage. Benign self-deceptions, at best. Be real… nobody from your org is ever going to follow up on that. And if I have no signal about why someone would listen to me again after a mere six months, I, the hopeful candidate, will not waste time re-applying. So don't set up those expectations.

Direct outcomes

Over the year or so that we ran our interview loop;

• We landed five hires out of about 300 inbound.
• Each person became productive quickly (about a month) in our corner of the little open plan office. Three lapped up Clojure, two moved to mobile SDKs (installed base of 2 Billion devices as of 2017).
• Each of them further grew way beyond our expectations and moved to other roles like back-end development and product management, when a company-wide re-org did away with a discrete QA function.
• A pretty good retention rate, for VC-funded high-growth startups… All of our people became tenured staff (four to ten years each).

Indirect outcomes

Every single such hire further converted their input into fast-growth startup careers in-house, and elsewhere, as well as startups of their own.

Most of the credit for career growth and retention goes to the engineering culture built and nurtured by all of m'colleagues.

Hiring people with potential is critical, but it's only the beginning. All the work they did, actively helped these early hires acquire professional skills that helped them step out and do what they are doing.

We also indirectly hired at least one kickass engineer (who also became tenured), because of the good word-of-mouth our approach earned (which we did not anticipate, but it happened). He was blown away by the learning support we offered his wife, whom we had screened, who opted into our office hours, and whom we didn't end up hiring.

Getting Lucky

With 20/20 hindsight, I wonder if our approach would run into some HR or legal quagmire (Like, would it creat an implied commitment to hire if <criteria> are met, and if we don't then would we run afoul of some arcane labour law?). Anyway, we just did it because we could, and I'm glad we took our chance. I wish this was the default way to hire.

Refusing to copy the Zero-sum FAANG Hiring Loop Playbook

Common experience is that typical recruiting funnels yield one final offer/acceptance for every 10 applicants (in that ballpark). This suggests that in a global market, if everyone uses typical recruiting methods—"We have 5 rounds because <insert FAANG> has 5 rounds; which is proof that it is best practice."—the small fry will always lose to the big fish because you simply cannot allocate enough resources to absorb the damage of such an abysmally noisy pipeline.

Said another way, the main problem of normal hiring practice seems to be that, we play it like a zero sum game even though it is an adverserial game.

• We have no objective standard to determine observed quality.
• We don't usually know the n a-prioiri… how many applicants should we assume constitute our applicant pool for a given role?
• The applicants do not reach us with uniform probability, and we cannot interview them all with the benefit of hindsight.

Running a tight, humane hiring loop: Max 24 hour SLA. Always Be Kind.

This one choice, I will take credit for being firm about at the outset, having hired people before, and having been fully convinced of speed as a core value of hiring, by Kayak's erstwhile CEO, Paul English ⁴. His "SLA" is seven calendar days (not seven business days) from getting to know about the existence of a candidate, to making a firm offer.

We got accustomed to receiving "Thank You" emails in response to "Regret" emails we sent out after people had passed through three of our four to five step filter pipeline (nb. a "step" in this filter pipeline does not equal "one to several hours long interview stage"). I attribute this mainly to our turn-around times, and clear communication. Any recruiter reading this, please meditate on this fact of life.

Here is an assorted recollection of the collection of rules we cooked up and/or evolved over the year we ran our loop:

• Set clear communication expectations with candidate: We promised a max 24 hours turn-around time to candidates. We explicitly asked them to ping us if they did not hear within 24 hours. ⁵
• Never ghost anybody. We both hate being ghosted in any sort of communication, not just hiring. Hearing a quick "no" is much, much better than hearing a "yes, but" after three months of being strung along multiple interview stages.
• Ruthlessly protect our own brain-cycles:
  • Context is King. Route all communication through the ATS. Immediately create a note of the "why" of our vote/decision at each stage for each candidate. Just like good commit message hygiene. If by-chance either one of us spoke with some candidate out-of-band, we would immediately drop a note in the ATS. A dedicated browser tab was always open on both our laptops.
  • Synchronous calls and/or on-site interviews cost us heavily, by interrupting our own ongoing QA work. Ensure we always do these last, and always individually. This forced us to figure out early on precisely what to communicate, what our stage-by-stage checklist should be, what signals to look for, how to set up our templates and stages in the ATS, how to register votes etc.
  • Run all individual phone calls by a playbook for that call. Terminate the call early if it is going badly. Have a practiced, clear, and kind way to end it. Having the "open door" policy helps. So does having a structured Q&A + decision-rubric for each question.
  • Keep tuning our workflow for maximally async communication and decision-making criteria. Our mutual, written-down, check-listed clarity helped us independently decide and act within seconds and minutes–vote / reject / move forward—of noticing some change in the ATS.
  • Be diligent about our 24 hour communication SLA. Slow turn around times create piles of "candidate inventory" that force us to stop what we are doing and attend to the queue. In a fast-paced startup this never happens because everything is always on fire, and so candidates languish in ghosting-hell.
• Protect their time:
  • Much of this benefit to candidates falls out of protecting one's own brain cycles.
  • Additionally, waiting for replies sucks for candidates. Therefore, never give out coding assignments that take candidates hours to do, because it is a burden on their time, and it also means we need hours of careful reading to review the code. Large coding assignments are horrible signal-to-noise ratio tools. Any coding assignment should be a fifteen-ish minute job for the career level of the candidate. Because as soon as they send it in, a reviewer will know in one look whether to issue a reject, or what to ask them next. By default, our immediate request would be to refactor the solution using a totally different design.
  • Some copying was assumed. Being transparent and up-front about said copying was demanded. A clearly-stated deal-breaker and firing criterion, to the candidate—if we ever learned at any point that they copied stuff and didn't disclose it (yes, even after they were hired and working full-time).
  • In practice, code review turn around time of minutes often let us make go/no-go decisions fully async within the same day. This is just concurrent programming 101.
  • As you will see this also feeds back into "ruthlessly protecting our brain cycles". It's great when these two mechanisms feed-back positively into each other.
• Prize written communication and reading comprehension:
  • All primary screening over email, including the coding assignment, Q&A about the coding assignment, and refactoring of the solution.
  • All emails would require the person to think about something relevant to that stage and provide us a short written answer.
  • Our very first email would enumerate potential deal-breakers the candidate should consider for themselves. Such as, mandatory requirement to learn Clojure programming, potentially arriving a rung or two lower in a reporting hierarchy ("programmer" instead of "manager"), ballpark compensation budget, growth and promotion prospects, on-call expectations, strictness of per-commit code review culture etc… Little aspects of live culture that add up and have people say "no". (nb. We were not allowed to disclose salary band and compensation up-front, but I firmly believe disclosing this would be best practice. Promising candidates drop out at salary negotiation, which is an incredibly expensive place to lose them.)
• Filter for initiative and curiosity:
  • Dig for legitimate self-driven study and project work, of any shape or form. Github portfolios and personal websites are not automatically-good signals. They are entry points to the digging.
  • Try to understand the "why" of the candidate. What animates them? Craft interview questions appropriately.
  • Offer "office hours" support to all maybe-yes candidates that we said "no" to.
• Async decision-making protocol:
  • Only two "Yes" votes progresses to the next stage.
  • One Yes, one No = Reject (Two nos = Obviously Reject).
  • Both votes must be registered in the ATS within 24 hours.
  • Immediately send +2's to the next stage. Whichever of us registered a second +1 immediately chose an email template we had crafted for the purpose, in the ATS. We also fixed templates on the fly and/or created alternate versions for subtly different responses needed at the given stage. No mutual permission necessary.
• Err on the side of false negatives:
  • After our rounds, we would chat for a few minutes and pass on only those we felt "hell-yes" about, to the "CXO interview and offer stage".
  • Our open-door policy let us say no safely, and quickly, to "oooh, almost… aaalmost there" candidates, because our approach respectfully held the door open to them to surprise us (and themselves) at any time thence.
  • We would often use the first-contact phone call to coach away people. Make them really think about why they are changing roles. Not infrequently it is because they had not even considered trying to change their situation at the current job. Prompting people to exercise initiative is a good thing. We wanted people who would not be the silent suffering type, because that way lies stagnation and decay. We wanted to have a professionally satisfying workplace. That means deliberately overcoming friction, and discomfort. If they really tried, and it didn't work out, then they knew our door was always open to talk again. (This feeds back into protecting their time + brain cycles and ours.)

This is all for now. I hope it helps someone out there. I'd love to discuss this, over email.

_\\// Live Long, and Prosper.