Hiring Process

EMS

13:51

Hi!

Chris Lively

Hey.

I figured this is better than the amount of comments on the site.

EMS

Yes, definitely.

Chris Lively

Which part would you like to start with?

EMS

One thing I am always interested in getting more data about is what hiring managers think when evaluating someone who is on the high-skill end of the spectrum. It seems these are the places where the subjective issues have the most potential to get it wrong, and having been in some weird hiring situations myself, I've always wanted to learn more about what people think in these cases.

Chris Lively

I can walk you through the process we go through for finding a sr. developer.

EMS

13:55

Sure. Can we assume in this case that the skill set is hard to find, and that it's important to get someone very good at that skill set? Or, actually, knowing how you determine "how much skill" would be really interesting too.

Chris Lively

Ads are placed and resumes come in. Things like names, hobbies and personal items are ignored. The devs we hire need to be able to handle everything from design through deployment and testing.

EMS

How are the job ads created, like which skills to ask for and so on?

Chris Lively

Because of that I'm looking for tech experience with .net code, sql, html and some basic server maintenance skills.

EMS

What do you think about the following claim, "If a developer is really good with an advanced technique like functional programming, then usually they can pick up a simpler technology (like .net or SQL) extremely quickly (less than a few weeks) and often in one or two months they are better at it than the best experts you already had."

Chris Lively

If we see those topics on their resume then we'll schedule a phone screen. The call is to ensure they can communicate clearly and effectively

It's been my experience that people have wildly varying levels of ability to pick up new things.

I've seen rock solid winforms based VB programmers who simply couldn't wrap their mind around doing web development - even when it used VB for the code piece - after months of trying.

EMS

14:01

Because, when I see job ads that focus a lot on specific technology, it makes me pass on them. It makes me feel like the organization is trying to hire to a specific standard, like SQL, instead of a high level of problem solving skill that can be implemented in whatever language is needed.

For example, in my last job, I had never written a line of SQL before I started, and a few months in even the most experienced DBAs were coming to ask me advice on data normalization and how to use views for encapsulation, etc. If the hiring manager had screened me for the particular buzzword ("SQL") I w…

Chris Lively

On the flip side I've seen people who can pick up just about anything and start being effective in it relatively quickly.

The reason why we ask for experience in those skills is simply that the language is only one aspect of it.

Taking a PHP programmer and throwing them into a .Net web app is going to result in a bunch of .net code that is created in a similar way to how php functions.

Yes, that project might work but it's going to be a complete mess

EMS

That depends.

Chris Lively

I have neither the time nor inclination to train them.

EMS

The programmers who do that sort of thing are the bad ones.

A smart programmer who has never used .net is going to ask questions, like "how do I solve this in .net's best idiomatic, clean manner."

and a person who would think to ask that question is worth hiring, usually, regardless of which random technologies they've happened to use before

whereas a person with 12 years of .net, but who would not think to approach a new tool in that way, is probably not worth hiring, unless your business will fail without super short term immediate .net output

I've seen the same thing when people cross over from Java to Python. They write clunky Python code with manual getters and setters in classes and try to make "private" variables in Python.

Their years of experience with Java doesn't tell me much about whether they are a good programmer versus just someone who got a job done well enough in one language for many years.

Chris Lively

I somewhat disagree. There's a time component to this. A programmer who is under time pressure to perform will approach things in ways that they are comfortable with. They know that along the way they'll figure out the right/wrong approach.

EMS

14:07

I'm not sure. The best programmers I know have the attitude, "You don't have time to do it wrong." It's the same with writing unit tests. If I have never used .NET before (I haven't) the first thing I will do is figure out how do I write clean tests and how do I design functions and interfaces to make them more easily testable.

Chris Lively

Getting back to my PHP example, this person will start off with a "web site" project and likely be editing code directly on a server (even if it's IIS built into their desktop). It's not going to be until they figure out that the model itself has changed that they move into a "web app" project model that deploys assemblies to the server.

EMS

I absolutely will not think, "welp, there's time pressure so I'd better just crank this out even if, given my low knowledge of .NET, this is going to look like crap."

The common thread here, at least to me, is that an important skill is for someone to care about and be curious about best practices.

And, very truly, I have always found this to be inversely related to years on the job.

Chris Lively

I've hired a couple hundred people in my time from all ranges of backgrounds. All I can say from personal experience is that people fall back onto concepts of what they know works from their prior job as soon as there is pressure to perform.

EMS

I totally believe that is true.

I think I am more interested in what it means statistically

Chris Lively

Which leads me back to my answer to the original question: I want to know that they have at least had the opportunity (via time on job) to acquire the skills I'm looking for.

EMS

14:12

One interpretation is: "good" developers are so rare, than even in hiring hundreds of people, you don't have a statistically significant sample of "good" software skill

Chris Lively

For devs, I have a test that has exactly 4 questions on it. The questions cover common situations in html, c# and sql. They also have multiple potential answers.

EMS

I would guess that it's something like 98% of coders do it the hard, bad way, and lean on frameworks and canned architectural patterns, and would definitely fall back on them in a pinch. And maybe only 2% will push themselves to constantly be learning what a given technology's best idioms are.

Chris Lively

We give this as soon as the show up for the in person interview. It should take any "expert" about 5 minutes. 10 minutes is average but if they haven't finished it at the 20 min mark then that is the end of the interview.

Probably 1 out of 20 applicants can actually answer all the questions - which is really really bad.

EMS

What if someone has never coded in C# before, but they would understand the concept behind your question in a different language?

Chris Lively

Yep.

However, if they had never used sql or never actually worked on a web site then they'd fail

html example:

<div class="r b">Hello World!</div>

question: what color is the text?

EMS

14:18

What happens when someone gives a pretty good, but not perfect, set of answers?

Chris Lively

That is a dead simple thing to answer for anyone who has worked in html. Yet only 1 out of 20 people I talk to can answer it.

Then I'm happy and move forward with other questions

EMS

In my case, I have literally never written HTML in my life (I do statistical computing and have never worked on web pages) so I do not know the answer.

Chris Lively

lol

EMS

I bet I could Google it in 10 seconds, and would understand "why" the answer is what it is forever after.

For me, it's difficult to understand how that question can possibly tell you something useful about the candidate.

Chris Lively

Some people claim it's an error. Some say red. The answer is blue.

EMS

14:20

It's like if I asked someone to solve some parochial coin flipping question, like how many expected flips of a fair coin before I will see the sequence "HTH"

The skill of the candidate is almost completely independent of their ability to answer that question in a timed setting.

Chris Lively

The reason why that's an important question is that it demonstrates a fundamental concept of html styling. If you've never run into it before you'd sit for awhile trying to think which one you think it ought to be. If you have then you answer in about 5 seconds.

EMS

Yes, but that is a test for trivia. "Have you seen this counterintuitive thing before?" that's very different from "are you a smart person who can learn new things."

Chris Lively

The c# and sql stuff is along the same lines. The sql part is to write a query that updates a set of records off of a join; which is incredibly common. The c# piece

EMS

Like in my case, I feel confident that whatever principles of HTML are behind that example, I can learn them cold in a few minutes of time. But, as it happens, I have never been exposed to that piece of trivia before.

It's almost like grilling someone on what the capital of Kazakhstan is, and if they don't know it, then declaring that they don't have "experience" with Asian geopolitics. The person might have a PhD in Nepalese foreign trade, but has just never had the occasion to worry about Kazakhstan.

Chris Lively

It's not counterintuitive if you work in html. The entire model is built on an idea of layers. Each one interacting with and overriding the other. Sure someone can pick up how to create an html document in a week, but to understand how that document actually interacts with the browser etc in order to be both performant and maintainable requires a much deeper understanding.

If I was looking for someone to know how to do trade negotiations with countries like Kazakhstan then I'd expect them to know the relevant details of the area.

EMS

14:25

But what about excellent people at trade negotiations who can easily pick up the idiosyncratic details about whatever particular country in a short time.

it's like you are asking for (good at trade negotiation) && (good at kazakhstan already)

Chris Lively

Actually, I'd probably question anyone who claimed experience with geopolitics of a region that didn't know the hot spots in that region.

EMS

where that second conditional doesn't seem very necessary

well, maybe my example was bad, i am trying to highlight the different between parochial knowledge that can be gained by a smart person in a short time, versus deeper skills that often can't be learned by parochial people even if they spend years trying

Chris Lively

Well, if I'm hiring a generalist in negotiations then I'll ask general questions. If I need someone with explicit application of those skills in a known area then they better understand the problem domain.

EMS

for example, i'd hire a Haskell expert who failed your HTML question in a heartbeat, because if they are good at Haskell, then once they try to be good at HTML, they will be better than the currently best HTML people in a short time.

Chris Lively

Which is fine ... if you're willing to wait for them to complete the on the job training.

At my current company, I don't have the luxury of doing that.

EMS

14:28

yes, i would be, because it would happen so fast that the training part would be irrelevant

for a good haskell developer to learn the "zen" of HTML, i'd say a few days tops

because the haskell skill implies a much more advanced structural approach to thinking about problems in general

Chris Lively

I think the web is littered garbage produced by people who thought that.

EMS

my opinion is different -- the web is littered with bad things because he people who create them were hired for their parochial knowledge of HTML or CSS trivia

Chris Lively

:)

EMS

and not for their abstract problem solving ability which would then be applied to web dev

Chris Lively

Disagreement is totally fine, I'm actually happy we live in such a world.

EMS

14:30

Yes, of course I agree we can disagree.

I am not looking to change your opinion, just to learn more.

The opinion of people in your position has always surprised me so much, because I have seen it lead to project failures, unmaintainable legacy code bases, etc., so many times, but people keep doing it and I don't understand why.

Chris Lively

Interesting. The web application that kicked this company off was initially written in 2001. It was expanded and maintained for a period of 10 years before I opted to completely rebuild it.

The reason for the rebuild wasn't that it was no longer maintainable. It was purely that it was no longer feasable to hire people that knew (or wanted to continue working in) the technology - classic asp.

EMS

Both in my experiences, and through a lot of the quant research on software projects, that is a very rare experience.

Chris Lively

We took 18 months and rebuilt it from the ground up. Every little lesson learned, every concept was looked at, discussed, and decided on.

Part of the reason for our success: we hired people that knew exactly what they were doing from the get go.

I'm not saying I am perfect in those decisions. I've had to replace quite a few people along the way.

EMS

That's where I disagree. The trouble is we don't have a counterfactual. You can't reset your company and re-do it as an experiment to see if it would have succeeded even more if you had hired people with higher abstract skill and less direct skill in the specific technologies being used.

It's almost a narrative fallacy to attribute the success to that, because it's easy to tell that story, but quite hard to know with any actual quantifiable certainty.

Chris Lively

However, we leave experimentation with new concepts and ideas to minor projects and only roll new stuff in when we know exactly how it's going to behave.

"but quite hard to know with any actual quantifiable certainty" You're likely right here.

EMS

14:37

It's similar to the way that current university professors suffer from the narrative fallacy that it was through their own hard work that they were awarded tenure. When in reality, it was more due to luck or circumstance, as there were many other equally qualified professors trying to do the same hard work, for whom the tenured position just randomly didn't work out.

This is also very true in start-ups.

Selection bias is a huge thing, like the books written by the guys at 37signals.

I like them, and often agree, but the principles in their books can hardly be claimed as the reasons for their success. There is just no sample size whatsoever to base those claims. It's really just that they were randomly successful and later, because they succeeded, were in a position to claim that it was due to x,y or z

In other words, I've heard of 10-year-old projects like yours, and those people did all of the exact same things you did, but their code became a monstrous garble of spaghetti and the rewrite of the project failed badly, even though they hired for the targeted specific expertise, and they carefully discussed every decision along the way, etc. etc.

Chris Lively

I've worked for a number of startups and large companies prior to founding this one.

EMS

And I am interested in learning whether, on average, it's better to build a team with a small number of expensive, high-skilled generalists, or a larger number of relatively cheaper hires who are more skilled in one or two core competencies that the project needs right away.

Chris Lively

What I've seen is that you have to have well thought out (and battle tested) internal standards and those have to be policed.

EMS

I definitely agree with that.

A focus on quality is very rare.

It usually requires very competent managers, in the tech areas the project uses, which is also very rare.

Chris Lively

thecodelesscode.com/case/94 is pretty clear

Yes, it does, and yes, those people are rare.

Harder still is once found, keeping them around long enough to ensure compliance

EMS

14:45

What do you find most motivates their desire to leave?

Chris Lively

Environment.

If the environment isn't one that promotes team work while recognizing individual contributions then you'll have issues.

EMS

Do you find that things like private working space, control over what OS to use, or other forms of "using the best tools" is a significant factor?

Chris Lively

Which gets back to part of my hiring criteria: I filter out those that, in my opinion, haven't had the time to make the stupid work mistakes that can cause problems.

EMS

What are those kinds of mistakes?

Chris Lively

Interesting: I've got a few open offices here that no one (except sales people for obvious reasons) has wanted to move into. Most of the offices have 3 people in them. They seem to like that.

We don't allow them control over OS and the primary tools are well defined: MS Office, Adobe, Visual Studio, SQL server.

Ancillary items such as jquery, etc are already in the project and it would take a titanic reason to include anything else.

EMS

14:50

What would you feel about a candidate that aced your test and had great interpersonal skills, but who wanted to work on a Linux machine and have a private office? Assume that all the conversations you've had indicate the person is a hard worker and a big time team player who likes to collaborate, but is just more sensitive to noise when programming than most are

Chris Lively

At one company I was at, we had such a person. They had a Mac that was running something which allowed them to emulate visual studio. Multiple times per week this person had technical issues whether it was reconnecting to our TFS server or just simply some sort of Mac issue that no one else could help with.

The productivity level in his case simply wasn't worth it.

So, no, I wouldn't hire that person. Private office? That's fine. As I said, we actually have a couple open that no one has been interested in.

In part, the big issue is that what we do requires a lot of interaction with the other team members. Everyone plays a part and we try to move very fast.

EMS

Let me try to summarize on those last points.

If the company already has free offices, then asking for an office would not be a dealbreaker for you. But if the company has already chosen a tech stack, then accomodating something outside the tech stack is a dealbreaker if it comes with associated overhead, like IT issues

Chris Lively

Covering mistakes: relatively young employee (early 20s) who decides their manager is an a** because he asked for daily status reports on outbound sales calls. Things get posted to FB that could be interpreted as a threat. Other employees see this and report it.

Good summary

EMS

Under these considerations, what is the usual efficiency with which you make an offer? Does it take your team a long time to find candidates who can do it (you mentioned 1/20 pass the test). Out of that 1/20th of the pool, how many more fail the team work and other parts of the process? How many interviews do you need on average for 1 quality hire? And how long does that hire typically last within the company?

Chris Lively

I know there are lots of companies who have no issues with bringing your own device in. We're good with ipads/phones etc for connecting to the email server. That said, we're a relatively small company. So spending time supporting people that deviate from the standards is just not financially feasible.

EMS

15:00

(it's funny to me to hear that about a small company, because at big companies they say the exact same thing, and then they tell you to go work at a small company if you want more customizability... so it leaves me wondering in what situation yo can work where you have some autonomy and control over the tool chain you use)

for me, autonomy is probably the overwhelmingly number 1 thing i want in a job. I like being told a high level abstract problem, asked to assess what the best ways might be to solve it, with cost trade offs, and then i get to implement that solution.

Chris Lively

The last dev we hired, I interviewed 10 people before hiring one. I'm not sure how many resumes were submitted as those were filtered out before they even got to me. The time from when ad was placed until offer made was about 10 days. We made the offer the same day as the interview of the guy I wanted. Basically called him about an hour after he left the interview.

EMS

roles where my abstract skills are tested heavily in the interviews, and then i am slotted into a fixed tech stack, a fixed language, and a fixed approach to the problem are a bad fit for me, but it's really rare to find anything else. it makes me wonder why the interviews don't skip the abstract problem solving part and instead test for willingness to go along with a fixed approach

Based on your 1/20 claim about the test, it would seem like this recent hire was a very very rare occurrence. Getting someone at a rate of 1/10 when 1/20 is the average is very very unlikely.

Chris Lively

The 4 problems we give during the interview, like the one I showed you earlier, are exactly ones that we commonly face.

That 1/20 is based on quite a few years. In '05 I interviewed around 100 before picking one. That's when I changed how I go about doing it.

EMS

You changed from something else to the current approach with the mini-test?

Chris Lively

Honestly, Id say if you want that level of freedom then you might be better off just picking a problem domain and building a company around it

EMS

15:05

I've thought about it a lot, but I am very very opposed to working with VCs and would insist on bootstrapping completely, which is not feasible for me at the momeny.

Chris Lively

yes. Initially I just did a phone screen to make sure I could understand them. Then I'd bring them in, ask questions about past problems they've solved while going into details of those issues.

I found that all of those questions were highly subjective and unlikely to be an indicator on whether they'd work out.

EMS

I also don't want to get myself into a situation where servicing all of the non-science parts of a business suddenly take up all of my time. I actually have written down a list of "wont-do" items for what kind of companies I would never found, and if i shared them with you, you would laugh, because they rule out almost all kinds of companies

Chris Lively

Nevermind the sheer amount of time I spent interviewing people.

Around that time I read an article by spolsky talking about keeping the interview to go-no go questions.

EMS

it's interesting, because Joel Spolsky advises that approach. He likes to laser in on tech details from a past experience and size up whether they are BS-ing about it, or if they know it, rather than quizzing them.

!

Chris Lively

So everything I ask at this point is either a yes, keep talking, or thanks for stopping by.

This has led to a much higher retention rate.

EMS

15:09

This is very interesting. As I read Spolsky's articles on it (I assume you mean the guerilla guide), my take away was that you should not ask quiz-like or trivia-like technology questions, and that you should ask them about the items from their past experience, but do it in a way where they are not allowed to ramble, but instead have to answer short, precise questions about the details of their past experience.

Chris Lively

I think part of your misunderstanding is categorizing the questions I ask as "trivia". The html question above isn't "trivia" in the sense of being a corner case that an html developer may not have run into. It's a core question that someone with html experience should know without thinking. As such it is a prime indicator of whether they are completely BS'ing me.

It's short and has only one right answer.

Same thing with the SQL questions. If you've actually been using that tech then you'd know the answer (even if you happen to get the syntax slightly wrong) without putting too much thought into it.

EMS

For me, this is the distinction. As Spolsky says in the interview article, you should hire for only two things, "Smart" and "Gets things done." For me, any question that tries to change those qualities into more idiosyncratic, specific versions that are about a single technology or a single topic is an example of trivia.

Chris Lively

To me: not knowing them means you aren't going to be effective day 1.

EMS

Exactly. That's what I mean. And, with the SQL example, you would have been wrong about me (and a lot of my friends) as an example.

Chris Lively

Smart - should be apparent during the interview while just chit chatting.

EMS

15:13

I didn't even know what a select statement was. But in less than a week I was writing stored procedures, and in less than a month I was re-architecting things.

Chris Lively

Gets Things Done - impossible to tell for sure until they come on board.

EMS

Look, look, that's all fine. I'm just saying you mentioned the Spolsky article, but it seems actually like your process is really different than what he describes.

That's not right or wrong, I mean it obviously works for you.

Chris Lively

Maybe

EMS

I was just trying to talk about how I had read that article really differently.

In fact, I would say SQL was one of the easiest things I've ever learned

I did math as my undergrad, so relational algebra was truly simple

thinking about joins has never thrown me off

sometimes i got tripped up with random weird syntax, like case logic or handling nulls

but i started reading about declarative programming, and even started programming in prolog, to learn more about how it differs from procedural programming

i read some o'reilly books on sql in the first week

i even went and read the mysql docs on the different partition properties between MyISAM and InnoDB, within the first couple of weeks of ever even using SQL

and figured out that we were storing data in a manner where the partition locking didn't make sense for the queries we were running

to me, thinking about those problems was very very simple by comparison to thinking about writing Cython-ized numerical linear algebra routines and creating good interfaces for them in Python

but i feel that modern interview practices could never have picked up on this. my bosses just got lucky, really

if they had asked be to write some SQL code in the interview, I would have looked like a dumbass

in the time since, i've tried to learn more and more about data modeling, NoSQL, and alternative data stores, as well as algorithms. I have a pet project on the back burner to implement my own b-tree in Python, just so I can understand better the algorithms that are used to create indexes in database systems

Chris Lively

Yep, you would have been passed over here which may have been our loss. The problem is it's hard to know when you're talking to a real rock star: those that can pick up anything quickly and be highly effective, work well with others, etc. All I can do as a manager is play the averages.

EMS

15:22

Yeah, I agree. In some areas I am in that 2%, and it is expensive in terms of time and money to search for a 2%-er

I agree that out of the other 98%, your methods are good and cost-effective at finding the better end of that spectrum.

Chris Lively

BTW, thank you. I've absolutely enjoyed this conversation. A side note is that one thing I look for when hiring are people who are not afraid to challenge my way of thinking. Whatever else, I will be considering what you've said. However I need to get prepped for a partner meeting this afternoon. If you have some follow up questions or whatever, then feel free to email me: [email protected]

EMS

(I'm not trying to toot my own horn or anything, as I am clearly not in the 2% when it comes to HTML or web design)

Of course, thank you too.

I have enjoyed it and found it very enlightening

Transcript for