Foreword
It’s well known that most programmers wannabes can’t code their way out of a paper bag. As a consequence, the tech industry is pushing for longer, harder and evermore extreme screening.
The whiteboard interview has been the standard for a while, followed by puzzles [now abandoned], then FizzBuzz.
The latest fad is HackerRank. It’s introducing automated programming tests to be done by the candidate before he’s allowed to talk to anyone in the company.
A lot of very good companies are using HackerRank as a pre-screening tool. If we can’t avoid it, we gotta embrace it.
What to find in a HackerRank test?
There are 3 types of questions to be encountered in a test:
- Multiple Choice Questions: “What is the time complexity to find an element in a red and black tree?” -A- -B- -C- -D-
- Coding Exercise: “<Long description of a problem to be solved>, <input data format>, <output data format>.” Start coding a solution.
- SudoRank Exercise: “Your ssh credentials are tester:QWERTUIOP@123.98.45.76 <long description of what’s wrong with that server>.” SSH to the server and start fixing.
Any amount of any question can be put together, in any order, to make a complete test. A company should give some indications on what to expect in its test.
HackerRank provides a library of hundreds of questions and exercises ready to use. It’s also possible for a company to write their own (and recommended).
Defeating Multiple Choice Question
The majority of the multiple choice questions can be solved by an appropriate Google search. Usually on the title, sometimes on a few select words from the text.
Defeating Coding Exercises
Searching for a 10 lines long paragraph in Google is not an acceptable option. Not to mention that the HackerRank website disables copy/paste in the description area.
The workaround is to search for the title of the exercise. A title uniquely identifies a question on HackerRank. It will be mentioned in related solutions and blog posts. Perfect for being indexed by Google.
The first result is the question, the second result is the solution. Well, that was easy.
Bonus: That google solution is actually wrong… yet it gives all the points.
// [boilerplate omitted]
int main() {
int N;
cin >> N;
int tmp, result = 0;
for (int i = 0; i < N; i++) { cin >> tmp;
result ^= tmp;
}
cout << result;
return 0;
}
This solution is only correct if duplicated numbers are in pairs. All the HackerRank unit tests happen to fit this criteria by pure coincidence [cf. Addendum].
Originally, we put this simple question at the beginning of a test for warm-up. We received that answer from a candidate in our first batch of applicants. It was quite puzzling, what are the odds that someone would come up with an algorithm that convoluted if given only the text from the question? A quick investigation quickly revealed the source.
Addendum:
The “Lonely Integer” question is worded slightly differently in the public HackerRank site and the private HackerRank library but the input, output and unit tests are the same. Hence why the solution is off but works. HackerRank is obviously copying questions from the community into the professional library. That’s another copy-cat busted!
Recruiter Insights: Cheating brought to the next level
We have a lot of candidates coming from recruiters. How are they comparing to candidates from other sources?
Let’s see the statistics on a hard question [i.e. dynamic programming trading algorithm].
Most candidates get 0 points: ran out of time, unable to answer, wrong algorithm, or incomplete/partial solutions (i.e. good start but not enough to pass any unit test yet).
We wanted to show the same distribution over our pool of candidates but HackerRank doesn’t provide that graph anymore. It used to.
Anyway, we remember approximate numbers. The distribution for our candidates is about 50/50% on each extreme. That’s significantly better than the 75/17% from the general population. We can correlate that number with the time spent on the question and a visual code review.
The result is that candidates coming from recruiters perform better, especially on hard exercises. In fact it is unbelievable how much better they perform! (Special kudos to the guys who are able to solve a problem -with a perfectly optimized solution- in less time than it would take to actually read it =D).
The conclusion is plain and simple: Our recruiters give away the test to the candidates.
Lesson learnt:
- For candidates: Remember to ask the recruiter for support before the test.
- For recruiters: Remember to coach the candidate for the test and instruct him to write down changes (if any).
- For companies: Beware high-score candidates coming from recruiters! In particular, don’t calibrate scoring based on extremes scores from a few cheaters.
Challenge: How long does it take you to solve a trading challenge? [dynamic programming, medium difficulty].
Custom HackerRank Tests
Companies can write custom exercises and they should. It’s hard and it requires particular skills but it is definitely worthwhile.
It is the only effective solution against Google, if done carefully. (It’s actually surprisingly difficult to make exercises that are both simple AND not easily found with Google on 1000 tutorials and coding forums).
Sadly, it won’t help against recruiters. (Excluding the first batch of candidates who will be sacrificed as scouts).
Conclusion
Did we just ruin HackerRank pre-screening? Of course not! There is a never-ending supply of bozos unable to tell the difference between Internet and Internet Explorer.
We could write a book teaching the answers to 90% of programming interviews problems, yet 99% of job seekers would never read it. Hell, it’s been written for a while and it had no impact whatsoever.
Only the handful of devs following blogs/news or searching for “What is HackerRank?” will be able to come better prepared.
If anything, this article makes HackerRank better and more relevant. Now a test is about looking for help on Google and fixing subtly broken snippets of unindented code written in the wrong language.
HackerRank is finally screening for capabilities relevant to the job!
A last word of advice. You’re gonna need training if you are trying to pass the interview bar at good tech companies and you didn’t have the chance to attend MIT. This book is the gold standard when it comes to data structures and algorithms questions. Highly recommended. It contains hundreds of questions that you will be asked over and over again, in every company. At a minimum, read the free preview on Amazon.
The HFT Guy 
what do we do when the test is in full-screen mode???
LikeLike
I think “full-screen” means full-screen on *one* display. But what if a second display is connected… 😉
If this is not possible, just have another computer (say a Raspberry Pi) that is connected to another mouse/keyboard/display on your desk.
LikeLike
“It’s well known that most programmers wannabes can’t code their way out of a paper bag. As a consequence, the tech industry is pushing for longer, harder and evermore extreme screening.”
Question is if that more and more extreme screening is making the companies recruit better employees for them or quite the opposite?
Problem is that most companies are looking for wrong type of employee.
Instead of looking for a guy who can do THAT particular job they seem to be looking for a fresh MIT graduates / programing competitions laureates / Mensa members etc.
Is that what is really needed in the job?
The industry should finally recognise that the point of job interview is to find a fit for a role and not to prove to 99.99999% of candidates that they are rubbish.
What the role involves should be in the job interview.
Instead of giving problems involving bit manipulation, graphs, trees, heaps, God knows what, give a simplified version of a problem that the candidate would REALLY face in the day to day work.
And most of the companies should finally admit they are NOT Google.
Let top companies recruit the top talent.
Let mediocre companies recruit the mediocre coders, just decent enough to do their job.
LikeLiked by 1 person
More and better screening make the companies recruit more qualified candidates.
Keep in mind that the majority of candidates will not complete a test to print numbers from 1 to 10. Harder doesn’t necessarily means what you think it does. See https://thehftguy.com/2016/07/20/hackerrank-a-glimpse-at-the-company-side/
LikeLike
Agree with better.
Do not agree with more.
What I am able to observe recently is that HackerRank/Codility is just another stage of the process, usually a “pre-screening” thing to check if the candidate can code before somebody from the company even bothers to speak to him. Then there is still “the initial phone screen” and then “an awkward 4h on-site interview” or a couple of them.
All this is down to the fact, that the whole process is a negative one, with “Keep in mind that the majority of candidates will not complete a test to print numbers from 1 to 10.” – yes of course, only coders in THAT particular company can code. Rest is just plain rubbish.
Some points why I do not think HackerRank/Codility adds any value introducing more noise instead:
1. Practice is needed upfront.
You can be a top engineer at a decent company but still you can fail miserably because e.g. you do not read from STDIN everyday (I am sure at THAT company everyone does that for breakfast).
2. Time limit.
This is something that can you hit hard even if you practice. On Codility practice challenges have 120 minutes limit each. On HackerRank there is none. Then you start your test and you learn that you have 90 for 3 tasks…
3. Hidden test cases.
You can write your code. Test it against the sample input/output. Then you even come up with your own test cases. Then your code still fails the hidden ones. Yes, coming up with good cases is a very valuable skill. Even more if you have about 30 minutes for that and everything else and you have an environment at your disposal which accepts one custom test case at a time…
4. Performance requirements.
The solution is expected to have some time complexity. Mentioned explicitely (Codility) or not ( HackerRank). Both of the platforms measure it in the way, that they execute some massive test case against your code and expect it to come up with an answer in some time (3 sec?). If not you fail the case. OK, fair enough, I can optimize my code. Wait. I have only 30 minutes for everything. I do not know what the test case exactly is. I can suspect it is hitting the max input size, but I still need a way to generate 10^5 array of input and feed it to my STDIN only accepting “solution”.
5. Often you need to know the particular algorithm/trick.
So you get a problem. Some of them are quite straightforward, some of them more difficult. Trouble is that the difficult ones often differentiate from easy ones by the expectation of knowing some specific algorithm/trick that allows you to solve the problem efficiently (to tick the performance requirements). Also there are boundary cases. As well as massive input size. The problem may resemble DFS or Fibbonacci-like sequence, but good luck going with default recursive solution for the input size of 10^9.
Yes you can test/optimise/pray…all within your 30 minutes that you have for a problem…
All that makes the HackerRank/Codility geared towards the regular users of these and similar platforms.
Now the question is are you really looking for a competitive programming champion who can solve the awkward algorithmic problem in O(n) in 30 minutes reading from STDIN?
LikeLiked by 1 person
It’s the company who decides on the duration, the questions, the unit tests and the input (stdin or not). The correcter can see how long you spent on each question, review your code and see slow tests.
You need to understand that none of your complaints are valid against HackerRank. It’s really the company that makes the test and decides on the rating.
LikeLike
Sorry but all of that complaints are valid against HackerRank.
Latest example, World CodeSprint 12 competition there that finished last Saturday. Was taking part too.
Take a look at what people are discussing in one of the problems:
https://www.hackerrank.com/contests/world-codesprint-12/challenges/factorial-array/forum
Basically there are problems with (hidden) test cases and meeting the performance expectations.
Which is caused by not using the right algorithm/trick and banging the head against the wall trying to improve it instead of going for different algorithm (the one the author of the task had in mind). And that is due to inexperience with the platform (not being aware that timing out test cases mean wrong algorithm) and also partially to the time limit for the competition, that makes one not wanting to start one of the tasks from the scratch again when there are others pending.
It is a programming competition, so it is kind of expected.
The problem is when that programming competition environment is used for interviewing with even more harsh time limits. This is wrong because it has nothing to do with the actual job.
Pre-screening is fine, but there are better ways to do it like custom homework problems to code. They take longer for a candidate than 90 minutes, but if attemted they are much closer to everyday expectations, allow a follow up (e.g. pair programming to extend the solution, discussion etc.), give much better insight into candidate (e.g. does he write tests, does he understand the problem, can he implement a simple formula, what is the usability of his solution etc.). For the candidate the benefit is to actually solve the problem in the way he preferes and with the tools he chooses than being limited to the one “chosen” algorithm / data structure that you guess or you fail.
LikeLike
I agree that the timeout error is the most unclear thing ever.
The companies make their screening tests. It’s up to them to make something reasonable.
LikeLike
i just hate these tests (in general, not just HackerRank) with a passion – most of them are not realistic as you get so little time to do them, you are already nervous because it’s an interview and then the questions are invariably mostly irrelevant to everyday coding or development problems.
As for all this talk of looking up the answers – sorry, how does that work when you’re put in a room, often with someone watching (or possibly on cctv), and usually on a PC that does not have an internet connection (i.e.: the test is hosted locally)? Every tech test I’ve had has been like this so cheating or “getting assistance” has not been an option.
I’ve missed out on two or three jobs in the last couple of years because of these insane tests – and not because I’m not a good developer as I have been doing C#, VB.Net, ASP.Net, SQL and Oracle for over 15 years – yet I still get so nervous when I have to do these tests because they’re usually stacked with curve-ball questions that, if you struck them just doing your everyday job, you would use the web to find a solution to – or consult a colleague or perhaps have a quick team pow-wow. Yet all of a sudden, inside some arbitrary time period, you’re just meant to be able to solve these without getting the time to *really* consider the problem(s) and therefore the solution(s).
In my current job (of over 3.5 years) get they gave me a “homework assignment” – designing and writing a fairly complex class structure and using it in a WPF app via MVVM. I had 24 hours to do it so I could take my time. I did not use the web once as all this is in my experience but I did go to my own codebase for design patterns etc – nothing wrong with that as I know it’s all good, well designed OO code. Now, *that’s* the way to do it IMHO.
I am going to be looking for a job in May 2018 as my current company has been sold and I am dreading it since, like other people, I often blow the tests because of the stress and time pressure. Yet companies who have taken me on have kept me on for extended periods of time and been very happy with my work – so go figure.
All of this is also an indication of the crazy world we live in – when I started my career if you weren’t good enough they got rid of you, simple as that. and in 25 years of development I’ve never been pushed once. My CV and references should speak for themselves but no, I still have to do these insane, hellish tests.
Maybe i’m just a dinosaur. Could be time to shutdiwn the compiler for good and become a bus driver. 😉
LikeLike
Mike, your last comment is the best… become a bus driver jjaaja
LikeLike
Yep so true- – shut down the compiler and become a bus driver (ofcourse driverless buses) !
LikeLike
Nice Post and funny too !!
LikeLike
99% of programmer jobs are “implement business functionality” and not “design perfect algorithm”. Modern prescreening asks candidates about Spring Controller without first asking Servlet Lifecycle… and candidates know difference between GET and POST but can’t use telnet to test HTTP. What about algorithms? Googling solution is an art too.
LikeLike