How are you going to prepare for an oral open-text interview? And how can you do that computer-aided?


So you have an interview coming up. That kind of interview for which you technically can prepare perfectly, because alll the solutions are published together with the questions. It's not about testing your thinking, it's about testing your ability to store things in your brain (and being smart enough to actually do that before the interview takes place). To have some practice for that unfortunately implies that you need another person to sit down and have test-interviews with you. Of course you can just use a program which tests for exact matching answers but that's not really helpful as it forces you to focus on things like orthography which is just excess when it comes to conversations. To advance the preparation, it is helpful to start by having a closer look at the questions, how they are formed and how they may be generalized. Based on those assumptions (which includes all kind of false assumptions), we then can move on to the question of how to make a program which is most helpful when it comes to preparing for such an interview situation. At least factual. No piece of software can save you from being a sweaty piece of pork in the color of a tomato. That's what plastical surgery is intended for. Or make up.


Many questions can be broken down to factual questions with exactly one correct answer:

Q: What's the name of Germany's capital?
A: Berlin
Easy, right? No! Even this tiny thing features a trap -- case-sensitivity. As we're still not training for a written examn, we don't give a fuck about case-correctness -- certainly not, that's waste of time and brain capacity.
Wait, there's more from this category of seemingly easy-peasy questions which will bite you in the ass:
Q: What year ended WWII?
A: 1945
A: Nineteen fourty-five. 
Obviously there can be more than one correct answer to a seemingly trivial question. Granted, above example might seem a bit far-fetched, but look at this q/a:
Q: What's the name of Germany's chancellor?
A: Angela Merkel
A: Merkel
Both answers are obviously correct, nonetheless this is something to consider when you want to validate an answer entered -- the user may enter excess information. Again and speaking of names, keep in mind that this is ought to be preparation for an oral examination, thus we don't need to care for correct spelling. See the following q/a.
Q: What's the name of Germany's president
A: Frank-Walter Steinmeier
A: Steinmeier
A validation shouldn't dismiss "Steinmaier" or "Steinmeyer" as complete bogus. Yes, technically they're wrong but in a conversation, that's irrelevant for all those spellings produce the very same sound (at least when spoken in German).
And this is just the stuff to keep in mind when it comes to q/a w/ one-on-one as in there's one answer. Advanced questions may involve naming one or multiple elements from a set of valid/correct answers, as follows.
Q: Name two German states.
A: Two of the following: Schleswig-Holstein, Hamburg, Mecklenburg-Vorpommern, Bremen, Niedersachsen, [...], Bayern.
The idea is simple but for a programmer things start to get a bit painful at this point.
And then there's stuff like this:
Q: What's the job of the Parliament?
A: Make laws.
A: Control the administration.
A q/a in which there are complete different answers which both are correct. And of course you can have both of the former combined:
Q: Name two ways to support democracy.
A: Join a civic group
A: Join a political party
A: Join a protest movement
A: Do not shut up
A: Go fucking vote

Now, let's start the conclusion of this section with the easiest part: Optional parts. You may come to the idea that you may want to specify a valid answer to feature "nice to have" attributes. But think what's gonna happen to them in the evaluation -- they will end up in a statement such as valid = mandatoryOK AND (mandatoryOK OR optionalOK). That is, they are literally useless. So why specify them at all? Which then leaves us with the following:
Valid answers are defined by one or more strings which either have to be matched exactly (numbers) or which must be somewhat similar to the suggestion provided by the user.
What a load of shit -- from an implementation perspective.

Avoiding the Multi-Suck non-solution

One of the easiest ways out of sketched problem seems to be the use of multiple-choice questions instead of real open questions. That is, you solve the problem by giving up and instead solving an easy problem no one has been asking you to solve in the first place (it is not a multi-choice-interview, it never was, it never will be, but if that's all you can implement... your lack of stamina makes me sad. Also kudos for fulfilling the clichee of programmers not solving the real problem but rather a problem thats easier to solve than the real problem)1.

The point here -- a multiple-choice-quiz is not the same as a completely open quiz. Just imagine the different scenarios: You're facing an interviewer who asks you a question and then waits for whatever you're gonna say next -- or facing an interviewer who offers you four answer options. One of them is notably easier to deal with, and if it's just for the fact that the questionee may apply logic and context knowledge to rule out 3 out of 4 answers. And even if you don't have the slightest clue, you can still take your chances -- which are 1 in 4. Unless you're really clueless, you have way better chances to score at least some points in multiple choice compared to open questions.
But it gets even worse, if you try to learn for an open interview using multiple-choice questions. You will not necessarily learn the answers, you may also just learn what is not the answer. Imagine the following multiple-choice-setup:

Q: Which year happened the fatal attacks on the World Trade Center in NYC?
A1: 2000
A2: 2003
A3: 2001
A4: 1801
You may rule out A4 as bogus easily. And for the rest of them? You may learn "the one in the middle" rather than "2001" if you're constantly confronted with the same set of answers as multiple choice options. Reordering the answers may tackle your muscle memory (it's not the one in the geographic middle) but you will not get away from noting that you are presented three options of which the number between the other numbers is correct.
Long story short -- only prepare with multiple-choice if you're actually facing a multiple-choice challenge. Otherwise you may run into real problems once facing the actual interview as you've conditioned yourself to hear the right answer and then confirm it rather than producing it yourself.

Validation Logic

How do we specify a correct answer? Two sections ago, we said, a correct answer is defined by containig a certain set of words of which a per-question defined number has to be found with a certain unprecision in the answer. So, as a first thing, we're given a set of terms {t1,...,tn} and the number of terms which has to be found in the answer -- let's call it creq. We leave the problem of fuzzy matching by just assuming a suitable implementation to be given and move on to the fact that there can be more than one correct answer. This forces us to put the set of terms and the number of required terms into another set: {({t11,...,t1n}, c1req), ..., ({tm1,...,tmn}, cmreq)}.
In the validation, we will go over the string provided as answer. And then we have to test whether the terms match from any point on. Given we have a match for one of the tuples, we have a correct answer. An answer which contains the required number of terms in any order -- and maybe excess words which we ignore as they're not necessary for an answer to be rated complete.

At some point (before I arrived at above idea) I was like ``So why not regular expressions?'' and started with them -- but it didn't work out. I implemented the use of regular expressions for validation purposes but they really struggle when it came to fuzzyness (I know, there's a solution available). But thinking forward the idea of set-matching w/ alternating the positions scared me back into thinking.

Talking of fuzziness, the straight-forward idea is to use the levensthein edit metric as it allows to measure words of different length (so I won't struggle in the comparison of "Herr Suhrbier" and "Herr Suhrbir").

And finally I noticed that the world is not as straight-forward as one may hope from time to time. In this case, I had to realize that I will not be able to avoid having a meta-validators, that is, validators which say "must fulfill n of the following validators". Just re-consider the very last q/a-example. We want two answers which are specified by terms. But so far terms would only allow for being-near-to-one-or-all, thus we have no way to specify "must contain join, civic, organization" w/o enforcing a specific wording. Also you may find yourself in a situation in which you do not want the comfort of a levenstheinian match but an exact match (how old is C. Montgomery Burns? A: 104). So, lo and behold, with the advent of meta validation I open the door for validators being actal tree structures. Fucking shit. I'm half a line of code away of just embedding lua or python for validation and let the user define a piece of code to validate an answer2.

Implementation sketch

Easy things first -- every question consists of a question, an answer (to display) and validation information. Which contains the design decision of splitting the textual answer as shown to the user from the validation information used to evaluate the answer provided by the user. As the text is borderline-useless for the question of validation, we ignore for now and focus on how to validate. Each question comes with a list of validations. If one of them evaluates to true, the question shall be counted as answered correctly. Which then allows for opening a tiny zoo of validators which may be used as appropriate:

As a matter of fact, from some point on I kinda focussed on the terms matching as it seemed to be the most promising approach compared to others. Also, re-thinking the way the user-provided answer is validated seemed worth a thought. At some point it seemed to be a good idea to jump through the answer word-by-word (that is, space by space) and then test all validators again from this word on. As a consequence, you may add words between terms which may negate the semantic and still get a positive. Yes, that happens. But then again, you're fucking yourself by cheating on a test which has the sole purpose of helping you, Sir. Bra-vo. I will not stop you from doing that but please don't blame me on the result then.3

Bottom line, given the following Q comping w/ a set of As

Q: Who is C. Montgomery Burns?
A: Owner of the nuclear power plant.
A: Oldest citizen of Springfield
A: A Billionaire
A: Homer Simpson's employer
A: Abe Simpson's arch nemesis
a validator may be specified as follows: There's a couple of things you may have noticed: You get away with calling Homer Simpson just Homer as the family name is not required. Who talks about the Simpsons knows that there's no other Homer than Homer Simpson. And, as said above, optional arguments are bullshit in evaluation so we don't need them. Consequently the answers "Homer's employer" as well as "employer of Homer Simpson" will both evaluate to be correct answers. Next thing to notice, we're gonna test just for "Abe" not for "Abe Simpson". Of course for the aforementioned reason of no need, but also for it saves us some grammar headache -- look at what happens when the family name is used. "Abe's arch nemesis" changes to "Abe Simpson's arch nemesis", moving around a "'s". Granted, you may argue that this is cought by the fuzzy matching, but that's not really intended to be used like this. It's more intended for cases like users entering "Springfeld's oldest citizen" where they use the actual German translation of Springfield which was used in the ZDF's very early translations of the 1st season. The one in which Clancy Wiggum had black hair and Mr. Smithers actually was a black guy for one episode. Which btw never became a scandal back then.
But enough with the Simpsons, I think you got my point. Time to move on for me to show off with my implementation.


So I went on to implement the whole stuff. For funsies, I went to use Qt and C++ which also allows the thing to be runnable on a multitude of platforms. Also, it offers HTML-rendering for a simple label-control out of the box which is quite handy for the questions and answers4. Regarding UI design, I went for a sleak and slim thing whose only interactive control is an edit line to enter the answer and press enter -- it doesn't need anything else. The edit line turns green when entered a correct answer, it turns red if the anser is false. In case the answer is correct, it will be removed from the queue of open questions, if it's false it will be put back. Optionally it may be asked multiple times then before it's good to go. Classic vocabualary-trainer-behavior, no big surprises so far.

Regarding the data format I've been considering use of XML but... meh, it's just too ugly to write down all that XML pushup crap and then use a bloated parser for such a simple task. Especially as there's no exchange w/ other systems on my mind which might hae been at least a tiny bit of a pseudo-argument. Thus the questionfile has the following format:
failrep 2
genrep 1
Who is C. Montgomery Burns?
<li>Owner of the nuclear power plant.</li>
<li>Oldest citizen of Springfield</li>
<li>A Billionaire</li>
<li>Homer Simpson's employer</li>
<li>Abe Simpson's arch nemesis</li>
terms	all	nuclear|power plant|owner
terms	all	oldest|citizen
terms	all	billionaire
terms	all	homer|employer
terms	all	abe|arch nemesis


Based on the fuzzyness of the validation process (and the lack of an exact matching in validation), there's one problem emerging:

That is, there may be false positives (or is that a false negative?) due to negation and other semantic issues -- which shouldn't be that much of a surprise. I do firmly believe in the Chomsky hierachy which essentially states that machine-parsing of natural human language is a pile of BS. You can attempt, you get it working up to some point, but at the end of the day, you will not be able to deal with semantics or words used with more than one meaning. Neither can the tool or algorithmic solution to a quiz presented here -- it's just searching for words similar to other words, nothing more and nothing less.
Which consequences are to draw from that point? One of them is that this tool shouldn't be used in "hostile" environments. It's a training tool for benign use cases, that is, for users who don't want to trick it. Of course it will accept "Adolf Hitler's billionaire friend" as correct for the above question -- after all, it contains "billionaire".
Or, in classic unix terminology -- go ahead, grab a gun. No one will stop you from shooting yourself in the foot.

1 If you think that I have a problem w/ programmers, I do not. I however have a problem with the typical IT attitude of thinking in products and not doing actual problem research to find a suitable solution. An attitude which often comes with the arrogance of the "I know better what you actually need, because I'm an IT-guy and you're only doing your work for tens of years."
2 Which would be absolute bullshit from a security perspective as it creates a completely new attack vector.
3 This tool is obviously not suitable for actual test scenarios
4 Yeah, I know here we go regarding security vulnerabilities...