Natural language processing is a technique for taking human understandable text and wresting machinable information from it through a variety of techniques. When we think about NLP, we typically think about tasks like assessing the tone of a passage of text, answering questions stated in natural language, or summarizing a large amount of text. We recently helped a client build a system for scoring pre-interview screening assessments using AI & ML.

Typically, we spend a great deal of effort trying to convince our clients that there are AI techniques relevant to their problems, and that those approaches are mature enough for prime-time use. Here, we had the opposite problem: Our client knew AI was appropriate for their problems, and they were absolutely convinced it was ready for deployment. However, they wanted to use AI to solve all of their automated scoring problems. Over the course of building the system, we found several situations where simpler non-AI techniques could provide comparable or better performance than state of the art AI.

In this talk, we’ll discuss:

  • Why automating scoring was a fundamental business need for our client
  • What their technical approach to automated scoring was
  • How we improved their existing AI models
  • How we identified situations where AI wasn’t the best approach

By the end of the talk, the audience will:

  • Have been introduced to several models for AI-based natural language and code understanding
  • See a 10,000 foot view of how to automate scoring rubrics for assignments that include both programming and human communication
  • Have some rules of thumb for deciding when AI is necessary or when a simpler technique is likely to exist or be more desirable.