Explaining AI — Auditing Machine Learning

The Hype

Softwares help us take a lot of decisions. From basic decisions, like which restaurant to visit or traffic direction to take to advanced decisions, like determining loan eligibility, detecting fraud, or influencing elections.

The majority of such decision-making software is built using machine learning (ML) algorithms. An ML algorithm creates a model by identifying and learning patterns from the past. Once code is in the software, a model uses these learned patterns as RULES to analyze the present and predict the future.

Models are complex to interpret and therefore are like a “black box” to people who use them. Because models are cryptic, it can be hard to tell when they make bad or sometimes harmful decisions. When challenged, the justification for such a decision usually becomes “the software said so”!

The Challenge

Unlike humans, whose decision-making is derived from experiences, subjectivity, ethics, values, and numerous other factors, the software makes decisions based only on the quality of data and the quality of the model built into it.

Since ML algorithms learn about things or people based on specific examples in the data, the quality and representativeness of the data can significantly impact the quality of decisions. For example, in 2015, Google got trolled on Twitter after its Photos app categorized a couple as Gorillas.

Among many reasons for this bad decision by google photos, one of the reasons may have been that the photos app learned how Gorillas look using a limited variety of data and not learned how Humans and Gorillas differ on otherwise similar-looking facial features. This is called adversarial training in machine learning.

Consequently, when the software analyzed Oluwafemi’s photos it may have successfully matched facial features such as the color, and position of the nose, lips, and eyes of the person to that of a Gorilla.

Also, the reach and scale of the internet makes the impact of bad learning spread faster and difficult to catch or eliminate. Microsoft’s AI-powered chatbot called Tay got corrupted by Twitter conversations and was shut down within a day of its launch.

Tay was learning twitter trends fast and used such learnings to tweet and reply.

Tay tweeted racist remarks, said Hitler was right, and agreed with Donald Trump’s stance on issues!

In such situations, bad decisions by software can become an embarrassment.

One approach to avoid bad decisions may be to limit the learnings of ML algorithms only to the expected results or good tweets in this case! However, such an approach to limit the data to favorable outcomes also does not completely eliminate bad decisions.

Predictive models can use proxies automatically. Proxies are instances that strongly correlate with those bad examples you want to avoid. For example, an ML algorithm called “Minority Report-esque” predicted the chances of a convict committing a crime again. Instead of using gender, race, or age, which may bias decisions, it used the data on whether the arrested person has a parent who has been imprisoned earlier. After several biased decisions and allegations, it was discovered that the algorithm’s approach also correlated with race. Since African-Americans are arrested more than whites “Minority Report-esque” disproportionately rated African-American arrestees as more likely to commit future crimes.

It will therefore not be an overstatement that ML algorithm creators may unintentionally discriminate if they fail to correctly translate use cases into the software.

What’s at stake

All decisions have a win-to-lose ratio. Losses sometimes are bearable sometimes are not. The criticality of making the right decision varies according to the possible consequences of a wrong decision. For example, a recommendation algorithm that suggests bad restaurants over good ones may not cause as much harm as algorithms that assist in decisions related to job opportunities, medical insurance, or criminal justice. A bad decision in such cases can become life-changing.

In 2014, Ben Bernanke, former chair of the U.S. fed reserve, was denied a mortgage for a house whose value was much less than his projected income. Upon investigation, it was discovered that since he changed his job a few months ago and since recently changing jobs makes you a credit risk, the software made a decision to deny. The decision taken was against Ben Bernanke and therefore the software was challenged. Usually, the software can deny eligibility, and people are not provided an opportunity to challenge the decision or the reasoning behind the decision.

When a software’s decision is misaligned with the user’s expectation, it generates a misplaced trust. When such dingy trust combines with the opacity of ML-based software, it confuses people and thus discourages them from trusting the decisions made by software. This means that people may perceive ML software products as damaging without giving them the credit for the unprecedented benefits they bring.

Therefore, the stakes for machine learning software are high and we as product professionals own the responsibility to ensure people trust software.

How to build trust

Traditionally, the software was tested for possible outcomes to ensure predictability and establish trust. In the case of machine learning software, possible outcomes are many, are of varying types and sometimes unknown, and therefore difficult to test. Had Microsoft been able to test how their chatbot will behave they would have never released it. But they did!

In my professional life so far, I have traveled to different parts of the world to meet product and business folks, and CxOs at events, offices, and conferences. The underlying theme of such meetings has been to quantify the business benefits of machine learning-based products and articulate how ML-based products can be trusted by giving an intuition of how such products make decisions.

From being in the trenches and witnessing what works and what does not, I can confidently state that trust is not synonymous with correctness in decisions. The trustworthiness of a software depends on the software’s ability to explain the decisions itself and the logic behind the decision. What matters is that software gives users enough information to assess how much trust they should place in its decisions.

A good ML-based software product that will build trust has to translate the software’s decision in one way or the other into an understandable and relevant explanation.

How it can be done

The ML-based software analyzes data in dimensions unimaginable by humans. We can visualize at best 3 dimensions (3D Movies, add color, shape, and size to increase the number of dimensions). However, an ML-based software analyzes data in hundreds or thousands of dimensions depending on the complexity of a problem. Such multi-dimension analysis renders to ML-based software the capability to make complex decisions.

So, in an ML-based software product, successfully building capabilities that reduce dimensions of a decision from machine-interpretable to human interpretable will be a big step towards establishing trust. This is MUCH MUCH easier said than done!

After spending numerous calories on this endeavor that resulted in a series of failed attempts and some success, my team and I productized an approach to make ML-based products explainable using an evidence pack. An evidence pack attempts to visually explain in a model what importance was given to evidence (from the past and present data) that led to a decision step by step.

Once presented to a user, an evidence pack renders the ability to understand the decision-making process. This comprehension gives a user the confidence to trust the decision and also communicate the rationale for a decision to the person who gets affected by the decision.

An approach like an evidence pack not only establishes trust with people who use software but also helps product folks like us to understand where programmers created a bias in the model and correct it to avoid bad decisions.

I imagine that using an approach similar to the evidence pack, Google Photos’ product team would have quickly figured out that while recognizing a Gorilla, the software is putting over-emphasis on features that resemble a Human. Further, the evidence pack may have also highlighted that almost no emphasis was put on skin texture which is an important and distinguishing feature between humans and Gorillas.

Also, using an evidence pack approach, Microsoft Tay’s product team may have quickly figured out that while responding to users, Tay will put emphasis on the context of a conversation merged strongly with the context of trending tweets on the same subject. This may have led to biased responses from Tay since there may exist a lot of varied opinions on a particular topic that may differ drastically from the context of the conversation that Tay may be having with a user. All of this led to gibberish conversations Tay had with users.

I hope the above examples establish the point that an approach like an evidence pack helps users and creators uncover and comprehend the functioning of an algorithm.

With references to customer interviews and engagements with customers who use ML-based software products, I will conclude that an evidence pack gives software tremendous UI capabilities and encourages users to trust the decisions. In the near future, I foresee refinement in product approaches like evidence pack that will ensure that ML-based software can solve real-world problems better and with people confidently trusting the decisions.