Trust, Teaming, Cognition & Dysfunction

What we are missing when we think about AI, Autonomy, & Humans

I’ve been mulling over several topics in my head this week, and they all seem to be merging around one simple but often overlooked fact: what “normal” looks like. This seems to be the biggest elephant in the room to me, at least from my little perch. Why is this important and why should you care? Well, if you are interested in all of the new fangled AI assistants being peddled this week, or if you are interested in human-autonomy teaming, or if you are interested in human-machine interaction, or if you are interested in discussions about “trust” and AI and autonomy, or if you are merely just interested, then this is for you.

How do I begin this rather convoluted and messy set of topics…. OK. So, I am interested in human-autonomy teaming (HAT). Cards on the table. I’ve researched this for a number of years, and what I can say is that when it comes to a broad understanding of what “good” or “right” looks like… we don’t have it. There is an ocean of literature out there, but here are a few of my favorite perspectives that show the difficulty of wrapping our minds around HAT. As Greenberg and Marble point out, semantics matter. The words we even use to describe what is necessary or sufficient for HAT are all value loaded. “Team” is a social entity, that has interdependence, multiple members, and vulnerabilities. In some instances, teammates need to “share a mental model” of the task, the mission, the goal or even the world. This requirement quickly punts us down the rabbit hole of whether or not AI agents need to have a Theory of Mind (ToM) to be “true” teammates. Perhaps not of themselves, but they at least require it of their human counterparts to be able to function appropriately. DARPA, awhile back, was exploring this very idea.

But, as Chakraborti et al point out, learning human mental models (which is maybe a lesser challenge than coming up with entire ToM) is no easy task because the human mental model is open-ended. AI planning agents “typically do not have sufficient knowledge about all task-relevant information” and so ultimately their models of people are incomplete. This is an open area of research and “we do not yet understand how these different human models interact,” i.e. we can have different people with different models of the task, mission, role or what-have-you. That’s because humans are weird and different.

So, we have this challenge. We want to create AI assistants, teammates, or whatever. But humans are messy for an AI agent. Due to our messiness, there is an almost 100% chance that there will be some sort of failure or error that results when the human and the AI agent are “working” together, whatever you want to define “working” as. This is where those pesky discussions about TRUST come into view.

Now, I have very strong feelings about the word “trust” as it applies to technological artifacts. David Danks and I wrote about this many moons ago, and I have still followed the technical and philosophical literature with both pain and pleasure. Here is the rub: we don’t have any standard measurements for trust (or HAT for that matter). There are a variety of technical approaches to getting at measuring trust in HAT, to be sure. Most of these approaches involve humans self-reporting how they “felt” about an AI/autonomous agent at the end of some experiment. Many experiments also happen to be “Wizard of Oz” style, where humans are actually engaging in an experiment with another human, who is pretending to be an AI agent.

And there are some very interesting approaches that delve a little deeper into what humans actually need from their AI teammates, given that we humans come with our own theories of mind. For example, Cohen et. al., looked at “verbal anthropomorphism” as a potential way to measure trust of between AI agents and humans in an HAT. They found that variations in the use of verbal anthropomorphisms “could be used to inform adaptive algorithms in autonomy AI towards strengthening team resilience” where the human use of anthropomorphic terms or more object-related terms could “be used as a trigger for a virtual agent to initiate automation trust repair mechanisms.” But… they warn that “a validation study o the use of verbal anthropomorphism as a measure of perceived anthropomorphism is recommended in earnest.”

While many individuals and researchers warn about anthropomorphizing, for good reasons, this may just be a social fact that cannot be ignored. Thus we could potentially use it, as Cohen’s team does, as a way to understand the functioning of the team. Who knows. We need more research.

The results from experiments that try to measure “trust” of robo-teammates also tend (not always) to vary according to human perceptions of reliability. Humans tend to “trust” AI agents/team mates/whatever, more when they perform how they are expected to perform and do not fail. But, experiments like this just mean that trust is nothing but a reliability rating. Clearly there is something more than that…

Maybe. Esterwood and Robert experimented on “trust repair” when an AI agent fails in the course of an iterated task. This trust repair was measured via an AI agent stating some form of apology, denial, or a promise (to do better). But the experiment was still ultimately a self-reporting of how humans felt prior to the “violation” of trust by the robot and their self-reported feelings of trust after the “violation” and potential “repair.” They find that “apologies and denials appear to be more effective when subjects ascribed the robot greater levels of conscious experience and less effective when subjects ascribed the robot lower levels of conscious experience.” «I literally have a spasm when I think about the word “violation” used in this paper. Violation of what? A right? But again… language matters.»

Let’s pause here for a moment. Trust is a human construct. It is a concept. It is dependent on all sorts of things. Moreover, there are different kinds of trust. One of my favorite kinds of trust, especially one that gets used often and every day (especially if you work in national security or defense) is transitive trust. In other words, trust becomes a transitive property. I trust Bob, Bob trust Jane, and so I trust Jane on Bob’s recommendation. This transitive property also works for institutions… I trust that the institution has put into place certain standards, expectations, evaluations, etc. and I trust the institution and its workings. Thus if you are a part of that institution, then I trust you (to some degree). An interesting paper on trust, blame and rank (I’m guessing as a proxy for hierarchical responsibility) in HAT by Gall and Stanton try to tease something out here, but aren’t really measuring anything to do with institutions (though they are implicitly using it in a weird way… ).

My point is that trust is multi-faceted, has different properties, and thus it has different dimensions depending upon context. So “trusting” an AI system or teammate is sort of like saying “I trust people.” Really? All people? Either you know all people (8 Billion+ of them), or you are saying that you “generally trust people” given a variety of experiences you’ve had with them. But the kicker is that even if you know someone, it does not guarantee that you can trust them. Sure, you may have had some experience that gives you one or more data points on which to say — nope! Or, like a freight train, you have no experiences of mistrust and you are suddenly hit by one.

As David Danks and I suggest, there are also different models - “thick” or “thin” - levels of trust too. Where maybe “thin” just amounts to reliability. The car turns on all the time, the worker shows up to the job on time (or late), etc. Or perhaps it is a mix of reliability and predictability. For I may be reliably late… and hence predictable. Or I could very unpredictable - late today, on time tomorrow, early the next day - but my performance is reliable.

The point being is that saying that we need to “trust” systems is crazy. We can have mathematical data on failure rates, sure. We can use our fancy econometric models to test their confidence intervals. Or, if the systems are too complex to thoroughly test and evaluate, then we try to create guardian systems around them to give us a sense of predictability.

Ok. So trust as a word sucks. Where does the “cognition” and “dysfunction” come into play? Stay with me on this… I know it is a long post.

AI Assistants (or teammates, etc.) are increasingly about cognitive tasks. While I like my physically embodied robots (thanks Roomba), I’m concerned here with cognition. If the AI Assistant can help me understand something, or do cognitive work for me… that kind of thing. Given the releases by Google and OpenAI this past week, I think we can all see where this is going.

OK. So cognitive stuff. Here is my big problem. What is the baseline for “normal” cognition? How do I measure what a normal human brain is doing at any given moment during any given task? How pray-tell, can I then measure if my assistant is actually assisting me?

Cognitive measurements to date are pretty lame. They typically involve subjective measures about perceived mental effort or task difficulty. For the past 30-odd years, this is done using a 9-point Likert scale (where the lower the score the easier the mental effort). There are numerous criticisms of this approach, and I will not delve into them here. A newer effort as attempted to change the Likert scale with pictoral representations rather than the numerical Likert scale. Their results weren’t that jaw-dropping.

So, cognitive measurements, trust measurements, all these measurements are subjective. There are certainly problems with that — the least of which is reproducibility.

But my question problem/objection is deeper. We keep assuming some “normal” baseline for everything, when we do not in fact have such a baseline. The only time we can identify something is if it is abnormal - or perhaps outside of our beloved Bell curve - and that is if it is significantly abnormal.

« So what would we be talking about then? Well, would it be someone with a cognitive disorder, dysphoria, or dysfunction? But do we even have data on that?! No. A brief scan of the literature on AI and assistive technologies is pretty bad. Most of the technologies are blatantly ableist. But, that is another topic. »

We assume that the *subjective* response is already in the norm. There is not a test for that. Sure, we can say the law of large numbers will take care of it for us. But, first we don’t have that data. Second, what we are even measuring is in fact a convoluted and socially constructed concept (trust) that varies with people, time, and circumstance. So, if we are really concerned with building systems that can team with people, that can “assist” them - whatever that means - then we are wading into a pool of (presently) unknowable things because we’ve assumed away everything important. Maybe in the future we can have better measurements for cognition, maybe we won’t, but we need better data first and foremost.

Ok.. enough for today. That’s all folks.