"We just let the AI decide—if someone is unhappy with it, they can appeal/request a human review" seems to be one of the ways people are arguing for the use of AIs by public bodies or courts.
But if the protection for those whose case is being decided is the ability to appeal/request human review, you don't neeed an AI, you can just have a random number generator tossing a proverbial coin.
`return random.choice(["claimant", "defendant"])` uses a lot less energy than an ML model.