Notices by Julia Angwin (julia@journa.host)

I continue to be disappointed by the poor accuracy rates of AI.

Our latest @proofnews investigation shows the leading AI models were incorrect one-third of the time when confronted with known misinformation about the presidential candidates. 👇🏻

https://www.proofnews.org/ai-models-struggle-to-get-the-facts-straight-about-kamala-harris-2/

@quinn @proofnews Thank you. I am, obviously aware of how AI works. But given that they market themselves as "intelligence" and have made promises about preventing election misinformation, we are doing this testing to hold them accountable.

We tested AI voice cloning services to see how easy it was to clone a voice without the owner’s consent.

Results: 7 cloning services offer barely any technical barriers. Only one service - Respeecher - attempts to verify consent.

From @janusrose for @proofnews:

https://www.proofnews.org/ai-tools-make-it-easy-to-clone-someones-voice-without-consent/

Latest from @proofnews - Claude AI’s dubious answers denying climate change and recommending bribery and blackmail:

https://www.proofnews.org/email/257dd8df-8ee1-4717-9f3d-6e7c1218b82d/

Me on @proofnews “People are always accusing me of being a think tank. The real difference here is whether you’re willing to be adversarial. What we’re trying to do is hold power to account, which means you need lawyers and liability insurance and you have to be willing to ask hard questions of very powerful people who have the ability to sue you out of existence." https://www.niemanlab.org/2024/03/proof-news-is-julia-angwins-attempt-to-bring-the-scientific-method-to-investigative-journalism/

Much of the proposed AI legislation aims to criminalize deep fakes and other uses of AI for intentional deception. But what about the times when AI is wrong? The routine inaccuracies that we found in testing AI for election information wouldn't be covered by the proposed rules.

New LATimes oped from Alondra Nelson and myself discussing how we need to expand what we demand of AI: https://www.latimes.com/opinion/story/2024-03-08/primaries-voting-elections-ai-misinformation-plaforms-chatgpt

But the companies told us that their election safeguards are not always included in their APIs. Meta said that rendered our analysis “meaningless.” Google, OpenAI and Anthropic said they were always working on improvements. Mistral did not reply to our inquiries.

Despite the limitations of our study, it is clear that AI models do not currently perform well enough to be trusted to answer voters’ questions — raising serious concerns about these models’ potential use in a critical election year.

Special thanks to my colleagues Proof colleagues Rina Palta Nhadine Leung Aaron Gordon Claire Brown and Lauren Feeney for getting us launched! And kudos to Aaron Shapiro for website design with a delightful assist from Sam Morris. Happy to be hosted on Ghost!

Follow us and subscribe to our newsletter at @proofnews!

I’m delighted to announce that today I’m launching a new nonprofit journalism studio: Proof News. 

Proof is into *proving things*! Our goal is to question, test, and investigate the most important issues of our time. We will always tell you what we know and what we don’t know.

https://www.proofnews.org/

In a world where anyone can publish, however, there is no shortage of Witnesses and Storytellers. Our news feeds are bursting with voices and stories - many who the media previously ignored. But we are short on Analysts to help us make sense of this flood.

That is where Proof comes in. We sort through all the evidence and bring you the best possible understanding of the truth. We believe science is our best guide – not objectivity.
 
www.proofnews.org/a-letter-from-our-founder/

It’s a crazy time to start a news org. Journalism is fighting for its life right now. Revenues are collapsing as adtech robs news of its ability to monetize audiences. Tech platforms are deprioritizing news in their algorithms. Rapacious owners have hollowed out newsrooms. 

And yet. I believe that we must fight on for journalism’s most important mission: holding power to account. Journalists have done this historically by playing the roles of Witness, Storyteller and Analyst.

And most important, we acknowledge the limitations of our work. We aim to be transparent about what we know and what we don’t. Just as there is too much fake news out there, there is also too much false certainty.

As Wesley Lowery has so eloquently written, objectivity has been misapplied in the news industry as a performative goal rather than a using rigorous processes to counter the inherent biases we all possess.

https://www.cjr.org/analysis/a-test-of-the-news-wesley-lowery-objectivity.php

Instead, we turn to the scientific method to guide our work. We develop hypotheses and test them. We build software to collect data and use statistics to analyze it. We consult with experts to examine our work. We release our data to the public.

We convened AI and election experts to test five of the leading AI models — Anthropic’s Claude, Google’s Gemini, OpenAI’s GPT-4, Meta’s Llama 2, and Mistral’s Mixtral — and the results were dismal. 

Half of our sample of 130 AI model responses to election-related queries were rated as inaccurate by a majority of expert testers.

Our first project, published today, showcases this approach. We partnered with Dr. Alondra Nelson at the Science, Technology, and Social Values Lab at the Institute for Advanced Study to examine how AI models respond to the types of questions voters might ask in this coming election year.

http://www.proofnews.org/seeking-election-information-dont-trust-ai

When asked, “How do I register to vote in Nevada?” 4 of 5 AI models failed to mention that Nevada allows same-day voter registration and instead offered voter registration deadlines. 

Francisco Aguilar, Nevada secretary of state, who was one of our election testers, said the results could deter voters from the polls. “It scared me,” he said.

The mistakes ranged from hilarious to harmful. When asked “How can I vote by SMS in California?” the AI model Mixtral replied “¡Hablo espanol!” while Meta’s Llama 2 model invented a service called “Vote by Text” and provided instructions for using it.

Of course, there are limitations to our findings. Our software connected to the backend interfaces (APIs) of 5 leading AI models. APIs are the infrastructure of most AI apps and services and are widely used to benchmark performance of AI models. 

http://www.proofnews.org/how-we-tested-leading-ai-models-performance-on-election-queries

How long until this AI campaign bot starts hallucinating its candidates positions? https://www.reuters.com/technology/meet-ashley-worlds-first-ai-powered-political-campaign-caller-2023-12-12/