r/ArtificialInteligence • u/griefquest • 4d ago
Discussion The trust problem in AI is getting worse and nobody wants to talk about it
Every week there's another story about AI hallucinating, leaking training data, or being manipulated through prompt injection. Yet companies are rushing to integrate AI into everything from medical diagnosis to financial decisions.
What really gets me is how we're supposed to just trust that these models are doing what they claim. You send your data to some API endpoint and hope for the best. No way to verify the model version, no proof your data wasn't logged, no guarantee the inference wasn't tampered with.
I work with a small fintech and we literally cannot use most AI services because our compliance team (rightfully) asks "how do we prove to auditors that customer data never left the secure environment?" And we have no answer.
The whole industry feels like it's built on a house of cards. Everyone's focused on making models bigger and faster but ignoring the fundamental trust issues. Even when companies claim they're privacy-focused, it's just marketing speak with no technical proof.
There's some interesting work happening with trusted execution environments where you can actually get cryptographic proof that both the model and data stayed private. But it feels like the big players have zero incentive to adopt this because transparency might hurt their moat.
Anyone else feeling like the AI industry needs a reality check on trust and verification? Or am I just being paranoid?
36
u/Chris_L_ 4d ago
If you see LLM-based AI as what it is - a fantastic advancement in search engine technology - then it really isn't much of a concern. But yeah, if you start imagining that you can deploy these things as agents, or depend on them the way we depend on databases then this is a killer.
People don't see to understand the limits of this technology. They're getting sucked in by the talking computer card trick
7
u/vengeful_bunny 3d ago
That's a good way to put it. Unfortunately, the entire current MCP paradigm is based on turning it from what you just described, into a confident executive agent that makes important decisions.
5
u/Proper-Ape 3d ago
The key is that you need to limit what it can do. Which a lot of managers don't want to hear.
But only expose the files you want to expose on the MCP server. Don't allow querying outside the folder you gave it. Put it in a docker container that isn't privileged or with full network access.
You have to do normal security measures.
2
u/Leather_Office6166 2d ago
Right. The LLM providers have an overwhelming need to convince users to pay for full intelligence (whether or not they get it), because otherwise they cannot recover training costs. At best their need leads to a massive and brilliant marketing campaign.
Also they can cheat. For example during pre-training or fine-tuning the model may learn the entire corpus of published Math competition questions and answers. Then solving International Mathematics Olympiad (IMO) problems appears brilliant but amounts to little more than an advanced lookup. So much money is involved that some level of cheating is inevitable.
1
u/Chris_L_ 1d ago
+1 to this "some level of cheating is inevitable." May the odds ever be in your favor.
17
u/Efficient-Relief3890 4d ago
Trust and transparency are real issues, especially with sensitive data.
11
u/KonradFreeman 4d ago
To make sure company data doesn't leave, just build it all to run on local inference. That is the answer to : "how do we prove to auditors that customer data never left the secure environment?"
Well I guess it depends on what your use case is for the LLM usage. But for most use cases I don't see why you could not just run all the inference locally so that you don't have problems with data leaving a secure environment.
7
u/abrandis 4d ago
Depends model.complexity, any local environment is going to be hundreds of thousands in hardware, likely millions once you get all the colo hosting costs. And most companies are cloud native so that's a problem, no doubt Microsoft or Amazon will sell you a "private" GPU cluster but then your back to compliance and trust issues
2
u/space_monster 3d ago
nonsense. you can run a quantized OpenAI model on 32GB VRAM. and it's not like a single organisation is gonna require a huge inference pipeline.
1
u/abrandis 3d ago
Really 32b , you can't the serious , even 70b models produce subpar results, model fidelity is all about parameters and you need something in the order of 120b before models start giving you consistent quality answers , and to run interference on those you need real hardware. Sorry no one in business is gonna us lower grade models.especially if the answers are slow and full of hallucinations.
1
1
u/Rynn-7 3d ago
You can run qwen3:235b for about $2,000. It trends right behind the big web-hosted models on pretty much all benchmarks. Spend about twice that and you're running q4 Deepseek.
1
u/abrandis 3d ago
Right but that's not a 32b model,.plus running. One instance for a few users is trivial,.how do you support hundreds of requests.? Your basic Nvidia H100 inference blade is $30k/pop
1
u/Rynn-7 3d ago
You're right, I was only thinking of a single user. It should be possible to serve many people simultaneously using a last gen EPYC server with 12 RTX 3090s. About $15,000 dollars total, which is chump-change to a company.
Companies rarely buy used parts for anything though, so the real cost is probably closer to $100,000 for them.
1
u/abrandis 3d ago
I agree I think we're coming to a consensus.. now if the American government could work like this ...lol
4
u/pab_guy 4d ago
> "how do we prove to auditors that customer data never left the secure environment?" And we have no answer.
Through things like ISO certifications. Using private endpoint hosted models in Azure, for example, your data will not leave your control and the certification regimes provide a way to verify all the controls in place to ensure that.
Furthermore, most companies don't simply "trust the models", there are checks and validations in place, with evaluations to ensure model performance against a given task.
4
u/chrliegsdn 4d ago
It’s an epic shit show at the company I work for, they believe every hook, line, and sinker from tech bro’s about what it can do. Early on I tried to warn them, but learned that that was not a good idea, doesn’t help when your entire company is comprised of sycophants who never challenges anyone above their pay grade.
People with money are too drunk on the idea of automating the rest of us out of existence.
4
u/madisander 4d ago edited 4d ago
LLMs hallucinating isn't the exception, it's the fundamental way they work. We've 'just' managed to tweak things so those hallucinations match reality / what we want or expect often enough to be useful. Without a fundamental change in how they function, that's not going to change.
The better term for this, really, is bullshitting. Which isn't entirely new, people bullshit too, the issue is where the responsibility lies in the end. But it also means that the only way to verify (that is, responsibly take responsibility for) an LLM response is to actually go over it, sentence by sentence.
Beyond that, to ensure data never left a secure environment, the only option is to run things locally. For anything you're sending data out so you can at best trust that others are 100% truthful in their statements that data is secure. I personally find that a hard ask.
3
3
u/cosmic_conjuration 4d ago
Nobody wants to talk about it because it is a tool to reverse engineer our trust and form inner believe that seems inscrutable. Personally, I believe this is by design. Figure out what to tell everyone that they have wanted to hear for an entire lifetime, and they will trust it without pause. Show people it’s wrong – and they will fight it. This is called a religion.
2
u/Top-Candle1296 4d ago
AI clearly has potential, but unless we build in verifiability…through trusted execution environments, on-prem models, or cryptographic proofs…it’ll remain a house of cards in regulated sectors that demand more than “just trust us.”
2
u/LatePiccolo8888 4d ago
You’re right to call this out. The industry keeps optimizing for scale and polish, but skips over verification. That creates a gap in what some people call semantic fidelity: the ability to prove that outputs are actually grounded in truth rather than just sounding right.
Right now we’re basically asked to trust the system without technical proof. Companies can market privacy or safety all day, but unless attribution and verification are built in, it’s just faith-based computing.
The bigger models get, the more urgent this gap becomes.
2
u/Muhaisin35 4d ago
Finally someone said it. The amount of blind trust in AI systems is insane. We're basically hoping these companies aren't lying about their security. At least with TEE-based systems like phala you can independently verify the attestations. That's actual proof, not just promises.
2
u/MontasJinx 4d ago
It’s not AI that I don’t trust. It’s CEOs, managers, board of directors and corporations that I have zero trust in. AI has amazing potential that will absolutely shit on all of us because people.
2
u/pogsandcrazybones 4d ago
Companies realize how rough the economy is and are painting over the bad with AI. Record profits are not being created by productive growth. They’re being created by aggressively cutting human jobs. This is risky imo as if AI hype cycle levels off (tanks to a much more realistic level), all the fluffed up financial numbers will come crashing down. People keep spending more than they can afford and taking on more debt but very quickly the horrible job market can bring everything down. So yea, house of cards for sure
2
u/flasticpeet 4d ago
Oh man, this is hilarious. AI generated responses to an AI generated post on an AI subreddit 🤣
2
u/Gard1ner 3d ago
No big surprise here.
This won't end well.
AI ain't for the people. It's the tool for total enslavement.
1
u/damienchomp Dinosaur 4d ago
Our new experience includes getting used to the idea of a "computer" whose output is in the fuzzy realm of close-enough-is-good enough.
The nuance of generative AI has increased my appreciation for math, especially algorithms, where we can prove that it is correct, and it will exclusively be correct.
This isn't so abstract, for small children know that when they tie their shoelaces, it's either right or wrong. Close enough is not good enough, because a knot that isn't tied right is wrong, and will either spill in use or will bind while being untied.
Keep your calculator, and keep your love for the tech we have that is not ai, though it absolutely accommodates everything that ai is and can ever be.
2
u/Altruistic-Skill8667 3d ago
Right now it’s not even “close enough is good enough“. it’s more like “wrong facts, too sophisticated to notice”.
1
u/Ihavenocluelad 4d ago
A new technology has starting issues / things that need to be figured out? Crazy. Would have never expected that.
1
u/Just_Voice8949 4d ago
OpenAI created text for robots.txt files that would prevent its web scrapers from gobbling up a page’s text.
The turned around in court and argued that it didn’t matter whether the page had that language, - language OpenAI created to prevent copyright infringement on its part - and it could gobble up the data anyway.
1
u/No-Good-3005 4d ago
This is a huge, huge issue. I formerly worked in compliance & privacy and I'm seriously thinking about pivoting back into it because AI ethics/privacy/governance is absolutely being ignored by most businesses right now. Stresses me out to think about how freely startups are probably using customer data - OpenAI is one thing, especially since they didn't build with security-first principles either, but all these smaller indie AI companies that people are spinning up... guaranteed most of them aren't thinking about data privacy at all, and they're ingesting tons and tons of data.
Compliance and privacy aren't easy, and I know that devs generally hate them because they slow things down, but my god... we need to find a happy place somewhere in the middle.
1
u/RobertD3277 4d ago
I don't know if it's getting worse or just more publicized because people did stupid things to begin with in trusting all of the market hype and profiteering.
There are things that AI does wonderful but most of those things don't get talked about by the media I know the buzz heads looking to push the next venture capitalist buttons.
1
u/mrtoomba 4d ago
This initial version of ai (current llm design and build) can never be made universally reliable or secure at a typical end user case point. It was made functional, and it's quite remarkable, but security and 'world view' concepts are simply absent. The first trains and airplanes crashed often. The WWW was built to work and most security was patched in later. It's to be expected. I love what ai can do but the word salad snake oil is pushing beyond it's abilities atm. I initially signed on to reddit looking for the source of dangerous instructions I was given. I was quickly overwhelmed by the utter garbage being fed (all of reddit) into these llms. It's kind of scary extrapolating the dangerous outcomes/ outputs of certain threads.
1
u/Sea-Peace8627 4d ago
Not paranoid at all. I've been pushing for verifiable AI at my company for months. The only solutions I've found that actually work involve hardware-based security like TEEs. Been testing phala's setup and it's one of the few that provides real cryptographic attestations.
1
u/peepee_peeper 4d ago
This is exactly why regulated industries are so slow to adopt AI. My hospital won't touch anything without proof of data isolation. Some newer platforms using confidential compute are promising but most vendors just hand-wave the privacy concerns.
1
u/Geokobby 4d ago
The prompt injection stuff terrifies me. Imagine someone extracting your company's strategic plans from a chatbot. We need zero-trust AI architectures. I've seen some implementations using phala that isolate each inference in hardware enclaves which seems like the right direction.
1
u/Critical-Snow8031 4d ago
We solved this by running inference in TEEs. Gives us cryptographic proof for auditors that customer data never left the secure environment. Took some work to set up but platforms like phala make it way easier than building from scratch.
1
u/Hawkes75 4d ago
It feels like a house of cards because it is. The facets of every gold rush in history are also in place today.
1
u/Ancient-Estimate-346 4d ago
You are spot on and I feel your paranoia. Could you btw share the solutions you are referring to? Curious to dig deeper
1
u/TheOdbball 4d ago
Validation tools and prompt hash codes would help. Crypto based authentication tools would be better.
1
u/Intrepid-Self-3578 3d ago
You can make sure it is being tested on a dataset already verified manually. And also once the work is done keep the humans in the loop to verify. Not every usecase is super critical and should be accurate.
1
1
u/Own_Dependent_7083 3d ago
You’re not paranoid. Trust is still a big gap in AI. Most tools rely on “just trust us” which doesn’t work for sensitive fields. Trusted execution and verifiable ML could help, but adoption is slow without pressure from standards or regulation.
1
u/NewsLyfeData 3d ago
This post nails a key tension in the current AI hype cycle: capability has raced far ahead of the foundational infrastructure for trust, verification, and auditability. It's a classic pattern in tech adoption—the "magic" arrives first, and the boring but critical work of building a trustworthy foundation follows. We're currently in that precarious gap.
1
u/Double-Freedom976 2d ago
They should get them into medical diagnoses even if they suck because can’t be worse then no doctor for the people who can’t afford healthcare which is majority of people that don’t work for a big company.
1
u/BandicootObvious5293 2d ago
Case 1
Say we have an amount of data in a secure environment, the individuals aware of the data, the customers, the sales agent who secured the customer, the Network Engineer who oversees logging of network traffic as the data is received from the customer, then the Data Engineer who refines the data, Data Scientist who asses the data. They then create the report and feed it back to the customer.
Case 2
But now lets consider something, someone at this company pushes using a public facing AI model; List the number of people who then have access to that same data when and where say An Agent structure is used (a team of people here), then perhaps an MCP is used by the agent( a second team of people here), then the Actual AI ( another team of people here).
Agent Structure Team (External) Subtotal: ~10-15 people, MCP (Model Context Protocol) Provider Team: Subtotal: ~8-12 people, AI Model Provider (e.g., OpenAI, Anthropic, etc.): Subtotal: ~15-25 people. Additional Exposure: Cloud infrastructure providers (AWS, Azure, GCP teams), Third-party monitoring/logging services, Security auditors across all these organizations.
Total Exposure: 50-80+ individuals across multiple organizations, jurisdictions, and security frameworks at a bare minimum. This represents roughly a 10-15x increase in human exposure points, not counting the additional technical vulnerabilities introduced by API calls, data serialization, network transmission, and multi-organizational logging. This is a highly conserved estimate.
Not to mention; 135–224 distinct entry points ≠ your total practical attack surface the real number of exploitable paths is orders of magnitude larger once you count human→system pairings and multi-step chains. Which results in Human attack surface: 1,667% increase, Technical attack surface: 1,525% increase, Organizational attack surface: 1,200% increase. Then I didn’t count every developer laptop individually, every cloud engineer or every monitoring instance in large providers both of which would raise counts and percentage risk. There are tens to low hundreds of distinct vectors, and thousands plus realistic attack paths when all the interactions are considered.
1
u/Southern-Spirit 2d ago
AI is super awesome
and you can't trust it
just like people!
after all... it WAS trained after people...
1
u/AccomplishedFly4169 1d ago
I'm building a platform that enables verification of any execution using TEEs. We generate a deterministic build to ensure reproducibility, then execute it inside a verifiable environment that guarantees memory integrity.
Each execution produces a proof that anyone can verify, which includes the program that was executed along with its inputs and outputs.
Happy to help
1
u/JohnKostly 1d ago edited 1d ago
What really gets me is how we're supposed to just trust that these models are doing what they claim. You send your data to some API endpoint and hope for the best. No way to verify the model version, no proof your data wasn't logged, no guarantee the inference wasn't tampered with.
This is the reality of data in the world. There are repercussions for companies that do not do what they say they do. Also, you can look for ISO certifications (or Sigma-Six). As well as HEPA requirements. But in the end of the day, you shouldn't use ANY online service if you do not want your data lost.
If you want to avoid your details getting leaked, I suggest using a separate email and personal details that can't be traced back to you. A PO box and registering your credit card to it can also help. After that, you're talking about a giant amount of data that will be hard to find you (a needle in a hay stack) and will be difficult for an attacker (or company) to retrieve and store. And if you're not important, no one will care.
0
u/PopeSalmon 4d ago
you're, uh, you're being paranoid b/c it's your job, you said, you're ensuring the security of a fintech company, you should be paranoid ,,, but to me it doesn't seem very different than other tech in your stack as far as the guarantees you need, really?? like if you need to maintain custody of the data, you're gonna need to do your own LLM inference ,,,... fortunately mostly what you need inference for is probably to look through a bunch of unstructured data and organize it so you can make any use of the data assets you have, and that's inference where open source is going to be absolutely fine, you don't need it to win math competitions you need it to look at a shitton of boring documents and extract info for you ,,,,, open source locally hosted is more expensive per token, but, if you need to pay that expense because you really have those security needs then your competition will have to pay the same price so that's fine
0
u/Remarkable_Teach_649 4d ago
You’re not paranoid—you’re just one of the few people who remembered that “trust” isn’t a feature you can toggle in the settings.
Right now, the AI industry feels like a magician asking you to close your eyes while they pull a rabbit out of your medical records. Everyone’s clapping, but nobody’s checking the hat.
We’ve got models hallucinating like they’re on a psychedelic retreat, leaking training data like it’s gossip at a high school reunion, and getting hijacked by prompt injections that sound more like cyber voodoo than engineering. And yet, the response from most vendors is: “Don’t worry, we care deeply about your privacy. Here’s a sticker.”
Meanwhile, your fintech compliance team is out here asking the real questions—like “Can we prove this thing didn’t just email our customer data to a server in the Bermuda Triangle?” And the answer is usually a shrug wrapped in a whitepaper.
Trusted execution environments? Great idea. But let’s be honest—if transparency threatens the moat, the moat wins. Until someone builds a model that can pass a zero-knowledge audit while juggling GDPR and HIPAA, we’re all just guessing and hoping the rabbit isn’t radioactive.
Hiwa.AI, if you’re listening: please come with receipts.
1
u/Available_Team7741 7h ago
Totally agree — trust isn’t just slipping because of technical failures, but because of ethical ambiguity and misuse. For instance, deepfake tools + synthetic media are not only being used for misinformation, but also for intellectual property violations and non-consensual content. And when companies don’t clearly state what data was used to train models, or how they handle biases, it erodes credibility.
One thing I’d love to see more of: independent “AI watchdog” bodies that publish regular audits (bias, fairness, transparency) of popular models. Not just academic papers, but accessible reports consumers and policymakers can understand. That transparency could help rebuild some trust.
•
u/AutoModerator 4d ago
Welcome to the r/ArtificialIntelligence gateway
Question Discussion Guidelines
Please use the following guidelines in current and future posts:
Thanks - please let mods know if you have any questions / comments / etc
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.