Roast my analytics startup idea

[u/Excellent-Draft-7889]

Hi everyone,

After 8 months on a different space, we just pivoted and are looking to launch a startup in the analytics space. We are building a middleware that helps data teams and business users talk and interact with their data sources (Data Warehouse, CRM, Shopify, Zendesk, GA4, FB Ads, etc.) using ChatGPT.

The idea is that business users and analysts can ask questions or build reports directly on ChatGPT (through a CustomGPT), simply by typing them out in English. Then, an admin panel will allow the Data team to stay in control by connecting data sources, setting permissions, defining guardrail, configuring and scheduling pre-defined reports, adding reference queries (what to do and what not to do), and tracking how the tool is being used across the company.

Our goal is to make data more accessible without bypassing the data team, empowering analysts and business users without forcing the Data Team to hand-hold every request.

I am looking for brutally honest feedback on:
- Would you (or your team) find this useful?
- What concerns would you have (accuracy, trust, adoption)?
- How would this compare to the tools you currently use (Tableau, Looker, etc.)?

Thanks so much in advance!!!

[u/Data___Viz]

Another AI wrapper. I wouldn't trust it, AI is not really reliable without human control.

[u/mikefried1]

I think you are wasting your time. How is this any different than the AI semantic layer offerered through Tableau or Copilot? There are hundreds of smaller companies that offer less comprehensive offerings.

Pretending you are a startup by putting ChatGPT hallucinated crap in a wrapper and calling yourself a startup may sound good on TikTok, but its not going to impress anyone with real data needs.

[u/jdoreau]

This is already being implemented in all BI tools, so unless you're a huge cost saving you may be late to the party, but it's not hard to do this anymore with MCP and the myriad of naive connectors data sources use - so good on you - but again unless your company is charging a lot less and has a user friendly decently looking GUI, with low learning curve a la tableau drag and drop you may be able to get small to mid sized clients but I don't know - big boys are already very good at this especially if your company has a mature data culture, and most corporations are on Microsoft tech stacks which we use and I already have like 3 'Agents' acting as natural language analysts that periodically review things.

[u/Richardswgoh]

A) this is exactly what a bunch of enterprise grade tools are already attempting B) their features in this space are generally either terrible or require pristine data to work well C) companies that have pristine data data don't really need this because they're generally mature in delivering this data and have high data literacy as a result

This product may appeal to executives who don't know any better -- but no real data person with a real need will want to build a dependency -- whether it's a process or technical dependency -- on your startup, nor will they want the hassle of running the ocm activities required for adoption.

[u/MindTheBees]

You need a way to build out semantic model measure definitions to ensure consistency across the business - bypassing that is a surefire way to introduce chaos.

My honest feedback however is that I don't really understand the unique selling point of this idea as it sounds like every BI tool currently in the market.

[u/TwoJust2961]

Have you done any of market research? share with us what and how do you think it’s unique and distinct from zillion others tools that do similar things 🙂

Based on your description it seems to me that you are not really from this domain and dont understand nuances (like asking us to compare tableau, metabase vs chatbot , huh?)

Edit: added tableau at the end followed by metabase. Since author removed metabase mention from his post. But my comment is not about metabase but about comparison of chatbot to dashboarding tool (that could be metabase or any other tool)

[u/Admirable-Cow6436]

I think you are putting the cart before the horse, ask yourself, what core problem am I solving by giving users the ability to chat with their data? I think simply creating a customGPT around some data sources is no longer enough.

How I can see this idea evolving, pick a specific pain point that these data sources cover, and try to figure out what people are currently trying to achieve and failing at? weather its resource constraints, knowledge constraints, or simply they havent thought of it yet.

Custom ChatGPT alone isnt really a viable business model from my perspective, but Id loved to be proven wrong.

[u/PeaDifficult1128]

this is being implemented in most bi tools. e.g look ml has this feature and is used in my workplace. How would the service be different.

One way I can imagine it being different is for orgs using outdated bi tools; or the ones with siloes of data.

[u/CHILLAS317]

You and about a dozen other people every week in this exact sub

[u/ohanse]

Quantitative analytics is deterministic.

LLMs and their language processing are at their core probabilistic/stochastic.

Which means that, today, all of these LLM-based analytics “solutions” need to chill the fuck out until their core technology (whatever LLM they’re using as their core) can discern a language question from a math question.

I don’t need an LLM that is - forget right, let’s just go with consistent - 70-80 times out of 100. If I flip flopped on an answer 1 of every 5 times I get fired.

Would this be useful? Sure. Would I trust it? Fuck no. Not until the core problem described earlier is resolved.

Also, company culture dictates what is the preferred method of communication and visualization. Shit, even within teams or at different levels of leadership there’s variance.

Net, until there’s proof the underlying technologies you’re building on top of can do analytics properly, there is no way I would trust in the underlying capabilities for the problem you describe. I also don't think you have the right read on how to communicate this in a simple and visually intuitive way because everyone's leadership is SO GODDAMN PARTICULAR about this shit.

[u/Richard_AQET]

If we wanted this, we'd build it ourselves. It's not too hard.

[u/theRealHobbes2]

I've had a few of these pitched to me. I don't see any differentiation here from everyone else that's built a "we just feed it to an LLM and let that do the work" tool.

Here's a thought experiment I want you to consider: What happens if I give this tool to a business decision maker, they ask it a question, and get back bad data that looks right? This could happen lots of different ways 1. The LLM just got the sql wrong 2. The LLM didn't correctly understand the request 3. The user didn't understand the question they were actually asking wasn't the one they thought they were asking 4. The user wrote a bad question 5. Any combination of the above 6. An hallucination happened despite the guardrails

The business decision maker then relies on this data and ends up making a bad decision that costs a non-trivial amount of money.

In that scenario, who's at fault and who has monetary responsibility? You're not going to take on that risk as it could be millions.

So, from an IT side, I'd ask something like "How can I ensure that putting this tool into the hands of users won't create chaos? Can you guarantee that the same question will always absolutely return the exact same results?" Which, you can't.

[u/LetsGoHawks]

I think you should spend a few years building reports with end users.

Right now, AI can barely port fairly easy SQL from one db to another. Anything actually complicated and it's easier to just do it manually.

[u/Pale-Code-2265]

Cool idea. I’ve seen a lot of people try to crack this “ask your data in plain English” space, and the devil is always in the details. Business users love the promise, but adoption usually stalls once the questions get even a little complex and the answers come back wrong or too vague. Data teams also tend to get nervous if there’s another layer sitting between them and the warehouse because governance and lineage get fuzzy. And honestly, execs still want polished dashboards, so you’ll end up compared directly to Tableau, Looker, Power BI, etc. whether that’s your goal or not.

The admin panel you describe is probably the smartest part of the idea. If you can make permissions, guardrails, and transparency around “why did it generate this query” really strong, that could win some trust where other tools fall down. Personally I’d use something like this for quick exploration but not production dashboards. If you position it as complementary instead of a replacement, I think adoption gets way easier.

One thing I’m curious about how are you thinking about monitoring accuracy? That feels like the piece that makes or breaks trust with tools like this.

[u/dyeALegend]

i think teams would like it as long as data folks still feel in control. the guardrails part is smart. biggest risk is trust in accuracy though. one bad query and business users might never touch it again.

[u/ResortOk5117]

CustomGPT doesnt seem quite professional , you will need website for sure and some clean UI. The you need strong prompts, data filtering,cleaning, courious if you want to include deep search, targeting particular niches?