I don't have good advice, unfortunately. You should quit as soon as you can line something else up. This project is an albatross around the neck of anyone working on it. More generally, it boggles...
Exemplary
I don't have good advice, unfortunately. You should quit as soon as you can line something else up. This project is an albatross around the neck of anyone working on it.
More generally, it boggles my mind how eager people (especially executives) are to do things without any understanding of what they're doing. Like, it's been obvious for decades that there's a deep strain of intellectual laziness that flows through American white-collar culture; but even my cynicism has been surprised by the degree to which people (again, especially executives for some reason) thought "hey, I might never have to think again", and promptly hurled themselves mind body and soul into the abyss.
Anyway, the real solution to the problem you've been told to solve by killing albatrosses is to hire a data analyst. So you've got it on the authority of some rando on the Internet that you're not crazy, lol. =) There's a truism in data analysis that you can't analyze data you don't understand, and that holds true for large neural network models as much as it does for more trivial stuff like regressions and averages.
This is beyond the scope of any individual. AI will not save you. This is the type of concept that entire companies are built to provide. There is no exploration into feasibility, it's simply not...
This is beyond the scope of any individual. AI will not save you. This is the type of concept that entire companies are built to provide. There is no exploration into feasibility, it's simply not possible. You can hire a five head data team and you'll be able to start doing basic questions in a few years.
Sounds management had a big sip of the AI kool-aid and isn't hindered by any practical knowledge. This blog post written by an actual data scientist is still very much relevant. So what you are...
We have one large database that is "relational" (it has keys that other tables reference but they rarely have proper foreign keys, this is a corporate accounting software specifically for commercial real estate and was not our design and is 30 years old at this point) and we have a couple of smaller databases for things like brokerage and some other unrelated things.
So what you are saying is that your data is fragmented, loosely (not) structured and also has been accumulating slowly over 30 years without anyway bringing structure to it? Data hoarding is a thing, I've seen companies keep records going back decades. This might be interesting to historians, but most of that data has no real relevance to the market or company in the present day.
Finally, what a lot of people might not realize is that the LLMs currently out there also are not trained by just throwing all the data in and hoping for the best. They, too, have been incredibly labor-intensive as the input needed to be vetted by humans. Because so many people are needed, a lot of it was done by cheap workers in African countries, which according to that article resulted in some interesting side effects on its output.
But I digress slightly. Of course you are not training a model from scratch, likely not even tuning a model. You already stumbled upon the use of RAGs which likely is part of the puzzle if what management requests is even possible (which I highly doubt). However also with the RAG approach it is still garbage in, garbage out.
If the data doesn't "make sense" so to speak for a human a LLM isn't magically bring structure to it.
We're a small dev team (3 people plus my boss, who is IT director)
If you are going to make this work you will be needing more people than this. Not even necessarily developers, but people to go over the base data.
Finally
Be wary of proof of concepts when working with LLMs, they are very deceptive.
One thing I have noticed with LLM products is that proof of concepts look much more promising than traditional software proof of concepts. Do not let that fool you, the devil is in the details and more importantly the ability to consistently reproduce results within a certain margin. Traditional software will throw blank results, errors, etc when you don't get results. LLMs will still output something, something that might be entirely wrong. Something you might not notice in a proof of concept phase but will effectively ruin your days when trying to actually turn it into a reliable production ready product.
Well, I mainly have condolences. Our senior staff get some random hair up there arse once or twice a year about tech jargon that they think will change the world, and I'm typically the person that...
Well, I mainly have condolences. Our senior staff get some random hair up there arse once or twice a year about tech jargon that they think will change the world, and I'm typically the person that bursts their bubble. So I get it, this is a shitty spot to be in.
But I've gotta ask, what is your actual job at this place? I'm an IT director and at first glance the warning sirens are going berserk in my brain. Considering their initial request (as you worded it) I don't think anything you can realistically deliver will hit the mark, especially not at version 1. Not to say it can't be done, but this sure sounds like it seriously exceeds what can be cobbled together without a more formal plan and budget. I'd say you need a small scale demo or proof of concept, then hit them with the reality that this shit does cost money, and if you really want this level of business intelligence you need to be willing to invest the man hours and dollars into it. There is no free lunch.
Hi Jambo, this is what I do. I work at a large, well known company, and we are struggling to build this. I track other players in this space closely. Everyone is struggling to build this. I've...
Hi Jambo, this is what I do.
I work at a large, well known company, and we are struggling to build this.
I track other players in this space closely. Everyone is struggling to build this.
I've given hour long talks at various conferences saying "don't do this, start simple."
There are two unstoppable forces here. The first is the technology and data isn't there yet.
The second is the c-suite doesn't care, because vendors such as myself are demoing total bs.
Gartner has a hype cycle, which talks about how this isn't the first time c-suite have bought into total bs.
But the c-suite aren't listening to Gartner. They aren't listening to you. They aren't listening to me.
They are listening to Salesforce who says "just buy Data Cloud & Agentforce. See? Shiny!"
They see ChatGPT. Which is magic for kids. And they want magic for the c-suite.
So. Get the various vendors to come in and do a demo with your data.
You will need someone with budgetary authority to sign off.
But vendors are hungry to sell you their solution.
Get them to prove to your c-suite it works.
Now. It wont work. But it will look like it works. You will want to ask their system a question not in the happy path demo script. Vendors will say this isn't an implementation, so you can't ask any questions they weren't prepared to demo. Your c-suite will completely buy whatever they are selling however. And then the vendors will quote a price. And your c-suite won't be happy with the price. But one thing the vendors should be able to hammer home, is that you should not DIY your AI. And they are all right.
Why should you not do this yourself?
Currently Generative AI is unable to navigate custom structured data. It can barely navigate well known data structures. The only way it knows how to navigate well published data structures, is by creating a well known query, and tailoring that query based on what the c-suite types into the chat bot. This will get better. But it isn't better yet. We are a year or three away from taking any data, throwing a chatbot on top, and getting any sort of sensible insights . You can take unstructured data in a very well known format, like a 10K, and upload the PDF and get some amazing insights. But you can't give structured data and expect an intelligent response.
Your data is a mess. Seriously. Even if you bung all your data into a highly relational data model, your data will still be a mess. You will still have duplicates everywhere. It's just some of the duplicates will be now related to some of the other duplicates. Even the worlds greatest AI will struggle with your incoherent mass that you call structured data. Don't talk about zero data copy. Nope. Don't even think about it. Now you have zero copies of messy, duplicative data from multiple duplicative source systems. Half your data is sitting in spreadsheets. Even Gen AI needs good data in, before it gives you good data out.
So what should you do?
Ideally, you need to pick the one real business problem. The one that will drive real business results. See if you can have vendors show you how they would solve that problem. Get them to show you how the system constructs reasonable queries. Get them to show you how the system detects bad data. Get them to show you how the system handles no data. Let them pretend they can solve world hunger. Let them lobby to your tight fisted c-suite to spend more money on this problem. But your focus should be to solve one real world problem with Gen AI. If you can do that, and the problem is beyond summarizing text or generating text or extracting text from text, then you are ahead of the world my friend.
Or. You can treat this as an R&D project. But you are fighting against tech that is rapidly changing, and by the time you have figured out Langchain, everyone will have switched from that to Autogen, to something else from Deepseek, and to something else even betterer.
Edit: Oh. I forgot the two most obvious answers. While your c-suite is chasing the latest chatbot tech, don't forget. We have other tools.
The worlds simplest heuristic will solve 80% of your problem. I am talking about a simple ratio like Price/Rent. Throw some generative AI explanation on top of it and it looks like magic. This is what they want. Magic. Give it to them.
Machine picked statistics is just simple math, but it can look like magic. Throw some Gen AI explanation of the trend line and you have something so close to magic that no one will care if it is just simple math. Throw a chatbot on top of that to ask additional questions, and no one will care if it hallucinates the answer. Lets be real. They aren't data driven. They are simply looking for data to support their pre-existing narrative.
ChatGPT is fantastic at solving my kids homework. What CEOs really want is the corporate version of that. Trouble is, all that homework was published on the internet. No one is sharing corporate...
ChatGPT is fantastic at solving my kids homework.
What CEOs really want is the corporate version of that.
Trouble is, all that homework was published on the internet.
No one is sharing corporate data and corporate strategy on the internet.
I think this is pretty legit. I mean I always kick vendors to the curb, but I guess this is a great opportunity to let management and vendors hang themselves on the same tree.
I think this is pretty legit. I mean I always kick vendors to the curb, but I guess this is a great opportunity to let management and vendors hang themselves on the same tree.
Congratulations on your new research project! If you build this thing, you will be a top-tier AI researcher contributing meaningfully to advancing the state of the art in the field! Now go make...
Congratulations on your new research project! If you build this thing, you will be a top-tier AI researcher contributing meaningfully to advancing the state of the art in the field!
Now go make sure your bosses understand that.
By combining things like LangChain and its tool use, DeepSeek and its chain of thought instincts, and RAG as you mentioned, and maybe giving it a page of text that explains how to do business strategy, or a pass of training on explanations of good and bad ideas in your problem domain along, with the database queries that fetch the evidence to support them, you might be able to make something.
It's not clear that this is a better investment of time than if you just built the data analysis tools that would help a human answer these questions themselves. Especially since you mostly have to build those tools anyway to feed the AI. A model can just write SQL queries, but the easier the tools are to use the smarter the system is going to be.
The sampler/framing code/driver logic is going to be where all the real work is, and it's going to be a lot of work. Have your boss read up on Williams syndrome and Broca's aphasia, because really with a language model you have a linguistics research project. Everyone involved needs a good understanding of how language is a special-purpose function of the human mind only loosely connected to the other parts of what one might call intelligence, and how a project like this amounts to "build the rest of the brain".
I did exactly this sort of thing for about a decade at my previous company and I concur. This sort of database almost certainly wasn't meant for this and will likely need someone (or a group of...
I did exactly this sort of thing for about a decade at my previous company and I concur. This sort of database almost certainly wasn't meant for this and will likely need someone (or a group of someones) that thoroughly know this product to get this data into a more suitable reporting or BI system.
It sounds like they want an oracle of Delphi. You should do what you can to find a different job or find an ally and kill this project. A good first step would be some structured data libraries...
It sounds like they want an oracle of Delphi. You should do what you can to find a different job or find an ally and kill this project.
A good first step would be some structured data libraries and dashboarding tools like powerbi.
This is what non tech powerful people do when something shiny comes along. You need a project manager. If they won't give you one or don't have one, you'll need to take it on yourself. Don't...
This is what non tech powerful people do when something shiny comes along.
You need a project manager. If they won't give you one or don't have one, you'll need to take it on yourself. Don't forget to write this up on your resume, it's a marketable skill.
I suggest setting up a meeting of stakeholders and force some priority on them. Write up the requested features on index cards before hand. Break them down into actual features with measurable goals based on business needs. "Areas where people between the ages of x and y live," "Areas with 3 existing ice cream shops at least x blocks away," "Available properties next to a gym," etc. Include a business goal with each one if possible. You've got to manage scope by yourself. Seriously, write out all the individual requirements like that so there's something they can touch.
Lay the cards out on a table, holding back some blank ones because they'll have a bunch of new ideas. Don't offer those up unless you have to.
These are the features you'd like as I understand them. I can only work on one at a time, so I need these put into order with the highest priority at the top. My expected outcome for this session is a stack of prioritized features.
If/when they add new ideas help them manage the scope to an actual single feature, even if that means wiring up several new cards. Let them go wild, as long as you keep them focused on the outcome of a single stack of cards.
It's ok for them to negotiate priority. Whoever is in charge gets to make the final call.
Whatever you end up with, put the top 3-5 in your project management software, don't spend a bunch of time adding all of them.
I strongly suggest asking for the help of a designer if your company has one. If not, find some kind of map standard for whatever kind of markers and selectors you'll need.
After you complete the first feature, do this again to make sure you're building the right thing. Different things are going to float to the top once they have an actual tool to play with. They will not want to part with things at the bottom of the pile, but you don't really need to worry about those unless they float to the top.
If you're lucky, the big stack may be a catalyst to getting some help.
A way that I have found effective to chop scope for AI projects targeting LLMs (currently working in a c-suite) is telling people to replace AI in their request with "plausible sentence generator"...
A way that I have found effective to chop scope for AI projects targeting LLMs (currently working in a c-suite) is telling people to replace AI in their request with "plausible sentence generator" and if the request still makes sense then I'll look at it.
The fact is this company obviously already has institutional knowledge on how to make real estate investments (for example) and they should first seek to mechanise that model which would then serve as the actual basis for an AI agent, or at least a good thing to benchmark your AI against. The project should be that specific: they need to pick a beachhead from which to launch their revolution otherwise you have exactly zero chance of success (rather than slim if you go with trying to create a slick agent interface to well-vetted mechanised knowledge).
This page from AWS might help curb your leaders enthusiasm:
You can think of the Large Language Model as an over-enthusiastic new employee who refuses to stay informed with current events but will always answer every question with absolute confidence.
You need serious work to get past that if you are going to use this to drive internal decision making. Personally I don't think this is fixable. Something something computers cannot be held accountable so computers should not make decisions.
Folks have touched on it but I wanted to chime in, having built something like what you are talking about. LLMs allow you to extract more value out of a mature data environment. They don't allow...
Folks have touched on it but I wanted to chime in, having built something like what you are talking about. LLMs allow you to extract more value out of a mature data environment. They don't allow you to side step or short cut your data maturity journey. Your organization needs to go through the data governance maturation process of formalizing systems of record, data integrations, and standardized definitions and sources in order to get value from an LLM in the way your company is asking. Without a solid foundation, you will chase endless problems with your implementation, including not being confident that the decisions are based on the latest authoritative data.
If your company balks at $10k/month, they should reconsider.
That said, I do have a technology I can DM you if you want to pursue things. I don't want to post it publicly, because it is part of a public/private tech transfer that I am associated with. But it would do what your company is asking, if you get your data universe in order.
Your situation is hitting pretty close to home as my current c-level has also contracted a severe case of AI obsession. I don't have much advice other than to reiterate what some others have said,...
Your situation is hitting pretty close to home as my current c-level has also contracted a severe case of AI obsession. I don't have much advice other than to reiterate what some others have said, you'll likely need more people for this, especially with the task of transforming the billing database into vetted useful data that an LLM can use.
I don't personally have a very high opinion of the current state of AI/LLM related tech, but I don't see this hype wave ending soon and if your c-suite is like mine, they aren't going to let it go. Good luck!
I don't have good advice, unfortunately. You should quit as soon as you can line something else up. This project is an albatross around the neck of anyone working on it.
More generally, it boggles my mind how eager people (especially executives) are to do things without any understanding of what they're doing. Like, it's been obvious for decades that there's a deep strain of intellectual laziness that flows through American white-collar culture; but even my cynicism has been surprised by the degree to which people (again, especially executives for some reason) thought "hey, I might never have to think again", and promptly hurled themselves mind body and soul into the abyss.
Anyway, the real solution to the problem you've been told to solve by killing albatrosses is to hire a data analyst. So you've got it on the authority of some rando on the Internet that you're not crazy, lol. =) There's a truism in data analysis that you can't analyze data you don't understand, and that holds true for large neural network models as much as it does for more trivial stuff like regressions and averages.
Its a condition of employment for these people I swear to God.
This is beyond the scope of any individual. AI will not save you. This is the type of concept that entire companies are built to provide. There is no exploration into feasibility, it's simply not possible. You can hire a five head data team and you'll be able to start doing basic questions in a few years.
Sounds management had a big sip of the AI kool-aid and isn't hindered by any practical knowledge. This blog post written by an actual data scientist is still very much relevant.
So what you are saying is that your data is fragmented, loosely (not) structured and also has been accumulating slowly over 30 years without anyway bringing structure to it? Data hoarding is a thing, I've seen companies keep records going back decades. This might be interesting to historians, but most of that data has no real relevance to the market or company in the present day.
Finally, what a lot of people might not realize is that the LLMs currently out there also are not trained by just throwing all the data in and hoping for the best. They, too, have been incredibly labor-intensive as the input needed to be vetted by humans. Because so many people are needed, a lot of it was done by cheap workers in African countries, which according to that article resulted in some interesting side effects on its output.
But I digress slightly. Of course you are not training a model from scratch, likely not even tuning a model. You already stumbled upon the use of RAGs which likely is part of the puzzle if what management requests is even possible (which I highly doubt). However also with the RAG approach it is still garbage in, garbage out.
If the data doesn't "make sense" so to speak for a human a LLM isn't magically bring structure to it.
If you are going to make this work you will be needing more people than this. Not even necessarily developers, but people to go over the base data.
Finally
Be wary of proof of concepts when working with LLMs, they are very deceptive.
One thing I have noticed with LLM products is that proof of concepts look much more promising than traditional software proof of concepts. Do not let that fool you, the devil is in the details and more importantly the ability to consistently reproduce results within a certain margin. Traditional software will throw blank results, errors, etc when you don't get results. LLMs will still output something, something that might be entirely wrong. Something you might not notice in a proof of concept phase but will effectively ruin your days when trying to actually turn it into a reliable production ready product.
Loved reading the article written by the data scientist.
It's such a realistic view on companies and upper management.
Well, I mainly have condolences. Our senior staff get some random hair up there arse once or twice a year about tech jargon that they think will change the world, and I'm typically the person that bursts their bubble. So I get it, this is a shitty spot to be in.
But I've gotta ask, what is your actual job at this place? I'm an IT director and at first glance the warning sirens are going berserk in my brain. Considering their initial request (as you worded it) I don't think anything you can realistically deliver will hit the mark, especially not at version 1. Not to say it can't be done, but this sure sounds like it seriously exceeds what can be cobbled together without a more formal plan and budget. I'd say you need a small scale demo or proof of concept, then hit them with the reality that this shit does cost money, and if you really want this level of business intelligence you need to be willing to invest the man hours and dollars into it. There is no free lunch.
Hi Jambo, this is what I do.
I work at a large, well known company, and we are struggling to build this.
I track other players in this space closely. Everyone is struggling to build this.
I've given hour long talks at various conferences saying "don't do this, start simple."
There are two unstoppable forces here. The first is the technology and data isn't there yet.
The second is the c-suite doesn't care, because vendors such as myself are demoing total bs.
Gartner has a hype cycle, which talks about how this isn't the first time c-suite have bought into total bs.
But the c-suite aren't listening to Gartner. They aren't listening to you. They aren't listening to me.
They are listening to Salesforce who says "just buy Data Cloud & Agentforce. See? Shiny!"
They see ChatGPT. Which is magic for kids. And they want magic for the c-suite.
So. Get the various vendors to come in and do a demo with your data.
You will need someone with budgetary authority to sign off.
But vendors are hungry to sell you their solution.
Get them to prove to your c-suite it works.
Now. It wont work. But it will look like it works. You will want to ask their system a question not in the happy path demo script. Vendors will say this isn't an implementation, so you can't ask any questions they weren't prepared to demo. Your c-suite will completely buy whatever they are selling however. And then the vendors will quote a price. And your c-suite won't be happy with the price. But one thing the vendors should be able to hammer home, is that you should not DIY your AI. And they are all right.
Why should you not do this yourself?
Currently Generative AI is unable to navigate custom structured data. It can barely navigate well known data structures. The only way it knows how to navigate well published data structures, is by creating a well known query, and tailoring that query based on what the c-suite types into the chat bot. This will get better. But it isn't better yet. We are a year or three away from taking any data, throwing a chatbot on top, and getting any sort of sensible insights . You can take unstructured data in a very well known format, like a 10K, and upload the PDF and get some amazing insights. But you can't give structured data and expect an intelligent response.
Your data is a mess. Seriously. Even if you bung all your data into a highly relational data model, your data will still be a mess. You will still have duplicates everywhere. It's just some of the duplicates will be now related to some of the other duplicates. Even the worlds greatest AI will struggle with your incoherent mass that you call structured data. Don't talk about zero data copy. Nope. Don't even think about it. Now you have zero copies of messy, duplicative data from multiple duplicative source systems. Half your data is sitting in spreadsheets. Even Gen AI needs good data in, before it gives you good data out.
So what should you do?
Ideally, you need to pick the one real business problem. The one that will drive real business results. See if you can have vendors show you how they would solve that problem. Get them to show you how the system constructs reasonable queries. Get them to show you how the system detects bad data. Get them to show you how the system handles no data. Let them pretend they can solve world hunger. Let them lobby to your tight fisted c-suite to spend more money on this problem. But your focus should be to solve one real world problem with Gen AI. If you can do that, and the problem is beyond summarizing text or generating text or extracting text from text, then you are ahead of the world my friend.
Or. You can treat this as an R&D project. But you are fighting against tech that is rapidly changing, and by the time you have figured out Langchain, everyone will have switched from that to Autogen, to something else from Deepseek, and to something else even betterer.
Edit: Oh. I forgot the two most obvious answers. While your c-suite is chasing the latest chatbot tech, don't forget. We have other tools.
This comment is full of pragmatic solutions and I may end up borrowing some of this for my own c-suite AI mania.
ChatGPT is fantastic at solving my kids homework.
What CEOs really want is the corporate version of that.
Trouble is, all that homework was published on the internet.
No one is sharing corporate data and corporate strategy on the internet.
I think this is pretty legit. I mean I always kick vendors to the curb, but I guess this is a great opportunity to let management and vendors hang themselves on the same tree.
I love this. I didn’t mention in my top level comment, but I am a director of analytics.
Sell good stats well and they don’t care what you call it.
Congratulations on your new research project! If you build this thing, you will be a top-tier AI researcher contributing meaningfully to advancing the state of the art in the field!
Now go make sure your bosses understand that.
By combining things like LangChain and its tool use, DeepSeek and its chain of thought instincts, and RAG as you mentioned, and maybe giving it a page of text that explains how to do business strategy, or a pass of training on explanations of good and bad ideas in your problem domain along, with the database queries that fetch the evidence to support them, you might be able to make something.
It's not clear that this is a better investment of time than if you just built the data analysis tools that would help a human answer these questions themselves. Especially since you mostly have to build those tools anyway to feed the AI. A model can just write SQL queries, but the easier the tools are to use the smarter the system is going to be.
The sampler/framing code/driver logic is going to be where all the real work is, and it's going to be a lot of work. Have your boss read up on Williams syndrome and Broca's aphasia, because really with a language model you have a linguistics research project. Everyone involved needs a good understanding of how language is a special-purpose function of the human mind only loosely connected to the other parts of what one might call intelligence, and how a project like this amounts to "build the rest of the brain".
I did exactly this sort of thing for about a decade at my previous company and I concur. This sort of database almost certainly wasn't meant for this and will likely need someone (or a group of someones) that thoroughly know this product to get this data into a more suitable reporting or BI system.
It sounds like they want an oracle of Delphi. You should do what you can to find a different job or find an ally and kill this project.
A good first step would be some structured data libraries and dashboarding tools like powerbi.
This is what non tech powerful people do when something shiny comes along.
You need a project manager. If they won't give you one or don't have one, you'll need to take it on yourself. Don't forget to write this up on your resume, it's a marketable skill.
I suggest setting up a meeting of stakeholders and force some priority on them. Write up the requested features on index cards before hand. Break them down into actual features with measurable goals based on business needs. "Areas where people between the ages of x and y live," "Areas with 3 existing ice cream shops at least x blocks away," "Available properties next to a gym," etc. Include a business goal with each one if possible. You've got to manage scope by yourself. Seriously, write out all the individual requirements like that so there's something they can touch.
Lay the cards out on a table, holding back some blank ones because they'll have a bunch of new ideas. Don't offer those up unless you have to.
If/when they add new ideas help them manage the scope to an actual single feature, even if that means wiring up several new cards. Let them go wild, as long as you keep them focused on the outcome of a single stack of cards.
It's ok for them to negotiate priority. Whoever is in charge gets to make the final call.
Whatever you end up with, put the top 3-5 in your project management software, don't spend a bunch of time adding all of them.
I strongly suggest asking for the help of a designer if your company has one. If not, find some kind of map standard for whatever kind of markers and selectors you'll need.
After you complete the first feature, do this again to make sure you're building the right thing. Different things are going to float to the top once they have an actual tool to play with. They will not want to part with things at the bottom of the pile, but you don't really need to worry about those unless they float to the top.
If you're lucky, the big stack may be a catalyst to getting some help.
A way that I have found effective to chop scope for AI projects targeting LLMs (currently working in a c-suite) is telling people to replace AI in their request with "plausible sentence generator" and if the request still makes sense then I'll look at it.
The fact is this company obviously already has institutional knowledge on how to make real estate investments (for example) and they should first seek to mechanise that model which would then serve as the actual basis for an AI agent, or at least a good thing to benchmark your AI against. The project should be that specific: they need to pick a beachhead from which to launch their revolution otherwise you have exactly zero chance of success (rather than slim if you go with trying to create a slick agent interface to well-vetted mechanised knowledge).
This page from AWS might help curb your leaders enthusiasm:
https://aws.amazon.com/what-is/retrieval-augmented-generation/
You need serious work to get past that if you are going to use this to drive internal decision making. Personally I don't think this is fixable. Something something computers cannot be held accountable so computers should not make decisions.
Folks have touched on it but I wanted to chime in, having built something like what you are talking about. LLMs allow you to extract more value out of a mature data environment. They don't allow you to side step or short cut your data maturity journey. Your organization needs to go through the data governance maturation process of formalizing systems of record, data integrations, and standardized definitions and sources in order to get value from an LLM in the way your company is asking. Without a solid foundation, you will chase endless problems with your implementation, including not being confident that the decisions are based on the latest authoritative data.
If your company balks at $10k/month, they should reconsider.
That said, I do have a technology I can DM you if you want to pursue things. I don't want to post it publicly, because it is part of a public/private tech transfer that I am associated with. But it would do what your company is asking, if you get your data universe in order.
Your situation is hitting pretty close to home as my current c-level has also contracted a severe case of AI obsession. I don't have much advice other than to reiterate what some others have said, you'll likely need more people for this, especially with the task of transforming the billing database into vetted useful data that an LLM can use.
I don't personally have a very high opinion of the current state of AI/LLM related tech, but I don't see this hype wave ending soon and if your c-suite is like mine, they aren't going to let it go. Good luck!