Hi all,

I have been tasked with the gargantuan task of understanding and eventually implementing what is effectively turning our database into an all-knowing human.

What they want at the base level is to be able to open up a chat bot or similar and ask "where can I put an ice cream shop in <x region of our portfolio>?" And the result should be able to reason against things like demographics in the area, how many competing ice cream shops are in the area, etc.

They also want it to be able to read into trends in things like rents, business types, etc., among many other "we have the data, we just don't know how to use it" questions.

You may be sitting there saying "hire a data analyst" and I agree with you but the ai bug has bitten c-level and they are convinced our competition has advanced systems that can give this insight into their data with a snap of a finger.

I don't know if this is true but regardless, here I am knee deep in the shit trying to find some kind of solution. My boss thinks we can throw everything into a datalake and connect it to chatgpt and it will just work, but I have my reservations.

We have one large database that is "relational" (it has keys that other tables reference but they rarely have proper foreign keys, this is a corporate accounting software specifically for commercial real estate and was not our design and is 30 years old at this point) and we have a couple of smaller databases for things like brokerage and some other unrelated things.

I'm currently of the opinion that a datalake won't do much for us. Maybe I'm wrong but I think cultivating several views that combine our various tables in a sensible way with sensible naming will help to give AI a somewhat decent chance at being successful.

My first entry point was onelake + powerbi + copilot, but that isn't what they're looking for and it's ridiculously expensive. I then looked at powerbi "q&a" which was closer but still not there. You can do charts and sums and totals etc but you can't ask it introspective questions, it just falls on its face. I don't think it was designed for the type of things my company wants.

I have since pivoted to retrieval augmented generation (rag-ai) with azure openai and I feel like I'm on the right path but I can't get it to work. I'm falling face first through azure and the tutorials that exist are out of date even though they're 3 months old. It's really frustrating to try to navigate azure and fabric and foundry with no prior understanding. Every time I try something I have to create 6 resource group items, permissions left right and center, blob stores, etc, and in the end it just...doesn't work.

I think I'm headed in the right direction. I think I need to make some well formatted views/data warehouses, then transform those into vector matrices which azure's openai foundry can take and reason against in addition to the normal LLM that 4o or o1 mini uses

I tried to do a proof of concept with an exported set of data that I had in a big excel sheet but uploading files as part of your dataset is painful as they get truncated and even if they don't, the vectorizing doesn't seem to work if it's not a PDF or image etc.

I need to understand whether I'm in the right universe and I need to figure out how to get this implemented without spending 10 grand a month on powerbi and datalakes that don't even work the way they want.

Anyone got any advice/condolences for me? I've been beating my head against this for days and I'm just overwhelmed by all the buzz words and over promises and terrible "demos" of someone making a pie chart out of 15 records out of the contoso database and calling it revolutionary introspective conversational AI

I'm just tired 😩

Hey friends,

I struggled hard to find a simple to-do list that would work the way I needed it to. Even the paid options weren't quite helpful for me. Eventually I just gave up and made my own.

I use it at work and at home every single day. I also made a backup copy to handle an individual project I'm working on.

Anyway, it's free. I just wanted to share it with you guys.

Small side notes

• To start using this, just copy it. It'll open in a read-only mode but you can quite easily pull it over to your own Google Drive.

• I threw in a donation link on the Sheet. Delete it or ignore it. It's 100% optional and I figured I'd put it there in case this ended up becoming someone's daily driver and they want to say thanks.

• You can somewhat easily make this sheet work with Google appsheet to use it on mobile as an app. That's what I do for mine so that I can use it on my phone and on desktop throughout the day. Just open appsheet from this sheet and start customizing your app. As long as you're using it for just yourself, you don't have to deploy it or anything. You just download the appsheet app on mobile and access it that way. You can even name it and choose a custom icon. The one caveat is that you'll have to create a time trigger in scripts to make it auto sort. If anyone is dying to do this, I'll post more specific details to help you through it.

https://docs.google.com/spreadsheets/d/1sIp4uPq3xwgudNA9tIOGmNXzETtyG421WbmvJPWgbsI/edit?usp=drivesdk