How do I convince my workplace we need SQL databases?
I work for a GIS company and our tools have not grown with our projects and client base. We use ArcPro personal geo databases (GDBs) for ALL data. We recently had a project where shit really hit the fan, one major issue was related to invalid values from poor version control. Everything uses personal GDBs and is just "version controlled" by dating filenames in Explorer. It would have been trivial to fix in a proper database. We also have operational constraints, like we can only have one person doing X job at a time since all the data for X job is in a personal GDB.
But I'm just an analyst. I've garnered some attention for my technical expertise beyond processing the data. PostGIS is a thing so it isn't as though we'd be recreating the wheel. How can I push for that sort of change? I'm thinking I can sell it using how much we lost on this project because of these avoidable failures. I'm also wondering if I can make this an opportunity to create a "database administrator" position for myself
I had a somewhat similar ride to create my career.
My best advice is show first. Mock up something on a free tier and demonstrate if you can. The question becomes how well your decision makers understand the benefits. Obviously if they really got it you wouldn’t need to tell them, but if you want to convince someone start with examples
Agreed - I also created my job/department by identifying a tech-related need and building on that.
Build a prototype, and be able to articulate the potential business value for what you’ve built. The steps after that sort of depend on the formality, structure, and size of your company, but typically from there you’d do some sort of pilot or limited launch. That’s when you’d measure the actual impact, get feedback from users, etc.
Using the information from your pilot (e.g. “we saved $X/delivered Y% faster/had Z% fewer errors), project what the impact at scale would be and use that to make the case for a company-wide rollout.
Whatever you do, make sure you understand the challenges of the business from the point of view of whoever is making the decisions, and frame your proposal in those terms. If the business’ #1 problem is costs, show how this can save on labor time. If the problem is that you have more demand than you can manage, show how this will tie directly to more revenue generation by allowing your teams to work through projects faster and support more customers.
I would agree with this. They need to see the benefits.
At the end of the day it comes down to money. How much money will it save to setup the server and maintain it? How much will it cost to migrate everyone over? How will this effect everyone's work flows?
I have experience working with a customer who used a SQLite DB on a shared drive as a multi-user DB for an app. So i totally understand the benefits of moving over. But companies with older tech will wait as long as possible to update. So they need a very good motivator.
In my company, the €€€ speak louder than anything else.
Whenever I need to argue for something, the battle gets easier if I justify my position with how much we will lose/gain if we do/don’t do X.
When you make your case, don’t start with how problematic or bad practice or bad idea or etc the current process is. In my experience, managers/leads/bosses couldn’t care less as long as things work. They don’t care if your life is easier or harder with doing things a certain way, as long as the thing gets done anyway.
But if you dangle €€€ in front of them, they’ll pay more attention.
In your case, I would attack with two points:
1- You already have an incident that you can use, so try to list and explain the damage that was done, how many people and hours it took to fix, and what worst case scenario could've been.
Next, do napkin maths. You likely don't have access to your colleagues salaries, so grab an average/estimate and calculate how much the hours wasted per person cost the company while they fixed the problem. This is hours wasted by people that could've been doing something else and bringing money to the company.
2- Argue how your proposed solution brings benefits in normal day to day operations. How much it would cost, how much time it would save you and your colleagues, and translate that into €€€ per month/year. Show how the investment pays itself back within X months.
Make the business case clear, fixing this costs X upfront but saves Y annually. If Y > X within a reasonable timeframe, you'll get their attention.
Not sure how feasible this is for your scale, technical context, and departmental culture, but:
You could try double-writing. Whatever goes into your existing "bad" tech stack also gets written to your proposed "good" tech stack. Doesn't have to be super efficient, or perfectly in parallel. It could be a delayed write, even in a batch at midnight, or whatever. Then you at least have a source that you can read from and perform SQL queries against, etc. It would let you compare both tech stacks, hopefully with a situation, project, or task that gives you some meaty numbers and facts to pitch your proposal. e.g. "In the emergency incident that happened in December, it took the team X hours to come up with the needed data, but I got it from the parallel SQL database in (X - n) hours."
The new DB doesn't even necessarily have to be a perfect 1:1 copy of the data. If it can just ingest roughly the same volume/scale of data, you might still be able to make your point.
I have no ability to convince, but I'm in the same boat in an entirely different field. I have no idea how our clients are handled on the backend, but it isn't a searchable SQL DB, and I am perplexed, confused, and upset by it. I will be reading with interest.
You could try being more familiar with PostGIS first, and then using it on your own alongside the regular work. Documenting hours it takes to do it the old way vs new. Then before next project, ask for permission to use only the new one if it's a solo project, and compare the time savings as well as any other advantages. Make a good presentation for why it'll save money: remember that not having panic catastrophic loss is a money saver too. Say those happen once a year, vs 0, that's X dollars saved. Project Regular labour savings stacked over one year + Catastrophe per year savings.
Edit
And then money is time. Tally up the hours saved, and estimate that if the whole team saves X hours, they could take on Y number of additional projects, which would bring in Z amount of money.
In addition to what others have said, I'd look into ways to make the transition reversible if it doesn't work out. That will lower risk and help to discover any shortcomings that need to be fixed before turning off the old system.
If the issue is really version control then I'm not totally sure postgis would solve your issue. People will just as easily do an UPDATE and muck up the data with postgis as GDB. That sounds more like a solution for kart, which can of course be backed by postgis (or geopackage).
But if you're in an Esri shop, beware the culture shock of suggesting a move to the OSGeo "darkside" (not that kart itself is OSGeo) even if it's much better (it is). The parent company of kart, Koordinate, offers managed solutions for Kart if you think you'd have a better chance pitching the "buy" instead of "build" option.
Thank you kindly for the additional technical input. Version control was just the simplest way to describe the problem but perhaps not the most accurate. One UPDATE here could have solved the issue where we were left with hundreds of disparate GDBs with outdated info, but I do see the risk you're mentioning. But this kart seems like a handy solution so I do appreciate you bringing it to my attention! Especially giving me an option of selling a buy over build.
Oh, so more of a centralization/data sharing issue. I see why your head is at database then. But keep in mind, a lack of coordination + centralization will quickly turn your problem into a version control problem. :)
You might want to do a little googling about data architectures at a generic level. Some of your issues might be something you can alleviate with a super light "data lake" approach, which really is just a fancy networked file system with tooling. And if you do go down the database approach, I could see you easily running into traditional data warehouse headaches depending on your company's scale. Geospatial data complicates and limits your options unfortunately as a lot of tools that claim geospatial support have huge limitations like missing raster support or forcing things into WGS-84. But a lot of the high level concepts and approaches could be useful depending on your specifics.
In addition to everyone else's comments, I'd suggest identifying transitional pain points and showing how they might not be all that bad. I'm guessing some of the pushback you'll get will be from people concerned about hassle / learning curve, even if they don't explicitly say it. I'd try to address those concerns when presenting your demo.
Also, I'm not sure if it's helpful, but I'm currently working for an OSGeo-adjacent company, and I can share a bit of our tech stack:
I'm on the big data / cloud-native side, so I'm used to a heavier stack, but I strongly recommend checking out DuckDB, at least.
Thank you kindly for the additional technical recommendations, having some choices will definitely help!
But yeah finding those points and making them easier transitions would help sway more production oriented folks too
Thank you kindly everyone for the input :) y'all have given me some solid ground to push forward here and a couple different perspectives I should present this from.