6
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I've been working on a pure-rust no-unsafe embedded on-disk key value store. I know, this sounds like a lot of buzzwords :) . For the user, the features would be the same as LevelDB or LMDB. But underneath, I would like use B-Espilon trees. All of this written only in rust, without any unsafe block, only using std. The goal is to offer similar read performance as LMDB, with batch writes as fast as RocksDB.
I have some future personal use for this project, but to be fair, redb, sled or even the rust bindings to LMDB would work just fine for these future uses.
This is a very selfish project where I would like to learn better how data is physically stored for database and file system. I would also like to show off what is possible with no-unsafe rust (and just whatever unsafe is wrapped by the Rust Standard Library)
I'm leaning on the shoulder of giants, such as KingDB's series of blog posts which explains how to implement a key value store from scratch, and I'm also studying the code of Tkrzw which is under Apache 2.0. I chose this because Tokyo Cabinet (the predecessor of Tkrzw) was revolutionary, and I would like to publish my future project under a non-restrictive license.
The project might never see the light of day, I hope it will, but my main goal is to learn along the way :)
I am working on a script to split an audiobook into audio segments corresponding to the sentence in the ebook version of the book. The eventual goal with this is for people to be able to a simply point the script at an mp3 of an audiobook and get a nicely formatted LJspeech-style dataset that can be used for future machine learning projects. My current goal is to finetine a text-to-speech model for PiperTTS using the output of the script.
The current script uses Whisper.cpp to transcribe the audio and then does a fuzzy search using Python's difflib library to try and match the transcription its corresponding sentence in the ebook replacing the transcription with the actual line from the book if it finds it.
This current solution only kind of works and I am open to ideas on how to do things better.
If I understand this correctly, you want to do this because it will give you audio/text paired data with an easily accessible ground truth taken from the text version of the book, to avoid needing manual verification of the transcript?
This is pretty closely related to a hobby project of mine where I am transcribing and correcting transcripts of video instructionals, except I don’t have a known good transcript. I’m making a lot of use of difflib and LLMs to assist in the corrections. What kind of problems are you having?
The biggest problem is that my transcription does not place punctuation in the same place as the eBook so when I go to match the transcription to the eBook, the pieces do not line up because the eBook sentence I am comparing to may overlap multiple audio segments or not cover the entire audio segment.
I am considering a resource intensive option where it tries adding one word at a time from the eBook until it either has a high enough matching percentage or the sentence its created is longer than the Audiobook transcription but this would be very resource intensive and I would hope there is a more clever method than simply bruteforcing it.
I think I would do some kind of sliding window method using edit distance to decide when it's appropriate to apply a patch. If you've got a sample of a transcript + known good text, I'd totally take a stab at trying to align the text.
That's cool! Keep me posted on how it goes.
I'm actually working on the reverse lol (epub to audiobook, except each character has a unique and configurable voice)
Will do. For the epub to audiobook project, what TTS are you using, something local, or ElevenLabs?
I'm thinking of checking out myshell.ai's model because I remember the demo being very impressive.
But if that's not good enough yeah I'm gonna have to shell out for eleven labs.
I've got my functions up in azure so now it's figuring out how the hell to interact with them in the ways i want to and where they store some of their data (i think they should be writing logs to a file, but they sure don't seem to be, but they absolutely are running).
Currently working on a project off of Frontend Mentor, I've never worked with Figma files like this, so it's been interesting having all the designs like this laid out, something I might mess with trying on my next personal project.
I'm not sure where to begin honestly, but for now I'm just making individual components for things like buttons, inputs, etc, based in the designs and then I'll begin building other parts I guess. 🤷
Still working on my "repeatTest" property-testing framework. I did a lot more refactoring, implemented some better ways of generating strings, and finally started on shrinking today.
It’s a very short & sweet project, but just the other day I finished completely rewriting
trl, my concise command-line translation script.It just wraps the DeepL API, which I found out to my surprise last year sports a very generous free tier, and using it feels way faster than opening a browser and having to navigate to their website and selecting the language(s) I want. And I can pipe into/out from it, redirect in- and output from/to files, use it in other software or workflows, etc.
The v1 I wrote last year was a very rudimentary bash script which couldn’t even do all basics of a “classic” UNIX utility. Now it’s written in Python, has gotten way more readable, extendable, and probably more portable too. :D