8
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I started writing my dream automation task runner, essentially a runtime that can combine various scripts and code from any programming language and have them all together without any ABI or direct linking into each other. Works relatively well when you drop it into a folder of various automation scripts.
Then I realized I could turn it into its own custom shell as a direct task runner (sort of like a REPL), then realized I could turn it into its own workflow language (similar to Modula-2), so now I am writing my own programming language (inspired by the syntax of Powershell) that is compiled without memory management (all memory is flushed from the allocator heap at the end of a functions use, that way I can entirely avoid using a GC, RC, or Borrow Checker), and the global dynamic (immutable) values are stored in a separate thread that stores a SQlite database.
The project has grown immensely, but I am absolutely in love with it. I didn't intend to write my own little micro-operatiing system and language, but its cool that it mostly works. I have a lot of plans for it, and transitioning all of my projects to use it.
I am terrified by the mere suggestion of Powershell being the inspiration for a DSL.
But that's probably because I only use it as a shell, and very rarely.
I thought so too, but recently I have really fallen in love with the way Powershell is designed. As a shell, it sort of sucks until you realize that "the shell" is just a REPL for the language.
The verbosity of Powershell is absolutely crazy, where most languages pride themselves on having very few keywords (in the dozens), Powershell went the other way and has over 1600 built in (they call them cmdlets). This makes developing applications, connecting to remote servers, compressing files, doing transfers, and so many more features incredibly easy as it is just a keyword with a parameter. The largest 'standard' library I've ever seen in a language.
I can do in 20 lines in Powershell, what would have taken me hundreds in other languages. As an example: the very 'keyword' of 'Get-ChildItems' can list all directories, files, or items (including registry and credentials, etc).
Not only that, but PowerShell has the most complete and useful documentation generator and documentation I have ever seen. It's as verbose as the language, which is saying something.
My only gripe? It relies on .Net, so I've ended up writing my own that is compiled to C.
Powershell is a deeply fucked up language in terms of syntax when compared to most others out there, but it's also really the only language that I feel anyone could pick up and build complex and massive programs with only a few hours to spare here and there. It was built for system administrators (not programmers or developers) to automate their tasks, its an automation language, but it can be a really solid language on its own. The downside is that the language isn't built to be a 'project', and to be short scripts, so I aim to solve that issue too.
I would argue its less of a DSL and more of a scripting language vis-a-vis JavaScript, and could be a GPL, but its limitations at the core of being single files or modules breaks its ability to be used as a large project (but there are others that have tried to ham-fist it into being a proper language).
Powershell's Hyphenated-PascalCase really freaks me out, but I will say that I'm starting to appreciate the arguments for a general bias towards verbosity more and more.
I'm spending a lot of time on Unix internals these days, and it's just annoying that everything is still incomprehensibly short... mostly because there were no GUIs or tab completions in 1970. There are so many C files that are just an obscure acronym which is never even expanded in a single comment in the file. Inconsistent concatenation also makes it impossible to to use search effectively. Is it conf or config? ctl or ctrl?
My “repeatTest” library is coming along nicely. (I’m thinking of calling it just “Repeat.”) I have basic shrinking working - at least, it works nicely for trimming junk from strings and arrays.
To make testing shrinking easier, I wrote some code to do the inverse operation - taking a value and parsing it into an array of picks (integers). So, now, in addition to Arbitraries, I have a parallel set of functions for defining Domains. A Domain is like an Arbitrary, but it converts in both directions. (It wraps an Arbitrary and adds the inverse operation.)
Encoding and decoding can be tested against each other by doing a round-trip test, which exactly the sort of thing that a property test library is good at.
Domains validate values by parsing them, so that means I have an API that looks like a simpler version of Zod, creating a validated copy of an object by doing a round trip. It will be slower, though, to convert all the way to integers and back again, and I don’t plan to make it into a viable Zod alternative.
Another feature I’ve been working on is generating arrays of unique values and tables that have unique columns. This is generating permutations instead of combinations - picking values without replacement. When generating permutations, it’s possible to run out of values - for example, a list of unique booleans can’t have more than two items.
To make that work, I needed a class to keep a history of what’s already been picked and whether any more values remain. The code I wrote for it had a silly bug in it that I didn’t find until I wrote a property test.
I've been working on Rust port of libdatrie, now with C2Rust. libdatrie is a C implementation of double-array trie data structure. It is used in libthai, the Thai word segmentation algorithm (written by the same author) used in pango. So if you're on Linux desktop, chances that you have that library linked. It also seems to have bindings in other languages such as Ruby, so I suppose people have been using it for other purposes.
There's not much reason behind this, other than that Rewrite it in Rust meme and yet I want to see how hard it actually is to do one.
The first attempt I port it manually line-by-line, but the archaic code keep tripping me. Arrays are almost always accessed with pointers, loops terminating condition oftentimes do not correlate to the variable they increment in each iteration, etc. After months of work I finally completed it and get it to run the first time. Obviously the tests failed; probably because of various off-by-one errors, or because I attempt to make it not use null-terminated string and end up breaking stuff.
The current attempt was started when I found C2Rust. I was curious whether it actually works. To my surprise, it actually works. In one swoop it managed to port the entire C codebase into Rust and is a drop in replacement for the C version (one could load the Rust version with
LD_PRELOADon top of existing C code). The code is written to exactly match the C's implementation so it was two weeks of work rewriting it in safe, idiomatic, object-oriented Rust. This time though, I could run tests as I go. The hardest part is probably the linked list, which become the easiest when I realize I could just replace the whole thing with the range_set crate.As of writing I've finished the main library, so if you want a Rust version of that library feel free to grab it from the GitHub. I'm not sure how should I get it adopted though, but perhaps I might consider replacing libdatrie.so in my machine with my version first.