15 votes

What programming/technical projects have you been working on?

This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?

18 comments

  1. [6]
    Moonchild
    (edited )
    Link
    I'm trying to make a formal system for video games. In regular programming, we have type systems, which allow for increased expressiveness, partial proof of correctness, and better performance....

    I'm trying to make a formal system for video games.

    In regular programming, we have type systems, which allow for increased expressiveness, partial proof of correctness, and better performance. There are also tests, which provide experimental verification for some subset of behaviour.

    Video games aren't helped very much by either of these, because the kinds of complexity they create and the kinds of bugs they are prone to are different than for traditional software. (One way you can see this concretely in terms of performance; video games are largely single threaded, because data dependencies are complex and implicit, and when they do use multiple threads, they usually calculate multiple frames in a pipeline, causing a lot of input lag.)

    In video games, what is interesting is interactions. My system makes these a first-class feature. It also theoretically allows games to scale to any number of cores (my PoC is not quite there yet, but it will be interesting to see benchmarks once more of the concepts are implemented).

    11 votes
    1. [5]
      LukeZaz
      Link Parent
      Sounds really interesting! Could you elaborate on how you're planning to accomplish this?

      Sounds really interesting! Could you elaborate on how you're planning to accomplish this?

      2 votes
      1. [4]
        Moonchild
        (edited )
        Link Parent
        Sure. What do you want to know? The parallelism comes from ensuring that each bit of data is associated with one function, and only that function is allowed to mutate it. The mutation is...

        Sure. What do you want to know?

        The parallelism comes from ensuring that each bit of data is associated with one function, and only that function is allowed to mutate it. The mutation is functional, kind of (think let newstate = update(oldstate) rather than update(&state), although it's a bit more complicated than that), so order of operations doesn't matter. The functions which update state can send each other messages, which lets you avoid some verbosity that would otherwise result. This ensures that all the code that deals with a given bit of state is centralized, so if you want to change the behaviour, or just understand it, you only have to go one place.

        Ultimately, what I'm trying to do is build a system that focuses on functions rather than data structures. In a traditional program, data structures are the most interesting part; they form the core, and everything happens around them, in terms of them. But in a video game, it doesn't matter so much; we don't care so much what things we're representing, we care how those things interact. And those interactions also create complexity in video games (both in code and in player experience), and can make the code harder to manage.

        Right now, I'm ironing the kinks out of a low-level formal system that's like untyped lambda calculus with mutation. After that, I hope to prove that there is a universal form to video game code, and that my model is more suitable to it than the traditional one. (First part shouldn't be too tricky, but the second might not be possible to formalize.)

        4 votes
        1. [3]
          wirelyre
          Link Parent
          Something like a calculus for entity-component-systems? What sort of logic or properties are you trying to get out of the formalization?

          Something like a calculus for entity-component-systems?

          What sort of logic or properties are you trying to get out of the formalization?

          3 votes
          1. [2]
            Moonchild
            (edited )
            Link Parent
            Not ECS, no. ECS wants to decouple code and data, following the traditional paradigm of many functions and primitives that are good at dealing with a single data structure. It also encourages...

            Something like a calculus for entity-component-systems?

            Not ECS, no. ECS wants to decouple code and data, following the traditional paradigm of many functions and primitives that are good at dealing with a single data structure. It also encourages homogenous types, so you avoid vtbl overhead and can allocate lots of objects contiguously, which makes cache happy. My system couples code and data almost* inextricably and encourages nested composition patterns over ECS's flat ones. It makes up for it on the performance end by (hopefully) allowing for lots of parallelism.

            What sort of logic or properties are you trying to get out of the formalization?

            There is one formal guarantee, which is that each state instance has only one mutator. But I'm much more interested in the informal properties. It's not possible to codify intent, but when you try to:

            1. It should not be difficult to transform your intent into the format your computing environment needs.

            2. It should be difficult for you to make a mistake in the codification of your intent.


            *The formal system contains a surprising (I was surprised...) escape hatch that lets you have functions. And the implemented version layers the calculus on top of functional units implemented in lisp, which does have functions.

            3 votes
            1. wirelyre
              Link Parent
              Thanks for explaining, I think I can sort of imagine what you're doing now. I hope you will share more in the future!

              Thanks for explaining, I think I can sort of imagine what you're doing now.

              I hope you will share more in the future!

              2 votes
  2. cadentius_aurelius
    Link
    I have been getting really into Rust, and have been making improvements on my small experimental tool, LSR, which is based on what I would prefer to be the behavior of ls -R. Essentially, to...

    I have been getting really into Rust, and have been making improvements on my small experimental tool, LSR, which is based on what I would prefer to be the behavior of ls -R. Essentially, to recursively list a directory tree, outputting only the relative paths, for direct use as arguments in other commands. For example:

    ~$ lsr top
    top/sub1/file1a
    top/sub1/file1b
    top/sub2/file2a
    top/sub2/file2b
    top/sub2/file2
    top/file3a
    

    So far, I really like Docopt. You write the help text as a &str constant first, and it gets automatically parsed and used to turn the command line input into usable values.

    Usage:  lsr [-adDLq] [<path>...]
            lsr -h
    
    Options:
        -a, --all           Include 'Hidden' Files and Directories.
        -d, --dirs          Print the Directories themselves, not just Files.
        -D, --dirs-first    Traverse Subdirectories before printing Files.
        -L, --dereference   Follow Symbolic Links to Directories.
        -q, --quote         Quote output Paths.
        -h, --help          Show this text.
    

    This, plus defining the struct and feeding the input into Docopt, results in a fully formed object with boolean attributes like args.flag_dereference. Not even Python has anything that makes this so easy.

    6 votes
  3. PetitPrince
    Link
    I've been working on a private video sharing site for some time now, and it's getting into shape nicely. The goal is to share some of my 300GB of martial arts videos (examination, workshop,...

    I've been working on a private video sharing site for some time now, and it's getting into shape nicely.

    The goal is to share some of my 300GB of martial arts videos (examination, workshop, etc...) with my dojo mates (30-ish users) , and put some access control and tagging capabilities on top of it.

    I had some vague experience in web dev (mostly front end, mostly leftovers from teenage experimentations), so everything I used was pretty new for me. I learned a lot.

    Here's the tech stack I ended up with:

    • Django as a CMS. It's "battery included" and there's tons of documentation on it. On top of it, I added Taggit for rudimentary tags and Django REST framework to communicate with other services (see below). Most of the page rendering is server side, and I use Bulma as a CSS framework. It's friction-less for my basic usage.
    • Hosted on a DigitalOcean Ubuntu droplet. Again everyone and their grandmothers use this and there's tons of documentations (including guide on how to install Django + Nginx + Gunicorn). I also get to learn how to admin my own Linux box, so it's a plus.
    • The whole thing revolve around Mux.com, which is a video converter and hosting service. It seems legit and the founder even engage with comments on HN (it's his second video SaaS; apparently he sold the first one). On a first prototype I used Amazon VOD solution, but this proved to be far too overkill, complicated and pricey for what I wanted (the sucker spun a load balancer that I had no usage of, resulting in unwanted bills :( ). The service require an URL as in input, so I used...
    • Backblaze B2 as backup host They recently added a S3 compatibility layer on top of their hosting stuff, so I can theoretically swap it with something else if need be.
    • I upload and send the REST request to the various web services with a custom Python+Qt app. It's a bit of a mess since it's the first time since uni I wrote a proper GUI app but it's less mystifying than I remember it was. I initially used PyQt but then switched to the more official PySide2 / "Qt for Python" for dubious reason (what if I wanted to sell my stuff ? even though I doubt this will ever happen). There's some gotcha (different signals, some typecasting that are not present), but the code is mostly compatible.

    I feel I need to up my testing / continuous integration game (currently there's none), but I find difficult to write tests when I have a poor grasp of the domain like it's the case here, and I don't find testing tiny function for testing sake very useful. Now that the project is sufficiently shippable (I'm opening the site to 2 trusted users at first, before releasing to the other 30ish persons), I can probably focus some hours on writing tests. I'm not sure on how to begin though, since Django is a careful but deep sandwich of classes (that helps with the ease of programming and magic, but it's a bit more difficult to grasp). I'm sure there's some documentation somewhere, anyway.

    Also: I'd like to use some deployment tools to automate some admin stuff (database dump, application update), but they all seems way overkill for my usage. I don't think I need with several layers of configuration and virtualization just to do a pg_dump mydatabase, git pull and python manage.py migrate. My test server runs locally, and I don't need a staging server. I feel I'm in the sour spot where I'd like to have some automation going, but don't need or want super enterprisey scalable solutions. Probably some bash script would be sufficient ?

    Anyway, it has been an valuable journey, and it will probably help understand some with my day job better (I do mostly configurations and server-side scripting for a lab management software; and it's (sadly?) a Java shop).

    6 votes
  4. [3]
    daturkel
    Link
    I'd like to move my personal website over to Hugo, but I'm learning some basic Go first so that I have some understanding of the tool I'm using. Right now I'm making my way through the Tour of Go...

    I'd like to move my personal website over to Hugo, but I'm learning some basic Go first so that I have some understanding of the tool I'm using. Right now I'm making my way through the Tour of Go but I'm open to suggestions on next-steps for learning the language. I come from Python and I'm a competent programmer, but Go is bringing me back to the single semester of C++ I took in college, which is about the only experience I have in a lower-level language than Python.

    5 votes
    1. [2]
      bonbon
      Link Parent
      Because you'll be using Hugo, it might be a good idea to learn go templates. I know they're used if you want to make your own website layout, but if you'd rather not go that route, then you'll be...

      Because you'll be using Hugo, it might be a good idea to learn go templates. I know they're used if you want to make your own website layout, but if you'd rather not go that route, then you'll be fine just picking a template someone else has made.

      Other than that, you should def checkout the http package. Go makes it too easy to start up a web server!

      4 votes
      1. daturkel
        Link Parent
        I definitely plan to make my own theme, so this looks like a great place to start, thanks! I'm familiar with Jinja templates, so I'm hoping that's somewhat transferable.

        I definitely plan to make my own theme, so this looks like a great place to start, thanks! I'm familiar with Jinja templates, so I'm hoping that's somewhat transferable.

        2 votes
  5. bonbon
    Link
    I recently pushed v1.1.0 of gmcts, my monte-carlo tree search implementation in golang. It includes a number of improvements to memory, speed, and decision-making. The main change to the api is...

    I recently pushed v1.1.0 of gmcts, my monte-carlo tree search implementation in golang. It includes a number of improvements to memory, speed, and decision-making. The main change to the api is the ability to set the exploration constant of a tree, which is useful for fine-tuning.

    Other than that, all of the changes come from within. It includes better support for concurrency, a better UCT algorithm for better decision making, and some memory improvements. v1.1.0 is also fully deterministic with/without concurrency (running the same program will produce the same results)! Overall, v1.1.0 uses less resources and runs much faster than before.

    I've already got ideas for a v2, but I still have features that I want implemented before I move onto v2. Namely, I want an option to set the UCT formula that gets used during simulations for v1.2.0 (the ones listed in this paper). I'm sure I had some other ideas, but with my lack of notes, I'll have to rely on my mind to remember them (o^_^o ).

    5 votes
  6. [3]
    Comment deleted by author
    Link
    1. [2]
      bonbon
      Link Parent
      What's the library called? Never heard of hypergraphs, but I'm always down for learning some new maths :^)

      What's the library called? Never heard of hypergraphs, but I'm always down for learning some new maths :^)

      3 votes
      1. JesusShuttlesworth
        Link Parent
        https://github.com/pnnl/HyperNetX Hypergraphs are just generalizations of normal graphs. Instead of an edge containing 2 vertices, an edge can contain up to n vertices.

        https://github.com/pnnl/HyperNetX

        Hypergraphs are just generalizations of normal graphs. Instead of an edge containing 2 vertices, an edge can contain up to n vertices.

        3 votes
  7. Apos
    Link
    I'm working on a static site generator for my programming project documentation. My goal is for the repos that use it to not include any dependency. It will just be a docs folder and markdown...

    I'm working on a static site generator for my programming project documentation. My goal is for the repos that use it to not include any dependency. It will just be a docs folder and markdown files. Then the CI can take that and generate the site for GitHub Pages.

    I've identified 3 types of documentations:

    • API for the public facing API. Function signatures, etc. (Might be nice to support versioning so previous API versions can still be read.)
    • Tutorials for teaching how the project can be used.
    • Articles such as design decisions, changelogs, etc.

    (Maybe example projects can also be useful, but that probably doesn't belong in the doc?)

    In practice, the API for my doc generator will just be the folder hierarchy. Markdown files in the tutorial folder would be compiled in the tutorial section with the tutorial page layout.

    Perhaps pages in the posts folder can be added to an rss feed. etc.

    My goal is for the HTML generation to be invisible.

    There's no public repo yet while I get the plumbing working.

    2 votes
  8. [2]
    Moonchild
    Link
    I also got back around to working on my APL interpreter. Writing in whitnified c has been interesting, and to be honest, I feel pretty neutral about it at this point. I'm sure most people's...

    I also got back around to working on my APL interpreter. Writing in whitnified c has been interesting, and to be honest, I feel pretty neutral about it at this point. I'm sure most people's initial reactions to it are horror, shock, and awe; and aficionados would say it's only way to make code readable. I feel about as productive in whitnish c as I do in regular c, and don't find either version significantly harder to write or read.

    2 votes
    1. Moonchild
      Link Parent
      Actually, it might be slightly misleading to say this. I'm significantly more reliant on syntax highlighting now, and I had to change my syntax highlighting configuration. Previously, the most...

      I feel about as productive in whitnish c as I do in regular c, and don't find either version significantly harder to write or read

      Actually, it might be slightly misleading to say this. I'm significantly more reliant on syntax highlighting now, and I had to change my syntax highlighting configuration. Previously, the most prominent highlighting was for keywords, types, and literals, but the specific highlighting didn't matter very much; its main role was as a guide in scanning. Now, because the code is so dense, I don't need to scan so much, but I have to have operators very prominently highlighted and multi-character operators underlined, so I don't miss anything.

      2 votes
  9. moocow1452
    Link
    Trying to figure out how to make a "soft mouse" in Android in order to play the web version of FTL, since the game is not touch compatible and I'm unwilling to plug in an external mouse because I...

    Trying to figure out how to make a "soft mouse" in Android in order to play the web version of FTL, since the game is not touch compatible and I'm unwilling to plug in an external mouse because I have Freaking Standards!

    1 vote