19 votes

What programming/technical projects have you been working on?

Automatically posted
Tags: ask.survey, recurring.weekly

This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?

10 comments

  1. xk3
    Link
    I switched from Xfinity Now back to Xfinity to lock in the $55/mo price for 5 years for 400Mbps (and 1 year of Xfinity Mobile free) and one hiccup that I wasn't expecting was that routing the...

    I switched from Xfinity Now back to Xfinity to lock in the $55/mo price for 5 years for 400Mbps (and 1 year of Xfinity Mobile free) and one hiccup that I wasn't expecting was that routing the coaxial through my CyberPower CP1500AVRLCD UPS was causing signal issues so it wouldn't activate (the light on the XB6 would stay blinking green instead of blinking white) so I had to connect it without the UPS intermediary. I guess that is what happens when you buy refurbished UPS at steep discounts... (I think it was around $50)

    On that note I'm miffed that the house across the street has gigabit fiber for about the same price as what I'm paying... It's been like that for about five years. guess I'll join the army...

    7 votes
  2. Sunbutt23
    Link
    I’ve got my “AWS free tier ec2 pihole with Tailscale all deployed via terraform and workflows” in a stable place! Works like a charm! In 2 minutes I can deploy it and have the terraform script...

    I’ve got my “AWS free tier ec2 pihole with Tailscale all deployed via terraform and workflows” in a stable place! Works like a charm! In 2 minutes I can deploy it and have the terraform script insert the new pihole Tailscale ip as my tailnet name server and set override local dns on joining nodes. This is all automated with stored secrets to access Tailscale and AWS. I’m pretty proud.

    Next step is to implement OIDC so that I don’t need an AWS access key stored as a secret in my repo.

    Next next step is to duplicate the work to GCP and Azure. All free tier of course.

    As always, advice is welcome!

    5 votes
  3. Diff
    Link
    I've been working on a 3D SVG Editor built on the Zdog library. So far it's coming along pretty well. Toiling away at undo/redo history, it's a little bugged for editing properties, but I think...

    I've been working on a 3D SVG Editor built on the Zdog library.

    So far it's coming along pretty well. Toiling away at undo/redo history, it's a little bugged for editing properties, but I think soon it'll be ready to start creating Things™ in Zoodle directly rather than manually stuffing them in the side. Biggest hurdle left is somehow crafting a 3D equivalent of the classic vector pen tool. Maybe I'll do it like Blender and just expose and allow users to directly manipulate the control points.

    5 votes
  4. [5]
    zestier
    (edited )
    Link
    Not particularly innovative, but I decided to get around to learning Vulkan. Its quite the process. I know its a joke that it takes a thousand lines of code to render a triangle, but I feel like...

    Not particularly innovative, but I decided to get around to learning Vulkan. Its quite the process. I know its a joke that it takes a thousand lines of code to render a triangle, but I feel like I'm basically already at that count and I have it just rendering a clear color. I've been making it hard on myself by not following a tutorial though because its more interesting this way.

    My long-term vision for this project (at least until I get bored of it) is to use 0 textures for gameplay elements. They'll be needed for UI and such, but for some reason I got the itch to explore exclusively using other coloring techniques such as vertex color, instance color, unique shaders, and lighting.

    The biggest tip I would have for anyone else that wants to learn Vulkan: install the Vulkan SDK and enable validation layers first. Like before you do anything else pick one of the routes to enable validation layers and get them on ASAP. Your segfaults will thank you.

    I've encountered a variety of fun stuff along the way though. Here are a few of the amusing (?) bits so far:

    1. Using the API correctly will not help you in the face of inconsistent driver support, even for functions that seem like they'd be used often. Like I found that https://registry.khronos.org/vulkan/specs/latest/man/html/vkAcquireNextImageKHR.html will just segfault on some machines if the supplied timeout is 0 (rather than behaving as non-blocking and returning VK_TIMEOUT if it could not be immediately satisfied). I'm a weirdo that develops in containers, even graphical applications, but a nice side effect of this is that since my host and container are compatible enough I can run the application built from the container on my host as an easy second test environment. On my guest machine, the one with the validation layers, setting it to 0 behaves as one would expect. On my host it just segfaults, which is super cool and weird. Seems like a driver bug since my host uses different drivers ("real" amd ones in host and mesa ones in guest).
    2. The rules it has for creating swapchains are so weird to me. Even if I have no desire to N-buffer my framebuffers I'm forced to create what feels like a lot of framebuffers anyway. The minimum swapchain/framebuffer set size on my configuration is 4, which is odd in that it kind of feels like a waste of vram. A minimum of 4 and a maximum of unlimited is weird when I feel like most applications would want at most like 3 frames in flight. Maybe the minimum makes more sense with additional target types I've not got around to yet, such as depth or stencil, but I've not looked into that yet so I'm not sure if that will make it any less strange.
    3. Swapchain recreate rules are a bit wacky. They're "fine", but not very elegant. Resizing a window invalidates the swapchain, but its not like it reports the desired dimensions when it does that. This feels like it is practically setting people up to introduce race conditions between the driver rejecting your existing swapchains and your application knowing what size to create the new ones. It seems like this specific condition makes it plausible that swapchains will sometimes be created for just a fraction of a frame before being destroy to be recreated again, depending on the order of events.
    4. https://registry.khronos.org/vulkan/specs/latest/man/html/vkQueuePresentKHR.html, at least on my machine, has a fascinating behavior that I did not find documented: it gets stuck blocking when the application is minimized and not exposed. If I minimize an application that call becomes long-blocking, although it switches back to non-blocking for as long as I hover over the icon to get the live preview. By "long-blocking" I just mean it blocks longer than would be expected from a vsync or something and converts to permanently blocked until the state changes. This is arguably useful because that will effectively block my main rendering thread automatically for me to save the CPU wasting submitting crap to the GPU for a hidden window, but it also feels a bit like a trap because it means I absolutely must move my rendering onto a separate thread from my event processing if I don't want it to risk blocking events that a hidden window can still receive. Honestly, I'm not sure why the spec even allows this function to be blocking in the first place, especially because it is a blocking function with an undefined length of blocking and no ability to set a timeout on it.
    5 votes
    1. [4]
      TangibleLight
      (edited )
      Link Parent
      The best advice I ever got was that smooth resizing is not worth the effort in Vulkan. I'm sure it's possible to do correctly but I never figured it out. I just drop whatever frames during the...

      Resizing a window invalidates the swapchain, but its not like it reports the desired dimensions when it does that

      The best advice I ever got was that smooth resizing is not worth the effort in Vulkan. I'm sure it's possible to do correctly but I never figured it out. I just drop whatever frames during the resize and recreate once it ends.

      This talk was a great resource for me: Vulkanised 2024: Common Mistakes When
      Learning Vulkan - Charles Giessen

      it gets stuck blocking when the application is minimized and not exposed

      Now that seems bizarre to me. You mentioned you're on AMD with official drivers on one machine and mesa drivers on the other - which drivers do you see that behavior? What display server and window manager/compositor do you use?

      Edit: Also, what present mode are you using? If you're in FIFO etc it might be waiting for vblank, and the display server or driver is clever enough to know it's not visible? That's my only guess. Try changing the present mode and see if behavior changes.

      1 vote
      1. [3]
        zestier
        (edited )
        Link Parent
        I do have something working for the resizing thing, as well as I had a rough plan for improving it later. But it all just seems so inelegant at a spec-design layer. At present the only place I...

        I do have something working for the resizing thing, as well as I had a rough plan for improving it later. But it all just seems so inelegant at a spec-design layer.

        At present the only place I ever create swapchains is right before rendering and it does that if it is flagged as invalid. The two places that seem able to invalidate the swapchain are vkAcquireNextImageKHR and vkQueuePresentKHR, which I currently have approximately at the start and end of my render processes. If the acquire fails on suboptimal I queue the recreate but let the render continue anyway, if the acquire fails on out-of-date I queue the recreate and also abandon the frame. The queue present though is so close to the end of the frame that it frankly doesn't matter what I do with its result (worst case it would handle the issue the next frame, but there aren't any more relevant Vulkan calls during the frame that could be affected), but I do queue a recreate if I get either of those anyway.

        I also flag queue recreates when I get an event from SDL about the window pixel dimensions changing.

        As of right now that all seems to be "fine" because I seem to get lucky enough that the events come in an order that happens to avoid excessive recreates. Something I was considering adding was changing my flagging from a boolean to a mask that describes the origin of the recreate(s). The logic being that if the origin of the recreate was the SDL event then I should only do it if the desired size is different from the last create, but if the origin of the recreate was Vulkan then I need to recreate it regardless. Or I guess there's also always the option of not even bothering to listen for the window event and just depending on the dropped frame.

        Something I'm not intending to do under any circumstances though is try to do the actual recreate during any part of rendering. I could try to acquire the image, if that fails immediately do a recreate, and then immediately try to acquire the image again. But I'd rather just drop the frame than deal with that.

        On the blocking thing, digging a little deeper confirms that it is a combination of Wayland and FIFO. I can't say I'm that surprised as Wayland has some interesting behaviors. I disagree with this particular approach though. In my opinion if they wanted to have this blocking it should be on acquire and respect its timeout, not be on present.

        In fact the way they implemented it has serious issues that lead to, potentially unavoidable, application hanging. The naive thing of leaving both input handling and rendering on the main thread, combined with this behavior, means that if I quit an application while its minimized it actually just hangs because the application never gets to process the quit signal unless I do something to give that window focus again.

        So the obvious thing is to just not do it all on one thread, right? Nope! Unless it behaves completely different when on a background thread it would still be broken! The only difference is that rather than the main thread being stalled and there not being a way to even know about the signal you just have an unjoinable thread. And since that unjoinable thread is locked in a Vulkan call even forcefully terminating the thread may leave your Vulkan resources in-use such that you're not allowed to clean them up. With that case in mind its possible the only "legal" thing to even do is to terminate the thread and then not even attempt any cleanup because attempts at cleanup could potentially all be illegal operations due to the in-use device.

        Alternatively could just refuse to use FIFO on Wayland, but that's just a different weird behavior.

        1 vote
        1. [2]
          TangibleLight
          Link Parent
          The closest thing I ever got to an "elegant" solution was something along these lines: Rough pseudocode-from-memory sci = swapchain create info { ... old_swapchain = .null_handle } sc =...

          At present the only place I ever create swapchains is right before rendering and it does that if it is flagged as invalid. The two places that seem able to invalidate the swapchain are vkAcquireNextImageKHR and vkQueuePresentKHR, which I currently have approximately at the start and end of my render processes. If the acquire fails on suboptimal I queue the recreate but let the render continue anyway, if the acquire fails on out-of-date I queue the recreate and also abandon the frame. The queue present though is so close to the end of the frame that it frankly doesn't matter what I do with its result (worst case it would handle the issue the next frame, but there aren't any more relevant Vulkan calls during the frame that could be affected), but I do queue a recreate if I get either of those anyway.

          The closest thing I ever got to an "elegant" solution was something along these lines:

          Rough pseudocode-from-memory 
          sci = swapchain create info {
    ...
    old_swapchain = .null_handle
}
sc = .null_handle

loop {
    if sc == .null_handle
        sc = create swapchain (sci)
        destroy sci.old_swapchain
        sci.old_swapchain = sc

    ...

    try to acquire image; catch: {
        sc = .null_handle
        continue (consider frame "dropped")
    }

    ...

    try to present image; catch: {
        sc = .null_handle
        continue (consider frame "dropped")
    }
    
    consider frame "complete". Increment frame counter etc.

    ...
}

          I seem to recall that it worked great on my laptop (XOrg with MESA intel drivers) but it caused some particular issues on my desktop (XOrg with proprietary Nvidia drivers) with certain present modes. I don't rememeber the details though.

          At some point I just gave up and simply do not attempt to render if the window is resizing. You still have to handle the invalidated swapchain, but it almost never occurs.

          In fact the way they implemented it has serious issues that lead to, potentially unavoidable, application hanging. [...]

          I think you are right.

          You're not guaranteed anything but FIFO, which is a portability issue in refusing to support it on Wayland.

          In most cases FIFO seems just fine; in my amateur usage there doesn't seem to be any reason not to use it unless you are building a microbenchmark. It is a shame that Wayland does this.

          1 vote
          1. zestier
            Link Parent
            After much digging I finally found the culprit's documentation lives in https://registry.khronos.org/vulkan/specs/latest/man/html/VK_KHR_wayland_surface.html: From the issues section: Honestly, I...

            After much digging I finally found the culprit's documentation lives in https://registry.khronos.org/vulkan/specs/latest/man/html/VK_KHR_wayland_surface.html:

            From the issues section:

            1. Should we require surfaces created with vkCreateWaylandSurfaceKHR to support the VK_PRESENT_MODE_MAILBOX_KHR present mode?

            RESOLVED: Yes. Wayland is an inherently mailbox window system and mailbox support is required for some Wayland compositor interactions to work as expected. While handling these interactions may be possible with VK_PRESENT_MODE_FIFO_KHR, it is much more difficult to do without deadlock and requiring all Wayland applications to be able to support implementations which only support VK_PRESENT_MODE_FIFO_KHR would be an onerous restriction on application developers.

            Honestly, I think this is absolute insanity. The one present mode the spec guarantees to be present, VK_PRESENT_MODE_FIFO_KHR, is also the one present mode that Wayland is allowed to just leave broken in a way that is even documented to potentially deadlock applications. If its so busted on Wayland I don't understand why they didn't just map it to mailbox with vsync on.

            To confirm I wasn't crazy and it wasn't just my usage of Vulkan having this issue I used a tiny sample app of SDL3 using SDL_GPU and it has this "hangs if trying to close a minimized window" issue. So I guess I could be nice and file a bug on SDL, but frankly this is a Wayland/driver insanity that they're just asking applications to work around.

            1 vote
  5. lynxy
    Link
    I finally gave up on the Elegoo Neptune 4 Plus- I'm getting Y-slipping in maybe 50% of prints no matter what I do. I'm not certain bed-slingers were made for beds this large. At this point I...

    I finally gave up on the Elegoo Neptune 4 Plus- I'm getting Y-slipping in maybe 50% of prints no matter what I do. I'm not certain bed-slingers were made for beds this large. At this point I really don't care about speed, I just want something reliable- and so I've ordered a Prusa. It will be coming in 10 weeks, which is a while!

    In the mean-time, I'm trying to get what prints I can out of the Elegoo, including parts for an easy install-uninstall bug-net frame I'm putting together. It's getting warmer and mosquito season will soon be upon us.

    Otherwise, plenty of little projects that don't seem to go anywhere! Fiddling with driving multiple 7-segment displays through a shift-register, replacing the fans in a second-hand Eaton UPS with 80mm Noctua, debating switching to ZFS over MDADM.

    Actually, that last one is one that I really have choice-paralysis on. I've used MDADM for the longest time for simplicity, but my partner has been pestering me about using ZFS, and I admit it does offer a few features which sound useful (non-homogeneous disk handling, metadata caching on faster media, etc). I've had a few data scares with MDADM recently in which the arrays failed to come back up after a power cycle (hence the UPS I just got!), and ZFS has a reputation for data safety. Unfortunately it also seems like ZFS is ideal for enterprise, in that it's best used when you know (and own!) the full set of hardware you will eventually be using in prod. This isn't ideal for home-use, as I'd like to grow the array over time, not buy everything up-front. It seems VDEV expansion has been added in limited capacity at the end of last year / start of this year, after a multi-year effort. I don't know. As usual.

    4 votes
  6. IsildursBane
    Link
    I have finally got all the parts for my Raspberry Pi audio project, after having some issues with some battery cells. I did a battery longevity test on the 14500 cells, and the batteries only...

    I have finally got all the parts for my Raspberry Pi audio project, after having some issues with some battery cells. I did a battery longevity test on the 14500 cells, and the batteries only lasted 35min. I am thinking the packaging on the battery listing 2500 mAh is incorrect. I will need to upgrade to using 18650 cells to get the battery life I want. I do not need it to last 5hrs, but I do want it to be able to last at least one full album, which the 14500 battery could not.

    3 votes