16
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I love these themes. Excellent work! And thank you for sharing. I'll be using the Sublime Text them for sure.
I am working on a Python project to scrape official COVID-19 case numbers for India from our government websites every day, archive them in JSON format at github.com/indiacovid19/indiacovid19, and present them as graphs and some key indicators like the growth percentage in the number of cases, doubling time, etc. here at indiacovid19.github.io. Every piece of data used in this website is supported by reference links to official government sources.
I started this project when the case numbers were still small in India and I wanted to see if the number of confirmed cases was growing exponentially and if so, determine the rate at which it was growing. The Wikipedia article on this subject did not have accurate case numbers at that time because the case numbers were not archived reliably well at any place. Troubled by this, I began digging into the old snapshots of our government websites archived here by archive.org and carefully began noting down the case numbers for each date, thus building a chronological archive of the numbers.
Once my archive of the numbers was good enough and my project became reasonably stable, I wrote another script to emit Wikipedia markup code to automatically generate the graphs used in the Wikipedia article on this subject. After a discussion with the frequent contributors to the case number charts on Wikipedia, we agreed to replace the data there with the data I had curated.
I am quite happy with the outcome because I went through a full circle here: looking for a good source of archived data, coming across inconsistencies in the data archived at Wikipedia, starting my own project to curate official data carefully from archived snapshots of government websites, then publishing my data as a project on GitHub, and finally contributing the data back to Wikipedia. These days, I am working on increasing the extent of automation to reduce the number of manual interventions required to pull new data updates from our government website, generate my project website, generate Wikipedia markup, and then update the data on Wikipedia.
I'm wondering if DoltHub would be interesting to you, maybe as another place to publish the data? I haven't used it, but I subscribed to their blog because it looks interesting.
Thanks for the great suggestion. I wasn't aware of DoltHub. It looks interesting indeed.
I probably will not publish data to it right now mainly because I cannot afford the time. With a full-time job and increased household chores due to lockdown, it is not easy for me to find the time to add more features. I am currently focusing on keeping the data up-to-date and supplying the data to Wikipedia charts here and here.
However DoltHub looks very promising and I would love to play with it sometime in future when I can afford the time. In the meantime, if someone is willing to fork my project and send a pull request to add support for DoltHub, I will gladly accept it.
I've been getting started on learning more deeply about computers over the last month and some change.
I began with the Nand to Tetris course. I deeply enjoyed this course and got to the end of Project 5, which involves implementing the virtual machine. It feels so cool to have built a virtual computer from tiny logic gates. Little did I realize, though, that I would be required to write an assembler in Chapter 6 using an established programming language. Oops.
I'd dicked around with an online Java course and Zed Shaw's C book years ago, but only ever got a few programs deep in each. I decided to rip the bandaid off and just learn C via K&R to write this assembler, so that's what I've been up to for the last couple weeks. I've nearly made it to the end of Chapter 1, doing all of the exercises as I go.
So far, I think I'm in love. Most of my life I've chosen things that really challenge me to spend my time on, but my brain feels pleasantly pre-wired for programming. Even the (fairly dry) example programs are enjoyable for me, so I can't wait to start writing stuff that will actually be useful for me (or others)!
I dusted off my accordion synthesizer project because I had another idea about how to do a bellows-like expression control. It didn't work, but if I keep trying, I guess I haven't failed yet.
I decided to try out writing some 3D code with Vulkan. I had made the switch from OpenGL to Metal, so I thought it wouldn't be too hard to also get to know Vulkan.
Boy was I wrong! Vulkan takes many of the bad habits of OpenGL and turns them up to 11! It even adds a bunch of new ones. I had thought they would take the opportunity to make an API that made it harder to screw things up, but they didn't. They still separate out creating an object from allocating it. They still require binding existing resources in order to use them. And now they have a bunch of huge data structures you need to fill out to make any calls, and they all have fields that are intended to be set to empty when you use them, but are obviously used internally for some purpose.
The basic "Hello World" style tutorial is broken into several steps. 15 of them, in fact. I got about halfway through before throwing up my hands in disgust. The amount of work required to get a single triangle on the screen is staggering. I can't believe that anyone would choose to use this API. For developers who can still target OpenGL, they'll probably stick with that. And if they can't stick with OpenGL, they'll probably target Metal or D3D12 because both require less work and are a lot easier to use. (They're not perfect, but wow! Vulkan is awful.)
Sorry to be so negative. I'm just shocked at how Vulkan turned out. I watched its creation from afar. It definitely feels like a design by committee. But the committee was made up of the same interests as the committees that designed Metal and D3D12, so it doesn't make sense!
Yikes. I had a lot of hope for Vulkan. Any idea if it at least gets better later on, or if it at least successfully removes most of OpenGL's cruft?
It's hard to say since I didn't go much further. I guess the answer also depends on what you consider to be OpenGL's cruft. For me, it's many of the things I mentioned above. The idea that you create an object, but it hasn't yet been allocated is bonkers to me. And it makes programming much more difficult. You don't know whether the memory for the object actually exists until you go to render, bind the object, and it fails. And just creating a texture requires 3 objects. You have, for example, VkImage, VkImageView, and VkDeviceMemory. I couldn't suss out why you need these 3 things for a single object, but it appears you do.
There are also lots of patterns it uses that are just irritating in this day and age. For example, since it's all straight C, it can't just return you an array of elements. You constantly have to make 2 calls to the same function to get stuff returned. Like:
But what's interesting is they have provisions for memory management. You can set up a set of routines to use for memory management. If it's there, Vulkan will call it, and if it's not, it sticks with the OS memory management routines. So they could have made this all work with a single call:
And then the caller would have to free the memory. But instead they went with the cumbersome version of call us once to get the number of things you'll need, then allocate the memory and call us again to get the actual things. And on top of that, if the second call sets the "numFoos" parameter to NULL (since you already know how many you need), it crashes.
I ran some of the pre-built examples, and it seems like it works reasonably well once you get it up and running. But I'm not sure I can bring myself to get that far. I may let it rest for a while and pick up where I left off. We'll see. The older I get, the less time I care to spend on frustrating things, though.
Actually, to be fair, at least 2 things have improved:
glGetError()if you want to know if something went wrong, every function now returns an error code. I have to give them props for finally cleaning that up.enumsinstead of#definesfor the constants so there can be some type safety and it can be clearer when you use an incorrect value. (GL_LINE vs. GL_LINES anyone? Ugh!)So yes, they did make amends for some of their past sins, but that doesn't mean they've fixed everything or even very much.
Still working on-and-off on an UNK rewrite. But more excitingly, I learned you can abuse /etc/manpath.config and put whatever man sections you want in there! So for my pubnix server I'm going to add a 'man bread' section for recipes :) :)
Finally got models rendering in my relativity simulation! Other work keeps intruding, and I still want to clean up the code, but it's getting easier and easier to just throw together visualizations. Next couple steps are probably code cleanup, first person view, and adding reticles indicating where tiny objects like the planets are, and their distortions.
One of the tricky parts I ran into recently was actually just taking video of it in action, cutting the video, and then putting it up somewhere.
So far it's a bit hard to come up with visualizations that are intuitive to people with no background in what's going on, but I think a first person view will help there, and good transitions from intuitive views to lesser-known ones like worldlines.
I am trying to learn how to utilize web services for Oracle's Taleo Enterprise product but have been hitting a brick wall. SOAPUI freezes when trying to create a new request so I have been trying to brute force my way through it using Python's zeep. However, its a bit over my head as I don't have a good grasp of SOAP APIs.
With that said, does anyone know of any good training materials for using SOAP?
Is this a work thing?
Yeah, I have been wanting to learn how I can utilize scripting to help with regression testing in our various HR systems. We have a ton of integrations that we hand off to my company's IT team that I would like to learn more about.
I started re-designing my personal website from scratch in some basic HTML/CSS with Bootstrap. I had it working with React before but I decided just to host it on GitHub Pages instead of the host I was using before. I decided to go to basic HTML/CSS because I realized that it'd be significantly easier for me since the main thing I put on my website is random tid-bits about school projects I've worked on and I didn't want it to be very complex at all.
I've also started my file system project for my OS class. It's the last project of the semester and I'm so ready for it to be over, but these next couple weeks are gonna blow. We have to set up subdirectories, go from basic extents-based inodes to indexed inodes with file growth, and get all of the synchronization working. I've learned a ton in the class so far but it is incredibly difficult.
I'm pretty happy with a website I built. A project like it is never finished, but I set up matomo for analytics, and wrote a few posts for it. It's been fun so far, especially writing about reverse engineering. It's hard, but fulfilling.
Nice and simple front page.
May I suggest aligning the home link ("ARCHWIZARD'S SITE") to center as well? It would put the whole header along the same axis. People like symmetry of alignment.
I am rescuing a failing hard drive (Seagate Barracuda 2TB ST2000DM001, bought in 2013) on my secondary/gaming PC (running Windows 10)
Acronis Disk Manager told me that the SMART status of this drive was worrying. I bought a bigger capacity hard drive (Toshiba X300 4TB, HDWE140XZSTA) because... damn storage is cheap nowadays.
It is no fun for several reasons:
--rescueoption, but somehow the disk was locked and required a chkdskAt least this prompted me to (at last) subscribe to a Backblaze account.
Does anyone know if there is any easy way to run python scripts without typing "python"? For example,
instead of
Been finishing a lot of shelved projects during lockdown: