Fortnightly Programming Q&A Thread

If you haven't already, download neovim and set it up with a plugins.

• I recommend you use a NvChad or another prebuilt config, but if you want to learn, you could create your own (just plugin manager + a few language servers and essential plugins). Even if you do create your own config, you could look at the NvChad source code for inspiration and good essential plugins, since it's very small.

• I also recommend adding Copilot for neovim

I don't have many tips for vim navigation/shortcuts, sorry. I think those are just muscle memory, eventually you get used to e to go to the end of a word, O to insert a line above the cursor, %s/search/replace/g to find and replace in a file, etc. Just use a Vim cheatsheet if you're not familiar with the key commands.

I've started using :norm to apply a series of keystrokes to each line in a selection.

For example, suppose you have a CSV file and wish to delete the first column. You could record a macro with q and then manually apply it to each line, but that's very slow.

Instead: :norm ^dt,x (use normal mode, jump to start of line, delete everything to the first comma, then delete the comma).

I'm sure it can do much more than just this.

The most useful 'trick' is learning to move around in the file efficiently and the more straightforward editor commands. A cheatsheet is a must; a physical copy may be better than having a spare window available so you can refer to it constantly and it does need to be constantly. In the beginning, it will be quicker to press the right arrow a bunch than to look up and use the specific one or two keystrokes to move to the the next word, or what have you, but persevere and always try to do whatever you are doing with the least number of keystrokes. Initially, everything will take so much longer and you will keep losing your focus on whatever it is you are actually trying to achieve but, eventually, it will become ingrained and you will be so much faster (it's like going from a hunt and peck typist to a touch typist).

There's some games that allow you to learn these keystrokes in a fun way - but you need to make sure you use vim frequently to maintain it, otherwise it'll all be forgotten a few days/weeks after you stop playing the game. e.g. https://vim-adventures.com/ and http://www.vimgenius.com/

I haven't used vim in quite a while, but I used to have a quite nice setup for doing development in golang with it. Have the dotfiles for it here if you're keen to see the setup. Haven't verified if it still works though (caveat emptor)!

What you described doesn't look too bad. So without knowing more details, it's hard (for me, at least) to see how to make the design better. Maybe just a couple of thoughts:

As others have pointed out, Python isn't necessarily well equipped for functional programming. This may be why something about your code feels off for you. When you start using things like immutable data structures and pure functions, you stray further away from object-oriented paradigm and more towards functional paradigm.

In the OO paradigm, what you would probably prefer is rich domain models that encapsulate the behavior and state (if it exists) for that model. For example, in your case B could be modeled as a class with a calculate_thing method as opposed to a free function. What you would probably also do is separate the file reading and writing part to its own class. So if the persistence format or logic changes, it can be easily swapped or modified in one place.

In the functional paradigm, you would do something similar to your example instead. Now I suspect it feels a bit off because the data and behavior are separated, and the language doesn't really encourage or support such programming style in any particular way. In many ways, it resembles the more traditional procedural paradigm. In languages like Clojure, it's more natural to write code in this way: If everything is an immutable list or map, you can use any functions that operate on lists or maps (many of which the language / standard library already provides for you), and combine and chain them together in all sorts of ways.

What you have is what most of our Python programs look like at my job. We use Python to process large sets of data, and we typically split functions into data access, engine, and controllers. There's no reason to get bogged down in object oriented design in Python, which can be hard to get used to if you started in the OO world.

I would be happy with your design if I saw it in a review. The most important thing for maintainable code is separation of concerns and encapsulation, which it looks like you have.

The other side of my job is embedded C, so we think very procedurally, which influences our Python.

In my experience, this is exactly what Python scripts naturally look like. I don't think there's much wrong with it, although if you tend to repeat parts of that often enough you could consider writing your own utility library.

There are a couple of things you might be interested in checking out:

1. A pure functional language, like Haskell. This would force you to handle your side effects in a clean way, but the languages have a reputation for being academic or difficult to learn.

2. Barring that, maybe try a language that lends itself more readily to that style of programming, but without being quite as restrictive as a pure functional language. I would suggest something like Elixir or maybe Racket.

I don't think that's particularly bad. Conceptually, it looks really readable to me.

I think the thing that would potentially make the data flow more obvious is the pipe operator, which is an operator like |> or %>% (depending on the language) that allows you to describe more of a pipeline for your data. The classic example is a UNIX shell pipeline, where you take some data from one command and pipe it into another command, and pipe the output of that command into another command, and so on.

For example, your code would look like this:

c = load_data(filepath)
    |> get_subset
    |> calculate_thing
...

Unfortunately, Python doesn't have the pipe operator, so this isn't going to work. But I think it's interesting (and hopefully encouraging for you) to see how much the pipeline version looks like the imperative version - both are essentially a series of commands, one after another, that process the data created by the previous line. What you're essentially doing here is manually doing something that might be only slightly clearer in a purely functional language.

The other thing you could have a look at if you're really interested in diving into the full functional rabbit hole is point free programming and the compose function. The compose function is a function that takes two other functions and essentially pipes them together, and returns a new function that does both of those things. So if I've got a function int which turns a string into a number, and a function greater_than_zero that returns true if a number is greater than zero, I can combine them together:

program = compose(int, greater_than_zero)

text: str = input()
program(text) # returns a boolean

This is sometimes called point-free programming, where "point" refers to the arguments to the functions. (See how we never look at the arguments to int or greater_than_zero, we just glue them together into a larger function.) Theoretically, you can build whole programs in this style.

That said, Python is not the best language for this sort of programming. It lacks a lot of the important primitives, as well as things like implicit currying that make it easier to use. The code is often harder to read if you're not familiar with it, so I would be careful about doing anything too complicated if you want your colleagues to like you! But composing pipelines together is probably a good example of where it could work well in Python, so maybe give it a go there and see if it's helpful?

I'm not familiar enough with the C# memory model, but are you sure that you really are waiting for a chunk to be finished and safely tucked away before another thread (behind another CPU cache) gets to muck with it? How are you synchronizing this? (Because a normal work queue/thread pool won't do it). Maybe visualizing which cells are being worked on and slowing it down would help, or recording cpu instruction clock for each thread/chunk/start/stop and playing it back if the above solves the race by debug mode, as is often the case. Remember that without volatile the optimizer has a lot of leeway when rearranging your instructions/memory accesses.

That certainly sounds interesting, I was looking for something like this recently to make serverless apps for small group sharing of simple stuff like chores, low critical finances and shopping lists. Supporting different schema versions is ambitious, though. I guess you require primary keys on everything?