4
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I’ve been working on a NLP neural network as part of something I want to build, and it turns out NLP is fairly difficult. Being unimpressed with word embeddings and also not having the ability to generate them (since my dataset is a postgres database of 100000 commoncrawl samples that is constantly having new samples added and an equal number deleted), I opted to use a character (well really UTF-8 bytes linearly transformed) level model that uses convolutions and a GRUCell to attempt to learn something similar to embeddings. It’s more complicated than that of course - there’s masks that work equivalently to padding, the forward pass of the model has packaging padded tensors taking place so the input data has to be sorted, and all sorts of complicated things have to be done to ensure awful edge cases don’t happen and disrupt the training. But it seems to be learning at the moment, so that’s good I guess?