pseudolobster's recent activity

  1. Comment on What programming/technical projects have you been working on? in ~comp

    pseudolobster
    Link
    I spent the better part of last weekend bashing my head against the wall trying to solve a simple problem with the wrong tools. Yesterday I got the right tools and got everything working in under...

    I spent the better part of last weekend bashing my head against the wall trying to solve a simple problem with the wrong tools. Yesterday I got the right tools and got everything working in under a minute.

    For the past month or two I've been preparing to pick morel mushrooms in the remote wilderness of British Columbia. To navigate the terrain I've been creating a map in QGIS using as much publicly available data as possible. After testing it out, it's fairly difficult to navigate the thousands of random forest service roads without GPS telling where you currently are. So I spent some time looking for ways to get GPS data to my laptop.

    I tried a few apps for Android that should be able to provide this functionality. It should be pretty straightforward, since GPS devices are typically just serial ports that spew NMEA data, which is a text-based format that gives your lat/long, speed, heading, altitude, current time, etc. It turns out recent versions of Android make this more difficult than it should be. There's a bluetooth serial profile, that provides a COM port over bt, so seeing as Android phones are just tiny linux computers this should be really simple. For whatever reason the app doesn't work, and people have been complaining about similar apps in the reviews, saying new versions of android don't let them run in the background, and there's a deep rabbit hole of Xposed framework root hacks to do things "the proper linux way".

    Anyway, I dug through my junk drawer and turned up a Sparkfun Venus GPS receiver. This is a really good GPS receiver with high output rates, and the external antenna should work a lot better than what's inside a phone. It's just a serial device, though it does talk at 3.3v levels, meaning you can't use a RS232 adapter; It needs to be TTL levels.

    I have a USB-TTL "FTDI" adapter, used for programming microcontrollers, but at the time it was 200km away in storage, so I decided I can make a USB-TTL serial adapter using an arduino I have laying around. This was a big mistake.

    Arduino Nano microcontrollers have a hardware serial port, wired up to its USB port for programming. You can also make a software serial port by bit-banging the protocol over a GPIO pin. In this way you can write a small program to send data from one port to the other, forming a bridge. HOWEVER, the last time I used this GPS device I was trying to measure the acceleration of vehicles at a drag race, so it was configured for the highest output rate, 30hz, which requires a serial connection at 115200 baud. Arudinos using their CPU to emulate a serial port in real time have a hard time at such high baud rates, which led to data corruption.

    So it ended up working, kinda. After an hour of figuring out that the baud rate is actually 115200, I'd connect to the serial port and I'd get something that looks suspiciously like NMEA data:

    $⸮⸮⸮⸮⸮,⸮,3,05,02,20,18,12,25,,,,,,,3.2,1.6,2.8*36
    $⸮⸮⸮⸮⸮,2,1,07,25,55,210,31,20,54,085,38,05,53,135,41,02,43,063,35*79
    $⸮⸮⸮⸮⸮,2,2,07,11,39,060,31,12,27,168,27,18,23,242,29*4⸮
    $⸮⸮⸮⸮⸮,012438.124,⸮,4909.9418,⸮,11934.4732,⸮,000.0,327.2,280322,,,⸮*76
    $⸮⸮⸮⸮⸮,327.2,⸮,,⸮,000.0,⸮,000.0,⸮,⸮*09
    

    This is... Almost usable. For whatever reason it's able to transmit numbers, but fails hard on letters. I'm still baffled why this is. This should be encoded as UTF-8 or ASCII or something, where numbers and letters are the same number of bits, just with the value offset by 20 or 30 or something.

    Anyway, the GPS unit accepts binary commands to set its data rate, which can be stored in flash memory, so I spent hours trying to send the correct sequence of bytes to tell it to drop into 1Hz refresh, 9600baud communications, but with the data corruption this was impossible. I tried sending the commands on a loop, letting it run five minutes to see if one of the thousands of commands got through intact, but alas it didn't work.

    Apparently arduino nanos with a ATMega 328p might be able to handle such a high serial load, so I tried some other knockoff arduinos I had laying around, but they failed too. Also apparently most knockoff arduino nano's on ebay use counterfeit Atmel chips, so that was another consideration. I could just keep trying the same thing with different parts forever, and maybe one of them might work.

    Anyway, that didn't work, and I really didn't want to drive 200km for a $10 part with gas the price it is, and I don't currently have an address to order something off amazon, so I tried working with the garbage data. Like I mentioned, the numbers are mostly intact, it's just the letters that are corrupted, and they don't contain much information. So I spent a while staring at these NMEA statements, figured if the 8th field is "000.0" then it's a $GPRMC string, which should be enough for navigation. I fudged the letters, since I know I'm always going to be in the northwest quadrant of the globe I can assume the letters N and W, etc.

    I came up with this really janky code to convert the garbage data into valid NMEA:

    void displayGPS()
    {
      char field[20];  
      getField(field, 7);
      if ( strcmp(field, "000.0") == 0) {
        String message = "GPRMC,";
        getField(field, 1);
        message = message + field + ",A,";
        getField(field, 3);
        message = message + field + ",N,";
        getField(field, 5);
        message = message + field + ",W,";
        getField(field, 7);
        message = message + field + ",";
        getField(field, 8);
        message = message + field + ",";
        getField(field, 9);
        message = message + field + ",";
        getField(field, 10);
        message = message + field + ",A";
        int checksum = 0;
        //Serial.println(message);
        for(int i=0; i<=message.length();i++){
          checksum = checksum^message.charAt(i);
        }
        Serial.println("$" + message + "*" + String(checksum,HEX));
      }
    }
    

    It's mostly janky because I couldn't be bothered to learn about Arduino's String datatype, and how to concatenate them with C++'s null terminated character pointer strings. ...But it worked! Or at least it gave me lat/long on a map, and after pointing gpsd to the serial port I could actually track myself on the map. I still had a few problems though. First of all the update rate was awful. It would only update once every 10+ seconds. Secondly, it'd sometimes be off in some direction by a lot. Presumably one of the numbers in the lat/long was being corrupted, yet my dumb checksum routine didn't care, and passed it off as valid data. So I'd get a GPS track that keeps jumping hundreds of kilometres north, west, east, or south, then jumping back.

    I considered using the checksum I got from the GPS and throwing out anything that's been corrupted, but I also have the problem where the checksum is in hex, and I have a problem transmitting letters, so only 10/16 of the checksums I receive have a chance of being valid.

    Sooo, long story short I was in the area of my storage, so I grabbed the FTDI serial adapter. Soldered the GPS module to it, plugged it in, and everything works perfectly, the first time, no hassle, running at 115200 baud, full refresh rate. Everything's smooth and nothing went wrong at all. I should have done this from the start.

    It's been a great lesson for me that even if I think I can get by with what I have on hand, using the right tools can save a ton of wasted time.

    6 votes
  2. Comment on What programming/technical projects have you been working on? in ~comp

    pseudolobster
    Link Parent
    The offer of processing power is compelling, and really generous of you. I'd be tempted to take you up on that, but I ended up realizing having 16 different pyramids for a single fire isn't so...

    The offer of processing power is compelling, and really generous of you. I'd be tempted to take you up on that, but I ended up realizing having 16 different pyramids for a single fire isn't so bad. I can import them as one layer in qgis, and I fleshed out my bash script to iterate over all the files generate the mapproxy configuration for all the layers, grids, and caches automatically.

    I was mostly just combining the tiles into one enormous file for the logistics of not having to deal with so many individual layers by hand, and that's the only part that needed absurd amounts of ram.

    Now I can just put all the tiles in their own directory and go: for a in *; do ./maketiles.sh $a; done and it'll process everything. It looks like it'll take a good long while to complete, perhaps days, but at least I don't have to do anything by hand. The bottleneck is now the fact that I have to use an external hard drive since I can't fit this all on my SSD.

    2 votes
  3. Comment on What programming/technical projects have you been working on? in ~comp

    pseudolobster
    Link Parent
    So, I've run into a bit of a hurdle on the data front. Yesterday I stumbled across some data that was too tantalizing to ignore, but it's proven to be moderately annoying to completely impossible...

    So, I've run into a bit of a hurdle on the data front. Yesterday I stumbled across some data that was too tantalizing to ignore, but it's proven to be moderately annoying to completely impossible to deal with. I hope by writing this out I'll figure out a solution, but I'm also looking for any advice anyone can give.

    I found a FTP server that has post-burn aerial imagery for most of the fires in BC last year, but they're in the form of enormous GeoTIFF files. The smallest fires are something like 2GB, but the ones I'm interested in are more like 32GB, split into 16x 2GB files. I can add these to QGIS directly, but naturally it grinds to a halt.

    My solution to this has been to make an image pyramid out of the GeoTIFF images, which provides lower resolutions for lower zoom levels as well as splitting everything into a bunch of smaller PNG tiles that compress better than TIFF. These tiles can then be served by mapproxy's WMS server that QGIS connects to.

    I've got this working, but it's been tedious. For a fire with 16x GeoTIFF images, I need to make 16 separate image pyramids, then 16 layers and cache sources in mapproxy's yaml config file, all with the correct bounding boxes.

    So, I've tried using gdal_merge to take all the giant 2GB GeoTIFF images and tile them together into one enormous one I can feed to gdal2tiles to create the pyramid. I did this on a smaller fire, it works, great! So now I can automate this, let's make a crummy bash script:

    #!/bin/bash
    gdal_merge.py -n 0 -a_nodata 0 -o g$1.tif /media/dingus/SEAGATE4TB/fire/imagery/$1/*
    mkdir $1
    gdal2tiles.py -p raster -s EPSG:3005 --xyz -z "0-7" --processes=12 -v -x $1.tif $1
    rm $1.tif
    echo "    "$1"_grid:
           base: GLOBAL_MERCATOR
           origin: ul
           srs: 'EPSG:3005'
           bbox_srs: 'EPSG:3005'
           bbox: `cat $1/openlayers.html |awk '/extent/ {print substr($2, 1, length($2)-1)}'`
           res: [128,64,32,16,8,4,2,1]
    ">>~/fire/mymapproxy/mapproxy.yaml
    

    Aaaand it crashes immediately as it tries to allocate 32gb of ram to create the merged GeoTIFF. I only have 16gb in this laptop, so the oom killer kicks in and kills my process. I don't know what to do except create a big swap partition and let it thrash my ssd. I'm not really sure how to deal with such enormous images, let alone the hundreds of them I'm going to have to deal with if I want all this imagery on my map.

    Bleh. I think I'll go for a walk and try and think about something else for a bit.

    3 votes
  4. Comment on What programming/technical projects have you been working on? in ~comp

    pseudolobster
    Link
    I've been dabbling with GIS stuff lately, and it turns out to be a pretty fun rabbit hole to explore. I'm planning on picking morel mushrooms this spring. It turns out to be insanely lucrative....

    I've been dabbling with GIS stuff lately, and it turns out to be a pretty fun rabbit hole to explore.

    I'm planning on picking morel mushrooms this spring. It turns out to be insanely lucrative. Last year I was able to pick 5lbs+ per hour, at a price of $25/lb. Essentially I'm getting a senior IT wage to hike through forests and climb mountains in some of the most beautiful wilderness in the world, and that was just my first year with no experience. Some people walk away from this with 6 figures cash in a season of picking.

    Morels grow like crazy in places where there was a forest fire the previous year. Last year, British Columbia had an insane fire season, with 465 wildfires covering 869688 hectares. That's an area almost the size of Puerto Rico, or more than 3x the size of Hong Kong. Here's what that looks like.

    Finding morels involves knowing where these fires are, what sort of geography you're dealing with, elevation, temperature, precipitation, tree types, soil drainage, solar exposure, etc. Mushroom hunters are traditionally a very secretive bunch, and their knowledge is passed on through word of mouth. I think I've found an edge though; I can use my geek powers to download vast amounts of data and bring those into the forest with me.

    So I've started making a map. A very detailed map. I've downloaded more than 50GB of datasets from a dozen different sources, covering everything from mean spring temperature to tree species to grizzly bear habitats. I have topographic data down to 5 metre resolution, aerial imagery from 4 different sources, down to 0.5m resolution in some cases. I've set up a caching proxy server called mapproxy so that all the normally online data is stored on my laptop for when I'm in the forest without internet.

    This whole project has turned out to be a much deeper rabbit hole than I expected. It's tested my skills in python, sql, graphic design, and a number of other fields. It's been really rewarding, too. It was a neat moment when some of this clicked for me. Like for example, elevation data is distributed by the BC government as a giant 500MB .TIF image. It's just an image file showing a greyscale picture of the province, but each grey level corresponds directly to an elevation in metres, so that can be used to generate accurate topographic contour lines or hillshading. Climate data is the same, and you can use vector analysis to calculate the average temperature within a polygon and things like that. I've been able to overlay that info onto the map for each fire, like so.

    It's just astounding how much data there is here, and it's been a lot of fun digging into it. OpenStreetMaps data for instance, shows things like individual park benches, and metadata about whether or not they have a backrest. I was able to download data about grizzly bear habitats, overlay that with the fire outlines, calculate area times population density for the overlapping areas, and output that as a skull and crossbones on the map showing estimated number of grizzly bears in that area. Here's what that looks like. I also got fairly depressed by some of this data, like the sheer amount of deforestation that happens in this province. Before, and after. It's really disheartening to see exactly how little old-growth we have left. There's virtually nowhere in this province that hasn't been logged or mined already.

    Anyway, I've been working on this for a week and I've gone from zero GIS knowledge to the Dunning–Kruger point where I realize how far in over my head I am. I'm learning shitloads about a topic I previously had no interest in, and it's absolutely fascinating.

    10 votes
  5. Comment on Steam Deck launch day megathread in ~games

    pseudolobster
    Link Parent
    Where on the site do you find these lists? Clicking your links give me "Due to Algolia limitations, we do not support currency switching in instant search", which is annoying since I'm not trying...

    Where on the site do you find these lists?

    Clicking your links give me "Due to Algolia limitations, we do not support currency switching in instant search", which is annoying since I'm not trying to switch currencies, in fact I really don't care about the prices, I just want to see the list of games.

    1 vote
  6. Comment on Fixing GRUB / Bad Kernels in ~comp

    pseudolobster
    (edited )
    Link
    This is usually caused grub not being able to find the initramfs. Oddly though, it found the kernel, which is typically on the same partition. In grub's configuration, you can specify which...

    This is usually caused grub not being able to find the initramfs. Oddly though, it found the kernel, which is typically on the same partition.

    In grub's configuration, you can specify which disk/partition houses your kernel and initramfs. Typically these days this is specified by a UUID which should never change, but older configs used to use the path to the device, eg: root=/dev/sda for the disk plugged into the first SATA port etc. If this is how it's configured then rearranging drives can cause sda to become sdb for example. At first I thought this is probably what happened, since you say the only difference was to add new drives, but if I'm reading this correctly it sounds like grub was able to load the kernel, but not the initramfs, which are, as I mentioned, typically on the same partition.

    Another possibility is the drives you added are a red herring, and your problem was actually caused by an update. Whenever you update your kernel, kernel modules, etc, it tries to generate a new initramfs and copy it to your /boot/ partition. Often times the /boot/ partition is really small, fills up, and fails to copy the initramfs. You probably would have seen an error if this happened, but it's a possibility.

    I'd probably start by just unplugging the drives you added. See if it boots fine without them. If it does, grub is freaking out that the order of your drives is different, and it's looking for a device by its location rather than its UUID. I fit doesn't, then it's an unrelated problem, possibly your /boot/ partition is full or for some other reason your initramfs got borked.

    In any case, if you need to reinstall grub, you'll want to generate the initramfs beforehand, so follow this one: https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/48516#48516

    4 votes
  7. Comment on Wordle | Game of words in ~games

    pseudolobster
    Link Parent
    I believe Wordle came first, and the imitators were mostly created to get around the 1-word-per-day limit. Here's a really interesting backstory about it in the New York Times:...

    I believe Wordle came first, and the imitators were mostly created to get around the 1-word-per-day limit.

    Here's a really interesting backstory about it in the New York Times: https://www.nytimes.com/2022/01/03/technology/wordle-word-game-creator.html

    It was created by former reddit admin /u/powerlanguage of /r/place fame as a gift to his wife. He apparently didn't expect it to go viral like this.

    3 votes
  8. Comment on Playing GTA5 on the original Game Boy in ~games

    pseudolobster
    Link
    Similar project: Reverse emulating the NES to give it SUPER POWERS Definitely not the same build quality and the overall result isn't as good, but everything runs off a raspberry pi inside the...

    Similar project: Reverse emulating the NES to give it SUPER POWERS

    Definitely not the same build quality and the overall result isn't as good, but everything runs off a raspberry pi inside the cartridge instead of streaming video from a playstation, which is more impressive in some ways. The making-of video goes into more technical details of how he pulled off such a janky hack.

    5 votes
  9. Comment on What's something that is, surprisingly, made with animal products? in ~enviro

    pseudolobster
    (edited )
    Link
    Vitamin D3 is derived from lanolin, which is extracted from wool. It's a pretty common ingredient in multivitamins. Another one that's really surprising to hear, but is really uncommon is some...

    Vitamin D3 is derived from lanolin, which is extracted from wool. It's a pretty common ingredient in multivitamins.

    Another one that's really surprising to hear, but is really uncommon is some artificial vanilla extracts are made from castoreum, which is extracted from anal glands of beavers.

    5 votes
  10. Comment on Making transparent wood in ~science

  11. Comment on What did you do this week? in ~talk

    pseudolobster
    Link
    I finally put a fitted sheet on a mattress correctly on the first try. I'm 38. I feel like I've been flipping a coin that's always come up tails for my entire life.

    I finally put a fitted sheet on a mattress correctly on the first try. I'm 38. I feel like I've been flipping a coin that's always come up tails for my entire life.

    7 votes
  12. Comment on <deleted topic> in ~music

    pseudolobster
    Link
    This digs up memories for me. I remember saying "If it's not love than it's 'da bum, da bum, da bum' that keeps us together'" with my first love. That must have been twenty years ago.

    This digs up memories for me. I remember saying "If it's not love than it's 'da bum, da bum, da bum' that keeps us together'" with my first love. That must have been twenty years ago.

    4 votes
  13. Comment on Microsoft reducing Windows store cut to just twelve percent in ~games

    pseudolobster
    Link
    The big thing I learned from this article was actually news from a couple years ago. I'm not sure how I missed hearing about this. A lot of the issues I had with the store were really issues with...

    The big thing I learned from this article was actually news from a couple years ago.

    Microsoft finally started supporting traditional win32 games in its store a couple of years ago (link), but this change alone hasn’t helped the Windows store compete with Steam.

    I'm not sure how I missed hearing about this. A lot of the issues I had with the store were really issues with UWP apps. It's probably worth another look, especially if the lower cut ends up translating to lower prices. I'm still opposed to linking my windows login to a Microsoft account though, and tbh I normally use Win10 LTSC, where installing Store is a giant PITA.

    Anyway, this is a big move to try and draw people to their store. I have to wonder if this is some desperate last-ditch all-or-nothing play in the last of Store's death throes. Aside from Forza and Office, I can't recall ever meeting anyone who bought anything from the Windows Store. It's funny to think now about how Valve ended up creating their own OS and encouraged a dozen PC manufacturers to make consolized PCs running SteamOS because they were so worried about the Windows Store being overwhelmingly successful.

    4 votes
  14. Comment on What does analog have that digital doesn't? in ~talk

    pseudolobster
    Link Parent
    I do get that, and I suppose it's the same as reading physical books versus e-readers, but to me the loss of fidelity, the finicky, fragile nature of the media, needles, etc is in no way worth it....

    I do get that, and I suppose it's the same as reading physical books versus e-readers, but to me the loss of fidelity, the finicky, fragile nature of the media, needles, etc is in no way worth it.

    It's just, when cassettes became available I ditched my 8-tracks and records. When I found a fancy Technics cassette player at a garage sale in the late 80's that had auto-reverse and could automatically fast-forward to the next track I could never use another tape player again. I did enjoy interacting with these old clunky machines, but I feel like the formats of the day did interfere with my enjoyment of the music.

    9 votes
  15. Comment on What does analog have that digital doesn't? in ~talk

    pseudolobster
    Link
    I have no nostalgia for analog media. As a media format, vinyl records are inferior to pretty much everything, and I can't wrap my head around the modern hipster LP scene at all. Cassettes are...

    I have no nostalgia for analog media. As a media format, vinyl records are inferior to pretty much everything, and I can't wrap my head around the modern hipster LP scene at all. Cassettes are making a bit of a comeback, and likewise I can't understand the appeal.

    Books, I get. It's a tactile thing, there's a smell to old books, a certain weight to them, etc. For me an e-reader is close enough, but I get that some people can't replace the physical object.

    What I actually miss most is physical buttons, switches, dials, and sliding potentiometers. I miss being able to control things without looking, just by feel. I miss tactile feedback where at least I can bottom out the pot to know where the end of travel is instead of dealing with rotary encoders or worse yet, touch controls.

    I fully support any future hipster movements that try and bring back tactile controls in lieu of touchscreens. If any of your friends are trying to bring back 8-tracks or something, remind them how cool analog, tactile buttons and switches and potentiometers used to be. Try and make this a movement. Call it the "Kinetic Movement" or something. Tell people that touching switches grounds you or something. I'll bet we can make it a thing.

    34 votes
  16. Comment on Recommend a piece of fiction that gives a specific feeling, regardless of genre or medium in ~talk

    pseudolobster
    Link Parent
    David Lynch is really good for that, imo. Blue Velvet (1986) in particular really touched that nerve for me. It's set in this quant picturesque 1960's small town, but as it goes on you start to...

    David Lynch is really good for that, imo. Blue Velvet (1986) in particular really touched that nerve for me. It's set in this quant picturesque 1960's small town, but as it goes on you start to see the seedy underbelly that was just below the surface the whole time. I found it deeply unsettling, and I ended up wishing I didn't smoke a joint before watching it.

    6 votes
  17. Comment on Song suggestions about: Cars, driving, trucking, racing, police chases, etc in ~music

    pseudolobster
    (edited )
    Link
    Gary Newman - Cars bonus: Fear Factory Remix, and bonuser: Live with NIN White Zombie - Black Sunshine Johnny Cash - General Lee Chuck Barry - Route 66 Carpenter Brut - Turbo Killer doesn't really...

    Gary Newman - Cars bonus: Fear Factory Remix, and bonuser: Live with NIN

    White Zombie - Black Sunshine

    Johnny Cash - General Lee

    Chuck Barry - Route 66

    Carpenter Brut - Turbo Killer doesn't really have any lyrics, but it does have a music video, which isn't even really about cars, but it's pretty awesome, and has cars in it.

    While we're on that aesthetic, the Drive Soundtrack isn't really about driving, but it's great music to drive to.

    edit: Bonus: Tracy Chapman - Fast Car

    2 votes
  18. Comment on SCOTUS sides with Google over Oracle in ~tech

    pseudolobster
    Link Parent
    Could you explain this a bit more? I'm not sure I understand. The API itself is nothing but a list of function names and expected behaviour from those functions, no? The implementation of an API...

    Could you explain this a bit more? I'm not sure I understand. The API itself is nothing but a list of function names and expected behaviour from those functions, no? The implementation of an API is kinda a no-brainer as far as being copyrightable, but I thought it was the literal names of the functions that was in doubt.

    As far as I understand it, google copied 14,000 lines of code declaring the names of those functions, and wrote the other 2mil+ lines of code themselves, which implement those functions. Taking 14,000 lines of code verbatim should be a no-brainer copyright infringement, but in this case it's considered fair use since it's just the names of functions, not how they work.

    I thought the real danger here was Oracle being able to say they own the names of variables and functions, and implementing your own String.toUpper() is copyright infringement. Even if your code is written from scratch without looking at Oracle's, the declaration itself would be infringing.

    3 votes
  19. Comment on This blog is now hosted on a GPS/LTE modem in ~comp

    pseudolobster
    Link Parent
    Haha, it's like you read my mind. I've been watching a lot of Adrain's Digital Basement on youtube lately, so oldschool hardware has been on my mind recently. Shortly after writing this I actually...

    Haha, it's like you read my mind. I've been watching a lot of Adrain's Digital Basement on youtube lately, so oldschool hardware has been on my mind recently. Shortly after writing this I actually thought about the 1541 and how it technically had more processing power than the C64 itself and ran its own OS, and thought about how I'd edit that into my comment. I ended up saying "fuck it" and hoped someone would come along mentioning that.

    2 votes