Handshape's recent activity
-
Comment on Is it possible to easily finetune an LLM for free? in ~tech
-
Comment on MIT report: 95% of generative AI pilots at companies are failing in ~tech
Handshape LinkI'm up to my armpits in this space; the failure rates are almost always tied to IM/IT debt, and the proponents are being sold magic beans by their vendors. "Wait... If I buy your product, I don't...I'm up to my armpits in this space; the failure rates are almost always tied to IM/IT debt, and the proponents are being sold magic beans by their vendors.
"Wait... If I buy your product, I don't have to fix the gaps in my historical data?"
Contracts get booked, salesdroids get paid, implementations fail. Wash, rinse, repeat.
-
Comment on What games have you been playing, and what's your opinion on them? in ~games
Handshape LinkAfter a hiatus from PC gaming that lasted most of a decade, I picked up Elite Dangerous Odyssey (after rage quitting on console back when Frontier abandoned PlayStation). It took me only a couple...After a hiatus from PC gaming that lasted most of a decade, I picked up Elite Dangerous Odyssey (after rage quitting on console back when Frontier abandoned PlayStation).
It took me only a couple of hours for all the old reflexes to come back. All the new content and game mechanics have been a pleasant path of discovery. Exploring and cataloguing xenobiology particularly scratches the itch for me.
-
Comment on What are your AI-generated guilty pleasures? in ~tech
Handshape LinkI probably waste more time than I should trying to figure out if/how we can deploy generative AI even remotely ethically. I've got a decent stack built out of vendor-agnostic open source and open...I probably waste more time than I should trying to figure out if/how we can deploy generative AI even remotely ethically.
I've got a decent stack built out of vendor-agnostic open source and open models running 100% on gear that was saved from a landfill. I have code that hot-swaps models to apply the smallest viable one to each task. I use the waste heat to keep my office warm for half the year or so.
But the elephant in the room is that I am only addressing inference. The practices around training are unconscionable, and each time I pull a new model for quant scaling and fitness testing, I'm creating demand for someone else to train yet another energy-guzzling monster.
The guilty part: I have truckloads of hubris about what I've put together. I bristle with nerd pride when the generated persona running in my basement smiles as it reads my personalized daily briefing.
I'm never sure if using this as a hobby is because I'm trying to make things better, or if I'm trying to make me feel better.
-
Comment on What's something that makes you feel like we're living in the future? in ~talk
Handshape Link ParentIt is, but hard to link without doxxing myself.It is, but hard to link without doxxing myself.
-
Comment on What's something that makes you feel like we're living in the future? in ~talk
Handshape Link ParentI've got a software rig that scales up or down with available compute, and runs happily on anything with about 8GB of headroom, trading off speed. The real magic is that I can keep several models...I've got a software rig that scales up or down with available compute, and runs happily on anything with about 8GB of headroom, trading off speed. The real magic is that I can keep several models "hot" in conventional ram for use cases that use more than one.
My daily driver is a 1080Ti that I'm hoping to supplement with a 3080 once I get the power situation sorted.
-
Comment on What's something that makes you feel like we're living in the future? in ~talk
Handshape LinkI can run reasonable simulacra of human intelligences on second-hand gear in my basement, and it's not really that difficult. I've got it to the point that if I have an idea I want to try, I can...I can run reasonable simulacra of human intelligences on second-hand gear in my basement, and it's not really that difficult. I've got it to the point that if I have an idea I want to try, I can hand-code a working instance in a couple of hours.
-
Comment on What did you do this week (and weekend)? in ~talk
Handshape LinkCelebrated a milestone birthday with friends and family. Ate too much, drank more than I ought to, and enjoyed having my home filled with happy people.Celebrated a milestone birthday with friends and family. Ate too much, drank more than I ought to, and enjoyed having my home filled with happy people.
-
Comment on Do you have games that you play (almost) exclusively? in ~games
Handshape Link ParentI was gifted the roomscale VR remake of Riven, and the folks at Cyan did an excellent job of adapting the core ideas to the medium. Visiting some of the places was almost sublime - the temple near...I was gifted the roomscale VR remake of Riven, and the folks at Cyan did an excellent job of adapting the core ideas to the medium.
Visiting some of the places was almost sublime - the temple near the beginning was the moment where I had to stop playing and swim in a little pool of nostalgia. I'm certain that the designers anticipated that some of those "set pieces" were what would attract original players to that world.
The solutions to some of the puzzles were also revised, with almost a wink and a wag of the finger.
There is a spot where you anticipate one of the biggest such set pieces, and they judo-flip your expectations... and it was there that I actually started grinning under the hat.
It was that same sense of initial "WTF?" turning into "Oh! I understand!" that made the original game so good... and the folks at Cyan delivered.
-
Comment on Do you have games that you play (almost) exclusively? in ~games
Handshape LinkPretty much every game I play, I play solo, and to the exclusion of all other games. The ones that drive me are those that involve exploration, and in which I gain capacity in-game from what I...Pretty much every game I play, I play solo, and to the exclusion of all other games.
The ones that drive me are those that involve exploration, and in which I gain capacity in-game from what I learn as a player, not just as a character. The Outer Wilds and The Witness have been mentioned elsewhere in this thread - they're masterworks, but the first one to really scratch this itch for me was Riven back in '97.
I'm currently enjoying Pacific Drive. My first four hours were essentially me shouting "Aah! What? No! Gitoffmycaryouwierdfuckingthing!" at the screen, and my wife and son laughing at me in the background.
I'm approaching the endgame now, and all the strange experiences have accumulated to make navigating the anomalous realities feel like being a cross between Steve McQueen and Hunter S. Thompson. Good times.
-
Comment on Anyone using Meshtastic/LoRA radio? in ~hobbies
Handshape LinkThere are only two active nodes in my community, and I've been thinking about investing in a few to see if I can get a few local businesses to agree to host them. I'm interested in word from folks...There are only two active nodes in my community, and I've been thinking about investing in a few to see if I can get a few local businesses to agree to host them.
I'm interested in word from folks deploying them about which devices offer the best value for money.
-
Comment on Building a personal, private AI computer on a budget in ~comp
Handshape LinkBeen working hard on getting this right for use cases with financial, ethical, and security constraints. In my professional life, I've been working carefully on getting inference for as many users...Been working hard on getting this right for use cases with financial, ethical, and security constraints.
In my professional life, I've been working carefully on getting inference for as many users on as many models as possible squeezed down onto as little hardware as I can.
There are potent optimizations that can be done by keeping the models (and logits) "as hot as possible" on VRAM, and using smaller models for many requests.
The sweet spot is somewhere around 9b parameters, with the ends of the model at full precision, and the middles quantized down to about 4 bits. Each user gets something like 8GB of VRAM for just the tiny slices of time when inference is actually happening, and then the rest gets interleaved for other users.
I have no doubt that many large vendors have done this work as well, and are reaping profits by "selling lamb, and then serving mutton".
-
Comment on Building a personal, private AI computer on a budget in ~comp
Handshape Link ParentMy favourite foible with the DS R1 model and distils is that you can detect when you've tripped an ideological guardrail by looking for the empty <think> block. No think. Only repeat. Good bot. 🙈🙉🙊My favourite foible with the DS R1 model and distils is that you can detect when you've tripped an ideological guardrail by looking for the empty
<think>block.No think. Only repeat. Good bot. 🙈🙉🙊
-
Comment on What possession(s) do you have that continue to delight you every time? in ~talk
Handshape LinkThe one I miss the most is my Pebble 2 watch. It hit the perfect sweet spot for ergonomics and features. Sadly, a -40°C day did it in. The one I still use daily is my FingerWorks iGesture pad. 25...The one I miss the most is my Pebble 2 watch. It hit the perfect sweet spot for ergonomics and features. Sadly, a -40°C day did it in.
The one I still use daily is my FingerWorks iGesture pad. 25 year-old pointing device, and there's still nothing that comes close to the functionality packed into that little beast.
For non-tech stuff, as dumb and as cringe as it sounds, a wooden Japanese practice sword (bokken). When my university martial arts club bought a cheap set in bulk in the 90s, I got the last one. It's straight, but the wood it blotchy
-
Comment on What video games have had you taking real-life notes? in ~games
Handshape Link ParentMy father was a commercial artist. The scribblings he made while we played Riven as a family are an anchor to a treasured memory. There's a little part of me that likes to imagine that he's out...My father was a commercial artist. The scribblings he made while we played Riven as a family are an anchor to a treasured memory.
There's a little part of me that likes to imagine that he's out there among the Ages, scribbling drawings of impossible places in a little book.
-
Comment on Any Kult players? in ~games.tabletop
Handshape LinkKult... That's a system I'd not thought about in a long time. I played the 1st Ed in the early 90s. Made good use of the vibe around Tool's Undertow album. Ran a campaign that felt like a...Kult... That's a system I'd not thought about in a long time. I played the 1st Ed in the early 90s. Made good use of the vibe around Tool's Undertow album.
Ran a campaign that felt like a police-investigation detective rpg to begin with, and let the players peel back the layers of the world.
-
Comment on What programming/technical projects have you been working on? in ~comp
Handshape Link Parentllama.cpp for inference, plain old python for orchestration, nginx for TLS termination and traffic balancing, redis for distributed session caching. With some careful decisions about what to cache...llama.cpp for inference, plain old python for orchestration, nginx for TLS termination and traffic balancing, redis for distributed session caching.
With some careful decisions about what to cache and when, you can minimize the contention quite a bit.
-
Comment on What programming/technical projects have you been working on? in ~comp
Handshape LinkA scalable hosting service for secure LLM inference. Lots of fun trying to serve the greatest number of concurrent sessions for the greatest number of LLMs with no lost conversations, and the...A scalable hosting service for secure LLM inference. Lots of fun trying to serve the greatest number of concurrent sessions for the greatest number of LLMs with no lost conversations, and the fewest trips to cold storage possible, with session multiplexing among lots and lots of users.
I've got density way, way higher than what I'd ever have guessed possible. Enough that my whole org can run on two consumer GPUs in separate enclosures, each at about 50% average load during peak hours, and acting as hot failovers for one another. There's still room for optimization, but we're right at the limit of whether the proverbial juice is worth the squeeze.
Some of the most fun optimization work of my career.
-
Comment on I can't cry for some reason in ~health.mental
Handshape LinkVery much so. Really wish I could find a context in which any such outlet would be socially acceptable. I know exactly where my issue comes from: I'm not young by any reasonable definition, but I...Very much so. Really wish I could find a context in which any such outlet would be socially acceptable.
I know exactly where my issue comes from: I'm not young by any reasonable definition, but I became the oldest living man in my extended family pretty suddenly, about 7 years ago.
Between a heavy grief load, a brutal dispute with my employer, and suddenly having just about everyone in the family looking to me for support and stability, I had to lock shit down hard, and keep it there.
The moments when I cry only happen in private, and only when I'm blindsided. There have been exactly two in those seven years.
-
Comment on What are your cooking experiments that haven't turned out well? in ~food
Handshape LinkI've been exploring various permutations of bread making - leavened/unleavened/soda/sourdough, loaves/miches/flatbread/tack, and the wild and wooly universe of flours. I've made undercooked blobs,...I've been exploring various permutations of bread making - leavened/unleavened/soda/sourdough, loaves/miches/flatbread/tack, and the wild and wooly universe of flours. I've made undercooked blobs, and overcooked curling stones. Inadvisable rosemary loaves that taste of medicine.
Of all the disappointments, the worst was the time I made parmesan-flavoured corn tortillas. The colour was good, the smell was heavenly... but as I went to lift it out of the pan, it looked at me and said "Mr. Stark, I don't feel so good."
The good/fast/cheap trifecta is in play. It'll be important to know exactly what you want the difference in your fine-tune to look like.
Like SloMoMonday, I encourage you to look to a smaller open-weight model as your starting point.
The kind folks at Unsloth have done some amazing work making the process of running parameter-efficient fine tunes faster and less painful.
Perhaps most of all, I encourage you to look at easier, cheaper options like RAG before jumping all the way to fine tuning.