9 votes

What programming/technical projects have you been working on?

This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?

14 comments

  1. skybrian
    Link
    I'm still working on my "repeatTest" property-testing framework. No new features, but I managed to unify two algorithms for tree search and delete a lot of code. I'm a lot happier with how it...

    I'm still working on my "repeatTest" property-testing framework. No new features, but I managed to unify two algorithms for tree search and delete a lot of code. I'm a lot happier with how it reads and more confident that it's correct.

    8 votes
  2. [3]
    xk3
    Link
    I put together a little script to help run LLMs over multiple files. It's all offline. No API keys needed. So far I've used it to suggest names for PDFs with poor names: library llm-map --rename...

    I put together a little script to help run LLMs over multiple files.

    It's all offline. No API keys needed. So far I've used it to suggest names for PDFs with poor names:

    library llm-map  --rename --text ./gemma2.llamafile ~/Downloads/*.pdf
    
    cat llm_map_renames.csv
    Path,Output
    /home/xk/Downloads/booka.pdf,/home/xk/Downloads/Mining_Massive_Datasets.pdf
    ...
    

    This model works well but it is 20GB and takes about 20s per file:

    wget https://huggingface.co/jartine/gemma-2-27b-it-llamafile/resolve/main/gemma-2-27b-it.Q6_K.llamafile
    chmod +x gemma-2-27b-it.Q6_K.llamafile
    ./gemma-2-27b-it.Q6_K.llamafile
    

    Gemini Nano might be another interesting one to play with

    I'm curious what other use-cases people could have with something like this? The only other thing I can think of right now I can only think of text summarization

    7 votes
    1. [2]
      brogeroni
      Link Parent
      Not sure if it's within scope but it'd be cool to run Microsoft Florence 2 on a bunch of files, so I can ask questions about a bunch of images.

      Not sure if it's within scope but it'd be cool to run Microsoft Florence 2 on a bunch of files, so I can ask questions about a bunch of images.

      3 votes
      1. xk3
        Link Parent
        That should definitely be possible if you can find a llamafile or GGUF version of Florence2... llamafile is easy to support and likely the most future-proof You can run a GGUF file with this: wget...

        That should definitely be possible if you can find a llamafile or GGUF version of Florence2...

        llamafile is easy to support and likely the most future-proof

        You can run a GGUF file with this:

        wget https://github.com/Mozilla-Ocho/llamafile/releases/download/0.8.9/llamafile-0.8.9
        chmod +x ~/Downloads/llamafile-0.8.9
        mv ~/Downloads/llamafile-0.8.9 ~/.local/bin/llamafile  # move it somewhere in your $PATH
        
        pip install xklb
        library llm-map --model ~/Downloads/llava-v1.5-7b-Q4_K.gguf --image-model ~/Downloads/llava-v1.5-7b-mmproj-Q4_0.gguf --prompt 'what do you see?' ~/Downloads/comp_*.jpg
        Saved results to llm_map_what do you see.csv
        

        This is the CSV that is generated:

        845 376s lb:/ (main|✔) 🌟 cat llm_map_what\ do\ you\ see.csv
        path,output
        /home/xk/Downloads/comp_a.jpg,"The image displays a collection of different drawings and paintings, all of which feature various scenes related to mountains and the ocean. These pictures vary in size and style, providing an interesting visual experience. Some of the images portray landscapes with mountains and trees, while others showcase the ocean and its surroundings. Overall, this compilation of artwork highlights the beauty of both nature and the artists' ability to capture and convey these captivating scenes."
        /home/xk/Downloads/comp_b.jpg,"The image is a book that features several different scenes of landscapes, including trees, hills, and water. The book also showcases the various letters of the alphabet, with each letter representing a different scene. For instance, the letter S displays a picture of a lake, while the letter O displays a picture of a mountain. Overall, the book is a creative and educational way to present the alphabet, incorporating both art and language.
         The image is a collection of black and white drawings or cartoons, possibly a comic strip or a series of illustrations. The drawings are arranged in a grid-like pattern, with each image taking up a square or rectangle section. The drawings vary in size and complexity, showcasing different subjects, such as people, animals, and trees.
        
        Some of the images include a person in a chair, a person walking on a field, and a group of people in a boat. There is also a large tree among the images. The overall composition of the drawings creates a visually engaging and diverse collection."
        /home/xk/Downloads/comp_c.jpg,"The image is a picture of a book, possibly a book of illustrations, with a focus on trees. The book is open to a page, and several trees are visible throughout the illustrations. The trees appear in various sizes and shapes, showing different perspectives and angles, providing a diverse representation of the subject matter. The scene captures the beauty and detail of the trees within the illustrations.
         The image is a collection of drawings, possibly depicting different types of lines or objects. There are 11 individual drawings, each showcasing different subjects and styles. Some of the subjects include a triangle, a cross, a circle, a square, and a letter. The drawings are presented in a square format, showcasing the variety of subjects and their arrangement within the collection.
         The image features a collection of 15 drawings of boats on water, arranged in a grid pattern. Each drawing captures a unique perspective and composition, showcasing different types of boats and water scenes. The artwork is likely a study of various elements of boat designs and the beauty of being on the water. The arrangement of these drawings is visually appealing and captures the essence of the subject matter."
        /home/xk/Downloads/comp_d.jpg,"The image is an open book with numerous artwork drawings of different sizes and shapes, showcasing various artistic interpretations of the letter ""S."" The drawings are spread across the pages, with some closer to the edges and others in the center of the page. There are at least 14 drawings of the letter ""S"" in different forms, providing a diverse collection of interpretations of the letter. The drawings are displayed in a visually engaging manner, likely for an art project or art exhibition.
         This is a collection of pictures or illustrations, possibly an old book or a comic book, showcasing various themes and subjects. The images include a mix of landscapes, people, and trees. There are also a couple of scenes with the number ""27"" written, which could be related to a specific theme or chapter within the collection. The illustrations vary in size and design, giving the collection a diverse and artistic appearance. Overall, the images display a range of different subjects and artistic styles.
         The image features an assortment of scenes and artwork displayed on a page. There are various drawings and paintings, including boats, sailing, and an island. The collection consists of 14 different images, each showcasing different artistic perspectives and styles. Some of the images depict a rock in the ocean, while others showcase sailboats, landscapes, and a group of people on a ship. The collection presents a diverse range of artwork, capturing different moods and emotions.
         The image features a page from a book, showcasing several illustrations of picturesque scenes, including landscapes, mountains, and other intricate artworks. The images are accompanied by a variety of descriptions, likely providing further context or information about each depicted scene. The artwork within the book appears to be drawn in black and white, adding to the nostalgic charm of the composition."
        

        Some images take 20s while others take 100s or more. Not sure why there is such a big difference. The speed also depends on the prompt. You can limit the number of tokens it generates with --custom-args='-n 20' (20 words)

        2 votes
  3. [4]
    Zorind
    Link
    Currently trying to start up a blog. I’ve owned a domain for about 3-5 years now and never had anything on it, and I figure a blog is a good enough place to start. Especially with the IndieWeb...

    Currently trying to start up a blog. I’ve owned a domain for about 3-5 years now and never had anything on it, and I figure a blog is a good enough place to start. Especially with the IndieWeb resurgence.

    Planning on using 11ty I think, though I’m a bit unsure what the best way to get started with templates is/if there are benefits to using one of the templating languages. (I followed a basic tutorial that just uses the .11ty.js for generating the html).

    6 votes
    1. [2]
      666
      Link Parent
      I know the question was about programming/technical projects but what is your main goal with this? Do you want to start a blog or do you want to use the blog as an excuse (maybe not the right...

      I know the question was about programming/technical projects but what is your main goal with this?

      Do you want to start a blog or do you want to use the blog as an excuse (maybe not the right word) to learn a specific technology? I have done both in the past and they are both interesting experiences but if this is your first time it's important to focus on just one of them if you want to achieve anything.

      If the main focus is starting a blog and creating content I suggest you use something that doesn't get in the way of your writing and posting, otherwise you end up spending a lot of time learning commands, reading docs and tweaking parameters until you eventually notice that you haven't been writing at all. If your goal is to learn those commands, docs, utilities, markup languages, etc then it's fine to choose a more "attention-demanding" blogging technology.

      In my opinion the best way to start a blog is to choose a platform/technology that lets you do just two things: write and publish, and that does them in the simplest and least obtrusive way possible (if publishing is anything more complex than the push of a button it's probably too much and not worth it). With practice and experience you can also eventually turn those more attention-demanding technologies into something that is as simple as the push of a button but I suggest leaving that for later unless you already have experience maintaining a blog and creating content.

      9 votes
      1. Zorind
        Link Parent
        Definitely agree with you about choosing something similar if my goal is to actually simply blog. But my “goal” is really the tinkering around with the website part more than the blogging part. So...

        Definitely agree with you about choosing something similar if my goal is to actually simply blog.

        But my “goal” is really the tinkering around with the website part more than the blogging part. So you’re right on about the blog being more of an excuse to tinker around with the technology. (I work in software, but currently doing more back-end work at my job, but I like front-end more. I’m not really doing this for job-experience either, just doing it before the tinkering is fun).

        3 votes
    2. mordae
      Link Parent
      I suggest you write your first post first. Then publish it by converting it with Pandoc from markdown to HTML with a plain index page linking to it. Then write your second post. Then implement a...

      I suggest you write your first post first. Then publish it by converting it with Pandoc from markdown to HTML with a plain index page linking to it. Then write your second post. Then implement a blog system. Also, your first posts shouldn't be about your blog.

      2 votes
  4. karsaroth
    Link
    Its been a while since I posted, but I've spent a bit of time working on some terrain generation code, which has turned into a Minecraft mod that I'm tinkering with. It's almost good to use, but a...

    Its been a while since I posted, but I've spent a bit of time working on some terrain generation code, which has turned into a Minecraft mod that I'm tinkering with.

    It's almost good to use, but a few remaining things like handing certain generation edge cases need to be dealt with, and adding a little UI data while the terrain is generating.

    4 votes
  5. Eji1700
    Link
    So help me God I will understand azure functions for F#

    So help me God I will understand azure functions for F#

    1 vote
  6. Liru
    Link
    Haven't been doing much coding lately, but I recently decided that I should make a plugin to auto-repay large loans in OpenRCT2, since I like watching big numbers go down. So I did just that. The...

    Haven't been doing much coding lately, but I recently decided that I should make a plugin to auto-repay large loans in OpenRCT2, since I like watching big numbers go down.

    So I did just that.

    The code is mostly quick and awful, but it does what I need it to. I'm happy enough with the result, especially considering that I haven't done Typescript stuff in forever.

    1 vote
  7. RheingoldRiver
    (edited )
    Link
    Wrote a post about a small thing in MediaWiki software that people mess up all the time because no one knows how it works (but it's actually pretty powerful and very nicely designed)

    Wrote a post about a small thing in MediaWiki software that people mess up all the time because no one knows how it works (but it's actually pretty powerful and very nicely designed)

    1 vote
  8. calla
    Link
    Long story short, I'm writing a program for Linux that will let you script any input device with lua. Long story long, you pass one or more devices evdev devices to the program. The program grabs...

    Long story short, I'm writing a program for Linux that will let you script any input device with lua.

    Long story long, you pass one or more devices evdev devices to the program. The program grabs the devices, and it will forward all events along with the device info for each event to a callback you implement in the lua config. It creates a virtual keyboard, mouse, and gamepad (I will perhaps explore other devices in the future and/or the ability for users to define custom virtual devices from their config) and from the lua script you can send events to these virtual devices which other applications will recieve the same as they would a physical device. So at the simplest, you can use it to remap devices such as remapping a gamepad to keyboard inputs for example, or you could use it to do things like consolidate multiple physical devices into a single virtual device. The ceiling for what you could do is limited only by your imagination (and willingness to actually implement it), layers, modes, context-sensitive inputs (e.g. "if you input X with your browser focused, perform action Y"), and whatever else you might want.

    The initial idea was just for remapping gamepads with more complex features because I get frustrated by the limitations of even the more advanced systems for remapping like Steam's input system. Then I realized that all input devices go through the same kernel interface (there's even an evdev device for my power button so I may use that as a silly demo when I publish the project) so supporting other devices was not only easy but completely effortless.

    Multiple devices at a time was just a random idea I had while working on it and I was initially not going to do it but I realized it would be make the program potentially much better as an accessibility tool, although I'm sure the overlap between "benefits from additional accessibility tools for games and such" and "wants to program their input device behavior with lua scripts" is probably pretty small, but I still hope it can be useful for people in that regard.

    1 vote
  9. tomf
    (edited )
    Link
    I was thinking that I need a way to quickly build spotify playlists. I asked chatGPT 4.x for some help. I provide tracks.txt -- the first line is the name of the playlist and the rest is artist -...

    I was thinking that I need a way to quickly build spotify playlists. I asked chatGPT 4.x for some help.

    • I provide tracks.txt -- the first line is the name of the playlist and the rest is artist - title
    • based on the playlist name, it'll either create a playlist or update an existing one
    • Its using the API with spotipy which has oAuth.
    • it doesn't add duplicates
    • It shows the progress. Not the prettiest thing, but it serves its purpose.
    Summary Report:
    Total tracks processed: 61
    Tracks added: 49
    Tracks skipped (already in playlist): 5
    Tracks not found: 7
    Tracks not found:
      - Les Baxter - Quiet Village Bossa Nova
      - Monster Rally - Coral
      - Lee Fields & The Expressions - Money Is King
    Playlist 'Dinner Jazz by chatGPT' updated with 49 new tracks.
    

    So that all works like a charm. I can do about 75 - 100 tracks without any issue. For the tracks themselves, I'm also asking chatGPT

    Can you do the same for various genres of jazz including lounge, exotica, space-age pop, etc. Include artists like Martin Denny, Les Baxter, but also new bands like Khruangbin and Monster Rally. You can also include soul from Colemine Records like Aaron Frazer, Lee Fields, Durand Jones.

    In this case it spits out the bands I referenced, so I ask it to branch out. The selection is pretty great.

    It isn't perfect with apostrophes all the time, but for the most part its great.

    Here's an sample list * Martin Denny - Quiet Village * Les Baxter - Jungle Flower * Khruangbin - Maria También * Monster Rally - Orchids * Aaron Frazer - Bad News * Lee Fields & The Expressions - It Rains Love * Durand Jones & The Indications - Don't You Know * Yma Sumac - Gopher Mambo * Esquivel - Mini Skirt * Les Baxter - The Enchanted Sea * Dorothy Ashby - Soul Vibrations * The Five Corners Quintet - Trading Eights * Les Baxter - Atlantis * Cal Tjader - Soul Sauce * Astrud Gilberto - The Girl from Ipanema * Sergio Mendes & Brasil '66 - Mas Que Nada * Walter Wanderley - Summer Samba (So Nice) * Antonio Carlos Jobim - Wave * Stan Getz & João Gilberto - Corcovado (Quiet Nights of Quiet Stars) * Martin Denny - Aku Aku * Les Baxter - Quiet Village Bossa Nova * Thievery Corporation - Lebanese Blonde * Bebel Gilberto - Aganju * Jazzanova - Bohemian Sunset * Bonobo - Days to Come (feat. Bajka) * Khruangbin - Evan Finds the Third Room * Monster Rally - Pelicans * St. Germain - Rose Rouge * Bonobo - Kiara * Koop - Summer Sun * Aaron Frazer - If I Got It (Your Love Brought It) * Quantic - Cumbia Sobre el Mar * Les Baxter - The Poor People of Paris * Gotan Project - Santa Maria (Del Buen Ayre) * Thievery Corporation - The Richest Man in Babylon * Khruangbin - Two Fish and an Elephant * Caravan Palace - Lone Digger * Parov Stelar - Catgroove * Tape Five - A Cool Cat in Town (feat. Brenda Boykin) * Caro Emerald - Back It Up * Martin Denny - Firecracker * Les Baxter - Rio * Delvon Lamarr Organ Trio - Close But No Cigar * The Haggis Horns - Hot Damn! * The Bamboos - On the Sly * Orgone - I Sold My Heart to the Junkman * The New Mastersounds - Miracles * Durand Jones & The Indications - Long Way Home * Lee Fields & The Expressions - You're What's Needed in My Life * Sharon Jones & The Dap-Kings - 100 Days, 100 Nights
    The script!
    import spotipy
    from spotipy.oauth2 import SpotifyOAuth
    import time
    import sys
    
    # Spotify API credentials
    client_id = 'xxx'
    client_secret = 'xxx'
    redirect_uri = 'http://localhost:8888/callback'
    
    # Authenticate with Spotify
    scope = 'playlist-modify-public'
    sp = spotipy.Spotify(auth_manager=SpotifyOAuth(client_id=client_id,
                                                   client_secret=client_secret,
                                                   redirect_uri=redirect_uri,
                                                   scope=scope))
    
    # Read tracks from the file
    with open('tracks.txt', 'r') as f:
        lines = f.readlines()
        playlist_name = lines[0].strip()
        tracks = [line.strip() for line in lines[1:]]
    
    # Function to search for a track and return its URI
    def search_track(sp, artist, title):
        query = f'artist:{artist} track:{title}'
        results = sp.search(q=query, type='track', limit=1)
        tracks = results['tracks']['items']
        if tracks:
            return tracks[0]['uri']
        else:
            return None
    
    # Check if the playlist already exists
    user_id = sp.current_user()['id']
    playlists = sp.current_user_playlists(limit=50)['items']
    existing_playlist = None
    
    for playlist in playlists:
        if playlist['name'] == playlist_name:
            existing_playlist = playlist
            break
    
    # Use the existing playlist or create a new one
    if existing_playlist:
        playlist_id = existing_playlist['id']
        print(f"Updating existing playlist: '{playlist_name}'")
        # Get the existing track URIs in the playlist
        existing_tracks = sp.playlist_tracks(playlist_id, fields='items(track(uri))')['items']
        existing_track_uris = {item['track']['uri'] for item in existing_tracks}
    else:
        playlist = sp.user_playlist_create(user=user_id, name=playlist_name, description="")
        playlist_id = playlist['id']
        print(f"Creating new playlist: '{playlist_name}'")
        existing_track_uris = set()
    
    # Add tracks to the playlist with rate limiting and progress display
    track_uris_to_add = []
    skipped_tracks = 0
    not_found_tracks = []
    
    for i, track in enumerate(tracks):
        try:
            artist, title = track.split(' - ')
            uri = search_track(sp, artist, title)
            if uri:
                if uri not in existing_track_uris:
                    track_uris_to_add.append(uri)
                else:
                    skipped_tracks += 1
                    print(f"Track already in playlist: {artist} - {title}")
            else:
                not_found_tracks.append(track)
        except ValueError:
            print(f"Skipping malformed track entry: {track}")
            not_found_tracks.append(track)
    
        time.sleep(1)  # Rate limit to avoid hitting Spotify's API rate limits
        sys.stdout.write(f'\rProgress: {i+1}/{len(tracks)} tracks processed')
        sys.stdout.flush()
    
    if track_uris_to_add:
        sp.playlist_add_items(playlist_id=playlist_id, items=track_uris_to_add)
        print('\nAll new tracks added to the playlist.')
    else:
        print('\nNo new tracks were added to the playlist.')
    
    # Summary report
    print(f"\nSummary Report:")
    print(f"Total tracks processed: {len(tracks)}")
    print(f"Tracks added: {len(track_uris_to_add)}")
    print(f"Tracks skipped (already in playlist): {skipped_tracks}")
    print(f"Tracks not found: {len(not_found_tracks)}")
    if not_found_tracks:
        print("Tracks not found:")
        for track in not_found_tracks:
            print(f"  - {track}")
    
    print(f"Playlist '{playlist_name}' updated with {len(track_uris_to_add)} new tracks.")
    

    Ultimately, I found over the last year or so that Spotify's recommendations are a bit of an echo chamber and I've been trying to pump new music into the algo... with little success. This has a lot of the stuff I already listen to, but I think attacking it via genre instead of specific artists will yield fresh music.

    To quickly clear tracks.txt I use head -n 1 tracks.txt > temp.txt && mv temp.txt tracks.txt

    1 vote