Even tech news moves a bit too fast for me to keep up. Did UMN ever get unbanned? I saw a half-hearted apology and then finally this [1], but never heard any update. Most recent article I've seen is this ZDNet article [2] from a couple of weeks ago discussing a related issue, but still mentions that UMN is still banned.

Anyone following this?

[1] https://cse.umn.edu/cs/statement-computer-science-engineering-confirming-linux-technical-advisory-board-findings-may-9

[2] https://www.zdnet.com/article/hard-work-and-poor-pay-stresses-out-open-source-maintainers/

This below is a summary of some real-world performance investigation I recently went through. The tools I used are installed on all linux systems, but I know some people don't know them and would straight up jump to heavyweight log analysis services and what not, or writing their own solution.

Let's say you have request log sampling in a bunch of log files that contain lines like these:

127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000
127.0.0.1 [2021-05-27 23:51:22.019] "GET /pl/player/123456/changelog/ HTTP/1.1" 200 16524 TLSv1.2 MISS-CLUSTER SessionID:(null) Cache:

You might recognize Fastly logs there (IP anonymized). Now, there's a lot you might care about in this log file, but in my case, I wanted to get a breakdown of hits vs misses by URL.

So, first step, let's concatenate all the log files with cat *.log > all.txt, so we can work off a single file.

Then, let's split the file in two: hits and misses. There are a few different values for them, the majority are covered by either HIT-CLUSTER or MISS-CLUSTER. We can do this by just grepping for them like so:

grep HIT-CLUSTER all.txt > hits.txt; grep MISS-CLUSTER all.txt > misses.txt

However, we only care about url and whether it's a hit or a miss. So let's clean up those hits and misses with cut. The way cut works, it takes a delimiter (-d) and cuts the input based on that; you then give it a range of "fields" (-f) that you want.

In our case, if we cut based on spaces, we end up with for example: 127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000.

We care about the 5th value only. So let's do: cut -d" " -f5 to get that. We will also sort the result, because future operations will require us to work on a sorted list of values.

cut -d" " -f5 hits.txt | sort > hits-sorted.txt; cut -d" " -f5 misses.txt | sort > misses-sorted.txt

Now we can start doing some neat stuff. wc (wordcount) is an awesome utility, it lets you count characters, words or lines very easily. wc -l counts lines in an input, since we're operating with one value per line we can easily count our hits and misses already:

$ wc -l hits-sorted.txt misses-sorted.txt
  132523 hits-sorted.txt
  220779 misses-sorted.txt
  353302 total

220779 / 132523 is a 1:1.66 ratio of hits to misses. That's not great…

Alright, now I'm also interested in how many unique URLs are hit versus missed. uniq tool deduplicates immediate sequences, so the input has to be sorted in order to deduplicate our entire file. We already did that. We can now count our urls with uniq < hits-sorted.txt | wc -l; uniq < misses-sorted.txt | wc -l. We get 49778 and 201178, respectively. It's to be expected that most of our cache misses would be in "rarer" urls; this gives us a 1:4 ratio of cached to uncached URL.

Let's say we want to dig down further into which URLs are most often hitting the cache, specifically. We can add -c to uniq in order to get a duplicate count in front of our URLs. To get the top ones at the top, we can then use sort, in reverse sort mode (-r), and it also needs to be numeric sort, not alphabetic (-n). head lets us get the top 10.

$ uniq -c < hits-sorted.txt | sort -nr | head
    815 /static/app/webfonts/fa-solid-900.woff2?d720146f1999
    793 /static/app/images/1.png
    786 /static/app/fonts/nunito-v9-latin-ext_latin-regular.woff2?d720146f1999
    760 /static/CACHE/js/output.cee5c4089626.js
    758 /static/images/crest/3/light/notfound.png
    757 /static/CACHE/css/output.4f2b59394c83.css
    756 /static/app/webfonts/fa-regular-400.woff2?d720146f1999
    754 /static/app/css/images/loading.gif?d720146f1999
    750 /static/app/css/images/prev.png?d720146f1999
    745 /static/app/css/images/next.png?d720146f1999

And same for misses:

$ uniq -c < misses-sorted.txt | sort -nr | head
     56 /
     14 /player/237678/
     13 /players/
     12 /teams/
     11 /players/top/
<snip>

So far this tells us static files are most often hit, and for misses it also tells us… something, but we can't quite track it down yet (and we won't, not in this post). We're not adjusting for how often the page is hit as a whole, this is still just high-level analysis.

One last thing I want to show you! Let's take everything we learned and analyze those URLs by prefix instead. We can cut our URLs again by slash with cut -d"/". If we want the first prefix, we can do -f1-2, or -f1-3 for the first two prefixes. Let's look!

cut -d'/' -f1-2 < hits-sorted.txt | uniq -c | sort -nr | head
 100189 /static
   5948 /es
   3069 /player
   2480 /fr
   2476 /es-mx
   2295 /pt-br
   2094 /tr
   1939 /it
   1692 /ru
   1626 /de

cut -d'/' -f1-2 < misses-sorted.txt | uniq -c | sort -nr | head
  66132 /static
  18578 /es
  17448 /player
  17064 /tr
  11379 /fr
   9624 /pt-br
   8730 /es-mx
   7993 /ru
   7689 /zh-hant
   7441 /it

This gives us hit-miss ratios by prefix. Neat, huh?

Last time I owned an inkjet was well over a decade ago. I had a nice HP color laserjet that Just Worked™for almost a decade (and PS, I bought it used), and then I just lived w/o a printer for the past 3-4 years. Now, I'm window-shopping for inkjets, it sounds like the whole "use-our-ink-or-die" business model has only gotten worse.

Are there any good inkjet printers where I can just use it like a normal printer, just buy ink (cheaper than the printer was) when I need it, yada? Or should I just write off the entire industry (again), and go straight to the laser printers?

And does anyone actually have a decent (color, all-in-one) printer that works reasonably well with their (YourDistroHere) Linux machine?

Danke

ETA: Thanks for all the feedback. I'm now prioritizing a Brother laser (maybe just mono), or possibly an Epson Ecotank.

Side-note ... how cool is it that we have so many Linux-folk in our midst!?

Thanks again.

I've put quite a bit of work into my i3 set up recently and I'm curious if the people here are interested in that kind of thing.

I'd be interested in looking through configs to get ideas, and sharing screenshots and such.

Here is what my desktop looks like right now. Let me know what you think.

I’m enough of a Linux lay user that I’m not even sure if I’m using the right terminology in the question (feel free to tweak it if needed!). Here’s what I mean:

I’m running Pop!_OS currently, and I have at least one app installed via each of the following methods:

• Deb app from the distro repositories
• Deb deb downloaded from program website
• Flatpak app downloaded from Flathub
• AppImage app downloaded from program website
• Snap app downloaded from the Snap store

As someone who doesn’t really know or necessarily even care to know what’s going on under the hood, these all pretty much work identically for me (with the exception of AppImage which doesn’t integrate into my regular programs menu, and the standalone Deb, which requires manual updating). In fact, for most of the programs on my computer I couldn’t tell you which one they’re sourced from. They all just run like they should.

I’ve looked up differences between all of the options and usually end up finding conversations that go well above my head and get deep into technical details. My question here is basically aimed at cutting through a lot of that depth: what is the important, need-to-know information about these different methods of installing apps? Is there anything I should be aware of if all I’m really going to be doing is running them as a standard, non-power user? Also, if an app is available via multiple methods — is there one that is preferred/better/safer/superior/etc.?

Hey Tildes!

I need some help with a specific tech issue, and I'm sure someone here can help me do it way quicker than I would be able to on my own.

General Request

I'd like to be able to scan a directory and find all of the largest files of a specific type (e.g. the largest .jpg files). I'm running Pop!_OS and I'm assuming there's some way to do this in the terminal, or alternately some utility I could use.

More Specific Details

I'm cleaning up my digital music library, and I realized in setting it up I made some errors by saving some very high res cover art. Many of my Bandcamp purchases come with a cover.jpg or cover.png file that is several megabytes large. I made the mistake of writing these into the files (adding, for some albums, an extra, say, 100 MB across all tracks). They also take a lot longer to load when I pull them up in my cloud music player. I'd like to be able to identify the albums with the largest cover.* files so that I can go in and replace the album art with a lower res version and gain back all that wasted space lost to unnecessary duplication.

I could go folder by folder and take a look at the sizes of each, but I figure there's an easier way to surface the ones that need my attention. Everything I've looked at online so far has helped me figure out how to identify the biggest files in general, but all that will do is surface the actual audio files, when it's the cover art that needs the specific attention.

Also, in case it's necessary information, the directory structure is Music/[artist]/[album]/cover.*

Any help will be very appreciated!

Ubuntu has this package installed by default:
network-manager-config-connectivity-ubuntu

It's only purpose is to provide settings for NetworkManager to send requests to connectivity-check.ubuntu.com , and based on the result (AFAIK) detect redirection by captive portals and open an ISP's page (think public WiFi, or hotel rooms, where you need to authorize to access the net).

Well, connectivity-check.ubuntu.com is hosted on Google cloud (you can check that by running:

dig connectivity-check.ubuntu.com
whois [the IP from previous query]

), so by default Ubuntu sends requests to a Google cloud page.
I don't say Google counts daily active Ubuntu users (because many of those have the same IP), or that Google actively logs and analyzes that data. But some of you guys may not like that behavior.

So what's the fix?

Purge the package

sudo apt purge network-manager-config-connectivity-ubuntu

If you do need a captive portal detection, create your own config file to query some HTTP (not HTTPS) page of your choice, in the example below I have a Debian page used for the same purpose. Use your favorite text editor to create and edit /etc/NetworkManager/conf.d/90-connectivity-custom.conf :

[connectivity]
uri=http://network-test.debian.org/nm

Restart NetworkManager

sudo systemctl restart NetworkManager

If you run an Ubuntu derivative, please report if you have network-manager-config-connectivity-ubuntu installed in the comments.

I still mainly use Windows, although I've dual-booted Linux a few times and I have Linux Mint on an old laptop right now. One thing I've never understood about Linux is all the different distributions - their different reputations and why they have them. What is the mechanical difference between using one distribution of Linux and another? Or are the differences usually not mechanical?

For example, Ubuntu and Debian seem to be large families, meaning that a lot of other distributions are based on them (using packages built for them in their package managers at least) as well as being popular distros on their own. But what's different between the two of them, and between each and the other distros based on them? (and what's similar? I gather they all use the Linux kernel at least!)

I also know that people are quite opinionated on their choice of distro, I wondered what reasons people had for their choice. What things are easier or harder for you in your distro of choice? Is it mainly day-to-day tasks that are important or more how the OS works underneath? How much difference does your preferred distro make?

For myself, I've only used Kubuntu (though not much) and Linux Mint, which was mainly for UI reasons, and particularly for the latter, ease of use for someone used to Windows (at least that was what I found years ago when I first looked into it).

Though I doubt I'll ever fully move away from Windows I would like / need to have access to a Linux OS, so maybe this will help me to know what is important to look for. But I also hope it'll be a useful and interesting discussion topic. Also, there are some previous discussions on the latter question so I'd be more interested in learning about the main topic.

also, please do add more tags

I've got a crappy Chromebook running GalliumOS (Xubuntu) and Chromium is slow as molasses. I tried a few other browsers like Otter and Falkon. They're alright for most sites -- not Tildes, but this seems consistent with QT5 browsers.

Anyway, outside of text browsers, anybody have any light weight browser suggestions?

My Linux desktop is having a bit of difficulty with bad sectors. Lately I've had to boot into recovery and run an fsck a few times to try to fix a problem where the OS drops into read-only mode at the drop of a hat. Today I tried copying some files from one directory to another and got the following error message:
cp: error reading "foo/bar": Input/output error

I've just booted into a live USB and run fsck /dev/sda1 -c and it fixed a load of bad sectors, but the above error message is still happening.

A bit of googling tells me that this is down to bad sectors on the SSD, and I'm not really sure what that means. Is anybody able to enlighten me? And as a follow-up question, would reformatting the hard drive resolve the problem, or are there any other things I can try to fix it?

Since I took so long to reply to Tips to use NixOS on a server? by @simao, I decided to create a new topic to share my configs. Hopefully this is informative for anyone looking to do similar things - I'll also gladly take critiques, since my setup is probably not perfect.

First, I will share the output of 'lsblk' on my VPS:

NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
vda       253:0    0   180G  0 disk  
├─vda1    253:1    0   512M  0 part  /boot
└─vda2    253:2    0 179.5G  0 part  
  └─crypt 254:0    0 179.5G  0 crypt 

That is, I use an unencrypted /boot partition, vda1, with GRUB 2 to prompt for a passphrase during boot, to unlock the LUKS encrypted vda2. I prefer to use ZFS as my file system for the encrypted drive, and LUKS rather than ZFS encryption. This is an MBR drive, since that's what my VPS provider uses, though UEFI would look the same. The particular way I do this also requires access through the provider's tools, and not ssh or similar. The hardware-configuration.nix file reflects this:

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/profiles/qemu-guest.nix")
    ];

  boot.initrd.availableKernelModules = [ "aes_x86_64" "ata_piix" "cryptd" "uhci_hcd" "virtio_pci" "sr_mod" "virtio_blk" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "rpool/root/nixos";
      fsType = "zfs";
    };

  fileSystems."/home" =
    { device = "rpool/home";
      fsType = "zfs";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/294de4f1-72e2-4377-b565-b3d4eaaa37b6";
      fsType = "ext4";
    };

  swapDevices = [ ];

}

# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, lib, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Hardware stuff
  # add the following to hardware-configuration.nix - speeds up encryption
  #boot.initrd.availableKernelModules ++ [ "aes_x86_64" "cryptd" ];
  boot.initrd.luks.devices.crypt = {
    # Change this if moving to another machine!
    device = "/dev/disk/by-uuid/86090289-1c1f-4935-abce-a1aeee1b6125";
  };
  boot.kernelParams = [ "zfs.zfs_arc_max=536870912" ]; # sets zfs arc cache max target in bytes
  boot.supportedFilesystems = [ "zfs" ];
  nix.maxJobs = lib.mkDefault 6; # number of cpu cores

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  # boot.loader.grub.efiSupport = true;
  # boot.loader.grub.efiInstallAsRemovable = true;
  # boot.loader.efi.efiSysMountPoint = "/boot/efi";
  # Define on which hard drive you want to install Grub.
  boot.loader.grub.device = "/dev/vda"; # or "nodev" for efi only
  boot.loader.grub.enableCryptodisk = true;
  boot.loader.grub.zfsSupport = true;

  networking.hostName = "m"; # Define your hostname.
  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.

  # The global useDHCP flag is deprecated, therefore explicitly set to false here.
  # Per-interface useDHCP will be mandatory in the future, so this generated config
  # replicates the default behaviour.
  networking.useDHCP = false;
  networking.interfaces.ens3.useDHCP = true;
  networking.hostId = "aoeu"; # set this to the first eight characters of /etc/machine-id for zfs
  networking.nat = {
    enable = true;
    externalInterface = "ens3"; # this may not be the interface name
    internalInterfaces = [ "wg0" ];
  };
  networking.firewall = {
    enable = true;
    allowedTCPPorts = [ 53 25565 ]; # open 53 for DNS and 25565 for Minecraft
    allowedUDPPorts = [ 53 51820 ]; # open 53 for DNS and 51820 for Wireguard - change the Wireguard port
  };
  networking.wg-quick.interfaces = {
    wg0 = {
      address = [ "10.0.0.1/24" "fdc9:281f:04d7:9ee9::1/64" ];
      listenPort = 51820;
      privateKeyFile = "/root/wireguard-keys/privatekey"; # fill this file with the server's private key and make it so only root has read/write access

      postUp = ''
        ${pkgs.iptables}/bin/iptables -A FORWARD -i wg0 -j ACCEPT
        ${pkgs.iptables}/bin/iptables -t nat -A POSTROUTING -s 10.0.0.1/24 -o ens3 -j MASQUERADE
        ${pkgs.iptables}/bin/ip6tables -A FORWARD -i wg0 -j ACCEPT
        ${pkgs.iptables}/bin/ip6tables -t nat -A POSTROUTING -s fdc9:281f:04d7:9ee9::1/64 -o ens3 -j MASQUERADE
      '';

      preDown = ''
        ${pkgs.iptables}/bin/iptables -D FORWARD -i wg0 -j ACCEPT
        ${pkgs.iptables}/bin/iptables -t nat -D POSTROUTING -s 10.0.0.1/24 -o ens3 -j MASQUERADE
        ${pkgs.iptables}/bin/ip6tables -D FORWARD -i wg0 -j ACCEPT
        ${pkgs.iptables}/bin/ip6tables -t nat -D POSTROUTING -s fdc9:281f:04d7:9ee9::1/64 -o ens3 -j MASQUERADE
      '';

      peers = [
        { # peer0
          publicKey = "{client public key}"; # replace this with the client's public key
          presharedKeyFile = "/root/wireguard-keys/preshared_from_peer0_key"; # fill this file with the preshared key and make it so only root has read/write access
          allowedIPs = [ "10.0.0.2/32" "fdc9:281f:04d7:9ee9::2/128" ];
        }
      ];
    };
  };

  # Configure network proxy if necessary
  # networking.proxy.default = "http://user:password@proxy:port/";
  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

  nixpkgs.config = {
    allowUnfree = true; # don't set this if you want to ensure only free software
  };

  # Select internationalisation properties.
  i18n.defaultLocale = "en_US.UTF-8";
  console = {
    font = "Lat2-Terminus16";
    keyMap = "us";
  };

  # Set your time zone.
  time.timeZone = "America/New_York"; # set this to the same timezone your server is located in

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  environment = {
    systemPackages = with pkgs; let
      nvimcust = neovim.override { # lazy minimal neovim config
        viAlias = true;
        vimAlias = true;
        withPython = true;
        configure = {
          packages.myPlugins = with pkgs.vimPlugins; {
            start = [ deoplete-nvim ];
            opt = [];
          };
          customRC = ''
            if filereadable($HOME . "/.config/nvim/init.vim")
              source ~/.config/nvim/init.vim
            endif

            set number

            set expandtab

            filetype plugin on
            syntax on

            let g:deoplete#enable_at_startup = 1
          '';
        };
      };
    in
    [
      jdk8
      nvimcust
      p7zip
      wget
      wireguard
    ];
  };

  # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
  # programs.gnupg.agent = {
  #   enable = true;
  #   enableSSHSupport = true;
  #   pinentryFlavor = "gnome3";
  # };

  # List services that you want to enable:

  # Enable the OpenSSH daemon.
  services = {
    dnsmasq = {
      enable = true;
      # this allows DNS requests from wg0 to be forwarded to the DNS server on this machine
      extraConfig = ''
        interface=wg0
      '';
    };
    fail2ban = {
      enable = true;
    };
    openssh = {
      enable = true;
      permitRootLogin = "no";
    };
    zfs = {
      autoScrub = {
        enable = true;
        interval = "monthly";
      };
    };
  };

  # Set sudo to request root password for all users
  # this should be changed for a multi-user server
  security.sudo.extraConfig = ''
    Defaults rootpw
  '';

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.users = {
    vpsadmin = { # admin account that has a password
      isNormalUser = true;
      home = "/home/vpsadmin";
      extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
      shell = pkgs.zsh;
    };
    mcserver = { # passwordless user to run a service - in this instance minecraft
      isNormalUser = true;
      home = "/home/mcserver";
      extraGroups = [];
      shell = pkgs.zsh;
    };
  };

  systemd = {
    services = {
      mcserverrun = { # this service runs a systemd sandboxed modded minecraft server as user mcserver
        enable = true;
        description = "Start and keep minecraft server running";
        wants = [ "network.target" ];
        after = [ "network.target" ];
        serviceConfig = {
          User = "mcserver";
          NoNewPrivileges = true;
          PrivateTmp = true;
          ProtectSystem = "strict";
          PrivateDevices = true;
          ReadWritePaths = "/home/mcserver/Eternal_current";
          WorkingDirectory = "/home/mcserver/Eternal_current";
          ExecStart = "${pkgs.jdk8}/bin/java -Xms11520M -Xmx11520M -server -XX:+AggressiveOpts -XX:ParallelGCThreads=3 -XX:+UseConcMarkSweepGC -XX:+UnlockExperimentalVMOptions -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:MaxGCPauseMillis=10 -XX:GCPauseIntervalMillis=50 -XX:+UseFastAccessorMethods -XX:+OptimizeStringConcat -XX:NewSize=84m -XX:+UseAdaptiveGCBoundary -XX:NewRatio=3 -jar forge-1.12.2-14.23.5.2847-universal.jar nogui";
          Restart = "always";
          RestartSec = 12;
        };
        wantedBy = [ "multi-user.target" ];
      };
      mcserverscheduledrestart = { # this service restarts the minecraft server on a schedule
        enable = true;
        description = "restart mcserverrun service";
        serviceConfig = {
          Type = "oneshot";
          ExecStart = "${pkgs.systemd}/bin/systemctl try-restart mcserverrun.service";
        };
      };
    };
    timers = {
      mcserverscheduledrestart = { # this timer triggers the service of the same name
        enable = true;
        description = "restart mcserverrun service daily";
        timerConfig = {
          OnCalendar = "*-*-* 6:00:00";
        };
        wantedBy = [ "timers.target" ];
      };
    };
  };

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "20.09"; # Did you read the comment?

}

Edit: Also, the provider I use is ExtraVM, who has been excellent.

I see some people using NixOs on their servers. I would like to try it out to self host some services and learn about NixOs.

I use hetzner and they have an NixOs iso available so I can just use that to install NixOs. But how do people manage remote instances of NixOs? They would just use ansible or something like it, to run nix on the host, or is there a better way?

Thanks