Hi Folks

I've been working on an autoconversion Bash script to pick up videos and convert them to AV1. Yes, I know converting a source to another source means degradation and yada yada yada, but it's something I can live with as most of my sources are of very high quality to begin with, and I'm going for space-saving. Plus, my eyes aren't what they once were.

The conversion into AV1 I'm mostly happy with. I'm currently going through some old 90s shows which are of lesser quality, so they would need a little help to look better with AV1 by adding some natural film grain, else AV1 makes them look a little bit too clean.

I can easily pop into the script and enable film grain in the variables, or add in a simple option, but that's boring and tedious. Why do that when we can automate the world :)

Where I have got to is using the signalstats filter. The issue I have is I don't know how to analyse it's output enough to work out whether I should or shouldn't enable film grain or not. I know it's subjective either way.

Does anyone have experience with this? The output per frame looks like:

frame:1438 pts:59977 pts_time:59.977
lavfi.signalstats.YMIN=0
lavfi.signalstats.YLOW=0
lavfi.signalstats.YAVG=60.6913
lavfi.signalstats.YHIGH=149
lavfi.signalstats.YMAX=239
lavfi.signalstats.UMIN=91
lavfi.signalstats.ULOW=108
lavfi.signalstats.UAVG=121.955
lavfi.signalstats.UHIGH=131
lavfi.signalstats.UMAX=148
lavfi.signalstats.VMIN=124
lavfi.signalstats.VLOW=128
lavfi.signalstats.VAVG=134.535
lavfi.signalstats.VHIGH=146
lavfi.signalstats.VMAX=154
lavfi.signalstats.SATMIN=0
lavfi.signalstats.SATLOW=0
lavfi.signalstats.SATAVG=9.74682
lavfi.signalstats.SATHIGH=27
lavfi.signalstats.SATMAX=43
lavfi.signalstats.HUEMED=147
lavfi.signalstats.HUEAVG=162.949
lavfi.signalstats.YDIF=0.737433
lavfi.signalstats.UDIF=0.642897
lavfi.signalstats.VDIF=0.162755
lavfi.signalstats.YBITDEPTH=8
lavfi.signalstats.UBITDEPTH=8
lavfi.signalstats.VBITDEPTH=8

I'm happy to analyse a random 60-second segment and then grab an average from a couple of these outputs, but I'm not sure if this is a good method or not. I'm asked a couple of the biggest LLMs, they have come back with older ways that no longer exist in ffmpeg 7.1.

I'm trying not to use too many other pieces of software in this script. The dependencies are fairly simple with ffmpeg, awk, grep, etc., the kind of thing you get on nearly every distro of Linux. Any thoughts and/or ideas?

I am currently preparing for a new job and cleaning up my dotfile repository. During the process, I had the idea that it would be nice to create a list of amazing tools, aliases, functions, and recommendations together.

I will start.

First, here is a list of nice tools to apt-get install or brew install that I can wholeheartedly recommend:

• nvim is just an amazing text editor.
• fzf is a very good fuzzy finder util. For example, you can quickly find files with it.
• eza is a good ls replacement (and the successor of exa).
• bat is a great replacement for cat with nice integrations and many options.
• stow is great for managing your dotfiles. Thanks to @TangibleLight for telling me about it some while ago. I really love it.
• tmux is a terminal multiplexer, i.e. you can have many sessions in one single terminal window. It's easy to use and super helpful. (When on a mac, I prefer iTerm tabs, though.)
• nvm is practically a must if you are working with Node.
• glow is an excellent markdown reader.
• tldr is a nice man replacement. (You must run tldr -u after installing it to update available texts.)
• z, an amazing tool for switching directories quickly.

Also, I can recommend Oh My ZSH! which I have been using for years.

Here is a small list of aliases I enjoy (I have 100+ aliases and I tried to pick some others may enjoy as well):

# Serve current dir
alias serve="npx serve ."

# What's my IP?
alias ip="curl --silent --compressed --max-time 5 --url 'https://ipinfo.io/ip' && echo ''"

# This should be the default
alias mkdir="mkdir -p"

# Nice git helpers
alias amend="git add . && git commit --amend --no-edit"
alias nuke="git clean -df && git reset --hard"

# Make which more powerful
which='(alias; declare -f) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot'

# This saves so many keystrokes, honestly
alias -- +x="chmod +x"

# Turns your path into a nice list and prints it
alias path='echo -e ${PATH//:/\\n}'

# Map over arguments and run a command
# Usage: map <command>
# Example: ls | map cat
alias map="xargs -n1"

And, finally, here are some fun functions:

# Get cheat sheets for almost anything!
# https://github.com/chubin/cheat.sh
cheat() {
    WITH_PLUS=$(echo $@ | sed 's/ /+/g')
    CAT_TOOL=$(command -v batcat || command -v bat || command -v cat)
    curl "cheat.sh/$WITH_PLUS" | $CAT_TOOL
}

# Send everything to /dev/null
nullify() {
  "$@" >/dev/null 2>&1
}

# Create a new dir and enter it
mk() {
  mkdir -p "$@" && cd "$_"
}

# Create a data URL from a file
# Source: https://github.com/mathiasbynens/dotfiles/blob/master/.functions
data-url() {
	local mimeType=$(file -b --mime-type "$1");
	if [[ $mimeType == text/* ]]; then
		mimeType="${mimeType};charset=utf-8";
	fi
	echo "data:${mimeType};base64,$(openssl base64 -in "$1" | tr -d '\n')";
}

Gonna try and put this into words.

I am pretty familiar with bash and python. used both quite a bit and feel more or less comfortable with them.

My issue is I often do a thing where if I want to accomplish a task that is maybe a bit complex, I feel like I have to wind up making a script, let's call it hello_word.sh but then I also make a script called .hello_world.py

and basically what I do is almost the first line of the bash script, I call the python script like ./hello_world.py $@ and take advtange of the argparse library in python to determine what the user wants to do amongst other tasks that are easier to do in python like for loops and etc.

I try to do the meat of the logic in the python scripts before I write to an .env file from it and then in the bash script, I will do

set -o allexport
source "${DIR}"/"${ENV_FILE}"
set +o allexport

and then use the variable from that env file to do the rest of the logic in bash.

why do I do anything in bash?

cause I very much prefer being able to see a terminal command being executed in real-time and see what it does and be able to Ctrl+c if I see the command go awry.

in python, you can run a command with subprocess or other similar system libraries but you can't get the output in real-time or terminate a command preemptively and I really hate that. you have to wait for the command to end to see what happened.

But I feel like there is something obvious I am missing (like maybe bash has an argparse library I don't know about and there is some way to inject the concept of types into it) or if there is another language entirely that fits my needs?

This below is a summary of some real-world performance investigation I recently went through. The tools I used are installed on all linux systems, but I know some people don't know them and would straight up jump to heavyweight log analysis services and what not, or writing their own solution.

Let's say you have request log sampling in a bunch of log files that contain lines like these:

127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000
127.0.0.1 [2021-05-27 23:51:22.019] "GET /pl/player/123456/changelog/ HTTP/1.1" 200 16524 TLSv1.2 MISS-CLUSTER SessionID:(null) Cache:

You might recognize Fastly logs there (IP anonymized). Now, there's a lot you might care about in this log file, but in my case, I wanted to get a breakdown of hits vs misses by URL.

So, first step, let's concatenate all the log files with cat *.log > all.txt, so we can work off a single file.

Then, let's split the file in two: hits and misses. There are a few different values for them, the majority are covered by either HIT-CLUSTER or MISS-CLUSTER. We can do this by just grepping for them like so:

grep HIT-CLUSTER all.txt > hits.txt; grep MISS-CLUSTER all.txt > misses.txt

However, we only care about url and whether it's a hit or a miss. So let's clean up those hits and misses with cut. The way cut works, it takes a delimiter (-d) and cuts the input based on that; you then give it a range of "fields" (-f) that you want.

In our case, if we cut based on spaces, we end up with for example: 127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000.

We care about the 5th value only. So let's do: cut -d" " -f5 to get that. We will also sort the result, because future operations will require us to work on a sorted list of values.

cut -d" " -f5 hits.txt | sort > hits-sorted.txt; cut -d" " -f5 misses.txt | sort > misses-sorted.txt

Now we can start doing some neat stuff. wc (wordcount) is an awesome utility, it lets you count characters, words or lines very easily. wc -l counts lines in an input, since we're operating with one value per line we can easily count our hits and misses already:

$ wc -l hits-sorted.txt misses-sorted.txt
  132523 hits-sorted.txt
  220779 misses-sorted.txt
  353302 total

220779 / 132523 is a 1:1.66 ratio of hits to misses. That's not great…

Alright, now I'm also interested in how many unique URLs are hit versus missed. uniq tool deduplicates immediate sequences, so the input has to be sorted in order to deduplicate our entire file. We already did that. We can now count our urls with uniq < hits-sorted.txt | wc -l; uniq < misses-sorted.txt | wc -l. We get 49778 and 201178, respectively. It's to be expected that most of our cache misses would be in "rarer" urls; this gives us a 1:4 ratio of cached to uncached URL.

Let's say we want to dig down further into which URLs are most often hitting the cache, specifically. We can add -c to uniq in order to get a duplicate count in front of our URLs. To get the top ones at the top, we can then use sort, in reverse sort mode (-r), and it also needs to be numeric sort, not alphabetic (-n). head lets us get the top 10.

$ uniq -c < hits-sorted.txt | sort -nr | head
    815 /static/app/webfonts/fa-solid-900.woff2?d720146f1999
    793 /static/app/images/1.png
    786 /static/app/fonts/nunito-v9-latin-ext_latin-regular.woff2?d720146f1999
    760 /static/CACHE/js/output.cee5c4089626.js
    758 /static/images/crest/3/light/notfound.png
    757 /static/CACHE/css/output.4f2b59394c83.css
    756 /static/app/webfonts/fa-regular-400.woff2?d720146f1999
    754 /static/app/css/images/loading.gif?d720146f1999
    750 /static/app/css/images/prev.png?d720146f1999
    745 /static/app/css/images/next.png?d720146f1999

And same for misses:

$ uniq -c < misses-sorted.txt | sort -nr | head
     56 /
     14 /player/237678/
     13 /players/
     12 /teams/
     11 /players/top/
<snip>

So far this tells us static files are most often hit, and for misses it also tells us… something, but we can't quite track it down yet (and we won't, not in this post). We're not adjusting for how often the page is hit as a whole, this is still just high-level analysis.

One last thing I want to show you! Let's take everything we learned and analyze those URLs by prefix instead. We can cut our URLs again by slash with cut -d"/". If we want the first prefix, we can do -f1-2, or -f1-3 for the first two prefixes. Let's look!

cut -d'/' -f1-2 < hits-sorted.txt | uniq -c | sort -nr | head
 100189 /static
   5948 /es
   3069 /player
   2480 /fr
   2476 /es-mx
   2295 /pt-br
   2094 /tr
   1939 /it
   1692 /ru
   1626 /de

cut -d'/' -f1-2 < misses-sorted.txt | uniq -c | sort -nr | head
  66132 /static
  18578 /es
  17448 /player
  17064 /tr
  11379 /fr
   9624 /pt-br
   8730 /es-mx
   7993 /ru
   7689 /zh-hant
   7441 /it

This gives us hit-miss ratios by prefix. Neat, huh?

EDIT: SOLVED

It looks like it was much simple than I thought and someone solved it on Reddit already. I won't delete, just leave the link if someone is interested.

Runtime Environment

• OS: MX Linux 18
• Result of Y: 4.19.0-5-amd64
• dotfiles
• i3 version: 4.13
• ~/.config/i3
• GNU Emacs: 27.0.50
• ~/.emacs.d

Issue

Sometimes I use "whereis" (aliased for "wh", but it doesn't make any difference...) for my own scripts.

I usually copy their paths manually (using tmux) and paste to the command line resulting in something like this:

nvim /home/my_username/my_scripts_folder/my_script

Could I make that into a single command?

Thanks in advance!

For my particular use case I share my home PC with my spouse and since I'm the more tech-savvy of the two I'll need to occasionally remote in and help out with some random task. They know enough that the issue will usually be too complex to simply guide over the phone, so remote control it is.

I'm also trying to improve my personal efforts toward privacy and security. To that end I want to avoid closed-source services such as TeamViewer where a breach on their end could compromise my system.

The following is the current state of what I'm now using as I think others may benefit from this as well:

Setup

Web

I use a simple web form as my first authentication. It's just a username and password, but it does require a web host that supports server side code such as PHP. In my case I just created a blank page with nothing other than the form and when successful the page generates a 6 digit PIN and saves it to a text file in a private folder (so no one can simply navigate to it and get the PIN).

I went the text file route because my current hosting plan only allows 1 database and I didn't want to add yet another random table just for this 1 value.

Router

To connect to my home PC I needed to forward a port from my router. I'm going to use VNC as it lets me see what is currently shown on the monitor and work with someone already there so I forward port 5900 as VNC's default port. You can customize this if you want. Some routers allow you to SSH into their system and make changes that way so a step more secure would be to leave the port forward disabled and only enable it once a successful login from the web form is disabled. In my case I'll just leave the port forwarded all the time.

IP Address

To connect to my computer I need to know it's external IP address and for this I use FreeDNS from Afraid.org. My router has dynamic DNS support for them already included so it was easy to plug in my details to generate a URL which will always point to my home PC (well, as long as my router properly sends them my latest IP address). If your router doesn't support the dynamic DNS you choose many also allow either a download or the settings you would need to script your own to keep your IP address up to date with their service.

Signal

Signal is an end-to-end encrypted messenger which supports text, media, phone and video calls. There's also a nifty command line option on Github called Signal-cli which I'm using to provide my second form of authentication. I just downloaded the package, moved to my $PATH (in my case /usr/local/bin) and set it up as described on their README. In my case I have both a normal cell phone number and another number provided by Google Voice. I already use my normal cell phone number with Signal so for this project I used Signal-cli to register a new account using my Google Voice number.

VNC

My home PC runs Ubuntu 18.04 so I'm using x11vnc as my VNC server. Since I'm leaving my port forwarded all the time I most certainly do NOT want to leave VNC also running. That's too large a security risk for me. Instead I've written a short bash script that first checks the web form using curl and https (so it's encrypted) with its own login information to check if any PIN numbers have been saved. If a PIN is found the web server sends that back and then deletes the PIN text file. Meanwhile the bash script uses the PIN to start a VNC session with that PIN as the password and also sends my normal cell the PIN via Signal-cli so that I can login.

I have this script set to run every minute so I'm not waiting long after web login and I also have the x11vnc session set to timeout after a minute so I can quickly connect again should I mess something up. It's also important that x11vnc is set to auto exit after closing the session so that it's not left up for an attacker to attempt to abuse.

System Flow

Once everything is setup and working this is what it's like for me to connect to my home PC:

1. Browse to my web form and login
2. Close web form and wait for Signal message
3. Launch VNC client
4. Connect via dynamic DNS address (saved to VNC client)
5. Enter PIN code
6. Close VNC when done

Code

Here's some snippets to help get you started

PHP for Web Form Processing

<?php
// Variables
$username = 'your_username';
$password = 'your_password_super_long_and_unique';
$filename = 'path_to_private_folder/vnc/pin.txt';

// Process the login form
if($action == 'Login'){
	$file = fopen($filename,'w');
	$passwd = rand(100000,999999);
	fwrite($file,$passwd);
	fclose($file);
	exit('Success');
}

// Process the bash script
if($action == 'bash'){
	if(file_exists($filename)){
		$file = fopen($filename,'r');
		$passwd = fread($file,filesize($filename));
		fclose($filename);
		unlink($filename);
		exit($passwd);
	} else {
		exit('No_PIN');
	}
}
?>

Bash for x11vnc and Signal-cli

# See if x11vnc access has been requested
status=$(curl -s -d "u=your_username&p=your_password_super_long_and_unique&a=bash" https://vnc_web_form.com)

# Exit if nothing has been requested
if [ "$status" = "No_PIN" ]; then
  # No PIN so exit; log the event if you want
  exit 0
fi

# Strip non-numeric characters
num="${status//[!0-9]/}"

# See if they still match (prevent error messages from triggering stuff)
if [ $status != $num ]; then
  # They don't match so probably not a PIN - exit; log it if you want
  exit 1
fi

# Validate pin number
num=$((num + 0))
if [ $num -lt 100000 ]; then
  # PIN wasn't 6 digits so something weird is going on - exit; log it if you want
  exit 1
fi
if [ $num -gt 999999 ]; then
  # Same as before
  exit 1
fi

# Everything is good; start up x11vnc
# Log event if you want

# Get the current IP address - while dynamic DNS is in place this serves as a backup
ip=$(dig +short +timeout=5 myip.opendns.com @resolver1.opendns.com)

# Send IP and password via Signal
# Note that phone number includes country code
# My bash is running as root so I run the command as my local user where I had registered Signal-cli
su -c "signal-cli -u +google_voice_number send -m '$num for $ip' +normal_cell_number" s3rvant

# Status was requested and variable is now the password
# this provides a 1 minute window to connect with 1-time password to control main display
# again run as local user
su -c "x11vnc -timeout 60 -display :0 -passwd $num" s3rvant

Final Thoughts

There are more secure ways to handle this. Some routers support VPN for the connect along with device certificates which are much stronger than a 6 digit PIN code. Dynamically opening and closing the router port as part of the bash script would also be a nice touch. For me this is enough security and is plenty convenient enough to quickly offer tech support (or nab some bash code for articles like this) on the fly.

I'm pretty happy with how Signal-cli has worked out and plan to use it again with my next project (home automation). I'll be sure to post again once I get that ball rolling.