Linux Fu: Failing Pipelines [Hackaday]

View Article on Hackaday

Bash is great for automating little tasks, but sometimes a little script you think will take a minute to write turns into a half hour or more. This is the story of one of those half-hour scripts.

I have too many 3D printers. In particular, I have three that are almost — but not exactly — the same, so each one has a slightly different build process when I want to update their firmware. In all fairness, one of those printers is heading out the door soon, but I’ll probably still wind up building firmware images for it.

My initial process was painful. I have a special directory with the four files needed to configure Marlin for each machine. I copy all four files and ask PlatformIO to perform the build. Usually, it succeeds and gives me a file that looks like firmware-yyyyddmmhhmm.bin or something like that.

The problem is that the build process doesn’t know which of the three machines is the target: Sulu, Checkov, or Fraiser. (Long story.) So, I manually look at the file name, copy it, and rename it. Of course, this is an error-prone process, and I’m basically lazy, so I decided to write a script to do it. I figured it would take just a minute to bundle up all the steps. I was wrong.

First Attempt

Copying the files to the right place was a piece of cake. I did check to make sure they existed. The problem came from launching PlatformIO, seeing the result on the screen, and being able to parse the filename out of the stream.

I thought it would be easy:

FN=$(pio run | grep '^Renamed to' | cut -d ' ' -f 3 )

That should do the build and leave $FN with the name of the file I need to rename and process. It does, but there are two problems. You can’t see what’s happening, and you can’t tell when the build fails.

Easy Problem First

The pipeline consumes the build’s output. Of course, a tee command can manage that, right? Well, sort of. The problem is that the tee command sends things to a file and standard out, but the standard out, in this case, is the pipe. Sure, I could tee the output to a temporary file and then process that file later, but that’s messy.

So, I resorted to a Bash-specific feature:

FN=$(pio run | tee /dev/fd/2 | grep ...

This puts the output on my screen but still sends it down the pipe, too. Sure, there are cases when this isn’t a good idea, and it isn’t very portable, but for my own use, it works just fine, and I’m OK with that. There are other ways to do this, like using /dev/tty if you know you are only using the script from a terminal.

Harder Problem

The bigger problem is that if the build fails — and it might —  there isn’t a good way to fail the whole pipeline. By default, the pipe’s return value is the last return value, and cut is happy to report success as long as it runs.

There are a number of possible answers. Again, I could have resorted to a temporary file. However, I decided to set a bash option to cause any failing item in a pipe to fail the whole pipe immediately:

set -o pipefail

So now, in part, my script looks like this:

set -o pipefail
FN=$(pio run | tee /dev/fd/2 | grep '^Renamed to' | cut -d ' ' -f 3 )
if [ $? -eq 0 ]
then
   echo Success...
   cp ".pio/build/STM32F103RC_creality/$FN" "configurations/$1"
   echo Result: "configurations/$1/$FN"
else
   echo Build failed
   exit 3
fi

Final Analysis

Is it brain surgery? Nope. But it is one of those bumps in the road in what should have been a five-minute exercise. Maybe next time you run into it, you’ll save yourself at least 25 minutes. This gets the job done, but it isn’t a stellar example of bash programming. I would hate to run it through a lint-like checker.