.
Sometimes machines need an eye. Easy. Attach a webcam, run a webserver.
Sometimes they need many eyes.
And sometimes they need to add/remove eyes during operation.
We are running on a Raspberry Pi.
We have all the shell scripts for running the services in /home/pi/scripts/ directory.
We are running the Lovecraftian abomination of systemd.
Weak assumption for now: we want to run a mjpeg_streamer, and we do not want to discriminate between the cams.
The cams are to run on OctoPrint, using the MultiCam
plugin.
As the cams are cheap, the wiring is lousy, and the software is experimental, it often seizes somewhere and has to be restarted. Preferably from the web interface (or calling a custom command API).
Let's run a webserver when the cam is attached.
First, detect the attach/detach events. For this, the udev system comes to help.
There are rules in /etc/udev/rules.d/. Let's put our ones to /etc/udev/rules.d/99-webcams.rules.
#
# rules for *all* /dev/video* devices
# attach cams by the order of their initiation
# todo: more granulated differentiation
#
# belongs to /etc/udev/rules.d/99-webcams.rules
ACTION=="add", KERNEL=="video[0-9]*" SYMLINK+="webcam%n" ENV{DEVNUM}="%n" TAG+="systemd" ENV{SYSTEMD_WANTS}="webcam@%n.service"
ACTION=="remove", KERNEL=="video[0-9]*" RUN+="/home/pi/scripts/webcam_server_kill.sh %n"
# do not forget to reload!
# udevadm control --reload-rules
We extract the number of the kernel video device as %n. We use it to start a systemd service via a template.
(The symlink is there just to see we ran through the rule, for now.)
The user "pi" the camservers run under is already in the group "video". Otherwise we could putz with permissions here.
We run the server for all webcams. Exceptions can be added here or in the service launch.
The "remove" action runs directly the camserver-kill script. Should be better through systemctl stop. It works for now, will be modified once we start having to use automatic daemon restarts on failure so systemctl knows what should NOT be running.
Also, after editing the file either do not forget to run udevadm control --reload-rules or do not wonder why the change did not work.
For listing all the attributes of the device attached to the /dev/something, run
udevadm info -q all --name=/dev/somethingFor /dev/video0 with certain cam attached, the output is
P: /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.3/1-1.3.4/1-1.3.4:1.0/video4linux/video0 N: video0 S: v4l/by-id/usb-Etron_Technology__Inc._USB2.0_Camera-video-index0 S: v4l/by-path/platform-3f980000.usb-usb-0:1.3.4:1.0-video-index0 E: DEVLINKS=/dev/v4l/by-id/usb-Etron_Technology__Inc._USB2.0_Camera-video-index0 /dev/v4l/by-path/platform-3f980000.usb-usb-0:1.3.4:1.0-video-index0 E: DEVNAME=/dev/video0 E: DEVPATH=/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.3/1-1.3.4/1-1.3.4:1.0/video4linux/video0 E: ID_BUS=usb E: ID_FOR_SEAT=video4linux-platform-3f980000_usb-usb-0_1_3_4_1_0 E: ID_MODEL=USB2.0_Camera E: ID_MODEL_ENC=USB2.0\x20Camera E: ID_MODEL_ID=0110 E: ID_PATH=platform-3f980000.usb-usb-0:1.3.4:1.0 E: ID_PATH_TAG=platform-3f980000_usb-usb-0_1_3_4_1_0 E: ID_REVISION=0201 E: ID_SERIAL=Etron_Technology__Inc._USB2.0_Camera E: ID_TYPE=video E: ID_USB_DRIVER=uvcvideo E: ID_USB_INTERFACES=:0e0100:0e0200: E: ID_USB_INTERFACE_NUM=00 E: ID_V4L_CAPABILITIES=:capture: E: ID_V4L_PRODUCT=USB2.0 Camera: USB2.0 Camera E: ID_V4L_VERSION=2 E: ID_VENDOR=Etron_Technology__Inc. E: ID_VENDOR_ENC=Etron\x20Technology\x2c\x20Inc. E: ID_VENDOR_ID=1e4e E: MAJOR=81 E: MINOR=0 E: SUBSYSTEM=video4linux E: TAGS=:seat:uaccess: E: USEC_INITIALIZED=1491093226745
This can be used in the streaming daemon launch script to select the proper daemon (and its setting).
Now it's getting thick. We need to define the service, a template file for multiple services of the same kind with different numbers.
Let's put the file to /etc/systemd/system/webcam@.service. The '@' is important so it is interpreted as the template. A little detail that can spoil a night that could've been spent chasing something more interesting.
# belongs to /etc/systemd/system/webcam@.service # called from /etc/udev/rules/webcams.rules # # do not forget to run systemctl daemon-reload [Unit] Description=webcam server on /dev/video%i [Service] User=pi Type=forking RemainAfterExit=yes ExecStart=/home/pi/scripts/webcam_server.sh %i ExecStop=/home/pi/scripts/webcam_server_kill.sh %i #ExecReload=/home/pi/scripts/webcam_server.sh %i [Install] WantedBy=multi-user.target
We are running under user "pi".
The type "forking" is important otherwise systemd will kill the process spawned by mjpeg_streamer -b leaving you wondering where did it go.
The "RemainAfterExit=yes" tells systemd to do the kill attempts properly on restart and to not be too smart.
Also, like before, do not forget to run systemctl daemon-reload to tell the machine we did changes.
So we now have a cam server that gets launched when the cam is attached, and killed off when we pull out.
But... the system is cheap and unreliable, and needs way too frequent restarts at times. Let's add a system command to OctoPrint, to /home/pi/.octoprint/config.yaml.
system:
actions:
- action: divider
- action: Restart Cam 0
command: systemctl restart webcam@0
name: restartCam0
- action: Restart Cam 1
command: systemctl restart webcam@1
name: restartCam1
- action: Restart Cam 2
command: systemctl restart webcam@2
name: restartCam2
- action: Restart Cam 3
command: systemctl restart webcam@3
name: restartCam3
And let's try. And fail.
Command for custom:Restart Cam 0 failed with return code 1: STDOUT: STDERR: Failed to start webcam0.service: Interactive authentication required. See system logs and 'systemctl status webcam0.service' for details.
Now we opened another layer of hell.
The thick becomes thicker. Policy kit, requiring complex rules where su or sudo or suid did the job well enough.
Long and frustrating and head-against-the-wall banging story short:
The polkit daemon talks over dbus. Everything is happening outside of reach of the eyes of the admin, the daemons are keeping their cards folded. No transparency at all, a junior admin is lost, a senior one just pissed off with the time waste unleashed by something that should've been a niche access handling system running somewhere where the complexity is actually warranted.
The documentation says that the /etc/polkit-1/rules.d/ directory should contain the new rules. On at least some configurations this directory does not exist at all.
The docs say it should be monitored, that the polkit daemon, /usr/lib/policykit-1/polkitd, should reload changes. strace attached to the process shown it does nothing. It was more responding on the /etc/polkit-1/localauthority/ subdirectories, where a change triggered some activity in the daemon.
So let's put the file to /etc/polkit-1/localauthority/50-local.d/.
A javascript (SRSLY?) .rules file was ignored. A .conf file was also ignored. Only the .pkla file was taken well.
The file was placed to /etc/polkit-1/localauthority/50-local.d/shad-pi-services.pkla, and written overly broadly to cover restarts of all the daemons from the octoprint interface than we can need (including octoprint itself), and then some.
# belongs to /etc/polkit-1/localauthority/50-local.d/shad-pi-services.pkla [webcam daemon restart permit] Identity=unix-user:pi Action=org.freedesktop.systemd1.manage-units ResultAny=yes ResultActive=yes ResultInactive=yes
More permissions can be placed here as needed.
Don't be too lax if the machine has some threat model (e.g. is visible from the Net or has more users).
Executed by systemd. Gets one parameter, the number of the video device ("1" for /dev/video1 or so).
Originally the process was to be found by lsof on the device name. But at the instant of removal, we already do not have the device in the filesystem. It however still exists in open state, listed by plain lsof as DEL, deleted.
So the open files of all the processes are queried and the device in question is grepped for.
If found, the process is mercilessly murdered.
A rule for sparing young children was added when systemd insisted on running the kill script concurrently with the start one, killing the camserver just after launching it. This was later solved properly by the Type=forking and RemainAfterExit=yes directives.
One day an "upgrade" of systemd comes and breaks this behavior. Then there will be crying and lamentation and wringing of hands and debugging.
The file is placed in /home/pi/scripts/webcam_server_kill.sh.
#!/bin/bash # $1 = num of /dev/videoX VIDEODEV=/dev/video$1 # process invulnerability to invocation of the FUCKING KILLER SPAWNPROTECT=2 #echo "Attempting to kill $VIDEODEV..."|logger -t $0 PROCNUM=`lsof -X |grep $VIDEODEV|tr -s ' ' |cut -d ' ' -f 2|head -n 1` echo Called from PPID=$PPID on $PROCNUM|logger -t $0 if [ "$PROCNUM" == "" ]; then echo "Nothing to kill."|logger -t $0 exit 0 fi PROCAGE=`expr $(date +"%s") - $(stat -c%X /proc/$PROCNUM)` if [ $PROCAGE -lt $SPAWNPROTECT ]; then echo "Not killing $PROCNUM (age $PROCAGE seconds), too young."|logger -t $0 exit 0 fi echo "Killing $PROCNUM (age $PROCAGE seconds) on $VIDEODEV..."|logger -t $0 kill $PROCNUM sleep 1 exit 0 #if [ ! -c $VIDEODEV ]; then # echo "Cannot find char device '$VIDEODEV'. Assuming not existing." # exit 0 #fi # #/bin/shad/killopenport $VIDEODEV 2>&1 |logger -t webcamkill
The camserver is launched from a start script. The script first calls the kill script, to sense and kill any other processes that may be sitting on the device. Then it runs the streaming software itself and forks it into background. Then it exists.
Assigns all the cams to CPU3, so they do not interfere with more important processes and hog resources of each other instead.
The file is placed in /home/pi/scripts/webcam_server.sh.
#!/bin/bash # $1 = device number # $2 = product, e.g. 1e4e/110/201 # check if it is in envvars? #set > /tmp/x #ps -ef > /tmp/x2 echo "Invoked for $1"|logger -t $0 NUM=$1 #if something already hogs the device, most likely old half-dead daemon, KILL KILL KILL!!! /home/pi/scripts/webcam_server_kill.sh $NUM if [ "$2" == "-k" ]; then echo "Stopped self. Not starting."|logger -t $0 exit 0 fi #NUM=0 BASEPORT=5100 VIDEODEV=/dev/video$NUM VIDEOPORT=$(( $BASEPORT + $NUM )) WEBCAMCPU=3 MDIR=/usr/src/mjpg-streamer/mjpg-streamer-experimental if [ ! -c "$VIDEODEV" ]; then echo "Device '$VIDEODEV' not found, refusing to start server." exit 1 fi echo "Starting webcam server for cam '$NUM' on '$VIDEODEV'..."|logger -t $0 #bruteforce avoid the fucking kill script that the fucking systemd insists on running!!! sleep 3 cd $MDIR taskset -c $WEBCAMCPU $MDIR/mjpg_streamer -b -i "$MDIR/input_uvc.so -d $VIDEODEV" -o "$MDIR/output_http.so -w $MDIR/www -p $VIDEOPORT"
- a snapshot of a version
Distribution scripts for saving the configuration from a remote machine (DO NOT overwrite the just-made change by running get instead of send!). The send script also auto-execs the commands to activate the changes. Run as a root to the target machines.