Latest

Much of the video that winds up on dash.ohn.sh is recorded in a very particular format (in OBS Studio), consisting of four 1080p tiles, each representing an “angle” of the same scene. Two of those angles are screen captures, which are difficult to read when limited to one quarter of a video player. Luckily, the web platform provides a bunch of tools to process video and create custom video players. The above demo shows a first pass at a solution, implemented as an interactive React component.

Concept

The basic idea is to display one quadrant at a time in a main view that occupies most of the video player, with the other three quadrants shown picture-in-picture style along one edge. Clicking a picture-in-picture view swaps it with the main view. Clicking the main view toggles all of the picture-in-picture views on/off as a group, which is important because they do obscure part of the main view when enabled.

For a new project, I have a video processing pipeline that prepares raw camera footage for publishing, extracts metadata, and syncs everything to a Cloudflare R2 bucket. The newest feature is voice activity detection with Silero VAD, which I’m running on the CPU via onnxruntime-node. Here it is taking around 20 seconds to analyze several hours of footage on my Mac Mini.

The model takes a tensor representing 512 samples of 16-kHz, single-channel audio (roughly 30 ms) and returns a score indicating the probability that the window contains speech (“voice activity”). In the pipeline, audio is extracted from source recordings and normalized with ffmpeg, then fed through the model piece by piece. The raw scores are filtered through a state machine (during a speech segment, a given sample is more likely to register as speech) to produce a list of timestamps representing segments of the video that contain speech.