Hardware-in-the-Loop Testing: CI/CD With Real Embedded Hardware

In short: host-side unit tests and mocked interfaces validate your logic, but they are blind to the driver, timing and integration bugs that only appear on real silicon — and manual on-device checks are too rare and too late to close that gap. Hardware-in-the-loop (HIL) testing wires the real device into an automated rig, stimulates its real inputs and measures its real outputs, and runs the whole loop on every commit, around the clock. That is what turns “we have CI” into genuine CI/CD for embedded products.

The blind spot in embedded test automation

Most embedded teams keep a respectable test pyramid on the host: unit tests for business logic, integration tests against mocked drivers, maybe a QEMU run in the pipeline. All of it is valuable — and all of it shares one limitation. It runs against a model of the hardware, not the hardware itself.

That model is exactly where a large class of embedded bugs lives. A display controller that latches a register one frame too late; a CAN driver that drops a message under bus load; a touch panel whose calibration drifts after a firmware update; an I2C transaction that races the kernel’s power-management code — none of these has a representation in your mocks, so none of them can turn a host-side test red. They surface on the bench, or worse, in the field.

The usual answer is manual on-device testing: someone flashes a build, clicks through the UI, wiggles a few buttons, signs off. It works, but it is rare (it happens at milestones, not on every commit), late (the regression is already several merges deep), and never complete (nobody manually exercises every screen and every bus message on every build). The result is a test strategy that is fully automated right up to the point where the interesting failures happen — and then falls back to a human.

Test layer	Runs against	Catches	Blind to
Host unit tests	Pure logic	Algorithms, state machines	Drivers, timing, real I/O
Mocked integration	A model of the driver	Interface contracts	Hardware behaviour, races
QEMU / simulation	An emulated SoC	Boot, gross regressions	Peripheral quirks, real timing
Manual on-device	The real device, by hand	Whatever is clicked, when it’s clicked	Everything else; runs rarely
Hardware-in-the-loop	The real device, automated	Drivers, timing, integration, UI as shipped	Very little

What hardware-in-the-loop testing actually is

Hardware-in-the-loop testing puts the real device-under-test (DUT) at the centre of an automated rig. The device runs the exact firmware image you would ship. Around it sits instrumentation that plays the role of the user and the surrounding system: it stimulates the real input interfaces and captures the real output interfaces, with no emulation layer in between.

On the input side that means driving the actual buses and controls — injecting CAN and CAN FD frames, sending serial traffic over UART/RS-232/RS-485, generating network requests, toggling GPIOs, and even actuating a capacitive touch panel so the device registers a genuine touch event. On the output side it means measuring what the device really does: capturing the display over its video link and comparing frames against a reference, reading back CAN and serial responses, watching LEDs and status lines, and checking network output.

The distinction from a mock is not academic. When the rig taps a button on the touchscreen and then reads the resulting frame off the display link, every layer between the two — the touch driver, the input stack, the application, the Qt scene graph, the GPU, the display controller — has been exercised exactly as it will be in the customer’s hands. There is nothing left to simulate.

The loop: commit, build, flash, stimulate, measure, verdict

HIL only becomes CI/CD when the whole thing is a closed, unattended loop. A single iteration looks like this:

1. A developer pushes a commit.
2. CI builds a complete, flashable firmware image.
3. The image is automatically deployed to the real hardware in the rack — flashed and rebooted into a known-good state.
4. The test suite stimulates the real input interfaces: touch, CAN, serial, network, GPIO.
5. The rig captures the real output interfaces: display frames, LEDs, CAN, serial, network.
6. Each observation is compared against an expectation, producing an automated pass/fail verdict.
7. The result is reported back into CI, with artifacts — captured frames, bus logs — attached.
8. Repeat, on the next commit, around the clock.

The payoff is threefold. Regressions are caught before shipping, because every merge is validated on the target. The loop runs unattended around the clock, so coverage is no longer rationed by how much manual testing a release can afford. And because the rig drives the real interfaces, you test precisely what your users experience, not a model of it.

Reproducible images: the foundation for deterministic runs

A HIL loop is only as trustworthy as the images it flashes. If two runs of “the same” firmware differ because they were built on different machines or pulled different dependency versions, a red result tells you nothing — you cannot separate a real regression from build noise. Deterministic, reproducible images are therefore not a nice-to-have; they are what makes the verdict mean anything.

This is where a well-structured Embedded Linux platform and a Yocto build pay off directly. With pinned layer revisions and a declarative build, the image that boots on the rack is bit-for-bit the image the pipeline produced, reconstructable months later. Every red or green in the HIL stage then maps to a specific, rebuildable commit — exactly the property you need when bisecting a timing regression across a series of merges.

Wiring HIL into your CI/CD pipeline

Mechanically, HIL is just another pipeline stage — one that happens to run on a runner physically wired to the hardware. The build stage produces the image as an artifact; a hardware-test stage picks it up, flashes it to the DUT, runs the suite over the real interfaces, and publishes a report. A minimal GitLab CI shape:

stages:
  - build
  - hardware-test

build-image:
  stage: build
  script:
    - kas build kas-project.yml
  artifacts:
    paths:
      - build/tmp/deploy/images/

hardware-test:
  stage: hardware-test
  tags: [embedded-testrack]        # runner physically wired to the DUT
  script:
    - testrack flash --image build/tmp/deploy/images/*/*.wic.gz
    - testrack run   suites/        # stimulate inputs, capture outputs
  artifacts:
    when: always
    paths:
      - results/frames/            # captured display frames for diffing
    reports:
      junit: results/report.xml    # verdicts surface in the CI UI

Because the verdicts come back as standard JUnit output, they show up next to your other test results, and captured frames and bus logs travel with the job. The pipeline does not care that this stage touches copper instead of a container — the contract is the same: consume an artifact, emit a pass/fail.

Practical considerations: flashing, stability, image diff, flakiness

Getting a HIL rig to run reliably is largely an exercise in engineering away non-determinism.

Provisioning and flashing should put the device into a defined, fresh state at the start of every run — controlled power, a reliable flashing path (fastboot, an SD-mux, network boot), and a boot check that fails fast if the image did not come up.

A stable test harness matters more than raw coverage early on. If touch coordinates, bus timing or startup delays are handled sloppily, tests fail for reasons unrelated to the code, and a flaky suite is quickly ignored. Deterministic waits — polling for a known UI state rather than sleeping a fixed time — go a long way.

Display capture and comparison is the subtlest part. A raw pixel-for-pixel diff is far too brittle: antialiasing, a blinking cursor or a one-pixel layout shift will trip it. Practical rigs compare within tolerances, mask dynamic regions such as a clock or an animation, and assert on structural properties or specific regions of interest rather than the whole frame.

Flakiness is the standing enemy. Quarantine and analyse intermittent tests rather than blindly retrying them; a retry that turns red-then-green hides exactly the timing bug HIL exists to find.

The Embedded Testrack: HIL as a turnkey system

Building this loop from scratch — power control, flashing, touch actuation, frame capture, a stable harness — is a project in itself. Our Embedded Testrack is a productised HIL system: we integrate your device physically into the rack, wire up its real input and output interfaces, and connect the loop to your existing CI/CD pipeline and test frameworks. Every build is flashed to the real hardware automatically and driven through touch, CAN, serial and network; the display is captured and compared, LEDs and buses are read back, and each release gets an automated pass/fail — 24/7, with no simulated mockups and no human in the loop.

Conclusion

Host-side tests and simulation are the fast, cheap first line of defence — but they cannot see the driver, timing and integration failures that define embedded quality, and manual on-device testing is too sparse to close the gap. Hardware-in-the-loop testing makes the real device a first-class citizen of your pipeline, so every commit is validated on the hardware your customers actually use. If you want genuine CI/CD on real embedded hardware, talk to us about the Embedded Testrack.