Real-Time Linux: What PREEMPT_RT Delivers – and What It Does Not

In short: Real-time means predictability, not speed. A system is real-time capable when it reliably meets a deadline – including in the worst case. PREEMPT_RT makes the Linux kernel suitable for that by making nearly all of it preemptible. That is half the job: the other half lies in system configuration and in the application, and that is where most hard-won microseconds get thrown away again.

Determinism is not throughput

The most common confusion in any real-time discussion concerns throughput. A standard kernel is optimized for average throughput: it may occasionally take a long time, as long as it gets a lot done overall. A real-time system inverts that. It cares about the worst case – about how late a response can possibly arrive.

The relevant metric is therefore never the mean. A controller that services 999,999 cycles in 50 microseconds and takes 8 milliseconds on the millionth has excellent average latency and is still useless if its deadline is 200 microseconds. From that perspective follows the central trade: determinism costs throughput. Wanting to maximize both gets you neither.

What PREEMPT_RT actually changes

Over two decades the RT project reworked the kernel wherever it was not preemptible. With Linux 6.12 the last core pieces landed in mainline – PREEMPT_RT is no longer a patch series to carry along but a configuration option. Four changes carry the effect:

Threaded interrupts. Interrupt handling largely runs in ordinary kernel threads rather than in hard interrupt context. That makes handlers schedulable – and a handler belonging to an unimportant device can no longer block a time-critical task for an arbitrary period.

Sleeping locks. Most spinlocks become mutexes that can sleep. A task that fails to acquire a lock no longer occupies the CPU but yields it.

Priority inheritance. When a low-priority task holds a lock that a high-priority task needs, it inherits that priority until it releases the lock. This defuses classic priority inversion, where a medium-priority task starves the work that actually matters.

High-resolution timers. Time-driven work is no longer tied to the system tick but woken at finer granularity.

The result is a kernel that, on suitable hardware and with clean configuration, holds response times in the range of tens to a few hundred microseconds. The spread is deliberate: it depends more on the platform than on the kernel.

The preemption models at a glance

Model	Behaviour	Typical use
PREEMPT_NONE	Kernel is not preempted	Servers, batch processing, maximum throughput
PREEMPT_VOLUNTARY	Defined preemption points	General systems with somewhat better response
PREEMPT	Kernel largely preemptible	Desktop, multimedia, many embedded devices
PREEMPT_RT	Almost fully preemptible, threaded IRQs	Control loops, motion control, audio

The step from PREEMPT to PREEMPT_RT is smaller than many expect. For devices whose deadlines sit in the range of several milliseconds, the PREEMPT model is often enough – a worthwhile first test before committing to the full RT route.

Measure, don’t assume

Latency figures without measurement conditions are worthless. The standard tool is cyclictest from rt-tests: it wakes a task at a fixed interval and measures how far the actual wake-up deviates from the intended one.

# One RT task on an isolated core, ten minutes, with a histogram
cyclictest -m -S -p 80 -i 200 -h 400 -D 10m

Two rules decide whether the result means anything. First, measure under load. An idle system produces attractive numbers that say nothing about operation. Generate the load that occurs in the field in parallel – network traffic, writes to flash storage, graphics output, fieldbus access. Second, measure long enough. Rare outliers are precisely the ones that bite in the field; runs over hours or days are normal. What matters is always the maximum deviation, never the average.

When an outlier shows up, ftrace takes it further: the kernel can record what caused the delay. Because such runs become a regression question the moment a product enters maintenance, they belong in automated verification – we run them on real hardware, as described in our article on hardware-in-the-loop testing.

The system around the kernel

An RT kernel on its own guarantees nothing. In practice these points decide the outcome:

• CPU isolation. One or more cores are withdrawn from the scheduler (isolcpus) and relieved of periodic work (nohz_full, rcu_nocbs) so that only time-critical work runs there.
• Interrupt affinity. Interrupts from irrelevant devices are steered away from the isolated cores, those of the time-critical device deliberately towards them.
• Frequency and power management. Dynamic clocking and deep sleep states are among the most common latency sources: waking from a deep C-state costs time that, in the worst case, lands directly in your response budget.
• Firmware interruptions. On x86 platforms, System Management Interrupts can stall the kernel invisibly. They are not controllable from the operating system, which makes platform selection itself a real-time decision.
• Memory. A page fault in the control path costs more than anything the scheduler will ever save.

The application has a vote

Even the best configuration cannot save an application that ignores real-time discipline. Four points carry most of the weight:

// Lock memory so the control path takes no page faults
mlockall(MCL_CURRENT | MCL_FUTURE);

// Real-time scheduling with a deliberately chosen priority
struct sched_param param = { .sched_priority = 80 };
pthread_setschedparam(thread, SCHED_FIFO, &param);

First the scheduling class: without SCHED_FIFO, SCHED_RR or SCHED_DEADLINE the task remains an ordinary process competing with everything else. Second the choice of priority – it has to sit below the important kernel threads, or the application starves exactly the interrupt threads it depends on. Third no dynamic memory in the hot path: allocation can block, so buffers are reserved up front. The language tools that help here are covered in our article on modern C++ for embedded systems. Fourth no blocking calls in the control cycle – no logging to a full filesystem, no network operation without a timeout.

When Linux is the wrong answer

Some requirements defeat even a perfectly configured RT Linux: deadlines in the single-digit microsecond range, strictly cyclic control in the high kilohertz range, or evidence obligations demanding a small, certifiable codebase.

The usual answer is not either-or but a split. Many modern SoCs pair the application cores with a microcontroller core that takes the hard control loop while Linux serves the user interface, networking and data handling. That division delivers both – hard deadlines where they are needed and a full operating system for everything else. Which split makes sense for a given product is something we work through regularly in technical consulting.

Getting there in your own product

In a Yocto-based setup the RT kernel is usually a matter of kernel selection in the BSP – many layers ship an RT variant, so recipes and configuration carry over. The effort rarely lies in the switch but in the validation that follows: drivers that were unremarkable on a standard kernel can stand out under RT because they linger too long in non-preemptible sections. That is exactly what measuring under load is for. How we cut and maintain BSPs is described under Yocto BSP & distributions and embedded Linux.

Conclusion

PREEMPT_RT is a mature tool that makes Linux viable for a large share of industrial control work – and since mainline inclusion, without the baggage of an external patch series. It does not replace system engineering, though: isolation, interrupt affinity, power management and a disciplined application ultimately decide the numbers. Measure on the target hardware, under realistic load, for long enough, and design against the maximum rather than the mean. If you would like an assessment of whether your deadlines are reachable with Linux or whether a split architecture is the better route, get in touch.