Open Source Summit + Embedded Linux Conference Europe 2020

Sometimes you find yourself looking at something a tiny bit incorrect, like your CI machinery reports a couple of more failures than you expect (surely you want zero failures). So you decide to fix it. You try to reproduce it outside the CI and... everything just works. OK, then you know who's guilty, right? That simple. And you ask your DevOps people to go fix their scripts. But apparently nothing helps, tests still fail in CI. And one fine day you decide to scratch that itch for real and start a journey down the rabbit hole. In this talk we'll reconstruct one very real debugging session which started from Zephyr RTOS tests failing in Jenkins-based CI flow and ended deep in the guts of the instruction set simulator (ISS). One by one we'll be inspecting possible faulty components (Jenkins, Shell & Python scripts used for test execution, Zephyr RTOS tests themselves and finally the simulator) until we may explain all the peculiarities observed before.