Open Source Summit + Embedded Linux Conference Europe 2020

AI training process is complex and invisible, when running the task, there are some monitoring blind spots by using the traditional tracing tools, which brings many difficulties to the developers to debug and tuning. For this reason, we choose eBPF to analyze the changes what we want to know in the real-time, such as: to understand whether a specific kernel function is called, short-lifetime processes, etc. With the data collected dynamically by eBPF, we choose the Prometheus to monitor and show them to the developers. In this topic, I will share the practice of eBPF in the observability of AI kernel. While running the AI training and reasoning tasks, we can dynamically inject the eBPF code into the kernel function to collect data, and report the data to the Prometheus in a unified format for visual management. The practice of the observability is currently in the experimental stage.