JoelFernandes.org

Age is of no importance unless you're a cheese -- Billie Burke

Blog posts by category.
Archive of all blog posts.
Presentations and other work.

Dumping User and Kernel stacks on Kernel events

| Comments

Dumping the native kernel and userspace stack when a certain path in the kernel or userspace occurs, can be useful to understand which code paths triggered a certain behavior that you’re trying to debug, such as an error you found in the log. One such case is when you notice Selinux denial messages in logs but want to know which path triggered it.

In this article we will show you how to use kernel instrumentation and BCC to dump the both the user and kernel stack. The article applies both to Android and regular Linux kernels.

Example: Understanding which path triggered an SELinux denial

Step 1: Add a tracepoint to the kernel

Apply the following diff to your kernel. It adds a tracepoint at precisely the point where an SELinux denial is logged. If not cleanly applying, patch it in manually.

Diff to add a tracepoint for selinux denials

diff --git a/include/trace/events/selinux.h b/include/trace/events/selinux.h
new file mode 100644
index 000000000000..dac185062634
--- /dev/null
+++ b/include/trace/events/selinux.h
@@ -0,0 +1,34 @@
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM selinux
+
+#if !defined(_TRACE_SELINUX_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_SELINUX_H
+
+#include <linux/ktime.h>
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(selinux_denied,
+
+	TP_PROTO(int cls, int av),
+
+	TP_ARGS(cls, av),
+
+	TP_STRUCT__entry(
+		__field(	int,		cls	)
+		__field(	int,		av	)
+	),
+
+	TP_fast_assign(
+		__entry->cls = cls;
+		__entry->av = av;
+	),
+
+	TP_printk("denied %d %d",
+		__entry->cls,
+		__entry->av)
+);
+
+#endif /* _TRACE_SELINUX_H */
+
+/* This part ust be outside protection */
+#include <trace/define_trace.h>
diff --git a/security/selinux/avc.c b/security/selinux/avc.c
index 84d9a2e2bbaf..ab04b7c2dd01 100644
--- a/security/selinux/avc.c
+++ b/security/selinux/avc.c
@@ -34,6 +34,9 @@
 #include "avc_ss.h"
 #include "classmap.h"
 
+#define CREATE_TRACE_POINTS
+#include <trace/events/selinux.h>
+
 #define AVC_CACHE_SLOTS			512
 #define AVC_DEF_CACHE_THRESHOLD		512
 #define AVC_CACHE_RECLAIM		16
@@ -713,6 +716,12 @@ static void avc_audit_pre_callback(struct audit_buffer *ab, void *a)
 	struct common_audit_data *ad = a;
 	audit_log_format(ab, "avc:  %s ",
 			 ad->selinux_audit_data->denied ? "denied" : "granted");
+
+	if (ad->selinux_audit_data->denied) {
+		trace_selinux_denied(ad->selinux_audit_data->tclass,
+				     ad->selinux_audit_data->audited);
+	}
+
 	avc_dump_av(ab, ad->selinux_audit_data->tclass,
 			ad->selinux_audit_data->audited);
 	audit_log_format(ab, " for ");

Step 2: Install adeb

Run the command:

adeb prepare --full

This also installs BCC on the Android device which contains the ‘trace’ utility we need for the next step. For regular Linux kernels, you may have to manually install BCC or find a package for it.

Step 3: Start tracing the user and kernel stacks

Running the following command:

adeb shell
trace -K -U 't:selinux:selinux_denial'

You should see something like this when denials are triggered:

2286    2434    Binder:2286_4   selinux_denied   

        avc_audit_pre_callback+0xd8 [kernel]
        avc_audit_pre_callback+0xd8 [kernel]
        common_lsm_audit+0x64 [kernel]
        slow_avc_audit+0x74 [kernel]
        avc_has_perm+0xb8 [kernel]
        selinux_binder_transfer_file+0x158 [kernel]
        security_binder_transfer_file+0x50 [kernel]
        binder_translate_fd+0xcc [kernel]
        binder_transaction+0x1b64 [kernel]
        binder_ioctl+0xadc [kernel]
        do_vfs_ioctl+0x5c8 [kernel]
        sys_ioctl+0x88 [kernel]
        __sys_trace_return+0x0 [kernel]
        __ioctl+0x8 [libc.so]
        android::IPCThreadState::talkWithDriver(bool)+0x104 [libbinder.so]
        android::IPCThreadState::waitForResponse(android::Parcel*, int*)+0x40
                                                            [libbinder.so]
        android::IPCThreadState::executeCommand(int)+0x460 [libbinder.so]
        android::IPCThreadState::getAndExecuteCommand()+0xa0 [libbinder.so]
        android::IPCThreadState::joinThreadPool(bool)+0x40 [libbinder.so]
        [unknown] [libbinder.so]
        android::Thread::_threadLoop(void*)+0x12c [libutils.so]
        android::AndroidRuntime::javaThreadShell(void*)+0x90 [libandroid_runtime.so]
        __pthread_start(void*)+0x28 [libc.so]
        __start_thread+0x48 [libc.so]

The same trick can be used for dumping the stack on syscalls, random kernel functions using kprobes and more! Just change the arguments passed to the ‘trace’ command.

Comments