• Introduction to FUSE
• How does FUSE work
• Use GDB with FUSE
• Trouble Shooting
• Further Read

During the past years, I have been involved in several storage related projects from which I have learned a lot about the world of storage. Definitely the journey is interesting. However, it is not awesome because almost all the work I have been involved is about applications of storage. Then I decided to dig more deeper in the storage world. It turned out that it is really wonderful to explore in the world of storage.

Actually I have heard about FUSE before but did not spend time to explore it because I thought FUSE would bring a lot of overhead and hence won’t get too much application in system design. During the past months, I have seen the implementations of a few storage solutions, both open source and commercial ones. To my surprise, FUSE is used in those solutions. It is time to get acquainted with it. Per my experience, debugging is the very very useful way to learn a new technology. In this post, I will show how to use GDB to debug the hello example. You can set break points, use single step execution to observe the behavior of the FUSE application. CLAIM: I am not an expert on GDB or FUSE and I can’t guarantee that it is practical in debugging your real world solution.

Introduction to FUSE

From the OS course in undergraduate school, your are told that file system is part of the kernel and it is almost true in the real world. That means typically if you want to create your own file system in a real world OS such as Linux, you have to program at the kernel space. However you can create your own file system totally in user space with the help of FUSE: Filesystem in Userspace. What is the difference? Well, there is much more restriction in kernel space programming than user space. For example, no call to third parties library such as glibc, limited recursive depth. Since Linux kernel is monolithic, a crashed kernel space program(kernel module) could make the whole OS crash too.

There is a simple hello file system from the homepage of the FUSE project. Use following command lines to test the hello file system. I tested it in a Ubuntu 14.04 x86_64 box.

1
2
3

gcc -Wall hello.c `pkg-config fuse --cflags --libs` -o hello
mkdir /tmp/fuse
./hello /tmp/fuse

If every thing is all right, you should see following result:

1
2
3
4
5

jeff@trusty:~$ ls -l /tmp/fuse
total 0
-r--r--r-- 1 root root 13 Jan  1  1970 hello
jeff@trusty:~$ cat /tmp/fuse/hello
Hello World!

How does FUSE work

Fig-1: Workflow of FUSE

Fig-1 is the workflow of FUSE fetched from the homepage of the FUSE project. The remaining in this section will try to explain the process.

1. Just like EXT4, EXT3, the hello file system much be mounted before it can be used. The command ./hello /tmp/fuse will mount it. It should be noted that this hello command will not only mount the hello FS, but also running as a daemon in the background. This is very important to understand the whole work process. Now you can verify whether the fs is mounted or not by issuing mount in the command line.
2. Users issue the command ls -l /tmp/fuse
3. The ls program will call functions lived in glibc
4. The glibc fucntions called in step 2 will send the requests to the kernel’s VFS subsystem with sys calls.
5. The VFS sub system will call FUSE module lived in the kernel like it does when dealing with other file systems such as NFS, EXT4. Util now, the workflow is the same with any other FS.
6. Unlike other FSes, the FUSE module in kernel will now call the the hello daemon created in Step 1.
7. The hello daemon then sends the result back to the FUSE kernel module
8. The rest work is the same with other FSes. The result will be passed in the path: FUSE kernel module -> VFS -> glibc -> ls command

With the explanation above, you should understand the overall process. So the FUSE is actually a framework which lives in the kernel space like other file systems. A FUSE file system must be built under that framework.

Next let’s look more deeper. The GDB will be leveraged to track the detail of function calls in next section. BTW, I have not figured out how the FUSE kernel module communicates with the hello daemon living in the user space yet.

Use GDB with FUSE

Since the hello daemon runs in the user space, we can attach it to the GDB like other user space applications. But before that we should make some extra preparations.

1
2
3

gcc -Wall -g hello.c `pkg-config fuse --cflags --libs` -o hello # Compile with debug symbols flag
./hello /tmp/fuse -s # Run the daemon in single thread mode
ps -ef | grep hello # Search the pid of the hello daemon

Now it is ready to enter the gdb by issue gdb64 in the command line. Attach the process to the gdb by attach PID_OF_THE_HELLO_DAMONE. Since we want to track which functions are called when the command ls -l /tmp/fuse is issued, we should set breakpoints for the 4 functions, namely hello_open, hello_getattr, hello_readdir, hello_read with following gdb commands.

1
2
3
4
5
6
7
8
9
10

(gdb) break hello_getattr
Breakpoint 1 at 0x40076d: file hello.c, line 24.
(gdb) break hello_readdir
Breakpoint 2 at 0x40082d: file hello.c, line 46.
(gdb) break hello_open
Breakpoint 3 at 0x4008b1: file hello.c, line 58.
(gdb) break hello_read
Breakpoint 4 at 0x400909: file hello.c, line 72.
(gdb) continue
Continuing.

Now the gdb should be in listen on the daemon. When a break point is hit, it will stop the daemon process immediately. Let’s open a new shell windows and issue the command ls -l /tmp/fuse. DO NOT use tab-complete feature when entering the command. Or the behavior it will be a little confusing though it could be still reasoning. Let’s check the first break point.

1
2
3
4
5

(gdb) continue
Continuing.

Breakpoint 1, hello_getattr (path=0x6d5100 "/", stbuf=0x7fff92c71080) at hello.c:24
24		int res = 0;

From the output, we can see that the command will first get the attributes of the hello file system’s root director, namely /tmp/fuse mount point. Enter continue to resume the daemon process.

1
2
3
4
5
6
7
8
9

Breakpoint 2, hello_readdir (path=0x6f5940 "/", buf=0x6d5100, filler=0x7f27f8e8e220, offset=0, fi=0x7fff92c710e0) at hello.c:46
46		if (strcmp(path, "/") != 0)
(gdb) continue
Continuing.

Breakpoint 1, hello_getattr (path=0x6f5940 "/hello", stbuf=0x7fff92c710b0) at hello.c:24
24		int res = 0;
(gdb) continue
Continuing.

When can see all the rest function calls for the ls command from the above output which is quite self-explanatory. You can also use similar way to track the function calls when the command cat /tmp/fuse/hello is issued.

Hope this posts helps you understand how FUSE works!

Trouble Shooting

• Fail to attach the daemon process in GDB. The error looks like this:

1
2
3

(gdb) attach 13387
Attaching to process 13387
"/home/jeff/workspace/fuse/hello": not in executable format: File format not recognized

This is because the hello executable file is a 64 bit binary but the gdb is 32 bit. Use the 64-bit GDB gdb64.

• Fail to link the executable file with error:

1
2
3

/tmp/ccIMWTZv.o: In function `main':
hello.c:(.text+0x268): undefined reference to `fuse_main_real'
collect2: error: ld returned 1 exit status

This is a silly problem. Make sure you use the the correct command

1
2

gcc -Wall hello.c `pkg-config fuse --cflags --libs` -o hello # work
gcc -Wall `pkg-config fuse --cflags --libs` hello.c -o hello # can't work

Further Read

1. FUSE for OS X : FUSE like framework in Mac OS X
2. Fuse on Solaris: Fuse on OpenSolaris
3. Opendedup: A deduplication solution built with FUSE