During the past years, I have been involved in several storage related projects from which I have learned a lot about the world of storage. Definitely the journey is interesting. However, it is not awesome because almost all the work I have been involved is about applications of storage. Then I decided to dig more deeper in the storage world. It turned out that it is really wonderful to explore in the world of storage.
Actually I have heard about FUSE before but did not spend time to explore it because I thought FUSE would bring a lot of overhead and hence won’t get too much application in system design. During the past months, I have seen the implementations of a few storage solutions, both open source and commercial ones. To my surprise, FUSE is used in those solutions. It is time to get acquainted with it. Per my experience, debugging is the very very useful way to learn a new technology. In this post, I will show how to use GDB to debug the hello example. You can set break points, use single step execution to observe the behavior of the FUSE application. CLAIM: I am not an expert on GDB or FUSE and I can’t guarantee that it is practical in debugging your real world solution.
Introduction to FUSE
From the OS course in undergraduate school, your are told that file system is part of the kernel and it is almost true in the real world. That means typically if you want to create your own file system in a real world OS such as Linux, you have to program at the kernel space. However you can create your own file system totally in user space with the help of FUSE: Filesystem in Userspace. What is the difference? Well, there is much more restriction in kernel space programming than user space. For example, no call to third parties library such as glibc, limited recursive depth. Since Linux kernel is monolithic, a crashed kernel space program(kernel module) could make the whole OS crash too.
There is a simple hello file system from the homepage of the FUSE project. Use following command lines to test the hello file system. I tested it in a Ubuntu 14.04 x86_64 box.
1 2 3
If every thing is all right, you should see following result:
1 2 3 4 5
How does FUSE work
Fig-1 is the workflow of FUSE fetched from the homepage of the FUSE project. The remaining in this section will try to explain the process.
- Just like EXT4, EXT3, the hello file system much be mounted before it can be used. The command
./hello /tmp/fusewill mount it. It should be noted that this
hellocommand will not only mount the hello FS, but also running as a daemon in the background. This is very important to understand the whole work process. Now you can verify whether the fs is mounted or not by issuing
mountin the command line.
- Users issue the command
ls -l /tmp/fuse
lsprogram will call functions lived in glibc
- The glibc fucntions called in step 2 will send the requests to the kernel’s VFS subsystem with sys calls.
- The VFS sub system will call FUSE module lived in the kernel like it does when dealing with other file systems such as NFS, EXT4. Util now, the workflow is the same with any other FS.
- Unlike other FSes, the FUSE module in kernel will now call the the hello daemon created in Step 1.
- The hello daemon then sends the result back to the FUSE kernel module
- The rest work is the same with other FSes. The result will be passed in the path: FUSE kernel module -> VFS -> glibc -> ls command
With the explanation above, you should understand the overall process. So the FUSE is actually a framework which lives in the kernel space like other file systems. A FUSE file system must be built under that framework.
Next let’s look more deeper. The GDB will be leveraged to track the detail of function calls in next section. BTW, I have not figured out how the FUSE kernel module communicates with the hello daemon living in the user space yet.
Use GDB with FUSE
Since the hello daemon runs in the user space, we can attach it to the GDB like other user space applications. But before that we should make some extra preparations.
1 2 3
Now it is ready to enter the gdb by issue
gdb64 in the command line. Attach the process to the gdb by
attach PID_OF_THE_HELLO_DAMONE. Since we want to track which functions are called when the command
ls -l /tmp/fuse is issued, we should set breakpoints for the 4 functions, namely
hello_open, hello_getattr, hello_readdir, hello_read with following gdb commands.
1 2 3 4 5 6 7 8 9 10
Now the gdb should be in listen on the daemon. When a break point is hit, it will stop the daemon process immediately. Let’s open a new shell windows and issue the command
ls -l /tmp/fuse. DO NOT use tab-complete feature when entering the command. Or the behavior it will be a little confusing though it could be still reasoning. Let’s check the first break point.
1 2 3 4 5
From the output, we can see that the command will first get the attributes of the hello file system’s root director, namely
/tmp/fuse mount point. Enter
continue to resume the daemon process.
1 2 3 4 5 6 7 8 9
When can see all the rest function calls for the
ls command from the above output which is quite self-explanatory. You can also use similar way to track the function calls when the command
cat /tmp/fuse/hello is issued.
Hope this posts helps you understand how FUSE works!
- Fail to attach the daemon process in GDB. The error looks like this:
1 2 3
This is because the hello executable file is a 64 bit binary but the gdb is 32 bit. Use the 64-bit GDB
- Fail to link the executable file with error:
1 2 3
This is a silly problem. Make sure you use the the correct command