Fuzzing Linux GUI/GTK Programs With American Fuzzy Lop (AFL) For Fun And Pr... You Get the Idea. Part One.

I've been trying to use American Fuzzy Lop (AFL) to fuzz stuff lately. Reading a bit about it inspired me to try it out as a file format fuzzer, though it can be instrumented to handle other kind of stuff. It's optimized for this kind of thing, so it shouldn't be difficult. Here is the general process for using AFL:

  1. Find some test cases for your program, these should be reasonably normal test cases, AFL will handle mangling them through smarty pants algorithms
  2. Compile the program with AFL's drop-in compilers afl-gcc and afl-gcc++ along with afl-clang and afl-clang++. I prefer using clang because it's a requirement for using another fuzzer, libFuzzer, which someday deserveres a post. The command looks something like CC=/path/to/afl-gcc CXX=/path/to/afl-g++ ./configure && make clean all usually (or optionally set the above as environment variables for other processes with export).
  3. Run AFL and point shit to other shit like so: ./afl-fuzz -i test_files_dir -o findings_dir /path/to/tested/program @@ where @@ is a placeholder for the input files from test_filles_dir

Pretty easy huh? Just compile the stupid program and run AFL on it. WRONG. Sorry, I mean, this assumption would be incorrect sometimes.

Target Selection and Problems

I went ahead and decided I wanted to fuzz xviewer - the default image viewer for Linux Mint 18.3 (current version at time of writing). Awesome, so I grabbed a few test cases from this handy little repo and went about my merry way. I started the fuzzing, instrumentation seemed OK, and then xviewer popped up and I sat there like an idiot. "Fly you beautiful bitch," I said to no avail.

Then I realized I'm an idiot. Of course this isn't working. xviewer, like most image viewing programs opens and remains open. AFL has no way of knowing that, and no instrumentation for closing the program itself. Thinking about it for a second this makes sense - how would it distinguish a slow-opening program from an infinite loop-type program like xviewer. Time is not an AFL's side, so sleeps and waiting are not really on the table (more on this later).

So I set about finding a solution to the problem.

Not Solving El Problema

Yes the above title is correct. I know problema ends in an A usually making it feminine, but it's not. It's el problema. Deal with it. /tumblr-rant.gif

Anyway, I have to admit my first thoughts on this were pretty stupid. What if I write a shellscript wrapper for the program that opens it, sleeps 0.1 seconds and closes it? Nerp. AFL does not take shell scripts as input:

[email protected] ~/exploit-dev/afl-2.52b $ ./afl-fuzz -o docs -i . whatever.sh @@
afl-fuzz 2.52b by <[email protected]>
[+] You have 8 CPU cores and 1 runnable tasks (utilization: 12%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[*] Checking CPU scaling governor...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning '.'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...

[-] PROGRAM ABORT : Program 'whatever.sh' not found or not executable
         Location : check_binary(), afl-fuzz.c:6854

And gives an error like so:

Oops, the target binary looks like a shell script. Some build systems will sometimes generate shell stubs for dynamically linked programs; try static library mode (./configure --disable-shared) if that's the case.

Another possible cause is that you are actually trying to use a shell wrapper around the fuzzed component. Invoking shell can slow down the fuzzing process by a factor of 20x or more; it's best to write the wrapper in a compiled language instead.

The code is even more telling:

/* Do a PATH search and find target binary to see that it exists and
   isn't a shell script - a common and painful mistake. We also check for
   a valid ELF header and for evidence of AFL instrumentation. */

Not only was my idea bad, it was just DEAD WRONG and by lcamtuf's view a "painful mistake", ouch. I thought about a wrapper in C with the sleep 0.1 idea or something and decided against it. I would run into other problems, like what if an image took more than 0.1 seconds to load? Is 0.1 the right delay or can it run faster/slower? It would also be quite inefficient, surely there's a part of the program that loads the image and won't require the GUI to actually show anything. The part that loads the image is what we want, that's where the overflows are at and we probably don't give a shit about the GUI very much, it's handled by a common and hardened library (GTK+).

Solving El Problema

Oh, did you come here thinking I'd have a magical solution? Sorry, I really don't have one. But I do have a general and repeatable-ish methodology I wanted to outline here and show how I solved the problem in the xviewer case specifically. It should generalize nicely, probably. Maybe. Just read the post yo.

So remember, I have xviewer that opens an image and then infinite loops until I close it. I can't have that, so somewhere in its mountain of C++ code (OK it's not that much, but enough to be annoying) I want to enter an exit(0); such that the program has loaded the image, but isn't doing a bunch of extra work to render it to the front-end and more importantly, doesn't loop infinitely until I close it. AFL wants several executions per second and I can't click that fast. Maybe some of you better Starcraft II players can, but you're probably busy playing Starcraft II.

Anyway, I didn't and don't know a ton about GTK+ applications, so I was a little blind. I started by reviewing the code in main.c, because wtf why not? I saw this in the main function:

set_startup_flags ();

XVIEWER_APP->priv->flags = flags;
if (force_new_instance) {
  GApplicationFlags app_flags = g_application_get_flags (G_APPLICATION (XVIEWER_APP));
  app_flags |= G_APPLICATION_NON_UNIQUE;
  g_application_set_flags (G_APPLICATION (XVIEWER_APP), app_flags);
  }

g_application_run (G_APPLICATION (XVIEWER_APP), argc, argv);
g_object_unref (XVIEWER_APP);

Ah-ha. So the part I saw as important is the g_application_run() function. This function kicks off several threads, returns almost immediately and continues on. This is a standard Gnome function. Hm, so I don't really want to get into editing source for Gnome or GTK+, so let's follow the breadcrumbs in xviewer. I'm interested in following the XVIEWER_APP definition:

[email protected] ~/exploit-dev/targets/xviewer/src $ find . -type f -exec grep -l "XVIEWER_APP" {} +
./xviewer-application-activatable.c
./xviewer-application.h
./xviewer-application-activatable.h
./xviewer-application-internal.h
./xviewer-application.c
./main.c
./xviewer-window.c

OK, awesome, I've got a lead in xviewer-application.h

#define XVIEWER_APP (xviewer_application_get_instance ())

And following this again to see where the function goes...

[email protected] ~/exploit-dev/targets/xviewer/src $ find . -type f -exec grep -l "xviewer_application_get_instance" {} +
./xviewer-main.o
./Xviewer-3.0.gir
./xviewer-application.h
./.libs/lt-xviewer
./.libs/libxviewer_la-xviewer-window.o
./.libs/libxviewer.ver
./.libs/libxviewer.exp
./.libs/libxviewer.so
./.libs/xviewer
./.libs/libxviewer_la-xviewer-application.o
./Xviewer-3.0.typelib
./xviewer-application.c

Sweet, so let's check out xviewer-application.c. Chosen from the list because it's the only human readable one I can see there and that I haven't looked at. I'm using this "methodology" so I can see where the higher-level functions are defined. I want to hook into (rather edit into) functions that are as high-level as possible because, well... that's the easiest route.

Anyway I wish I had some 1337 hacker trick I used here, but I don't... all I have is a little elbow grease and a fuckton of fprintf statements to give. In some of the more interesting-looking functions I added frprintf like so:

static void
xviewer_application_startup (GApplication *application)
{
    fprintf( stderr, "%s", "xviewer_application_startup\n", 30);

  XviewerApplication *app = XVIEWER_APPLICATION (application);
        GError *error = NULL;
        GFile *css_file;
        GtkSettings *settings;

Printing to stderr what function I was in. I also checked out the includes on xviewer-application.c and saw #include "xviewer-window.h" which is interesting. In particular, my logic went that once the window was ready to show, the image was obviously already processed. This might be a good place to prematurely exit the program. Another interesting header was the inclusion of #include "xviewer-image.h". It seemed to me that xviewer-image.c would be a good candidate to have a function that loaded images.

So I popped open xviewer-window.c and xviewer-image.c to check out the functions they had and fprintf'd them pretty hard as well. Interesting functions got the frprintf treatment:

static void
xviewer_window_display_image (XviewerWindow *window, XviewerImage *image)
{
  fprintf( stderr, "%s", "xviewer_window_display_image\n", 30);

        XviewerWindowPrivate *priv;
	GFile *file;

        g_return_if_fail (XVIEWER_IS_WINDOW (window));
        g_return_if_fail (XVIEWER_IS_IMAGE (image));

	xviewer_debug (DEBUG_WINDOW);
    
    ...

and

gboolean
xviewer_image_load (XviewerImage *img, XviewerImageData data2read, XviewerJob *job, GError **error)
{

  fprintf(stderr,"%s", "xviewer_image_load\n", 30);

  XviewerImagePrivate *priv;
        gboolean success = FALSE;

        xviewer_debug (DEBUG_IMAGE_LOAD);
        
        ...

The above looks pretty promising :-). So I ran the program - and it fell into the swamp. Then I ran it again... it burned down, fell over and THEN fell into the swamp. But the third time! The third time it ran just fine with this super clean output:

[email protected] ~/exploit-dev/targets/xviewer/src/.libs $ ./xviewer /home/punk/Images/politics.png
1111111111111111
xviewer_application_get_instance
xviewer_application_init

(xviewer:14668): XVIEWER-WARNING **: Error loading Xviewer typelib: Typelib file for namespace 'Xviewer', version '3.0' not found

2222222222
xviewer_application_get_instance
xviewer_application_startup

(xviewer:14668): libpeas-WARNING **: Type not found in introspection: 'XviewerApplicationActivatable'

(xviewer:14668): libpeas-WARNING **: Method 'XviewerApplicationActivatable.activate' was not found
xviewer_application_open
xviewer_application_open_file_list
xviewer_application_get_file_window
xviewer_application_get_empty_window
xviewer_application_get_instance
xviewer_application_get_instance
update_ui_visibility
xviewer_application_get_instance

(xviewer:14668): libpeas-WARNING **: Type not found in introspection: 'XviewerWindowActivatable'

(xviewer:14668): libpeas-WARNING **: Method 'XviewerWindowActivatable.activate' was not found
xviewer_image_load
xviewer_application_show_window

...program never returns, we're stuck...

Awesome. The last function I see running is xviewer_application_show_window after xviewer_image_load. This is promising! Pretty much what I was hoping to get from this exercise.

OK so now I've got two functions I can nab and insert exit(0);. This will allow the program to startup, load the image, then exit (ideally). So I had two options, insert the exit call at the end of xviewer_image_load or at the beginning of xviewer_application_show_window. Out of an abundance of caution (I really want to make sure that image loads or else I'm just wasting my time), I chose to go with exiting at the beginning of xviewer_application_show_window in case there was some additional image processing code I wanted included in the fuzzing process. So I popped open xviewer-application.c and made this small, but important change:

static void
xviewer_application_show_window (XviewerWindow *window, gpointer user_data)
{
    fprintf( stderr, "%s", "xviewer_application_show_window\n", 30);

     	//ACEDIT                                                                                                                                                                                   
        exit(0);

        gtk_window_present_with_time (GTK_WINDOW (window),
                                      GPOINTER_TO_UINT (user_data));

}

This should make the program open, load the image and then exit before showing the window. Now time to compile with AFL :-)!

[email protected] ~/exploit-dev/targets/xviewer $ export CC=/home/punk/exploit-dev/afl-2.52b/afl-clang
[email protected] ~/exploit-dev/targets/xviewer $ export CXX=/home/punk/exploit-dev/afl-2.52b/afl-clang++
[email protected] ~/exploit-dev/targets/xviewer $ make clean && make

...barely readable compilation and afl stuff...

Then running the program to see how our changes did...

[email protected] ~/exploit-dev/targets/xviewer/src/.libs $ ./xviewer /home/punk/Images/politics.png
1111111111111111
xviewer_application_get_instance
xviewer_application_init

(xviewer:14668): XVIEWER-WARNING **: Error loading Xviewer typelib: Typelib file for namespace 'Xviewer', version '3.0' not found

2222222222
xviewer_application_get_instance
xviewer_application_startup

(xviewer:14668): libpeas-WARNING **: Type not found in introspection: 'XviewerApplicationActivatable'

(xviewer:14668): libpeas-WARNING **: Method 'XviewerApplicationActivatable.activate' was not found
xviewer_application_open
xviewer_application_open_file_list
xviewer_application_get_file_window
xviewer_application_get_empty_window
xviewer_application_get_instance
xviewer_application_get_instance
update_ui_visibility
xviewer_application_get_instance

(xviewer:14668): libpeas-WARNING **: Type not found in introspection: 'XviewerWindowActivatable'

(xviewer:14668): libpeas-WARNING **: Method 'XviewerWindowActivatable.activate' was not found
xviewer_image_load
xviewer_application_show_window
[email protected] ~/exploit-dev/targets/xviewer/src/.libs $

Ahhhhhhh, it worked (for some (good) reason I never think any of my C code will work)! The application appears to have loaded my image, processed it, and just before showing it, closed, as I see no window appear. That's perfect. This is ready to take to AFL.

I got my corpus ready, shoved it in a directory and cd'd on over to my afl-fuzz directory. Getting ready to run:

[email protected] ~/exploit-dev/afl-2.52b $ ls testcases/images/all_images/
not_kitty.bmp  not_kitty.gif  not_kitty.ico  not_kitty.jpg  not_kitty.png  not_kitty.tiff
[email protected] ~/exploit-dev/afl-2.52b $ ./afl-fuzz -o out-xviewer/ -i testcases/images/all_images/ /home/punk/exploit-dev/targets/xviewer/src/.libs/xviewer @@

LET'S ROCK!!!!!!

afl-fuzz 2.52b by <[email protected]>
[+] You have 8 CPU cores and 1 runnable tasks (utilization: 12%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...

[-] Hmm, your system is configured to send core dump notifications to an
    external utility. This will cause issues: there will be an extended delay
    between stumbling upon a crash and having this information relayed to the
    fuzzer via the standard waitpid() API.

    To avoid having crashes misinterpreted as timeouts, please log in as root
    and temporarily modify /proc/sys/kernel/core_pattern, like so:

    echo core >/proc/sys/kernel/core_pattern

[-] PROGRAM ABORT : Pipe at the beginning of 'core_pattern'
         Location : check_crash_handling(), afl-fuzz.c:7275

...and fuck. Fixing this

punk-dtsp afl-2.52b # echo core >/proc/sys/kernel/core_pattern
punk-dtsp afl-2.52b # exit

LET'S GO!!!!!!!!!

afl-fuzz 2.52b by <[email protected]>
[+] You have 8 CPU cores and 1 runnable tasks (utilization: 12%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[*] Checking CPU scaling governor...

[-] Whoops, your system uses on-demand CPU frequency scaling, adjusted
    between 781 and 3417 MHz. Unfortunately, the scaling algorithm in the
    kernel is imperfect and can miss the short-lived processes spawned by
    afl-fuzz. To keep things moving, run these commands as root:

    cd /sys/devices/system/cpu
    echo performance | tee cpu*/cpufreq/scaling_governor

    You can later go back to the original state by replacing 'performance' with
    'ondemand'. If you don't want to change the settings, set AFL_SKIP_CPUFREQ
    to make afl-fuzz skip this check - but expect some performance drop.

[-] PROGRAM ABORT : Suboptimal CPU scaling governor
         Location : check_cpu_governor(), afl-fuzz.c:7337

...seriously? Is this ever going to run? This has gotta be it:

[email protected] ~/exploit-dev/afl-2.52b $ ./afl-fuzz -o out-xviewer/ -i testcases/images/all_images/ /home/punk/exploit-dev/targets/xviewer/src/.libs/xviewer @@
afl-fuzz 2.52b by <[email protected]>
[+] You have 8 CPU cores and 1 runnable tasks (utilization: 12%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[*] Checking CPU scaling governor...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning 'testcases/images/all_images/'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:not_kitty.bmp'...
[*] Spinning up the fork server...

[-] Hmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. There are two probable explanations:

    - The current memory limit (50.0 MB) is too restrictive, causing an OOM
      fault in the dynamic linker. This can be fixed with the -m option. A
      simple way to confirm the diagnosis may be:

      ( ulimit -Sv $[49 << 10]; /path/to/fuzzed_app )

      Tip: you can use http://jwilk.net/software/recidivm to quickly
      estimate the required amount of virtual memory for the binary.

    - Less likely, there is a horrible bug in the fuzzer. If other options
      fail, poke <[email protected]> for troubleshooting tips.

[-] PROGRAM ABORT : Fork server handshake failed
         Location : init_forkserver(), afl-fuzz.c:2253

@!#^#@. RUNNING AGAIN:

[email protected] ~/exploit-dev/afl-2.52b $ ./afl-fuzz -m 1000 -o out-xviewer/ -i testcases/images/all_images/ /home/punk/exploit-dev/targets/xviewer/src/.libs/xviewer @@
afl-fuzz 2.52b by <[email protected]>
[+] You have 8 CPU cores and 2 runnable tasks (utilization: 25%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[*] Checking CPU scaling governor...
[*] Setting up output directories...
[*] Scanning 'testcases/images/all_images/'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:not_kitty.bmp'...
[*] Spinning up the fork server...
[+] All right - fork server is up.
    len = 630, map size = 1040, exec speed = 150453 us
[*] Attempting dry run with 'id:000001,orig:not_kitty.gif'...
    len = 198, map size = 1040, exec speed = 152591 us
[!] WARNING: No new instrumentation output, test case may be useless.
[*] Attempting dry run with 'id:000002,orig:not_kitty.ico'...
    len = 367, map size = 1061, exec speed = 119392 us
[*] Attempting dry run with 'id:000003,orig:not_kitty.jpg'...
    len = 413, map size = 1137, exec speed = 148920 us
[*] Attempting dry run with 'id:000004,orig:not_kitty.png'...
    len = 218, map size = 1139, exec speed = 152403 us
[*] Attempting dry run with 'id:000005,orig:not_kitty.tiff'...
    len = 448, map size = 1060, exec speed = 149141 us
[!] WARNING: Instrumentation output varies across runs.
[+] All test cases processed.

[!] WARNING: The target binary is pretty slow! See docs/perf_tips.txt.
[!] WARNING: Some test cases look useless. Consider using a smaller set.
[+] Here are some useful stats:

    Test case count : 5 favored, 1 variable, 6 total
       Bitmap range : 1040 to 1139 bits (average: 1079.50 bits)
        Exec timing : 119k to 152k us (average: 146k us)

[*] No -t option specified, so I'll use exec timeout of 300 ms.
[+] All set and ready to roll!

Fucking finally AFL starts running:

afl-xviewer

Problemas and ideas

It's a bit slow, capping at about 10 execs/sec, and I'm not too thrilled about that, but such might be the reality of image fuzzing a GUI app. Checking out the process list I see that Cinnamon (Mint's front-end) is getting pounded pretty hard with high CPU usage - that sucks, I was trying to avoid that. One solution may be to hook into that xviewer_image_load function instead of the one I chose - but then again, I bet it's standard GTK+ stuff to initialize windows and all of that as these other processes (like image processing) go on. So maybe that will help, maybe it won't.

However it IS running and perhaps I can split this into multiple fuzzes - one for each image type that xviewer can handle. Whatever though, it's working ¯_(ツ)_/¯.

Side notes

A common theme in the GTK apps I've worked with are to generate a directory like this:

[email protected] ~/exploit-dev/targets/xviewer/src $ ls -alh
.libs
...snip...
-rw-rw-r--  1 punk punk  90K Feb  6 00:02 Makefile
-rw-rw-r--  1 punk punk 6.2K Feb  4 16:48 Makefile.am
-rw-rw-r--  1 punk punk  99K Feb  4 17:02 Makefile.in
-rwxrwxr-x  1 punk punk 8.7K Feb  7 15:13 xviewer
-rw-rw-r--  1 punk punk 189K Feb  7 15:13 Xviewer-3.0.gir
-rw-rw-r--  1 punk punk  38K Feb  7 15:13 Xviewer-3.0.typelib
...snip...

Looks like xviewer is the executable huh? Well it's not, it's an autogenerated stub shell script to run xviewer. The actual executable is in that .libs directory:

[email protected] ~/exploit-dev/targets/xviewer/src $ cd .libs/
...snip...
-rwxrwxr-x 1 punk punk 2.1M Feb  7 15:13 libxviewer.so
-rw-rw-r-- 1 punk punk 9.0K Feb  7 15:13 libxviewer.ver
-rwxrwxr-x 1 punk punk  23K Feb  7 15:13 lt-xviewer
-rwxrwxr-x 1 punk punk  23K Feb  7 15:13 xviewer
...snip...
[email protected] ~/exploit-dev/targets/xviewer/src/.libs $ file xviewer 
xviewer: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
[email protected] ~/exploit-dev/targets/xviewer/src/.libs $ ./xviewer 
./xviewer: error while loading shared libraries: libxviewer.so: cannot open shared object file: No such file or directory

Ahh, there's the real executable. But it won't run, we need to add an LD_LIBRARY_PATH so it knows where to find the shared lib it needs:

[email protected] ~/exploit-dev/targets/xviewer/src/.libs $ export LD_LIBRARY_PATH=/home/punk/exploit-dev/targets/xviewer/src/.libs
[email protected] ~/exploit-dev/targets/xviewer/src/.libs $ ./xviewer 
1111111111111111
xviewer_application_get_instance
xviewer_application_init

(xviewer:5907): XVIEWER-WARNING **: Error loading Xviewer typelib: Typelib file for namespace 'Xviewer', version '3.0' not found

2222222222
xviewer_application_get_instance
xviewer_application_add_platform_data
xviewer_application_get_instance
xviewer_application_finalize

There it is :-). She's running beautifully now (note output is slightly different because I didn't specify an image to open).

Conclusion and More Problems

Fuzzing GUI apps that take input and stay open is a little bit of a pain in the ass. The key is finding that right point where you can exit(0); IMHO. So get ready to hunker down and understand the code a bit/use some "hacker street smarts" to find the functions you're looking for where you can exit. There are some preprocessor tricks I've found that might be more useful than peppering print statements, but hell, I got the job done (mostly) and can probably do it again (xreader - you're next).

Anyway, this process should work for stuff like image viewers, readers, browsers and more and I wanted to get it on "paper" before I forgot it all. Thanks for reading!

Oh and why is this part 1? Because now I have to deploy it to our fuzzing server, which is running a different OS (Debian 9). This should be fun and I may do a write-up of it. I also need to come back with results! However, if they're boring/lame this will remain Part 1 (like History of the World!) and be done, probably.

Happy hacking :-) ☠️☠️☠️,

Alex