Anatomy of Qemu Dynamic Translation Using GDB Part 1

Purpose:

Show the steps to find what micro code the emulated CPU's instructions are being translated to.

Background:

Qemu is a fast and portable emulator. It currently can emulate several CPUs such as the x86, PowerPC, ARM, and Sparc. One of the reasons for Qemu's speed and portability is due to how the system translates emulated CPU instructions into the instructions of the host machine. This is accomplished by performing portable dynamic translation of the emulated instructions to the host instructions. The process begins by parsing the emulated CPU instructions into micro code. Micro code is an intermediary set of methods written in C that allows Qemu to be portable and fast. From the micro code, the emulated instructions are then translated into the instructions used by the host machine.

Procedure:

1. Boot up Linux image and navigate to qemu_images directory

2. Copy parameters in run.sh (everything from -m on) to run emulated OS
- vim run.sh (in the terminal to open text editor with run.sh loaded in)

- highlight contents in picture -> Edit -> Copy

- exit vim (ESC -> SHIFT + : -> q -> ENTER)

3. Type "sudo gdb qemu-system-i386" in the terminal window and press <ENTER> 

4. Input password and press <ENTER>

5. Type "b disas_insn" and press <ENTER> to set a breakpoint at the disas_insn function located in translate.c of the qemu source code

- The disas_insn function is responsible for handling the instructions of the emulated CPU by going through a giant switch case that passes off the instruction to the appropriate handler and translates it to the correct micro code operation

6. Type "handle SIGUSR1 noprint" and press <ENTER>

7. Type "run " and then paste the parameters that you copied previously from run.sh and press <ENTER>

- your screen should look like the following picture

- you are now in the disas_insn function. Here you can use the gdb command, list, to display code, use n to go to the next instruction and run the program line by line

8. Type "n" and press <Enter> 25 times until you get up to the instruction in the below picture:

- this function is partly responsible for translating the emulated CPU instructions into micro code

9. Type the gdb command "step" to step into the function call

10. Type "n" to go to the first line of the gen_op_movl_T0_im method

11. Step into this function (use the gdb command "step") and then type "list" to view the source code at this point

12. Type "n" to go to the first line of the method

- INDEX_op_movi_i32 is micro code for the particular emulated OS instruction

13. Now lets find where this particular micro code is declared. Type "shell" and press <ENTER>

14. Perform a search using grep: 

- the grep command to use is in the yellow box

- everything in the below picture is a result of the search for that particular micro code

As you may be able to see, none of the contents returned in the grep search is a definition for this particular piece of micro code. This is because the designers used a "tricky" piece of C code to create the micro code. Lets try performing a search for "INDEX" instead of the whole micro code.

15. Perform a grep search for "INDEX_op" 

- it may help to pipe the results of the grep search into the linux command "more" because there will be too much content to display (grep -R "INDEX_op" * | more)

- after paging through the results of the grep search (by pressing <SPACEBAR>) we will see that INDEX_op is actually defined in a macro in the file tcg/tcg.h

Let us open up tcg/tcg.h and look closer at the definition of INDEX_op

16. Open tcg/tcg.h into a text editor (vim ~/qemu/qemu-1.4.0/tcg/tcg.h) and perform a search for the macro (<SHIFT> + :  --> /DEF( --> <ENTER>) 

- you should find the contents of the below picture

Let us break down this code segment from the inside out. Line 105 declares a macro definition containing 5 parameters and joins "INDEX_op_" with the name parameter of the DEF macro by using the ## operator. (If you are unfamiliar with the ## operator, click here)  This is how the names of the micro code are generated (i.e. INDEX_op_movi_i32) . You may be wondering where are the names coming from though. Just below the DEF definition there is an #include statement for the file tcg-opc.h. It is in that file that the above macro is being used to generate an enum called TCGOpcode of type TCGOpcode containing all the micro code names. This way each piece of micro code gets a name like the one above and a corresponding number (index) in the enum. Then on line 107, DEF is undefined so it cannot be used anywhere else except this code segment. Following that on line 108, it appends NB_OPS to the enum. Let us investigate further into the tcg-opc.h file and see how the micro code is defined.

17. Open tcg/tcg-opc.h in a text editor (vim ~/qemu/qemu-1.4.0/tcg/tcg-opc.h) and go to line 51

As you can see, the name parameter of the DEF macro matches that of the micro code we found earlier when it is appended to the end of "INDEX_op_" as done by the DEF macro. The entirety of this file contains definitions for all of the micro code generated by the qemu emulator. Let us now look into what the remaining of the parameters of the DEF macro do (oars, iargs, cargs, flags). From the definition of DEF in tcg.h, it would appear that the other parameters are not used however, that is not the case.

18. Exit the file and do a grep search for DEF( again 

- grep -R "define DEF(" *     (NOTE: do this in qemu-1.4.0 folder)

We can see from our search that there is another definition of DEF located in tcg/tcg.c file that utilizes all the parameters passed to it. 

19. Open tcg/tcg.c in vim and do a search for the macro

- on line 97 we can find the macro definition

Similarly to the macro definition of DEF in tcg.h, this macro also imports all the data from tcg-opc.h in the same manner. The difference between the two, is that instead of storing the data in an enum, it is being stored in an array of type TCGOpDef as a structure indicated by the curly braces. Within those curly braces, its extracting each of the parameters passed to the macro as well as adding up all the arg parameters. At this point, it is unclear exactly what the parameters stand for, but we can speculate that oargs stands for output arguments, iargs for input arguments, flags for CPU flags, and I am currently unsure about cargs. Let us now find where TCGOpDef is defined to give us better insight about how this data is used.

20. Once again exit the file and do a grep search for TCGOpDef

- grep -R "TCGOpDef" *

It appears that TCGOpDef is defined in tcg/tcg.h as a structure. Let us go to that and look at the definition.

21. Open tcg/tcg.h in vim and search for "struct TCGOpDef"

- We can find the definition at line 581

Let us analyze the members of this struct:

• name: a variable to hold the name of micro code. If we look back at the tcg-opc.h definitions of the micro codes, we can see that the first parameter is the name of the micro code (i.e. movi_i32) therefore name would hold the value "movi_i32" when used in conjunction with the DEF macro in tcg/tcg.c. 
• nb_oargs, nb_iargs, nb_cargs, nb_args:  each of these variables are an 8-bit unsigned integer. I believe that the variables names are short for number output arguments, number input arguments and so on with the last variable being number (total) arguments. 
• flags: a variable to hold the flag value of the micro code definition
• The rest of the variables are not used in the above macro definition so we will take a look at those later.

So let me summarize what is happening in the above macro from step 19: The definitions contained in tcg/tcg-opc.h are being imported into the array much like the definitions were imported into the enum from step 16. Instead of being imported into an enumerator, each definition's parameters are being placed into a a struct of type TCGOpDef (above). So if we look at the definition of movi_i32 in tcg/tcg-opc.h 

the values contained in the definition are used to populate the parameters in the macro definition. For instance, #s will contain movi_i32, oargs gets the value 1, iargs gets 0, cargs gets 1, flags gets 0. Each of those values are stored in name, nb_oargs, nb_iargs, nb_cargs, flags respectively in the TCGOpDef struct. The part of the macro that looks like iargs + oargs + cargs adds the values of the parameters and stores it in nb_args of the TCGOpDef struct. The remaining members of the struct at this point are not initialized. Let us see where this struct is used.

22. Quit vim and lets do a grep search for TCGOpDef again

- grep -R "TCGOpDef" *

From our search, we can see that tcg.h declares an array called tcg_op_defs for use in other files that import it. Furthermore, we can see that TCGOpDef is used in tcg/optimize.c, tcg/tci/tcg-target.c, and tcg/tcg.c. Also, we can see that directly under the search results for tcg/optimize.c, tcg/tcg.c uses the declared tcg_op_defs array declared in tcg/tcg.h. If we delve into those files, you will see that the data is used in many functions that handle the details of the particular micro code. 

23. Now that we have seen in detail how micro code is created and used, let us see what micro codes are generated for specific x86 instructions and how many are generated for that specific instruction. 

• Exit current gdb session (exit shell by typing "exit" and then press <ENTER> and the type q into gdb and press <ENTER>)
• Restart a new gdb session with qemu loaded and once again set a breakpoint on the disas_insn function. (restart instructions below)
• sudo gdb qemu-system-i386 (in qemu_images directory)
• r -m 512 -drive file=winxp.img,cache=unsafe -net nic,model=rtl8139,name=nic1 -net tap,ifname=tap0,script=no,downscript=no -no-kvm -monitor stdio -rtc clock=vm
• handle SIGUSR1 noprint
• b disas_insn
• r (type "y" after your press <ENTER> to restart

24. Now lets ignore the first 100 hits on the breakpoint just to get a set of different instructions that the ones we worked with

• ignore 1 100
• the above gdb command will ignore the first breakpoint (1) 100 times
• c
• continue until the breakpoint is hit 101 times

25. Keep hitting the next line until you hit the big switch case like we did in step 8

• print the opcode and take note of its number
• p b (or p/x b for the opcode in hexadecimal) 

26. By typing the gdb command "bt", we could view the stack trace that brought us to the disas_insn function. If we go up one stack frame (by typing "up"), gdb will take us to its calling location. Let us go up one stack frame and set a breakpoint right after the call to disas_insn (line 8226).

By placing a breakpoint right after the call to disas_insn, we will be able to determine the amount of bytes of micro codes generated for a specific x86 instruction. We will accomplish this by using Qemu's global variable tcg_ctx.gen_opc_ptr. This variable keeps information about the emulated OS and struct member gen_opc_ptr points to the current location in the buffer that stores the opcodes. 

27. Now let us view the x86 instructions being executed by the emulated OS with a function that we inserted into the Qemu emulator, print_instrRange(unsigned int start_addr, unsigned int end_addr, CPUX87State *env). This function takes the starting address and ending address of the range of instructions you want to display as well as CPU state of the emulated OS (env variable in Qemu). We will use pc_start as the starting point, pc_start+30 for the ending point, and the env variable for the CPU state. (NOTE: pc_start is a variable that acts as the program counter)

From the output of this function, we can see that it displayed the next 30 bytes of Intel x86 instructions. You can verify this by adding up all the values in the parenthesis (i.e. (2), (4), (5), etc.). A side note to remember is that Intel x86 instructions are known as a complex instruction set and can vary in the number of bytes needed to represent them, but the max bytes of a single instruction is 15. As we can see from the picture, the first x86 instruction to be execute is a jump instruction (jz). Let us now see how many bytes of micro code it takes to emulate this specific instruction. Something to take note of is that each micro code instruction is represented with 2 bytes, so each 4 byte word contains 2 micro code instructions. 

28. Let us get the starting address of the opcode buffer pointer (tcg_ctx.gen_opc_ptr) and display its original contents using gdb's x command (display command) using the starting address of the buffer pointer. After doing so type "c" to continue and you should hit the breakpoint we set at line 8226.

Take note of the buffers contents.

29. Now that the breakpoint at line 8226 has been hit, we know that one x86 instruction has been translated (in this case the instruction jz as we saw above). Let us now get the address of the opcode buffer one more time and see if the buffer contents have changed.

New pointer location. 

New buffer content.

For the most part, the buffer content has completely change after the jz instruction was translated.

30. Let us now see how many bytes of micro code the one x86 instruction (jz) was translated to by subtracting the new buffer location from the old.

From the picture we can see that 26 bytes of micro code were generated from the one x86 instruction translation. Now we can see each opcode generated by viewing the buffers contents. We can use these opcodes and see the corresponding micro code by looking them up in the tcg-opc.h file we found earlier (the first declaration in the file corresponds to opcode 0, the second declaration to 1 and so on). 

Summary:

We saw that when an emulated OS is running in Qemu, its instructions are converted into micro code.  This micro code is defined in tcg/tcg-opc.h file where it is imported by the DEF macro in tcg/tcg.h to create the micro code names we saw in the debugger (INDEX_op_movi_i32). A different DEF macro uses the values specified in tcg/tcg-opc.h file to set values for a struct of type TCGOpDef which contains the micro code name, arguments, and number of arguments. This struct is placed in an array that the files tcg/optimize.c, tcg/tci/tcg-target.c, and tcg/tcg.c use to help carry out the proper operations of the micro code. Remember that the order of the definitions in tcg/tcg-opc.h correspond to the enum value when imported with the DEF macro in tcg/tcg.h. For instance, DEF(movi_i32, 1, 0, 1, 0) corresponds to 11 in the TCGOpCode enum and the very first definition, DEF(end, 0, 0, 0, 0) corresponds to 0. When we translated a single x86 instruction, we were able to see that one instruction corresponds to multiple bytes of micro code and that the buffer that tcg_ctx.gen_opc_ptr points to contains the opcodes of the micro codes that the instruction was translated into. An important fact to remember is that each micro code is 2 bytes so a 4 byte word contains 2 micro codes.