wiki:doc/TorBrowser/Hacking/DebuggingOnWindows

Debugging Tor browser on Windows

Debugging on Windows is tricky. I'm not aware of a mechanism to get symbols to work. There are several approaches and failed attempts. I'm going to detail them in an Appendix, but I never got any to work. In theory, it should be possible to debug on Windows, with symbols, using gdb. The mingw gcc toolchain can produce debugging information in the DWARF format. When we switch to mingw-clang, it might be possible to generate pdbs in which case we could generate symbols understandable with WinDBG.

General

You'll want to get familiar with the following environment variables for debugging:

  • MOZ_DEBUG_CHILD_PROCESS - should breakpoint child processes, but doesn't really work in Windows
  • MOZ_DEBUG_CHILD_PAUSE - will pause child processes upon start and print their pid so you can connect to them
  • MOZ_LOG - for outputting logging information
  • MOZ_IPC_MESSAGE_LOG - more other logging relating to IPC

You'll also want to be debugging a build with MOZ_ASSERTs - these are usually what we use to orient ourselves. So make sure you're building with --enable-debug, which is not the default for tor.

Debugging with gdb

The most important thing, if you're going to attempt to debug with gdb is to use a MinGW compiled gdb. You _cannot_ use the cygwin gdb. If you just want a pre-compiled gdb, you can get one here: http://www.equation.com/servlet/equation.cmd?fa=gdb

Debugging with WinDBG

WinDBG commands make no sense. Get http://windbg.info/doc/1-common-cmds.html open and ready.

Working Backwards from a Crash

First off, we have to figure out if the crash is in the parent or child. Try launching the process through WinDBG and see if you get the crash. Otherwise try attaching to the child (using MOZ_DEBUG_CHILD_PAUSE). Once you have a process broken in WinDBG, g will tell it to continue. ('g' for 'go')

Okay, we've crashed:

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\cygwin64\home\Tom\working\ff-transfer\debugging\14 svg logging\xul.dll - 
xul!XRE_GetBootstrap+0x2508847:
00000000`071da737 c7007a000000    mov     dword ptr [rax],7Ah ds:00000000`00000000=????????

Why did we crash? This code is trying to move 7Ah into the memory address stored in rax. What's in rax? We can look at registers with r (for registers.)

0:000> r
rax=0000000000000000 rbx=0000000000000001 rcx=00000000ffffffff
rdx=0000000000000002 rsi=0000000000000001 rdi=0000000000000000
rip=00000000071da737 rsp=00000000006596e0 rbp=0000000000659740
 r8=00007ffc56c14920  r9=0000000000653280 r10=0000000000000000
r11=0000000000000246 r12=0000000000000016 r13=00000000000000b0
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
xul!XRE_GetBootstrap+0x2508847:
00000000`071da737 c7007a000000    mov     dword ptr [rax],7Ah ds:00000000`00000000=????????

Note that the command I type starts with '0:000>'.

Okay. rax is 0x0000000000000000 - yea trying to store memory to the nullptr is going to crash. Why is it doing that?!??! (There's an easy answer below, you should learn this signature, but for now let's keep going.)

Once you've crashed, you can get a stacktrace with k.

Now you'll have a symbol-less stack frame, like:

0:000> k
 # Child-SP          RetAddr           Call Site
00 00000000`006596e0 00000000`071da80c xul!XRE_GetBootstrap+0x2508847
01 00000000`00659750 00000000`05a1be2f xul!XRE_GetBootstrap+0x250891c
02 00000000`006597e0 00000000`05a1db47 xul!XRE_GetBootstrap+0xd49f3f
03 00000000`00659850 00000000`053a1dc0 xul!XRE_GetBootstrap+0xd4bc57
04 00000000`00659880 00000000`0659877e xul!XRE_GetBootstrap+0x6cfed0
05 00000000`00659920 00000000`0558e20e xul!XRE_GetBootstrap+0x18c688e
06 00000000`006599c0 00000000`0a750e80 xul!XRE_GetBootstrap+0x8bc31e
07 00000000`00659cf0 00000000`0678b8bd xul!XRE_GetBootstrap+0x5a7ef90
08 00000000`00659d20 00000000`0653ee6b xul!XRE_GetBootstrap+0x1ab99cd
09 00000000`00659d50 00000000`059eedb4 xul!XRE_GetBootstrap+0x186cf7b
0a 00000000`00659d80 00000000`07cf7eac xul!XRE_GetBootstrap+0xd1cec4
0b 00000000`00659dd0 00000000`07cf59a8 xul!XRE_GetBootstrap+0x3025fbc
0c 00000000`00659f20 00000000`059ece67 xul!XRE_GetBootstrap+0x3023ab8
0d 00000000`00659fc0 00000000`059ee15d xul!XRE_GetBootstrap+0xd1af77
0e 00000000`0065a490 00000000`05a18cd1 xul!XRE_GetBootstrap+0xd1c26d
0f 00000000`0065a690 00000000`0a173d1c xul!XRE_GetBootstrap+0xd46de1
10 00000000`0065d9b0 00000000`05c8d432 xul!XRE_GetBootstrap+0x54a1e2c
11 00000000`0065dc70 00000000`05c8d998 xul!XRE_GetBootstrap+0xfbb542
12 00000000`0065dd00 00000000`05c8cecf xul!XRE_GetBootstrap+0xfbbaa8
13 00000000`0065ddc0 00000000`05ed18ea xul!XRE_GetBootstrap+0xfbafdf
14 00000000`0065de00 00000000`07da1f11 xul!XRE_GetBootstrap+0x11ff9fa
15 00000000`0065e3d0 00000000`07da243a xul!XRE_GetBootstrap+0x30d0021
16 00000000`0065e450 00000000`07da2ae3 xul!XRE_GetBootstrap+0x30d054a
17 00000000`0065e4f0 00000000`07ebf569 xul!XRE_GetBootstrap+0x30d0bf3
18 00000000`0065e580 00000000`07ebfaf2 xul!XRE_GetBootstrap+0x31ed679
19 00000000`0065e630 00000000`07ebf8bf xul!XRE_GetBootstrap+0x31edc02
1a 00000000`0065e6e0 00000000`09f406f8 xul!XRE_GetBootstrap+0x31ed9cf
1b 00000000`0065e7b0 00000000`09f41465 xul!XRE_GetBootstrap+0x526e808
1c 00000000`0065e800 00000000`0950b127 xul!XRE_GetBootstrap+0x526f575
1d 00000000`0065e8f0 00000000`09447b25 xul!XRE_GetBootstrap+0x4839237
1e 00000000`0065ec20 00000000`09445996 xul!XRE_GetBootstrap+0x4775c35
1f 00000000`0065ec80 00000000`0944352e xul!XRE_GetBootstrap+0x4773aa6
20 00000000`0065ee00 00000000`0944381e xul!XRE_GetBootstrap+0x477163e
21 00000000`0065ee50 00000000`0a4915eb xul!XRE_GetBootstrap+0x477192e
22 00000000`0065eea0 00000000`04f08cd1 xul!XRE_GetBootstrap+0x57bf6fb
23 00000000`0065f560 00000000`09435c16 xul!XRE_GetBootstrap+0x236de1
24 00000000`0065f5a0 00000000`09542e02 xul!XRE_GetBootstrap+0x4763d26
25 00000000`0065f5f0 00000000`055ace38 xul!XRE_GetBootstrap+0x4870f12
26 00000000`0065f640 00000000`055ac6d8 xul!XRE_GetBootstrap+0x8daf48
27 00000000`0065f690 00000000`055adc35 xul!XRE_GetBootstrap+0x8da7e8
28 00000000`0065f6d0 00000000`05d1ec56 xul!XRE_GetBootstrap+0x8dbd45
29 00000000`0065f720 00000000`054cff1b xul!XRE_GetBootstrap+0x104cd66
2a 00000000`0065f770 00000000`04dbe9cc xul!XRE_GetBootstrap+0x7fe02b
2b 00000000`0065f7b0 00000000`09542cb4 xul!XRE_GetBootstrap+0xecadc
2c 00000000`0065f810 00000000`055ace38 xul!XRE_GetBootstrap+0x4870dc4
2d 00000000`0065f860 00000000`055ac6d8 xul!XRE_GetBootstrap+0x8daf48
2e 00000000`0065f8b0 00000000`055adc35 xul!XRE_GetBootstrap+0x8da7e8
2f 00000000`0065f8f0 00000000`04f28756 xul!XRE_GetBootstrap+0x8dbd45
30 00000000`0065f940 00000000`07afa1bc xul!XRE_GetBootstrap+0x256866
*** WARNING: Unable to verify timestamp for C:\cygwin64\home\Tom\working\ff-transfer\debugging\14 svg logging\firefox.exe
*** ERROR: Module load completed but symbols could not be loaded for C:\cygwin64\home\Tom\working\ff-transfer\debugging\14 svg logging\firefox.exe
31 00000000`0065fca0 00000000`0040a745 xul!XRE_GetBootstrap+0x2e282cc
32 00000000`0065fcd0 00000000`0040a5bc firefox+0xa745
33 00000000`0065fd10 00000000`0040cffe firefox+0xa5bc
34 00000000`0065fd90 00000000`0040cdfc firefox+0xcffe
35 00000000`0065fe10 00000000`004013c8 firefox+0xcdfc
36 00000000`0065fe60 00000000`004014eb firefox+0x13c8
37 00000000`0065ff30 00007ffc`55f31fe4 firefox+0x14eb
38 00000000`0065ff60 00007ffc`5839efc1 KERNEL32!BaseThreadInitThunk+0x14
39 00000000`0065ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Okay, time to explore the frames. Our primary goal is to understand where we are. To do that, we're going to go up the frame until we find something with an ASSERT or a LOG. Basically, we're looking for something that's referencing a hardcoded string we can search for. ASSERTs tend to be more common.

First, let's look at our crashing frame. That's 'xul!XRE_GetBootstrap+0x2508847'. We want to disassemble the code we are crashing at. We can do what with u <addr> (u for unassemble.) This will show the assembly code that we crash on:

0:000> u xul!XRE_GetBootstrap+0x2508847
xul!XRE_GetBootstrap+0x2508847:
00000000`071da737 c7007a000000    mov     dword ptr [rax],7Ah
00000000`071da73d e846e69ffd      call    xul+0x4e8d88 (00000000`04bd8d88)
00000000`071da742 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da746 488b5518        mov     rdx,qword ptr [rbp+18h]
00000000`071da74a 4889c1          mov     rcx,rax
00000000`071da74d e82e89ce03      call    xul!XRE_GetBootstrap+0x61f1190 (00000000`0aec3080)
00000000`071da752 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da756 4883c460        add     rsp,60h

Notice that although I ask it to unassemble a relative address, the memory address it shows me is absolute: 071da737.

This shows us the crashing instruction and the next few we would execute, but really we'd like to get more context. So we're going to unassemble the entire function of this frame with uf.

0:000> uf xul!XRE_GetBootstrap+0x2508847
xul!XRE_GetBootstrap+0x2508680:
00000000`071da570 55              push    rbp
00000000`071da571 4889e5          mov     rbp,rsp
00000000`071da574 4883ec60        sub     rsp,60h
00000000`071da578 48894d10        mov     qword ptr [rbp+10h],rcx
00000000`071da57c 48895518        mov     qword ptr [rbp+18h],rdx
00000000`071da580 4489c0          mov     eax,r8d
00000000`071da583 884520          mov     byte ptr [rbp+20h],al
00000000`071da586 e8b5bc2703      call    xul!XRE_GetBootstrap+0x5784350 (00000000`0a456240)
00000000`071da58b 4889c2          mov     rdx,rax
00000000`071da58e 488b4518        mov     rax,qword ptr [rbp+18h]
00000000`071da592 41b800000000    mov     r8d,0
00000000`071da598 4889c1          mov     rcx,rax
00000000`071da59b e8d036ce03      call    xul!XRE_GetBootstrap+0x61ebd80 (00000000`0aebdc70)
00000000`071da5a0 488945f8        mov     qword ptr [rbp-8],rax
00000000`071da5a4 48837df800      cmp     qword ptr [rbp-8],0
00000000`071da5a9 7461            je      xul!XRE_GetBootstrap+0x250871c (00000000`071da60c)  Branch

xul!XRE_GetBootstrap+0x25086bb:
00000000`071da5ab 488d0d9eb7fb04  lea     rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
00000000`071da5b2 e859eb9600      call    xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
00000000`071da5b7 488945f0        mov     qword ptr [rbp-10h],rax
00000000`071da5bb 488b45f0        mov     rax,qword ptr [rbp-10h]
00000000`071da5bf ba04000000      mov     edx,4
00000000`071da5c4 4889c1          mov     rcx,rax
00000000`071da5c7 e804c6b002      call    xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
00000000`071da5cc 84c0            test    al,al
00000000`071da5ce 7427            je      xul!XRE_GetBootstrap+0x2508707 (00000000`071da5f7)  Branch

xul!XRE_GetBootstrap+0x25086e0:
00000000`071da5d0 0fb65520        movzx   edx,byte ptr [rbp+20h]
00000000`071da5d4 488b4d18        mov     rcx,qword ptr [rbp+18h]
00000000`071da5d8 488b45f0        mov     rax,qword ptr [rbp-10h]
00000000`071da5dc 89542420        mov     dword ptr [rsp+20h],edx
00000000`071da5e0 4989c9          mov     r9,rcx
00000000`071da5e3 4c8d058e078905  lea     r8,[xul!workerlz4_maxCompressedSize+0x906a98 (00000000`0ca6ad78)]
00000000`071da5ea ba04000000      mov     edx,4
00000000`071da5ef 4889c1          mov     rcx,rax
00000000`071da5f2 e81932b302      call    xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)

xul!XRE_GetBootstrap+0x2508707:
00000000`071da5f7 488b45f8        mov     rax,qword ptr [rbp-8]
00000000`071da5fb 4889c2          mov     rdx,rax
00000000`071da5fe 488b4d10        mov     rcx,qword ptr [rbp+10h]
00000000`071da602 e829b44e00      call    xul!XRE_GetBootstrap+0x29f3b40 (00000000`076c5a30)
00000000`071da607 e946010000      jmp     xul!XRE_GetBootstrap+0x2508862 (00000000`071da752)  Branch

xul!XRE_GetBootstrap+0x250871c:
00000000`071da60c 488d0d3db7fb04  lea     rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
00000000`071da613 e8f8ea9600      call    xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
00000000`071da618 488945e8        mov     qword ptr [rbp-18h],rax
00000000`071da61c 488b45e8        mov     rax,qword ptr [rbp-18h]
00000000`071da620 ba04000000      mov     edx,4
00000000`071da625 4889c1          mov     rcx,rax
00000000`071da628 e8a3c5b002      call    xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
00000000`071da62d 84c0            test    al,al
00000000`071da62f 7427            je      xul!XRE_GetBootstrap+0x2508768 (00000000`071da658)  Branch

xul!XRE_GetBootstrap+0x2508741:
00000000`071da631 0fb65520        movzx   edx,byte ptr [rbp+20h]
00000000`071da635 488b4d18        mov     rcx,qword ptr [rbp+18h]
00000000`071da639 488b45e8        mov     rax,qword ptr [rbp-18h]
00000000`071da63d 89542420        mov     dword ptr [rsp+20h],edx
00000000`071da641 4989c9          mov     r9,rcx
00000000`071da644 4c8d05ad078905  lea     r8,[xul!workerlz4_maxCompressedSize+0x906b18 (00000000`0ca6adf8)]
00000000`071da64b ba04000000      mov     edx,4
00000000`071da650 4889c1          mov     rcx,rax
00000000`071da653 e8b831b302      call    xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)

xul!XRE_GetBootstrap+0x2508768:
00000000`071da658 488b4518        mov     rax,qword ptr [rbp+18h]
00000000`071da65c 4889c1          mov     rcx,rax
00000000`071da65f e8fcd73bff      call    xul!XRE_GetBootstrap+0x18c5f70 (00000000`06597e60)
00000000`071da664 84c0            test    al,al
00000000`071da666 7412            je      xul!XRE_GetBootstrap+0x250878a (00000000`071da67a)  Branch

xul!XRE_GetBootstrap+0x2508778:
00000000`071da668 0fb64520        movzx   eax,byte ptr [rbp+20h]
00000000`071da66c 83f001          xor     eax,1
00000000`071da66f 84c0            test    al,al
00000000`071da671 7407            je      xul!XRE_GetBootstrap+0x250878a (00000000`071da67a)  Branch

xul!XRE_GetBootstrap+0x2508783:
00000000`071da673 b801000000      mov     eax,1
00000000`071da678 eb05            jmp     xul!XRE_GetBootstrap+0x250878f (00000000`071da67f)  Branch

xul!XRE_GetBootstrap+0x250878a:
00000000`071da67a b800000000      mov     eax,0

xul!XRE_GetBootstrap+0x250878f:
00000000`071da67f 84c0            test    al,al
00000000`071da681 0f84bb000000    je      xul!XRE_GetBootstrap+0x2508852 (00000000`071da742)  Branch

xul!XRE_GetBootstrap+0x2508797:
00000000`071da687 e8140d2803      call    xul!XRE_GetBootstrap+0x57894b0 (00000000`0a45b3a0)
00000000`071da68c 4889c2          mov     rdx,rax
00000000`071da68f 488b4518        mov     rax,qword ptr [rbp+18h]
00000000`071da693 41b800000000    mov     r8d,0
00000000`071da699 4889c1          mov     rcx,rax
00000000`071da69c e88f31ce03      call    xul!XRE_GetBootstrap+0x61eb940 (00000000`0aebd830)
00000000`071da6a1 488945e0        mov     qword ptr [rbp-20h],rax
00000000`071da6a5 488d0da4b6fb04  lea     rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
00000000`071da6ac e85fea9600      call    xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
00000000`071da6b1 488945d8        mov     qword ptr [rbp-28h],rax
00000000`071da6b5 488b45d8        mov     rax,qword ptr [rbp-28h]
00000000`071da6b9 ba04000000      mov     edx,4
00000000`071da6be 4889c1          mov     rcx,rax
00000000`071da6c1 e80ac5b002      call    xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
00000000`071da6c6 84c0            test    al,al
00000000`071da6c8 7440            je      xul!XRE_GetBootstrap+0x250881a (00000000`071da70a)  Branch

xul!XRE_GetBootstrap+0x25087da:
00000000`071da6ca 48837de000      cmp     qword ptr [rbp-20h],0
00000000`071da6cf 7509            jne     xul!XRE_GetBootstrap+0x25087ea (00000000`071da6da)  Branch

xul!XRE_GetBootstrap+0x25087e1:
00000000`071da6d1 488d0580078905  lea     rax,[xul!workerlz4_maxCompressedSize+0x906b78 (00000000`0ca6ae58)]
00000000`071da6d8 eb07            jmp     xul!XRE_GetBootstrap+0x25087f1 (00000000`071da6e1)  Branch

xul!XRE_GetBootstrap+0x25087ea:
00000000`071da6da 488d057f078905  lea     rax,[xul!workerlz4_maxCompressedSize+0x906b80 (00000000`0ca6ae60)]

xul!XRE_GetBootstrap+0x25087f1:
00000000`071da6e1 0fb65520        movzx   edx,byte ptr [rbp+20h]
00000000`071da6e5 4c8b4518        mov     r8,qword ptr [rbp+18h]
00000000`071da6e9 488b4dd8        mov     rcx,qword ptr [rbp-28h]
00000000`071da6ed 4889442428      mov     qword ptr [rsp+28h],rax
00000000`071da6f2 89542420        mov     dword ptr [rsp+20h],edx
00000000`071da6f6 4d89c1          mov     r9,r8
00000000`071da6f9 4c8d0568078905  lea     r8,[xul!workerlz4_maxCompressedSize+0x906b88 (00000000`0ca6ae68)]
00000000`071da700 ba04000000      mov     edx,4
00000000`071da705 e80631b302      call    xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)

xul!XRE_GetBootstrap+0x250881a:
00000000`071da70a 48837de000      cmp     qword ptr [rbp-20h],0
00000000`071da70f 0f95c0          setne   al
00000000`071da712 0fb6c0          movzx   eax,al
00000000`071da715 85c0            test    eax,eax
00000000`071da717 7429            je      xul!XRE_GetBootstrap+0x2508852 (00000000`071da742)  Branch

xul!XRE_GetBootstrap+0x2508829:
00000000`071da719 41b87a000000    mov     r8d,7Ah
00000000`071da71f 488d156a058905  lea     rdx,[xul!workerlz4_maxCompressedSize+0x9069b0 (00000000`0ca6ac90)]
00000000`071da726 488d0dbb078905  lea     rcx,[xul!workerlz4_maxCompressedSize+0x906c08 (00000000`0ca6aee8)]
00000000`071da72d e8eeb3a7fd      call    xul!DumpJSStack+0x58fc0 (00000000`04c55b20)
00000000`071da732 b800000000      mov     eax,0
00000000`071da737 c7007a000000    mov     dword ptr [rax],7Ah
00000000`071da73d e846e69ffd      call    xul+0x4e8d88 (00000000`04bd8d88)

xul!XRE_GetBootstrap+0x2508852:
00000000`071da742 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da746 488b5518        mov     rdx,qword ptr [rbp+18h]
00000000`071da74a 4889c1          mov     rcx,rax
00000000`071da74d e82e89ce03      call    xul!XRE_GetBootstrap+0x61f1190 (00000000`0aec3080)

xul!XRE_GetBootstrap+0x2508862:
00000000`071da752 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da756 4883c460        add     rsp,60h
00000000`071da75a 5d              pop     rbp
00000000`071da75b c3              ret

Note that the first line after the 'uf' instruction shows us the _start_ of the function. xul!XRE_GetBootstrap+0x2508847 is where we crashed; but xul!XRE_GetBootstrap+0x2508680 is where the function began. And the absolute address of xul!XRE_GetBootstrap+0x2508847 is 071da737 - if you control+f for 071da737 on this page you'll see the same instructions we unassembled.

Also: why does it break the function up with newlines in between? Each of those sections of the function is part of a branch. In IDA you'd get a nice graph view like so: http://hexblog.com/ida_pro/pix/idaqt_preview_100310_1.html (picture is an illustrative example, not this particular function). Take 'xul!XRE_GetBootstrap+0x2508852' for example. If you control+f on this page for 0x2508852, you'll see there are a couple of places in this function we jump to this location.

Now, let's pretend this frame doesn't tell us anything useful. We'd need to look at the address of one frame up: xul!XRE_GetBootstrap+0x250891c

0:000> u xul!XRE_GetBootstrap+0x250891c
xul!XRE_GetBootstrap+0x250891c:
00000000`071da80c 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da810 488b4810        mov     rcx,qword ptr [rax+10h]
00000000`071da814 488d45d8        lea     rax,[rbp-28h]
00000000`071da818 488b5518        mov     rdx,qword ptr [rbp+18h]
00000000`071da81c 4989c8          mov     r8,rcx
00000000`071da81f 4889c1          mov     rcx,rax
00000000`071da822 e8892cce03      call    xul!XRE_GetBootstrap+0x61eb5c0 (00000000`0aebd4b0)
00000000`071da827 488d55b0        lea     rdx,[rbp-50h]

In this situation we see the instruction immediately following the 'call' we made to go one stack frame down. Again, we need more context, so disassemble the whole function with uf.

0:000> uf xul!XRE_GetBootstrap+0x250891c
xul!XRE_GetBootstrap+0x2508870:
00000000`071da760 55              push    rbp
00000000`071da761 4889e5          mov     rbp,rsp
00000000`071da764 4883c480        add     rsp,0FFFFFFFFFFFFFF80h
00000000`071da768 48894d10        mov     qword ptr [rbp+10h],rcx
00000000`071da76c 48895518        mov     qword ptr [rbp+18h],rdx
00000000`071da770 488d0dd9b5fb04  lea     rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
00000000`071da777 e894e99600      call    xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
00000000`071da77c 488945f8        mov     qword ptr [rbp-8],rax
00000000`071da780 488b45f8        mov     rax,qword ptr [rbp-8]
00000000`071da784 ba04000000      mov     edx,4
00000000`071da789 4889c1          mov     rcx,rax
00000000`071da78c e83fc4b002      call    xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
00000000`071da791 84c0            test    al,al
00000000`071da793 7435            je      xul!XRE_GetBootstrap+0x25088da (00000000`071da7ca)  Branch

xul!XRE_GetBootstrap+0x25088a5:
00000000`071da795 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da799 488b4018        mov     rax,qword ptr [rax+18h]
00000000`071da79d 483b4518        cmp     rax,qword ptr [rbp+18h]
00000000`071da7a1 0f94c0          sete    al
00000000`071da7a4 0fb6d0          movzx   edx,al
00000000`071da7a7 488b4d18        mov     rcx,qword ptr [rbp+18h]
00000000`071da7ab 488b45f8        mov     rax,qword ptr [rbp-8]
00000000`071da7af 89542420        mov     dword ptr [rsp+20h],edx
00000000`071da7b3 4989c9          mov     r9,rcx
00000000`071da7b6 4c8d0573058905  lea     r8,[xul!workerlz4_maxCompressedSize+0x906a50 (00000000`0ca6ad30)]
00000000`071da7bd ba04000000      mov     edx,4
00000000`071da7c2 4889c1          mov     rcx,rax
00000000`071da7c5 e84630b302      call    xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)

xul!XRE_GetBootstrap+0x25088da:
00000000`071da7ca 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da7ce 488b4018        mov     rax,qword ptr [rax+18h]
00000000`071da7d2 483b4518        cmp     rax,qword ptr [rbp+18h]
00000000`071da7d6 7516            jne     xul!XRE_GetBootstrap+0x25088fe (00000000`071da7ee)  Branch

xul!XRE_GetBootstrap+0x25088e8:
00000000`071da7d8 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da7dc 488b5020        mov     rdx,qword ptr [rax+20h]
00000000`071da7e0 488d45b0        lea     rax,[rbp-50h]
00000000`071da7e4 4889c1          mov     rcx,rax
00000000`071da7e7 e844b24e00      call    xul!XRE_GetBootstrap+0x29f3b40 (00000000`076c5a30)
00000000`071da7ec eb1e            jmp     xul!XRE_GetBootstrap+0x250891c (00000000`071da80c)  Branch

xul!XRE_GetBootstrap+0x25088fe:
00000000`071da7ee 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da7f2 0fb64038        movzx   eax,byte ptr [rax+38h]
00000000`071da7f6 0fb6c8          movzx   ecx,al
00000000`071da7f9 488d45b0        lea     rax,[rbp-50h]
00000000`071da7fd 488b5518        mov     rdx,qword ptr [rbp+18h]
00000000`071da801 4189c8          mov     r8d,ecx
00000000`071da804 4889c1          mov     rcx,rax
00000000`071da807 e864fdffff      call    xul!XRE_GetBootstrap+0x2508680 (00000000`071da570)

xul!XRE_GetBootstrap+0x250891c:
00000000`071da80c 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da810 488b4810        mov     rcx,qword ptr [rax+10h]
00000000`071da814 488d45d8        lea     rax,[rbp-28h]
00000000`071da818 488b5518        mov     rdx,qword ptr [rbp+18h]
00000000`071da81c 4989c8          mov     r8,rcx
00000000`071da81f 4889c1          mov     rcx,rax
00000000`071da822 e8892cce03      call    xul!XRE_GetBootstrap+0x61eb5c0 (00000000`0aebd4b0)
00000000`071da827 488d55b0        lea     rdx,[rbp-50h]
00000000`071da82b 488d45e0        lea     rax,[rbp-20h]
00000000`071da82f 4889c1          mov     rcx,rax
00000000`071da832 e8f9b14e00      call    xul!XRE_GetBootstrap+0x29f3b40 (00000000`076c5a30)
00000000`071da837 488d45c0        lea     rax,[rbp-40h]
00000000`071da83b 488d4dd8        lea     rcx,[rbp-28h]
00000000`071da83f 488d55e0        lea     rdx,[rbp-20h]
00000000`071da843 4989c8          mov     r8,rcx
00000000`071da846 4889c1          mov     rcx,rax
00000000`071da849 e8d24f2002      call    xul!XRE_GetBootstrap+0x470d930 (00000000`093df820)
00000000`071da84e 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da852 488d5028        lea     rdx,[rax+28h]
00000000`071da856 488b4510        mov     rax,qword ptr [rbp+10h]
00000000`071da85a 4883c028        add     rax,28h
00000000`071da85e 488d4dc0        lea     rcx,[rbp-40h]
00000000`071da862 4989c8          mov     r8,rcx
00000000`071da865 4889c1          mov     rcx,rax
00000000`071da868 e813b14e00      call    xul!XRE_GetBootstrap+0x29f3a90 (00000000`076c5980)
00000000`071da86d 488d45c0        lea     rax,[rbp-40h]
00000000`071da871 4889c1          mov     rcx,rax
00000000`071da874 e8a7b24e00      call    xul!XRE_GetBootstrap+0x29f3c30 (00000000`076c5b20)
00000000`071da879 488d45e0        lea     rax,[rbp-20h]
00000000`071da87d 4889c1          mov     rcx,rax
00000000`071da880 e89bb24e00      call    xul!XRE_GetBootstrap+0x29f3c30 (00000000`076c5b20)
00000000`071da885 488d45b0        lea     rax,[rbp-50h]
00000000`071da889 4889c1          mov     rcx,rax
00000000`071da88c e88fb24e00      call    xul!XRE_GetBootstrap+0x29f3c30 (00000000`076c5b20)
00000000`071da891 90              nop
00000000`071da892 4883ec80        sub     rsp,0FFFFFFFFFFFFFF80h
00000000`071da896 5d              pop     rbp
00000000`071da897 c3              ret

Now, what we're looking for is a sequence of instructions like the below. We actually hit this in the first stack frame, I just wanted to demonstrate going up a stack frame.

xul!XRE_GetBootstrap+0x2508829:   <----- START OF A BRANCH, USUALLY PRESENT
00000000`071da719 41b87a000000    mov     r8d,7Ah                      <------ MOV OF A CONSTANT INTO A REGISTER
00000000`071da71f 488d156a058905  lea     rdx,[xul!workerlz4_maxCompressedSize+0x9069b0 (00000000`0ca6ac90)]   <-------- LEA OF A MEMORY ADDRESS INTO A REGISTER
00000000`071da726 488d0dbb078905  lea     rcx,[xul!workerlz4_maxCompressedSize+0x906c08 (00000000`0ca6aee8)]   <-------- LEA OF A MEMORY ADDRESS INTO A REGISTER
00000000`071da72d e8eeb3a7fd      call    xul!DumpJSStack+0x58fc0 (00000000`04c55b20)                          <-------- CALL
00000000`071da732 b800000000      mov     eax,0       <-------- MOV 0 INTO eax OR rax
00000000`071da737 c7007a000000    mov     dword ptr [rax],7Ah   <-------- MOV THE CONSTANT INTO THE MEMORY ADDRESS OF THAT REGISTER (WHICH IS A nullptr)

This pattern is a MOZ_ASSERT. Here's what it means:

00000000`071da719 41b87a000000    mov     r8d,7Ah                      <------ THIS CONSTANT IS THE LINE NUMBER (in hex)
00000000`071da71f 488d156a058905  lea     rdx,[xul!workerlz4_maxCompressedSize+0x9069b0 (00000000`0ca6ac90)]   <-------- THIS STRING IS THE FILE
00000000`071da726 488d0dbb078905  lea     rcx,[xul!workerlz4_maxCompressedSize+0x906c08 (00000000`0ca6aee8)]   <-------- THIS STRING IS THE ASSERT LINE
...
...
00000000`071da737 c7007a000000    mov     dword ptr [rax],7Ah   <-------- INTENTIONALLY CRASH THE PROCESS

Those lea instructions are loading a hardcoded string. Let's examine that. To do so we're going to use the d command for reading memory.

db <addr> - shows raw memory

0:000> db xul!workerlz4_maxCompressedSize+0x9069b0
00000000`0ca6ac90  2f 62 75 69 6c 64 73 2f-77 6f 72 6b 65 72 2f 77  /builds/worker/w
00000000`0ca6aca0  6f 72 6b 73 70 61 63 65-2f 62 75 69 6c 64 2f 73  orkspace/build/s
00000000`0ca6acb0  72 63 2f 6c 61 79 6f 75-74 2f 73 76 67 2f 6e 73  rc/layout/svg/ns
00000000`0ca6acc0  53 56 47 49 6e 74 65 67-72 61 74 69 6f 6e 55 74  SVGIntegrationUt
00000000`0ca6acd0  69 6c 73 2e 63 70 70 00-21 6d 46 69 72 73 74 43  ils.cpp.!mFirstC
00000000`0ca6ace0  6f 6e 74 69 6e 75 61 74-69 6f 6e 2d 3e 47 65 74  ontinuation->Get
00000000`0ca6acf0  50 72 65 76 43 6f 6e 74-69 6e 75 61 74 69 6f 6e  PrevContinuation
00000000`0ca6ad00  28 29 00 00 00 00 00 00-57 65 20 77 61 6e 74 20  ()......We want 


da <addr> - interpret data as an ASCII string

0:000> da xul!workerlz4_maxCompressedSize+0x9069b0
00000000`0ca6ac90  "/builds/worker/workspace/build/s"
00000000`0ca6acb0  "rc/layout/svg/nsSVGIntegrationUt"
00000000`0ca6acd0  "ils.cpp"


da /c100 <addr> - ASCII string but this time show 100 chars per line

0:000> da /c100 xul!workerlz4_maxCompressedSize+0x9069b0
00000000`0ca6ac90  "/builds/worker/workspace/build/src/layout/svg/nsSVGIntegrationUtils.cpp"

At this point, we know what file and line we are dealing with. By comparing the callsites in the code with the 'call' instructions, we are almost always able to identify which function we have traversed into, until we finally have a decent idea what the crash is.

Breaking where we want to break

The above section is good if you have a crash and want to go backwards. What if you want to break someone in particular, not related to a crash? Without symbols, how do you do it? There's a trick. Here's the code you'll want:

    if(getenv("XXX_FINDME1"))
#ifdef __MINGW32__
      __builtin_trap();
#else
      DebugBreak();
#endif

This will let you break (conditionally, depending on the enviornment variable) right where you want.

You'll need to do the following dance:

  1. Go into System -> Advanced System Settings -> Environment Variables and set your env var
  2. THEN START WINDBG. This is very important. The Env Var change will not be picked up unless you start WinDBG after you set the variable (and close the dialog boxes I think)
  3. Attach/Execute the program and get the address (as seen below)
  4. CLOSE WINDBG
  5. Delete the environment variable
  6. REOPEN WINDBG
  7. Attach/Execute the process and set your breakpoint

In MinGW the fault will look like this:

(3ec8.3a58): Illegal instruction - code c000001d (first chance)
(3ec8.3a58): Illegal instruction - code c000001d (!!! second chance !!!)
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for firefox.exe - 
firefox!TargetSetOPMSigningKeyAndSequenceNumbers64+0x4086f:
00000000`0046490f 0f0b            ud2

Disassemble that function, and the first line will give you the address of the function entry point, which you can then give to 'bp'

0:024> uf firefox!TargetSetOPMSigningKeyAndSequenceNumbers64+0x4086f
firefox!TargetSetOPMSigningKeyAndSequenceNumbers64+0x407f0:

If you're comparing to a non-MinGW build using DebugBreak(); the fault will look like the below. Note that I need to go one frame up in the call stack to get the correct function.

(7430.6d44): Break instruction exception - code 80000003 (first chance)
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for firefox.exe - 
KERNELBASE!wil::details::DebugBreak+0x2:
00007ffe`42ce6482 cc              int     3
0:023> k
 # Child-SP          RetAddr           Call Site
00 00000060`bafff448 00007ff7`5ad7c436 KERNELBASE!wil::details::DebugBreak+0x2
01 00000060`bafff450 00007ff7`5ad996fe firefox!GetHandleVerifier+0x4d96
02 00000060`bafff4d0 00007ff7`5ad99ddb firefox!IsSandboxedProcess+0x3a0e
03 00000060`bafff790 00007ffe`45edc3bb firefox!IsSandboxedProcess+0x40eb
04 00000060`bafffa50 00007ffe`45e96be3 ntdll!RtlpTpWaitCallback+0x9b
05 00000060`bafffac0 00007ffe`45e94c30 ntdll!TppExecuteWaitCallback+0xab
06 00000060`bafffb00 00007ffe`45ac1fe4 ntdll!TppWorkerThread+0x8d0
07 00000060`bafffe90 00007ffe`45ecf061 KERNEL32!BaseThreadInitThunk+0x14
08 00000060`bafffec0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
0:023> uf firefox!GetHandleVerifier+0x4d96
firefox!GetHandleVerifier+0x4d40:

A few other notes

If you see an address like xul!XRE_GetBootstrap+0x2508847 - that's a relative address (it's relative to xul.dll!XRE_GetBootstrap - It's not the _function_ XRE_GetBootstrap, it's 0x2508847 bytes after the start of that function). If you see an address like 00000000`071da837 or 071da837 - that's an absolute addresses.

If a relative address points to code (it usually does), then that address will be the same across process executions. xul.dll will move around in memory due to ASLR, but as long as you're running the same build, the instructions at xul!XRE_GetBootstrap+0x2508847 will be the same.

The same is not true for absolute addresses. These addresses are not going to be the same across process execution.

Symbols Appendix

Currently, building without stripping symbols for Windows produces a xul.dll over 2 GB. This is larger than is acceptable by Windows and thus it won't load the dll and firefox.exe will error very early on.

Using objcopy --add-gnu-debuglink one can move the debug sections to another file and tell gdb the symbols are there. This is explained in https://stackoverflow.com/questions/866721/how-to-generate-gcc-debug-symbol-outside-the-build-target/866731#866731

Once that it done, we can run firefox, and in theory tell gdb where the symbols are. However gdb choked on reading the symbols. This is detailed at https://ritter.vg/misc/ff/dwarf-error.html . Further investigation (in https://ritter.vg/misc/ff/dwarf-error-2.html ) revealed the culprit was yasm passing -dwarf2. The bypass for this is to comment out the -g dwarf2 line in https://searchfox.org/mozilla-central/rev/08df4e6e11284186d477d7e5b0ae48483ecc979c/python/mozbuild/mozbuild/frontend/context.py#375 Once that is done, you will get a symbol file gdb can, supposedly, read.

All is not well. Following this fix, objcopy segfaulted: https://sourceware.org/bugzilla/show_bug.cgi?id=23061 As explained in the bug, the suspicion is that something in gcc is overflowing (probably because it's a signed int and we're overflowing that) and resulting in corrupt DWARF information. At this point I ceased investigation.

Besides investigating the suspected gcc bug(s), there are other ways that are worth investigating for symbols:

  • gz (not supported in MinGW??)
  • gsplit-dwarf - This is supposed to move the dwarf data to separate files.
  • g1 - This produces less debugging information. This produces a xul.dll that is within the size limits, but didn't run. The dwarf2 fix may be needed.
Last modified 3 days ago Last modified on Jul 12, 2018, 2:36:15 PM