Working Around the Inherent Slowness of Debugger Conditional Breakpoints

Posted | Modified
Author

Why Do Conditional Breakpoints Slow Down Debugging?

When a breakpoint is hit a context switch between the target process and the debugger takes place. The debugger evaluates the expression of the conditional breakpoint, and based on the result of evaluation, the execution of target either continues or halts.

If the debugger keeps evaluating the expression of the conditional breakpoint to continue execution because of too many breakpoint hits, this will result in too many context switches between the target process and the debugeer. The cost of the switches add up, leading to slow execution of target.

Besides nuisance, the slowdown can be extremely inefficient, often to the point that makes the approach of using conditional breakpoint entirely unpractical.

Unless you can change the use of the conditional breakpoints towards acceptable performance. That is to reduce the number of context switches, which can be done by minimizing the number of breakpoint hits.

Are There Redundant Breakpoints That Can Be Removed?

Breakpoints that used to be useful but which now are no longer useful may have been accidentally left there. These unneeded breakpoints can cause unnecessary hits when running the target in the debugger and so causing slow execution. While it is obvious to remove or disable unneeded breakpoints, it is sometimes unnoticed.

Can the Conditional Breakpoint Be Replaced With One or More Non-Conditional Breakpoints?

Practically, there are numerous locations to place breakpoints to break near the desired program state. However, the location which more often hit is less advantageous than the location which less often hit during execution. If you can discover suitable location which less often hit, and put the breakpoint there, you can achieve better or perhaps significantly better performance.

You can replace a conditional breakpoint with one or more non-conditional breakpoints, or with another conditional breakpoint at a location which is less often hit during execution.

For example, if you want to place a conditional breakpoint which is using the return value of the function but the function is called way too many times before the desired value is returned, and this causes performance hit you can try with the followings. Find the location in the code where the desired return value is set, and place a simple non-conditional breakpoint there, rather than where the function returns the value. If there are few of those locations that should not be an issue either because you can place non-conditional breakpoints at each location.

Can the Breakpoint Be Split Into Two Breakpoints?

Another way to reduce the unnecessary breakpoint hits is to issue two breakpoints from within different program states, in two steps. That is to place the second conditional breakpoint, after the first conditional breakpoint has been reached.

For example, when there is a loop inside another loop, and you want to place the breakpoint inside the inner loop, but by doing so the breakpoint gets too many hits leading to excess slowdown that makes debugging unpractical, consider to use two breakpoints. The first breakpoint is placed inside the outer loop, before the entry point of inner loop with the conditions to halt execution that we reached the right inner loop. Additional research might be needed to what are the right conditions for the breakpoint. When the breakpoint is hit, from this place we issue the second breakpoint for the inner loop. And the next breakpoint hit we should reach the right program state within the inner loop.

Why does it work? Image if the outer loop has 1000 iterations and the inner loop has 1000 iteration as well. Imagine the wanted breakpoint is at outer loop 500 and inner loop 500. If we only put breakpoint in the inner loop that costs us 500500 condition checks or context switches to reach the right program state. However, if we use two breakpoints, that will costs us only 1000 condition checks or context switches. That is more than 500 times more efficient.

Can the Breakpoint Be Limited to a Specific Thread?

The code location of the breakpoint may or may not be executed by different threads.

If the code location of the breakpoint is executed by different threads and it can be determined that the breakpoint should be applied to one specific thread then it is reasonable to apply the breakpoint to that specific thread rather than to all threads which is normally done by debuggers unless specified.

Can the Breakpoint Be Replaced With Data Access Breakpoint?

Data access breakpoint or hardware breakpoint is supported by processor and so it allows to break into the debugger not just on execution but also when a specific address is being read or written.

If the execution breakpoint or the traditional INT 3 breakpoint can be ditched for a data access breakpoint that might be something worth considering to reduce the number of breakpoint hits.

For example, let’s assume there is a loop, and the loop incrementally iterates over a memory range to read every single byte of that range to do some computation on the bytes. For whatever reason, you want the debugger to break-in when the 10000th byte is being read. You can put a conditional breakpoint inside the loop that would monitor each read and compare the iteration count, which is sub-optimal. Instead a more optimal solution would be to put a data access breakpoint on the right memory address that is the address of the 10000th byte. This solution is more optimal because the breakpoint would only hit when needed.

Do You Consider Setting a Breakpoint on Function Entry Point?

Sometimes it is useful to think about why you want to set the conditional breakpoint on the entry point of the function. This is because setting the breakpoints on the callers rather than on the entry point of calle is advisable to reduce unnecessary breakpoint hits. And if you can narrow down the callers you might be able to reduce the unnecessary breakpoint hits.

Is Modifying the Binary Code in Memory a Viable Option?

The idea is to inject code which performs the condition check. If the conditions meet the debugger breaks in.

This approach involves to find a location for the condition check at which point the code needs to be patched to redirect execution to a yet unused executable region. The patch should not cross basic block boundaries for stability reasons. The executable region should contain the condition check and of course the original instructions overwritten by the patch as well as the restoration of register context. The non-conditional breakpoint should be put on the code path which is reached when the conditions meet.

Since the condition check is being done by the target, there are no unnecessary breakpoint hits and context switches, and so no significant slowdown.

Is the Source Code of the Target Available?

If the source code of the target is available to modify and the debugging can be performed on the modified source code, that can be advantageous.

Simply, add the break condition to the source code, rebuild the code, and when launching the debugger just add the non-conditional breakpoint to the location which is reached when the condition evaluates true.

Can Changing the Input Speed Up Execution?

If you understand the input data at some level, by modifying it, you may or may not be able to influence that how many times the location of conditional breakpoint is being executed.

For example, if normally the conditional breakpoint would hit 10000 times, and you may be able to modify the input in a way that after the modification the breakpoint would hit only 10 times that might be a great news.

Is Debugging From Within a Virtual Machine Is a Viable Option?

Being able to perform debugging from within a virtual machine has its advantages.

Virtual machines allow to take and restore snapshots. If a program state is reached on the expense of performance, it might be a good idea to take a snapshot of that program state. The snapshot will allow to restore the same program state without performance bottleneck when it is needed.

Can You Ditch the Debugger for a DBI Framework?

If the workaround does not provide acceptable debugging performance, which can happen specially when automating debugging tasks, you might want to consider ditching the debugger for a DBI framework. Pin, DynamoRIO and Frida are the most well-known DBI frameworks in the industry and the latter is gaining popularity recently due to its usage via scripts.

Final Note

There is no silver bullet to find the right approach, and often the combination of ideas work better than a single idea.

Identifying great breakpoint locations can be time consuming. It is up to the researcher’s discretion to invest further time into researching better breakpoint locations and conditions, or bear the longer execution. It is possible that sometimes it is worth bearing the one-off slow execution for an overall faster project completion.