DynaTrace OneAgent causing dbFront WebApp failure with FastFail

Question

The dbFront web application is being shut down hard by DynaTrace OneAgent. The application fails with a FastFail. Debugged using instructions at Debugging IIS FastFail

The eventlog appears to point to w3wp.exe terminating on a FastFail. Triggering a memory dump using WER and examining the content shows that the w3wp.exe process called into dbFront.dll!Unknown.. and was terminated with a FastFail. The final portion of the faulting thread is below.

• clr!SystemNative::GenericFailFast+156
• clr!SystemNative::FailFast+8e
• 0x263a68d2 <= Unknown
• 0x263a04a9 <= Unknown
• clr!CallDescrWorkerInternal+34
• clr!CallDescrWorkerWithHandler+6b

There was no indication on the server that DynaTrace OneAgent was involved. The only clue was that it was recently updated. Once we uninstalled DynaTrace then the issue was resolved.

Answer 1

Yes, DynaTrace OneAgent will crash the dbFront WebApplication if DynaTrace is configured with Process Injection enabled.

One of the ways that DynaTrace OneAgent works is by injecting monitoring logic into processes:

• OneAgent auto injection in Infrastructure Monitoring
• DynaTrace Dynamic injection of agents...
• StackOverFlow: FailFast Issue

The DynaTrace OneAgent fails when it attempts to gain Debug Access to dbFront and inject its monitoring logic. This failure triggers an unrecoverable FastFail error. The FastFail error kills the dbFront IIS AppPool, blocking any access to dbFront. After multiple failures the IIS Rapid-Fail protection may stop the application completely. (IIS Rapid-Fail Protection)

Although DynaTrace's purposes are good, it should be noted that Debug Access is dangerous if abused because it allows runtime Memory Inspection, Code Injection and much more...

• Memory Inspection allows hackers to capture passwords and other high-value information,
• Code Injection gives hackers complete application control for their own malicious ends.

Because of this, gaining Debug Access is a very tempting exploit vector on web servers. The dbFront executables and libraries are purposely configured and signed to block debugging and code injection.

The solution is to disable Process Injection within DynaTrace. See: DynaTrace Configure Automatic Injection

Likely due to the way it raises the error (FastFail), DynaTrace OneAgent does not create any local log of the failure so there is no way to tie the application crash to DynaTrace. The only way to verify that DynaTrace OneAgent was the source of the issue, is to disable the Code Injection within DynaTrace or Uninstall DynaTrace OneAgent.