Crash on Mac (VRS 2.4.0)

Bug reports and questions about running Virtual Radar Server on Linux
Post Reply
mgunther
Posts: 17
Joined: Sun Aug 21, 2016 6:45 pm

Crash on Mac (VRS 2.4.0)

Post by mgunther » Wed Aug 24, 2016 9:43 am

VRS 2.4.0.30532
I'm seeing a crash on my Mac laptop every now and then but I'm not sure if and how it is related to anything I'm doing at the time. I think it may happen when I'm not doing anything at all but have the VRS page open with aircraft moving, auto-selecting, etc.

Code: Select all

$ mono -V
Mono JIT compiler version 4.4.2 (mono-4.4.0-branch-c7sr1/f72fe45 Wed Jul 27 16:20:13 EDT 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           normal
        SIGSEGV:       altstack
        Notification:  kqueue
        Architecture:  x86
        Disabled:      none
        Misc:          softdebug
        LLVM:          yes(3.6.0svn-mono-master/a173357)
        GC:            sgen
The crash message is:

Code: Select all

Assertion failed: (!"reached"), function _cairo_hash_table_lookup_exact_key, file cairo-hash.c, line 506.
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Drawing.GDIPlus.GdipDeleteGraphics (intptr) <0x00012>
  at System.Drawing.Graphics.Dispose () <0x00057>
  at VirtualRadar.WebSite.WebSiteGraphics.ResizeForHiDpi (System.Drawing.Image) <0x0016f>
  at VirtualRadar.WebSite.ImagePage.DoHandleRequest (VirtualRadar.Interface.WebServer.IWebServer,VirtualRadar.Interface.WebServer.RequestReceivedEventArgs) <0x00117>
  at VirtualRadar.WebSite.Page.HandleRequest (object,VirtualRadar.Interface.WebServer.RequestReceivedEventArgs) <0x000ad>
  at VirtualRadar.WebSite.WebSite.RequestContent (VirtualRadar.Interface.WebServer.RequestReceivedEventArgs) <0x00097>
  at VirtualRadar.WebSite.WebSite.Server_RequestReceived (object,VirtualRadar.Interface.WebServer.RequestReceivedEventArgs) <0x00017>
  at (wrapper delegate-invoke) System.EventHandler`1<TEventArgs_REF>.invoke_void_object_TEventArgs (object,TEventArgs_REF) <0x000c7>
  at VirtualRadar.Interface.EventHelper.RaiseQuickly<TEventArgs_REF> (System.EventHandler`1<TEventArgs_REF>,object,TEventArgs_REF) <0x00024>
  at VirtualRadar.WebServer.WebServer.OnRequestReceived (VirtualRadar.Interface.WebServer.RequestReceivedEventArgs) <0x0002b>
  at VirtualRadar.WebServer.WebServer.GetContextHandler (System.IAsyncResult) <0x00647>
  at System.Net.ListenerAsyncResult.InvokeCallback (object) <0x00050>
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () <0x00035>
  at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x001d8>
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x00017>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) <0x0004c>

Native stacktrace:

        0   mono                                0x0019f3d6 mono_handle_native_sigsegv + 342
        1   mono                                0x001f22a1 sigabrt_signal_handler + 145
        2   libsystem_platform.dylib            0x99fcd79b _sigtramp + 43
        3   ???                                 0xffffffff 0x0 + 4294967295
        4   libsystem_c.dylib                   0x96601c38 abort + 156
        5   libsystem_c.dylib                   0x965c9ff9 basename + 0
        6   libcairo.2.dylib                    0x047da55b _cairo_hash_table_lookup_exact_key + 299
        7   libcairo.2.dylib                    0x047da3d4 _cairo_hash_table_remove + 36
        8   libcairo.2.dylib                    0x048571d6 _cairo_toy_font_face_destroy + 214
        9   libcairo.2.dylib                    0x047d06b6 cairo_font_face_destroy + 214
        10  libcairo.2.dylib                    0x047d3b9a _cairo_gstate_fini + 42
        11  libcairo.2.dylib                    0x047cc603 _cairo_default_context_fini + 99
        12  libcairo.2.dylib                    0x047cc956 _cairo_default_context_destroy + 38
        13  libcairo.2.dylib                    0x047bef40 cairo_destroy + 192
        14  libgdiplus.dylib                    0x04609fe4 GdipDeleteGraphics + 196
        15  ???                                 0x045f14f4 0x0 + 73340148
        16  ???                                 0x045f11c8 0x0 + 73339336
        17  ???                                 0x045efea8 0x0 + 73334440
        18  ???                                 0x0452b608 0x0 + 72529416
        19  ???                                 0x04506ebe 0x0 + 72380094
        20  ???                                 0x04506d88 0x0 + 72379784
        21  ???                                 0x04506ce0 0x0 + 72379616
        22  ???                                 0x0375d4d0 0x0 + 58053840
        23  ???                                 0x03648fe5 0x0 + 56922085
        24  ???                                 0x045068f4 0x0 + 72378612
        25  ???                                 0x045032d0 0x0 + 72364752
        26  ???                                 0x04502c49 0x0 + 72363081
        27  mscorlib.dll.dylib                  0x01971c86 System_Threading_QueueUserWorkItemCallback_System_Threading_IThreadPoolWorkItem_ExecuteWorkItem + 54
        28  mscorlib.dll.dylib                  0x01970459 System_Threading_ThreadPoolWorkQueue_Dispatch + 473
        29  mscorlib.dll.dylib                  0x01971ad8 System_Threading__ThreadPoolWaitCallback_PerformWaitCallback + 24
        30  ???                                 0x00713c3d 0x0 + 7420989
        31  mono                                0x000e5487 mono_jit_runtime_invoke + 951
        32  mono                                0x002bdb26 mono_runtime_invoke + 150
        33  mono                                0x00297b58 worker_thread + 1880
        34  mono                                0x00294979 start_wrapper + 569
        35  mono                                0x0034e9bd inner_start_thread + 349
        36  libsystem_pthread.dylib             0x93054780 _pthread_body + 138
        37  libsystem_pthread.dylib             0x930546f6 _pthread_body + 0
        38  libsystem_pthread.dylib             0x93051f7a thread_start + 34

Debug info from gdb:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Abort trap: 6
Does the above make sense to anyone?

Regards,
Marcus

agw
Posts: 2241
Joined: Fri Feb 17, 2012 3:20 am

Re: Crash on Mac (VRS 2.4.0)

Post by agw » Mon Aug 29, 2016 9:54 pm

It's saying that the graphics library that mono is using (libcairo) threw an exception after an aircraft picture had been resized. The problem with libcairo crashes is that I'm not calling it directly, it's going through mono, and they're typically intermittent and difficult to reproduce.

I've got an item on the list to look at the graphics handling code to see if I'm doing anything that I shouldn't - hopefully something will fall out of that :) We'll see.

mgunther
Posts: 17
Joined: Sun Aug 21, 2016 6:45 pm

Re: Crash on Mac (VRS 2.4.0)

Post by mgunther » Tue Aug 30, 2016 8:43 am

Is there any extra logging or debugging that I could enable to help track down the issue?

mgunther
Posts: 17
Joined: Sun Aug 21, 2016 6:45 pm

Re: Crash on Mac (VRS 2.4.0)

Post by mgunther » Tue Aug 30, 2016 10:43 am

Mr Google found this bug in Cairo:
https://bugs.freedesktop.org/show_bug.cgi?id=69470

and related in mono:
https://bugzilla.xamarin.com/show_bug.cgi?id=2426

more info:
https://lists.cairographics.org/archive ... 24667.html

Looks awfully similar to the messages I'm seeing.
Bugzilla Cairo bug:
/build/buildd/cairo-1.12.14/src/cairo-hash.c:506: _cairo_hash_table_lookup_exact_key: Assertion `!"reached"' failed.

My output:
Assertion failed: (!"reached"), function _cairo_hash_table_lookup_exact_key, file cairo-hash.c, line 506.

The bug reports also talk about cairo_font_face_destroy and cairo_toy_font_face_destroy and this is also what I'm seeing.

The Cairo version where this bug is fixed seems to be 1.12.14. But on the mono documentation I see that mono uses its own internal copy of Cairo version 1.6.4 (http://www.mono-project.com/docs/gui/libgdiplus/)
I'm not sure if and how the Cairo version can be upgraded independently.

Assuming that my findings above are correct and we are indeed running into this bug, is there a way you can work around it in your code?

agw
Posts: 2241
Joined: Fri Feb 17, 2012 3:20 am

Re: Crash on Mac (VRS 2.4.0)

Post by agw » Thu Sep 01, 2016 12:58 am

It looks like the libcairo bug is triggered when you have multiple threads performing graphics actions. I got rid of all multi-threaded graphics handling some time ago and forced it all onto a single thread. It's possible that either that code isn't working and multiple threads are still able to do graphics operations concurrently or, more likely, that I missed a graphics call when I was changing all of the instances to the new mechanism. The item I have on the list is to basically redo the searches and see if I overlooked anything.

I'll take another look at the weekend, see if I can see anything.

Post Reply