Out of memory exception error

Is Virtual Radar Server not behaving itself? If so then please report it here.
Sketco
Posts: 6
Joined: Thu Sep 29, 2016 12:38 pm

Out of memory exception error

Postby Sketco » Mon May 15, 2017 10:46 am

Hi,
Running VRS 2.4.0. I've always had random CB status 4 errors from dump1090,which are a simple shut down restart solution, but now VRS is throwing up "out of memory exception caught" errors.
I can shut VRS down normally from the menu, but attempting to restart uses 100% disk and 60% memory for anything up to an hour before starting, and sometimes it doesn't start at all requiring a reboot of the laptop (windows 8.1).
The occurences started off randomly a week or so ago, but are becoming more frequent (at least once every other day). The malware, windows registry etc. routes have been largely eliminated.
Are there any pointers to getting this fixed?
Cheers

Sketco
Posts: 6
Joined: Thu Sep 29, 2016 12:38 pm

Re: Out of memory exception error

Postby Sketco » Mon May 15, 2017 12:12 pm

Update my last. Instead of tearing my hair out, I should have taken a closer look through the postings.
I'm trying the sqlite file repair solution with fingers crossed.
I'll be back for some sage advice if this doesn't fix it. However, any readers with knowledge of the CD status 4 error on dump1090 please feel free to jump in, as this is an ongoing PITA that I haven't been able to resolve.
Cheers

Sketco
Posts: 6
Joined: Thu Sep 29, 2016 12:38 pm

Re: Out of memory exception error

Postby Sketco » Thu May 18, 2017 12:25 pm

Further to my last posts.
I have used the sqlite file repair (although it ran for 24hours and I finally closed it). So far so good VRS has been up for 4 hours which is a record since the exception error began popping up. Dump 1090 died with an CB staus 4 error 1hour and 20 minutes in but has remained up for 2 hours plus since (not a record yet as the status error is random).
What now transpires is that my receivers feed is now showing 692 aircraft tracked (I only count 48 on the dump 1090 output). All appears to be running normally otherwise. [UPDATE..the feed just dropped back to 50]
Questions:
How long should sqlite file repair run? (roughly 7.35Gb file size)
What is the cause of the Dump1090 CB status error?
VRS is using 606Mb of memory, 24% CPU and I am showing 100% disk usage is this normal? (Toshiba core i3, 4gb ram, Windows 8.1, Chrome browser as firefox doubles the memory usage)
Any ideas please?
Cheers

Sketco
Posts: 6
Joined: Thu Sep 29, 2016 12:38 pm

Re: Out of memory exception error

Postby Sketco » Sat May 20, 2017 8:00 am

Ok, so I thought things were sorted. A couple of days up and it was looking good then VRS threw another exception.
The log is below. Any thoughts please.


[2017-05-20 02:28:20.822 UTC] [t1] System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.Dictionary`2.Resize()
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at VirtualRadar.Interface.CollectionHelper.ShallowCopy[TKey,TValue](Dictionary`2 dictionary)
at VirtualRadar.Library.BaseStation.BaseStationAircraftList.ProcessMessage(BaseStationMessage message, Boolean isOutOfBand, Boolean isSatcomFeed)

[2017-05-20 02:56:15.480 UTC] [t22] Caught an exception on a fast tick heartbeat event: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Deserializeb9ae24ac-a5f3-4c33-9699-ca343436a5c5(IDataReader )
at Dapper.SqlMapper.<QueryImpl>d__58`1.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType)
at VirtualRadar.Database.BaseStation.Database.AircraftAndFlightsCount_GetByIcaos(IEnumerable`1 icaos)
at VirtualRadar.Database.BaseStation.Database.GetManyAircraftAndFlightsCountByCode(IEnumerable`1 icao24s)
at VirtualRadar.Library.AircraftDetailFetcher.DoFetchManyAircraft(IEnumerable`1 fetchedDetails)
at VirtualRadar.Interface.AircraftFetcher`2.FetchAllAircraft(IEnumerable`1 fetchedDetails)
at VirtualRadar.Interface.AircraftFetcher`2.Heartbeat_FastTimerTicked(Object sender, EventArgs args)
at lambda_method(ExecutionScope , Object , Object , EventArgs )
at VirtualRadar.Interface.EventHelper.Raise[TEventArgs](Delegate eventHandler, Object sender, Func`1 buildArgsCallback, Action`1 exceptionCallback, Boolean throwEventHelperException)

Sketco
Posts: 6
Joined: Thu Sep 29, 2016 12:38 pm

Re: Out of memory exception error

Postby Sketco » Fri Jun 02, 2017 4:30 am

Hi,
Still having problems with VRS throwing exceptions. I can't get the VRS to remain up for more than 24 hours. I did notice that the ADSBExchange Feed shot up to 116,000 aircraft tracked and dump1090 over doubled its input (tried to post screenshot but got a http error).
Does anyone have any ideas where I might start to troubleshoot this?..the log doesn't mean anything to me at all but \i've posted it below in the hope someone might be able to offer some advice.
Cheers


[2017-06-02 01:23:41.802 UTC] [t1] System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.Dictionary`2.Resize()
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at VirtualRadar.Interface.CollectionHelper.ShallowCopy[TKey,TValue](Dictionary`2 dictionary)
at VirtualRadar.Library.BaseStation.BaseStationAircraftList.ProcessMessage(BaseStationMessage message, Boolean isOutOfBand, Boolean isSatcomFeed)

[2017-06-02 01:36:46.307 UTC] [t27] Database writer plugin caught exception on message processing: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.CurrentSystemTimeZone.GetDaylightChanges(Int32 year)
at System.CurrentSystemTimeZone.GetUtcOffsetFromUniversalTime(DateTime time, Boolean& isAmbiguousLocalDst)
at System.CurrentSystemTimeZone.ToLocalTime(DateTime time)
at VirtualRadar.Plugin.BaseStationDatabaseWriter.Plugin.DefaultProvider.get_LocalNow()
at VirtualRadar.Plugin.BaseStationDatabaseWriter.Plugin.TrackFlight(BaseStationMessage message, Boolean isMlat)
at VirtualRadar.Plugin.BaseStationDatabaseWriter.Plugin.MessageQueue_MessageReceived(BaseStationMessageEventArgs args)
[2017-06-02 01:42:14.346 UTC] [t23] Caught an exception on a fast tick heartbeat event: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at VirtualRadar.Database.SQLiteExecute.<>c__DisplayClass10_0.<ExecuteReader>b__0()
at VirtualRadar.Database.SQLiteExecute.CallActionUntilUnlocked(Action action)
at VirtualRadar.Database.SQLiteExecute.ExecuteReader(IDbCommand command)
at VirtualRadar.Database.Sql.RunSql(IDbConnection connection, IDbTransaction transaction, String commandText, Dictionary`2 parameters, Func`2 readerFunc, Boolean isScalar, TextWriter log)
at VirtualRadar.Database.StandingData.StandingDataManager.FindRoute(String callSign)
at VirtualRadar.Database.StandingData.CallsignRouteFetcher.DoFetchAircraft(FetchedDetail fetchedDetail)
at VirtualRadar.Interface.AircraftFetcher`2.FetchAircraft(FetchedDetail fetchedDetail)
at VirtualRadar.Interface.AircraftFetcher`2.Heartbeat_FastTimerTicked(Object sender, EventArgs args)
at lambda_method(ExecutionScope , Object , Object , EventArgs )
at VirtualRadar.Interface.EventHelper.Raise[TEventArgs](Delegate eventHandler, Object sender, Func`1 buildArgsCallback, Action`1 exceptionCallback, Boolean throwEventHelperException)

Sketco
Posts: 6
Joined: Thu Sep 29, 2016 12:38 pm

Re: Out of memory exception error

Postby Sketco » Sun Jun 04, 2017 3:51 am

Many Thanks to those who offered assistance.
The database has now locked itself, so I've had to start a new one. I'll see how long this one lasts
Cheers

agw
Posts: 1862
Joined: Fri Feb 17, 2012 3:20 am

Re: Out of memory exception error

Postby agw » Fri Jun 09, 2017 12:52 am

Sorry for the delay in replying! Out of memory exceptions are usually caused by VRS pushing messages or objects into a queue faster than it can process them. In the current release build there are mechanisms in place to make it drop old items from queues that are growing out of control. You can see the state of its internal queues by going into Help | Diagnostics | Show Queues.

However, from the sounds of it I suspect you're not going to see out of control queues on that screen. It sounds like your database file is (or was) knackered. All the file repair tool does is run SQLite's "sqlite3" tool against your database to do one of two things:

1. Check the file for errors and
2. Rebuild the file by dumping its contents, creating a new and empty file and then loading the dump from the old file.

It's basically automating the usual procedure that you'll see all over Google when you search for "repair sqlite file".

Neither of those operations should take 24 hours on an 8GB file, and if you force-quit the program before it finished then it wouldn't have done anything useful. I imagine that now that you've started a new database file you should be alright.

One thing to note is that SQLite's locking doesn't work very well over networks, you'll want to make sure that BaseStation.sqb is on the same machine that VRS is running on... particularly if other programs are interacting with it while VRS is writing to it.


Return to “Bug Reports”

Who is online

Users browsing this forum: No registered users and 1 guest