Bugs in 2.3.1

Is Virtual Radar Server not behaving itself? If so then please report it here.
dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Bugs in 2.3.1

Post by dsfh2992 » Mon Jan 18, 2016 5:39 pm

agw wrote:
The program tries to read aircraft details in batches, but there is no cap on the size of the batch. It looks like your instance of VRS has so many aircraft to lookup in one go that it's exceeding some limit imposed by SQLite. I should be able to reproduce that and get it fixed in a preview.

I don't think that would cause the OOM exception though. I suspect what may be happening is that messages are coming in faster than VRS can process them. You'll be able to tell if this is the case by opening the queues window and watching for message queues that seem to keep on growing. That was the problem with OOM exceptions in 2.3.0, I had done something to slow message processing down and large feeds were putting messages into the queue faster than VRS could take them out. Version 2.3.1 addressed the slow message processing problem but did not address the problem of messages coming in faster than they can be processed. I'll have a think about that one, but I suspect the only thing that VRS can do in that situation is to start dropping messages at the head of the queue.
Cool, thanks for the info.

Regarding the messages coming in faster than VRS can handle them, I'd say dropping messages and logging some type of alert would be preferable to crashing. Here's an idea, let's say a message from a certain aircraft comes in. If the queue gets big, maybe drop all subsequent messages regarding that aircraft for the next "x" seconds? That way, aircraft that may have excessive info coming from overlapping receivers will not drown out aircraft with only occasional data from a single receiver. (am I making sense?).

Also, here is what my queues look like. The "MergedFeedListenerMessages" jumps around a lot. Between 0 - 100,000, with the peak being about 1.1 million.

What resource should I be adding to allow VRS to process more messages? Lower latency storage (i.e. SSDs) for the basestation DB? More processing threads? (or if VRS is not multi-threaded, just a higher clocked processor?)

Thanks again,
Dan
adsbexchange.com

agw
Posts: 2241
Joined: Fri Feb 17, 2012 3:20 am

Re: Bugs in 2.3.1

Post by agw » Thu Jan 21, 2016 10:58 pm

The code to record the message counts and times from aircraft so that it can prefer to drop messages from chatty aircraft over messages from quieter aircraft would add more overhead to the message processing code. I'll add some code to start dropping messages if the message processing queue gets to an excessive size, say .25 million messages, but I'll also take a look at the message processing with a profiler and see if things can be sped up a bit there.

VRS is multithreaded and it normally doesn't do any direct disk reads during message processing, the lookups get handed over to another thread to deal with. The more cores you can give it the better.

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Bugs in 2.3.1

Post by dsfh2992 » Fri Jan 22, 2016 4:47 pm

agw wrote:The code to record the message counts and times from aircraft so that it can prefer to drop messages from chatty aircraft over messages from quieter aircraft would add more overhead to the message processing code. I'll add some code to start dropping messages if the message processing queue gets to an excessive size, say .25 million messages, but I'll also take a look at the message processing with a profiler and see if things can be sped up a bit there.

VRS is multithreaded and it normally doesn't do any direct disk reads during message processing, the lookups get handed over to another thread to deal with. The more cores you can give it the better.
OK, thanks. I see your point about the overhead. I think dropping messages would be a big help. Can't wait for the preview release that implements that! ;)

I realize you're just doing it on your own time though, so I'm grateful for it whenever it arrives.

Another thing I have been considering is that on my main VRS server, it is taking 3 feeds (US-West, US-East, and Europe/other) and merging them, then sharing that merged feed. Perhaps if I had another VRS instance do the merging, and fed that into the user-facing instance that would reduce the load.

agw
Posts: 2241
Joined: Fri Feb 17, 2012 3:20 am

Re: Bugs in 2.3.1

Post by agw » Sun Jan 24, 2016 10:57 pm

Okey dokey - I've just uploaded a preview that should hopefully fix the SQLite exception when fetching > 999 records from the lookup cache in one go. I've also done the automatic dropping of old messages if the message processing queue gets up to something like 200,000 messages. The queues window should show the number of dropped messages for each queue.

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Bugs in 2.3.1

Post by dsfh2992 » Mon Jan 25, 2016 4:13 am

Awesome! Downloading new preview version now!

Thanks a ton.

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Bugs in 2.3.1

Post by dsfh2992 » Mon Jan 25, 2016 4:49 am

Looking great so far! Plane data populates almost instantly now! No more LOOOONG wait... ;)

The queue issue, I'll let you know after a couple days. If it gets through the next 24 hours, especially the peak period around mid-morning pacific US time, I think we should be good.


Thanks again!
Dan
adsbexchange.com

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Bugs in 2.3.1

Post by dsfh2992 » Mon Jan 25, 2016 8:27 pm

OK, now instead of crashing, it just gets pegged at 200,000, and starts dropping. This morning it dropped 271 million messages.

I am not seeing any resource contention on the host and it only seems to use 50% of the CPU. Is there a way to determine the bottleneck?

Thanks,
Dan

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Bugs in 2.3.1

Post by dsfh2992 » Mon Jan 25, 2016 8:30 pm

BTW, the queue that's maxing out is "MergedFeedListenerMessages".

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Bugs in 2.3.1

Post by dsfh2992 » Mon Jan 25, 2016 9:12 pm

I just changed all my feeds to JSON (instead of Compressed VRS) and the queue seems to be more under control now.

I still wonder why it seems to run out of resources at 50% CPU though....

agw
Posts: 2241
Joined: Fri Feb 17, 2012 3:20 am

Re: Bugs in 2.3.1

Post by agw » Tue Jan 26, 2016 12:23 am

Well, at least the drop code is working :) There's less effort involved with the JSON feed, it's not forwarding on every message from the aircraft. I have an item on the list to profile the aircraft list and merged feed handlers, and I know already that the aircraft list does more locking than is necessary so I think I should be able to improve performance there. The under-utilisation of the processor sounds like it might be a locking issue. There was a problem in the past where merged feeds ended up inadvertently blocking on a lock held by the receivers, which slows down the merged feed processing - instead of being able to run in parallel the merged feed processing becomes interleaved (at least for a while) with the message processing across all of its receivers. Maybe something like that is happening here. I'll take a look into it.

Locked