Receiver losing connection, with no error

[kallig]

Hi, one of my reveivers keep loosing its connection but VRS still shows them as connected,
I can see when this happenes when the "Aircraft Tracked" goes to 0, and the "Bad Messages" has "1up"
Can it be that a badly formatted message is crashing the connection ?

The receiver that does this is located on a moutntain and is using a 4G router to connect to the internet and the connection is lost only after a only a few hrh's
This is my most efficiant location so it is a real bummer that it is not keeping the connection.

It does not seem the LOG has anything on this untill I start reconnecting the feed, then it shows this:

Code: Select all

014-08-03 06:27:08.990 UTC] [t1] Disconnected from Blafjoll
[2014-08-03 06:27:08.990 UTC] [t1] Connecting to Blafjoll
[2014-08-03 06:27:09.038 UTC] [t28] Connected to Blafjoll

[jonfear]

Hi Kallig

Not directly helpful but in experiments at work, we found we needed to get an industrial grade router AND a SIM that the ISP did not kick off after a few hours. Perhaps your first port of call is the ISP and see if they can make the SIM truly 24/7. They will argue it is already but do not accept that! I now have a network of seven 3G sites around the UK running 24/7 for traffic information screens.

The bad message issue along with showing connected is one for Andrew. I too would like some more information here!

Jon

[agw]

I imagine that the bad message is just a side-effect of a message getting truncated as the feed goes down.

At the moment VRS uses heartbeat packets to test the connection to see whether it's gone down - if the server doesn't send anything for something like 30 seconds the server sends a heartbeat, if three of those ten seconds apart go unanswered then it drops the connection and tries to re-establish it. However it appears that under some circumstances the program that's sending the feed can stop sending but still reply to the heartbeat packets.

I already have code to timeout a connection and just break it and reconnect if so-many seconds go by without a message from the source. At the moment that's automatically used if you're running under Mono but you can switch it on with a command-line switch in the Windows version (start the program from the command-line instead of the Start Menu icon and use the -listenerTimeout switch. For example if VRS is installed on a 64-bit Windows system then these commands would start it with a timeout of 60 seconds):

Code: Select all

cd %PROGRAMFILES(X86)%\VirtualRadar
VirtualRadar -listenerTimeout:60

I might add a configuration option for the listener timeout in 2.0.3.

[jonfear]

Hi Andrew

I think that would be a great option to have. If you could impliment it in 2.0.3 even better. I have applied the command line to VRS World.

Many thanks

Jon

[Skibox]

Hi,

I had very similar problems to the extent that it was impossible to let VRS run unattended on my server. Some of these issues were discussed here: http://forum.virtualradarserver.co.uk/v ... =351#p1764

Not directly helpful but in experiments at work, we found we needed to get an industrial grade router AND a SIM that the ISP did not kick off after a few hours. Perhaps your first port of call is the ISP and see if they can make the SIM truly 24/7. They will argue it is already but do not accept that! I now have a network of seven 3G sites around the UK running 24/7 for traffic information screens.
Jon

My connections had problems depsite being very stable DSL and even some fiber connections, some with VPN tunnels and business-grade firewalls.

While it is of course better with a more stable connection, the internet is by design a "no-guarantees" transmission medium. There will always be cases of interruptions, packet loss, DNS problems or site downtime, and the software must be able to handle such events without manual hands-on to be truly scalable. I have many years experience from hosting similar remote equipment and my experience is that you need several layers of checks to keep systems running and re-connecting without intervention.

At the moment VRS uses heartbeat packets to test the connection to see whether it's gone down - if the server doesn't send anything for something like 30 seconds the server sends a heartbeat, if three of those ten seconds apart go unanswered then it drops the connection and tries to re-establish it. However it appears that under some circumstances the program that's sending the feed can stop sending but still reply to the heartbeat packets.

I already have code to timeout a connection and just break it and reconnect if so-many seconds go by without a message from the source. At the moment that's automatically used if you're running under Mono but you can switch it on with a command-line switch in the Windows version (start the program from the command-line instead of the Start Menu icon and use the -listenerTimeout switch.

As stated earlier, I don't trust heartbeat packets, as I had almost one feed hang every 24 hours. After switching to the listener-timeout method, stability has improved greatly and now I have maybe one hang per month! So I can really recommend implementing this setting, and maybe considering using it as default as well.

BR /M

[kallig]

Thank you Andrew, I will see if this works

[kallig]

Too bad, the code you gave me did not work.
I am still loosing the connection :/

[jonfear]

Kallig

I find it strange that you are seeing so many connectivity issues, especially when you are not getting the feed to auto-recover. As you will know, we have a quite few feeds now on VRS World and I just do not see the problems you describe. I will admit that we have no "mobile" data feeds but we do have all sorts of other feed, either from server farms or direct off of DSL. Those on dynamic IP are using a no-ip style service, those with static are obviously direct.

AFAIK we are not using anything special in the setups, just home routers etc. The server is virtualised on an anonymous server in France.

Please don't shot me for this but is your network setup "over" engineered? The old saying "keep it simple" works for VRS World.

Not having a push system is a pain but we have to live with that for the time being. I am talking to the software team at work and we think there maybe a possible standalone app we could write until Andrew can give us the push system. I am trying to get the team to write it cross platform and open source, it may even help Andrew.

Jon

[kallig]

I do not think my network is engineered at all, and it is certanly not over engineered I am not sure what gave you that impression.

I can list for you how my setup looks like:

I have a a simple Mode-S Beast receiver (ethernet) located in a mountainhut connected to a friends basic 4G router from our local internet company.
Conveniently a 4G antenna from the same internet company is located on the same mountaintop. So the 4G internet connection is solid.
We can access the router it remotly and open ports and change some basic settings if we need.

On my end I also have an basic home router connected to a 100MB/S Fiber connection. The server is an old PC running windows 7, VRS and nothing else.

There is nothing more to it, Just couple of routers and couple of lan cables, plug and play

But regarding my problem, This setup is almost indentical to all the other receivers I have access to (Mode-S Beast on a basic router)
Andrew's suggestion did not work and I an not requesting more for him, I am really thankfull to his contribution.

I have tested the feed with Planeplotter and I have no cutoffs there.
There is one workaround that I can think off, but that would deffinetly be considured over engeneering in both of our books that would be using a PP setup to forward a stable feed to VRS

But for now I just reconnect the feed now and then, The connections drops once per day on average.
And to be honest, My network is pretty solid, even without this feed. so I am not too worried about this at the moment.

-K