Virtual Radar+Apache

The "to-do" list for Virtual Radar Server is as long as my arm, but if you want to pile more work onto it then this is for you.
madrigal
Posts: 13
Joined: Sat Mar 10, 2012 10:44 pm

Re: Virtual Radar+Apache

Post by madrigal » Fri Jul 20, 2012 4:25 pm

After a long delay I'd like to present you a new version of VR+Apache+Perl. It has support for writes in BaseStation.sqb database. In order to work properly some new tables were added which does not interfere with standard database schema at any way. As an addition to that there is a table which stores information for the whole airplane's routes intercepted by ADS-B receiver.
The all kind of reports can be performed but my goal is to add an ability to display the whole aiprlane's route on the map when the link "Show path taken on map" is clicked.
Since that requires rework of the JavaScript code (in Report*.js files) and I'm not so good in writing JavaScript, I'd be glad if Andrew can give me a hint how the request for the database query can be started when the link "Show path taken on map" is clicked and how to create a route after receiving the result. It can be very useful if some player can be attached to the map at that point in order to play the route step-by-step, fastforward, rewind etc.
Some screenshots:
ss-1.jpg
ss-1.jpg (240.26 KiB) Viewed 7384 times
ss-3.jpg
ss-3.jpg (182.35 KiB) Viewed 7384 times
ss-5.jpg
ss-5.jpg (198.39 KiB) Viewed 7384 times
You can check how it is work on:
http://87.97.249.103:8001/vr/

Please note - the software is in testing stage, I'll be glad if someone points out my mistakes, bugs etc.
The old site is active too but has a new address - http://vr.telesys.bg (Telesys is the company I'm working for).

agw
Posts: 2249
Joined: Fri Feb 17, 2012 3:20 am

Re: Virtual Radar+Apache

Post by agw » Sat Jul 21, 2012 3:00 am

Very nice - sergsero will be pleased to have the full path shown :)
madrigal wrote:if Andrew can give me a hint how the request for the database query can be started when the link "Show path taken on map" is clicked and how to create a route after receiving the result
I did type this out but it's rather long and it'll just spam up the thread - I'll PM it to you.

g0hww
Posts: 11
Joined: Sun Jul 22, 2012 9:16 pm

Re: Virtual Radar+Apache

Post by g0hww » Mon Jul 23, 2012 12:21 pm

This looks very interesting. I'm having some issues on Linux with the regular VRS, so I'd love to try this version. I can't find a link though. Happy to help test, even eager in fact, as I have the day off and have nothing better to do than play with ADS-B databases :P

madrigal
Posts: 13
Joined: Sat Mar 10, 2012 10:44 pm

Re: Virtual Radar+Apache

Post by madrigal » Mon Jul 23, 2012 12:55 pm

g0hww wrote:This looks very interesting. I'm having some issues on Linux with the regular VRS, so I'd love to try this version. I can't find a link though. Happy to help test, even eager in fact, as I have the day off and have nothing better to do than play with ADS-B databases :P
The link is
http://87.97.249.103:8001/vr/
as stated in my previous post. The link may not be online 24/7 because the server is behind a router with dynamic address.

g0hww
Posts: 11
Joined: Sun Jul 22, 2012 9:16 pm

Re: Virtual Radar+Apache

Post by g0hww » Mon Jul 23, 2012 5:10 pm

I actually want to run it myself. Its the source I'm after, please :)

Edit. OK, perhaps I was being stupid. I grafted the path from the earlier post onto the other IP address in the link you just posted, and got me a tarball, from here http://87.97.249.103:8001/vr/vr-linux.tar.gz

Now to see what I can do with it. Thanks :)

madrigal
Posts: 13
Joined: Sat Mar 10, 2012 10:44 pm

Re: Virtual Radar+Apache

Post by madrigal » Mon Jul 23, 2012 11:08 pm

OK, I understood.
I'll release the code under GPL, of course. Till now, however, I think I'm going to reorganize the database structure at least once. The programs which tasks are to store the data in database shall be reorganized too. Nevertheless, I'll try to prepare the necessary setup files for recruiting the database and to establish it's normal filling with data and to give access to them. But be warned - at the moment the table for storing the flight's routes (I named it Routes) tend to grow constantly - for a month of work I have about 1 200 000 records in it at an average of 50 - 85 different flights at any moment - you can calculate how the table will grow in your conditions. That's why I think the table should be split to different tables (for any month for example). This will lead, however, to difficulties in fetching a flight's route which spans between two months (yes, I know there are UNIONs, VIEWs etc. but I haven't implemented them yet). The other problem is updating the tables which hold the information about Airlines and Air Routes - I'm using Andrew's approach in finding them since we have not access to flight plans. The problem is: let say a particular Callsign corresponds to a flight route Sofia - Plovdiv (LBSF-LBPD). In some time in the future the Callsign can be assigned to a different route (for example LBSF-LTBA). If I update the corresponding tables in database according to new routes the information about the old ones will be changed too - a thing we won't want to happens. I know there would be a solution but I didn't found and implement it yet.
As a final conclusion I think the structure should be as follows: one process for fetching the ADS-B receiver's data, converting it to port 30003 compatible output and filling the Flights table with information for every new flight, the other which accepts port 30003 data, analyses it and fill the Routes table when thresholds are reached. A CGI script which works in mod_perl who serves requests for displaying the real-time data and another two for serving post-flight date/icao/registration reports and displaying the routes for chosen flight. One of them should ensure updates in Airlines table as long as to reroute the records to new Routes table at the beginning of every month. well that is for now - I hope I'll update the situation in next days.

g0hww
Posts: 11
Joined: Sun Jul 22, 2012 9:16 pm

Re: Virtual Radar+Apache

Post by g0hww » Tue Jul 24, 2012 7:43 pm

Hi Madrigal,

I haven't taken a look at his yet after downloading it last night. I did read the readme, and I figured that I had a fair bit of leg work to do to prepare my machine to run it, as I don't even have apache installed yet :) Hopefully I'll have a go with it later.

I thought for a minute that you may be using json (something I'm entirely unfamiliar with, even more so than perl) instead of SQLite, but I suspect that you are just using it 'over the wire' to the browser clients.

Regarding invariant historical data, I asked my friend about this very issue a few days ago. He recommended that the best approach is to perform a query on the routes as part of the insertion of the Flight info. The idea is that the Routes table in the database would hold all the old routes associated with a Flight Number, as well as the latest, and that when inserting a Flight in the Flights table, the most recent Route record corresponding with the Flight Number would be referenced in the Flight record. That way, old Flight records continue to reference the correct old route. Even better, histories about Flight Numbers and Routes can be queried.

Of course, this assumes that flight info is stored in the database instead of in CSV files, along with, probably everything else in CSV files currently. Or in separate databases which can be attached.

I may give the impression that I know what I'm talking about, but I am a bit of a newbie in the sqlite (and databases in general) world, so I might be barking up the wrong tree here, but I'm keen to get more data into the database(s), and to increase the scope of the queries that can be run via the web interface. For a start, wildcard string matching with the % symbol would seem to be a must have for me. I'm hoping that your perl implementaion will let me play with stuff like that.

madrigal
Posts: 13
Joined: Sat Mar 10, 2012 10:44 pm

Re: Virtual Radar+Apache

Post by madrigal » Thu Jul 26, 2012 7:46 pm

Hi g0hww!

As a mater of facts there are two different implementations - the first (and older one) which works on vr.telesys.bg does not have a history at all. It just parses the data from port 30003 and generates a JSON file which is sent by Apache to the clients as a response to their requests. The information from Andrew's CSV files (for Airlines, Countries, Airports etc.) are loaded in hashes when the program starts up. Since there is no history if there are new versions of CSV files the easiest way to load them is simply to restart the program.
The second implementation (the new one) which currently works on http://87.97.249.103:8001/vr/ is completely database driven - even before mentioned CSV files are loaded to the database tables along with the mentioned tables Routes (which stores the history data for all flights) and the table named "curpos" for real-time data for every flight (lat, lon, alt, track, ground speed, vert. speed, squawk, timestamp, ICAO address). The data in curpos table is refreshed when the new info from given plane is received and after the plane goes out of receiver's range the particular row is deleted. My thoughts and concerns in the previous post was about the situation when the new CSV files are generated from Andrew and the corresponding tables are updated. In such a situation if you create a report for given plane (based on ICAO address) or for some period of time there might be a flights for which the Airline data would differ from the old one (I'm not sure if I clearly explain the problem - you should know the mechanism for finding the route info from Callsign assigned to every flight and it is not so simple - it took me a day or two to decode the Andrew's approach from it's very, very well documented source code as opposite to my).
Anyway - the tarball you downloaded contains the older implementation and if you are not so familiar with ADS-B and Linux (and particular with Apache) I suggest you to stick to it until you get a clearer picture how the stuff works and how the JSON file is generated.
After that you can try a new implementation which is more powerful (and more complex). I hope that at that moment its source will be released after some additional bugfixing and database reorganisation.
Of course I'll take into account all suggestions concerning my code and style if you (or anyone else) have enough Perl skills to understand what I have done.
Now I'm struggling with some JavaScript code (GoogleMap InfoBox-es in particular) - I'm not a good JS coder but don't want to bother Andrew again.

g0hww
Posts: 11
Joined: Sun Jul 22, 2012 9:16 pm

Re: Virtual Radar+Apache

Post by g0hww » Fri Jul 27, 2012 1:15 am

madrigal wrote:Hi g0hww!

As a mater of facts there are two different implementations - the first (and older one) which works on vr.telesys.bg does not have a history at all. It just parses the data from port 30003 and generates a JSON file which is sent by Apache to the clients as a response to their requests. The information from Andrew's CSV files (for Airlines, Countries, Airports etc.) are loaded in hashes when the program starts up. Since there is no history if there are new versions of CSV files the easiest way to load them is simply to restart the program.
I haven't finished getting it working yet, but I've been trying to figure out what the code seems to do (and for this I cheated and skyped a friend who is more familiar with the javascript and json stuff, as my 'talents' lay elsewhere).
The second implementation (the new one) which currently works on http://87.97.249.103:8001/vr/ is completely database driven - even before mentioned CSV files are loaded to the database tables along with the mentioned tables Routes (which stores the history data for all flights) and the table named "curpos" for real-time data for every flight (lat, lon, alt, track, ground speed, vert. speed, squawk, timestamp, ICAO address). The data in curpos table is refreshed when the new info from given plane is received and after the plane goes out of receiver's range the particular row is deleted.
I'd be interested in storing full track history, I think. Interested enough to be thinking about a MySQL backend, running on my NAS RAID.
My thoughts and concerns in the previous post was about the situation when the new CSV files are generated from Andrew and the corresponding tables are updated. In such a situation if you create a report for given plane (based on ICAO address) or for some period of time there might be a flights for which the Airline data would differ from the old one (I'm not sure if I clearly explain the problem - you should know the mechanism for finding the route info from Callsign assigned to every flight and it is not so simple - it took me a day or two to decode the Andrew's approach from it's very, very well documented source code as opposite to my).

Anyway - the tarball you downloaded contains the older implementation and if you are not so familiar with ADS-B and Linux (and particular with Apache) I suggest you to stick to it until you get a clearer picture how the stuff works and how the JSON file is generated.
After that you can try a new implementation which is more powerful (and more complex). I hope that at that moment its source will be released after some additional bugfixing and database reorganisation.
I'm quite familiar with Linux and ADS-B - I'm using a £25 DVB-T dongle with gnu-radio to get my data :) I'm not familiar with the web-server side of things though. I have a couple of posts on my blog http://www.g0hww.net/
Of course I'll take into account all suggestions concerning my code and style if you (or anyone else) have enough Perl skills to understand what I have done.
Now I'm struggling with some JavaScript code (GoogleMap InfoBox-es in particular) - I'm not a good JS coder but don't want to bother Andrew again.
Can you give any more hints about the database you are using now, please? Is it still sqlite3? Similar to the schema used by the original VRS? I've collected quite a bit of data in my database and have been manually tweaking it too.

sulamit
Posts: 1
Joined: Wed Jan 30, 2013 12:51 pm

Re: Virtual Radar+Apache

Post by sulamit » Wed Jan 30, 2013 8:53 pm

Hi! Is somebody working at this project VR+Apache+Database?

Post Reply