Database Writer Options..?!!??

Are you having problems with using or developing a plugin? Let me know here.
lcova76
Posts: 13
Joined: Mon Jun 01, 2015 9:16 pm

Database Writer Options..?!!??

Post by lcova76 » Mon Jan 04, 2016 7:11 am

I just downloaded and installed version 2.3.1 and installed the Database Writer 2.3.0..

What does the "Save Online Lookups in Database" and "Overwrite details on existing aircraft" do??

How do I do an Online Lookup?

Is there any info on this feature?

agw
Posts: 2249
Joined: Fri Feb 17, 2012 3:20 am

Re: Database Writer Options..?!!??

Post by agw » Tue Jan 05, 2016 12:12 am

Aircraft detail online lookups was a feature that was added in 2.3. It's switched on by default, you can switch it off in Tools | Options. It caches the lookups in its own file, the options in the database writer plugin tell VRS whether you want to write the aircraft details to BaseStation.sqb instead. That option is switched off by default. You can either have it write details for new / missing aircraft or you can overwrite aircraft that have details which were last updated more than 28 days ago.

paradiselost
Posts: 103
Joined: Sun Apr 06, 2014 10:22 am
Location: Philippines

Re: Database Writer Options..?!!??

Post by paradiselost » Thu Jan 07, 2016 8:17 pm

Aircraft detail online lookups was a feature that was added in 2.3. It's switched on by default, you can switch it off in Tools | Options. It caches the lookups in its own file, the options in the database writer plugin tell VRS whether you want to write the aircraft details to BaseStation.sqb instead. That option is switched off by default. You can either have it write details for new / missing aircraft or you can overwrite aircraft that have details which were last updated more than 28 days ago.
To flesh out Andrews comments, the lookup information is served by Andrew's server a work in progress.

Database Writer option 1 enable or disable lookups, verification of enabled previous 30 minute segment is Help/About/Diagnostics/Aircraft Online Lookup Log.

Database Writer option 2 enabled allows looked up data to be displayed on the live view list area for all configured columns. Example Year Built. Reports configured to show Year Built column.

Effects of Option 1 and Option 2 and the combination of the two.

Option 1 Online Lookup enabled fetches and writes to a local file but not basestation database. Online Lookup does not display on the live view list area nor the reports. Verify with Help/About/Diagnostics/Aircraft Online Lookup Log.

Option 1 disabled and Option 2 enabled. Build Date shows on the live view list area but not in the reports. The build date is displayed but not written to the basestation database.

Checking both Option 1 & 2 Allows the Online Lookup overcoming the last seen limitation and displays looked up records both on the live view list area and in the reports and writes to the basestation database.

In my example not all of the information is available from Andrew's server and will require manual entry using the Database Editor and using one of the other services like http://www.airframes.org/ to get the missing information and saving it with the Database Editor.

paradiselost
Working Example of Version 2 Beta Virtual Radar Server http://dgteflyovers.ddns.net/virtualradar/
VRS 2 Help Files http://dgteflyovers.ddns.net:8080

paradiselost
Posts: 103
Joined: Sun Apr 06, 2014 10:22 am
Location: Philippines

Re: Database Writer Options..?!!??

Post by paradiselost » Mon Jan 11, 2016 11:50 pm

Enabling Options 1 and 2 will fix most folks in stable areas where airlines and call signs don't change dependent in the freshness of the information in Andrews lookup DB. There are cases where the information is older than the information we have added only to have our information overwritten by a lookup.

There has to be a way for the lookup to only write missing information excluding information already in our local database. I have the feeling the lookup database is an initial proof of concept. Andrew has too many irons in the fire to continually refresh the lookup database.

An option which would be a proposed Option 3 would limit the scope of Option 2 to only writing missing information to the database.

Turning off Option 1 denies us the benefits of option 2, sort of like throwing out the baby with the bath water.

paradiselost
Working Example of Version 2 Beta Virtual Radar Server http://dgteflyovers.ddns.net/virtualradar/
VRS 2 Help Files http://dgteflyovers.ddns.net:8080

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Database Writer Options..?!!??

Post by dsfh2992 » Wed Jan 13, 2016 6:05 pm

Is there a throttle on the amount of aircraft that can be "looked up" via the online database in a given period? I am tracking a few thousand aircraft, and notice that it takes maybe 30-60 minutes for all the make/model data on all the aircraft to get populated on the radar screen when VRS is restarted.

Thanks,
Dan

agw
Posts: 2249
Joined: Fri Feb 17, 2012 3:20 am

Re: Database Writer Options..?!!??

Post by agw » Sat Jan 16, 2016 2:49 pm

There's no throttle on the number of lookups fetched from the local cache of lookups. Lookups that have to go to the server are sent in batches, with a pause between each batch. The maximum size of the batch and the pause between sends is configured at the server, VRS fetches the configuration at startup and once every hour so things might change depending on server load, but at the moment the pause is 5 seconds and the maximum batch size is 1000.

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Database Writer Options..?!!??

Post by dsfh2992 » Mon Jan 18, 2016 5:45 pm

Does the local cache of lookups survive a restart?

In your example, if there are, say 3000 aircraft to be looked up, does it go something like this:

1. Aircrafts 1-1000 are looked up

2. Pause 5 seconds.

3. Aircrafts 1001-1999 are looked up

4. Pause 5 seconds

5. Aircrafts 2000 - 2999 are looked up.

or, does it do something like:

1. Lookup aircraft 1-1000, hit limit

2. Wait 5 seconds.

3. Lookup aircraft 1-1000, hit limit

4. Wait 5 seconds.

.......?

agw
Posts: 2249
Joined: Fri Feb 17, 2012 3:20 am

Re: Database Writer Options..?!!??

Post by agw » Thu Jan 21, 2016 11:08 pm

It takes the first batch of ICAOs off the queue, looks them up, writes the results to the local cache (either the standalone cache or BaseStation.sqb, depending on whether you have the database writer plugin installed and told it to write lookups there). Then it waits 5 seconds and repeats. If you start with 3000 ICAOs before the first lookup then it'll do 1-1000, pause, 1001-2000 and so on.

If the aircraft has been looked up within the last 24 hours (for aircraft that came back as having no results) or the last 28 days (for aircraft that did return results) then they won't go into the lookup queue.

If the server doesn't respond, or responds with an error status, then the ICAOs stay at the head of the queue and the pause between fetches extended (it'll reset back to 5 seconds once there's a successful fetch). If they can't be looked up within 30-ish minutes then they get flushed from the queue.

Also yes, the cache survives a restart - if you're not writing to BaseStation.sqb then it caches to a standalone SQLite file in your configuration folder.

dsfh2992
Posts: 156
Joined: Tue Oct 06, 2015 9:57 pm

Re: Database Writer Options..?!!??

Post by dsfh2992 » Fri Jan 22, 2016 4:44 pm

OK, thanks for the info. Something is up because it takes a LOOOONG time (like 30 minutes) for all the data on all planes to be refreshed. Are we sure the airport-data.com server are able to handle a query of 1000 simultaneous planes?

Also, why 1000 per 5 seconds? Isn't that way more than the vast majority of VRS servers even track at a time?

agw
Posts: 2249
Joined: Fri Feb 17, 2012 3:20 am

Re: Database Writer Options..?!!??

Post by agw » Sun Jan 24, 2016 9:26 pm

I think the looooooooong time is down to the problem with the SQLite stuff crashing when it's being asked to fetch thousands of records simultaneously. Once I've fixed that I expect your data will refresh a lot quicker. I'll have a preview version for that this week.

Airport-data.com aren't serving the aircraft data, it's coming from my servers using (mostly) PlaneBase's data.

1000 per 5 seconds was just pulled out of thin air, it's configurable server-side so if it needs tuning I can do it without having to re-release the program. That's the maximum batch size, if you only have one aircraft in the queue when the 5 seconds are up then it'll send a batch of one. It won't wait until it's got 1000 ICAOs to lookup.

Post Reply