Home PageFacebookRSS News Feed
PocketGPS
Web
Read the current newsletter! Weekly
Newsletter
SatNav,GPS,Navigation
Pocket GPS World - SatNavs | GPS | Speed Cameras: Forums

Pocket GPS World :: View topic - Programatically removing bad points from GPS tracks (.gpx)
 Forum FAQForum FAQ   SearchSearch   UsergroupsUsergroups   ProfileProfile   Log in for private messagesLog in for private messages   Log inLog in 

Programatically removing bad points from GPS tracks (.gpx)

 
Post new topic   Reply to topic    Pocket GPS World Forum Index -> Advanced GPS Lounge
View previous topic :: View next topic  
Author Message
FishSlice
Occasional Visitor


Joined: 23/02/2003 18:04:16
Posts: 44
Location: United Kingdom

PostPosted: Mon Sep 08, 2008 11:09 am    Post subject: Programatically removing bad points from GPS tracks (.gpx) Reply with quote

hey Guys,
I need to borrow some of the brainpower going spare in this forum Smile
I'm writing a free Skiing/boarding/mountain biking GPS translator for Google Earth. Basic premise is that it splits your entire GPX file/s into days and runs/lifts using custom algorithms to give you max speeds on runs down, tells you how long you were on lifts versus actually skiing, how far you skied, provides altitude profile graphs for each run and a daily summary etc etc. it all works well and outputs to KMZ file format to display in 3D etc.

Now i've made it public, lots of poeple are uploading GPX files I've not seen before with lots of very bad sat info. When doing the 3D trig on the GPX points, some of the data comes out to 200+ MPH !!! (and some worse), now looking at the data in Garmin Mapsource, its obvious that the erroneous data is down to bad locks etc, not proper data. I've tried implementing a normal distribution analysis on speeds between points and then ignoring anything too far out of range, but thats not quite working.

So, so the point of my post, does anyone know of a proven method to programatically remove BAD data from GPS log files, so I can provide users the useful correct data, ie their PROPER max speed for the day, rather than 1056MPH which is obviously wrong. I really don't want to hard code anytyhing which says ignore anything above 60mph etc as its not flexible enough.

Please help, I'm going mad Smile

Si.
http://www.chuffnuts.com/pistemaps
Back to top
View user's profile Send private message Send e-mail Visit poster's website
philpugh
Lifetime Member


Joined: Dec 28, 2005
Posts: 2003
Location: Antrobus, Cheshire

PostPosted: Mon Sep 08, 2008 12:06 pm    Post subject: Reply with quote

This is a perennial problem with handling tracklogs. The only suggestion I can make is to make calculations based on instantaneous speed changes. i.e Is it possible to get a change in speed of more than xmph (or y%) from one sample (= one second) to the next?

But the whole area is fraught with issues. There is always a small variation in position (and therefore speed) between samples - working in a very hilly area can make this worse as some of the satellites will be eclipsed by the hills/mountains - making for less accurate fixes and more 'wandering' leading to more significant errors at low (e.g. near walking) speeds.

Also, GPS speed calculated from positional change does not take into account slopes - so you can get under-calculation of true speed over the ground especially on steep slopes. The extreme example is that the top of a cliff and the bottom of a cliff will have the same GPS co-ordinates - but the speed of descent would not be zero, as would be indicated on the device. Laughing
_________________
Phil
Back to top
View user's profile Send private message
FishSlice
Occasional Visitor


Joined: 23/02/2003 18:04:16
Posts: 44
Location: United Kingdom

PostPosted: Mon Sep 08, 2008 12:29 pm    Post subject: Reply with quote

Thanks for the reply. With regard to your first suggestion, yes I'm currently looking at monitoring instantaneous acceleration from one point to the next and perhaps using that as an indication, bearing in mind gravity is 9.8m/s/s I should be able to get some idea on that one.

the only data I'm taking from the GPX file is lat, lon, alt and time, the rest I'm calculating using some 3D trig, this allows for calculating speed on a slope correctly rather than using a basic flat earth model, so its ok on that front.

Cheers for the suggestions, will see how I get on.

Si.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
philpugh
Lifetime Member


Joined: Dec 28, 2005
Posts: 2003
Location: Antrobus, Cheshire

PostPosted: Mon Sep 08, 2008 5:06 pm    Post subject: Reply with quote

But the altitude you get from GPS based calculations vary even more than positional fixes do. I get variations of +/- 50 metres or more at my home location over a short (10 min) period (putting me below sea level some of the time). Just take a log of altitude at your home location - put a unit in clear sky and see what results you get and compare to your actual altitude. You will get best results with reception of satellites near the horizon as well as above. The problem in the mountains is that you just don't get visibility of the lower elevation sats.

{Edit to add} Don't forget that the altitude reported by the GPS is above the theoretical ellipsoid approximation to the earths surface - so the maths gets additionally tricky. You can get away with a 'flat' baseline for very short distances - but you start to get noticable errors with tracks over a mile long. {End Edit}

I would never rely on instantaneous GPS altitude for anything serious. Averaging over a long period (1 hour +) will improve accuracy dramatically. Even atmospheric barometers in some GPS units (like my Colorado 300) whilst more accurate short term, are still prone to variations.
_________________
Phil
Back to top
View user's profile Send private message
robertn
Frequent Visitor


Joined: Feb 06, 2005
Posts: 564

PostPosted: Thu Sep 11, 2008 2:29 am    Post subject: Reply with quote

I would start by keeping it simple, do some science to understand where the errors are coming from, you will quickly dump the 3D and elipsoid calculations - the errors due to a flat earth model and sloping terrain is orders of magnitude less than all other errors individually, let alone when they are combined.

If you really want altitude - use Google earth to get the altitude at a lat/long. It will be more accurate. If it is not good enough for you needs, then you do not have an accurate enough altitude. The speed error caused by a relitively steep load is unlikly to be significant anyway, as very rarely do you go straight down a very steep slope, with no turns (Think skiing - when was the last time you pointed your ski's down the fall line of a 45 degree slope and let it rip,- and if so, how big was the slope?) - and if you are on a bike, and care, you will be measuring speed off wheel rotations (front wheel is good - if it stops you crashed so the max for the day don't count Very Happy )

GPS receivers reports speeds by a rolling average. With SiRF you can even program the "Mode" - walking, car, biking, aviation etc. This determines the amount of rolling is used in the averaging. (Planes don't go from 200mph to 0 in a few seconds, and if they do, no one looks at the GPS logs, as the GPS is now made up of lots of tiny bits, Cars take a few seconds and skiers and walkers can stop and change direction instantly). If they need to do this, then almost certainly you will too.

At best you program is going to providing interesting, but not particularly accurate data. When I am skiing, I can change direction several times a second. The GPS reports a reading once a second, so what happend to the positions at the extreme edges of my turns - also with the rolling avergae and positional prediction they use, the reported position may even be an over shoot.

Hope I didn't dampen your enthusiasim, it's a great idea.
Back to top
View user's profile Send private message
robertn
Frequent Visitor


Joined: Feb 06, 2005
Posts: 564

PostPosted: Thu Sep 11, 2008 2:36 am    Post subject: Reply with quote

I forgot to explain position prediction - most GPS receievers use current direction and speed of travel as in input into the postion calcualtion - Therefore positions reported by skiing, with tight, fast turns, can overshoot. The GPS only reports a postion every second, but a skiier can throw a couple of turns in that time, so even more error can be introduced.

Some work has been done on this, and in the end, it requires custom GPS hardware that uses doppler shift and such like, so rather than the focus if "where am I", it becomes "How fast in what direction"
Back to top
View user's profile Send private message
FishSlice
Occasional Visitor


Joined: 23/02/2003 18:04:16
Posts: 44
Location: United Kingdom

PostPosted: Thu Sep 11, 2008 9:58 am    Post subject: Reply with quote

Thanks for the info Robert, I kinda knew most of it, but its nice to read it all together and with the rationale Smile

What I'm producing is more of a debrief tool for google earth after a days skiing. It was born out of my wife asking me "where have you skied today" at the end of a day with the guide Smile I started adding all the stats and graphs when I saw what data was captured.

The main stats I was after was time spent on decents versus acents (runs versus lifts) which it does pretty well now, as well as total distances, the speeds thing is kinda interesting, but not as much as seeing the totals and seeing the runs in GE split into runs/lifts.

I've managed to use the ideas from Phil and implemented some basic rules based on instantaneous acceleration, that seems to do the job nicely and has removed the majority of "spikes" due to bad locks etc when you assume a max of around 6m/s/s.

I've not put the new code live yet as I still need to work out graph axis based on time properly, but its on the Development version of the site if you're interested.

Main site (old code) - http://www.chuffnuts.com/pistemaps
New Development Code - http://www.chuffnuts.com/pistemaps/index_dev.php

Thanks for all the pointers chaps, I'll stop looking for perfection (given the errors we're talking about) and keep it simple Smile ).

Cheers
Si.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
philpugh
Lifetime Member


Joined: Dec 28, 2005
Posts: 2003
Location: Antrobus, Cheshire

PostPosted: Fri Sep 12, 2008 6:06 pm    Post subject: Reply with quote

robertn wrote:

If you really want altitude - use Google earth to get the altitude at a lat/long. It will be more accurate. If it is not good enough for you needs, then you do not have an accurate enough altitude.


I don't know how accurate the Google (or similar) spot altitudes are but if they use something like the NASA derived SRTM 0.3 arc-second data sets then they are going to be well out.

Even the trusted OSGB maps are suprisingly "inaccurate". Individual contor lines are +/- 5.4metres and spot heights (non-trig points) are +/- 3.4metres. (Taken from information published by the OSGB).

The most accurate way of getting height is to use a decent calibrated aneroid barometer as are found in some commercial units - if set correctly at the start of a day then they will be reasonably accurate for a couple of hours or so but will tend to drift due to changes in temp and weather systems.
_________________
Phil
Back to top
View user's profile Send private message







Posted: Today    Post subject: Pocket GPS Advertising

Back to top
Display posts from previous:   
Post new topic   Reply to topic    Pocket GPS World Forum Index -> Advanced GPS Lounge All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB 2.0.11 © 2001 phpBB Group
phpBB port v2.1 based on Tom Nitzschner's phpbb2.0.6 upgraded to phpBB 2.0.4 standalone was developed and tested by:
ArtificialIntel, ChatServ, mikem,
sixonetonoffun and Paul Laudanski (aka Zhen-Xjell).

Version 2.1 by Nuke Cops 2003 http://www.nukecops.com

Make a Donation



CamerAlert Database

Click here for the PocketGPSWorld.com Speed Camera Database

Download Speed Camera Database
19.121 (01 Dec 21)



WORLDWIDE SPEED CAMERA SPOTTERS WANTED!

Click here to submit camera positions to the PocketGPSWorld.com Speed Camera Database


12mth Subscriber memberships awarded every week for verified new camera reports!

Submit Speed Camera Locations Now


CamerAlert Apps



iOS QR Code






Android QR Code







Terms & Privacy


GPS Shopping