Home PageFacebookRSS News Feed
PocketGPS
Web
SatNav,GPS,Navigation
Pocket GPS World - SatNavs | GPS | Speed Cameras: Forums

Pocket GPS World :: View topic - Geocoded PostCode Database Leaked Online
 Forum FAQForum FAQ   SearchSearch   UsergroupsUsergroups   ProfileProfile   Log in for private messagesLog in for private messages   Log inLog in 

Geocoded PostCode Database Leaked Online
Goto page Previous  1, 2, 3, 4  Next
 
Post new topic   Reply to topic    Pocket GPS World Forum Index -> News And Latest Information
View previous topic :: View next topic  
Author Message
NickG
Frequent Visitor


Joined: Nov 09, 2003
Posts: 357
Location: UK

PostPosted: Thu Sep 17, 2009 10:38 am    Post subject: Reply with quote

I'm not sure this is the real Royal Mail database. It seems to be incomplete, and some (but not all) postcodes seem to be rounded to the nearest 100m grid (common in cheaper datasets used to implement "find my nearest store" type features). Perhaps it's been built by 'scraping' data from websites.

I've compared it to the latest real database (received yesterday by coincidence) that powers my venuefinder.com website and there are lots of differences.

Oddly, the real database seems to contain fewer postcodes - I have 1,649,713 rows in mine. But I know for example, I do not have PO Box postcodes which do not have coordinates anyway.

To the people talking about making this into a POI file, do you realise the file is larger than the amount of RAM in any current SatNav device? You cannot do this. It can only be used if it's compressed into a properly indexed database. It's no use to you as a POI text file.
Back to top
View user's profile Send private message Visit poster's website
Jellyroll
Lifetime Member


Joined: May 17, 2006
Posts: 54

PostPosted: Thu Sep 17, 2009 11:14 am    Post subject: Reply with quote

No problems importing into a database (of course), but 1.8 million records will overflow most spreadsheets.

If anyone wants, I can try to get it into something generic (dBase II, perhaps) that can be queried using derived (e.g. first part of Post Code, alpha portion of part of Post Code, alpha portion of OS GridRef) and existing (e.g. indexed fields such as PZ_County [37 codes incl. blank], PZ_District [220 codes incl. blank], PZ_NHS_Code [37 codes incl. blank] and _Region [11 codes incl. blank], PZ_Country [England, Northern Ireland, Scotland, Wales, Isle of Man, Channel Islands, but many blanks too]) fields. That would allow people who want to use spreadsheets to get the data via queries, and allow those using database applications other than M$ Access to get at the data without difficulty.

If I do that, can anyone (including the PGPSW site itself, subject to any legal restrictions) host the resulting file(s)?

Some fields probably contain errors. There is a St Neots post code, "PE19 6WF", that has the same PZ_County code as Stevenage, Milton Keynes and Luton post codes - it was still in Cambridgeshire last time I looked - and over half of the entries have a county code that is blank or "00".

As for whether this is of use for POI generation, I'd say that it might help those whose devices don't include post code navigation, generating a subset POI for local post codes only, for example.
Back to top
View user's profile Send private message
pcaouolte
Frequent Visitor


Joined: Dec 27, 2006
Posts: 998
Location: South Lincs, UK.

PostPosted: Thu Sep 17, 2009 12:49 pm    Post subject: Reply with quote

NickG wrote:

To the people talking about making this into a POI file, do you realise the file is larger than the amount of RAM in any current SatNav device? You cannot do this. It can only be used if it's compressed into a properly indexed database. It's no use to you as a POI text file.


If we can extract the PostCode, Lat and Long fields, then delete the rows with no Lat or Long data, then convert the result to an ov2 file it will be much smaller than the current text file. It may need splitting up, perhaps by county or by the first letter of the postcode. I am going to try when I get the time. I have a satnav with no postcode search facility and like to play with little projects like this.
_________________
Paul
Back to top
View user's profile Send private message
NickG
Frequent Visitor


Joined: Nov 09, 2003
Posts: 357
Location: UK

PostPosted: Thu Sep 17, 2009 2:45 pm    Post subject: Reply with quote

A TomTom (for example) can only handle a couple of thousand rows in a file before it becomes dog-slow. This database has over 1.5 MILLION rows. There is nothing useful you could do with that. It won't even be able to load it as a TomTom only has about 4MB of free memory when it's running, so you couldn't do any searches on it. I don't really see how it would be useful unless it was made into an indexed database you could search on. You can't search the raw text file with no index on it (even my 4GB Intel Quad core PC takes a few seconds do that!) and that has disk caching - which a sat-nav does not. Still, I'd be interested to know if it's at all usable, just out of curiosity.
Back to top
View user's profile Send private message Visit poster's website
pcaouolte
Frequent Visitor


Joined: Dec 27, 2006
Posts: 998
Location: South Lincs, UK.

PostPosted: Thu Sep 17, 2009 7:27 pm    Post subject: Reply with quote

NickG wrote:
I'd be interested to know if it's at all usable, just out of curiosity.

So would I! I'll let you know if I get it working. Very Happy
_________________
Paul
Back to top
View user's profile Send private message
navver
Regular Visitor


Joined: Dec 12, 2005
Posts: 111
Location: Chipping Sodbury

PostPosted: Thu Sep 17, 2009 8:51 pm    Post subject: Reply with quote

my old navman POI database I bought on ebay for about 2 quid was split something like
southwest 1 southwest 2
wales1 wales2
midlands1 midlands2 midlands3
london 1 london 2 london 3
anglia 1 anglia 2
North West 1 North west 2
Scotland 1 Scotland 2 Scotland 3
etc

They were .txt or .asc files (both the same) of about 1 or 2 or 3 MB each. Navman compiled these and downloaded them to the device, each taking about an hour or more. I manged to get about 40% of the country on the card but Navman had a limit of about 24 POI files in total, but I wanted to keep space for speed zoned camera files.

The system did work well for a device that only had 5 digit postcodes built in.
_________________
Tomtom Go520, App 8.010, Map UK&ROI 810.1870
Back to top
View user's profile Send private message
navver
Regular Visitor


Joined: Dec 12, 2005
Posts: 111
Location: Chipping Sodbury

PostPosted: Thu Sep 17, 2009 8:56 pm    Post subject: Reply with quote

You can still buy the post code POIs on Ebay for 3.49

http://tiny.cc/eBayLink99


edited by MaFt to shorten link
_________________
Tomtom Go520, App 8.010, Map UK&ROI 810.1870
Back to top
View user's profile Send private message
Oldboy
Pocket GPS Moderator
Pocket GPS Moderator


Joined: Dec 08, 2004
Posts: 10640
Location: Suffolk, UK

PostPosted: Fri Sep 18, 2009 6:48 am    Post subject: Reply with quote

I don't know why users would want to pay for them. They're still available (free) in 12 files, alphabetically listed, and take up a total of 66MB as CSV Files.

In this, and probably the E-Bay case, they are a little out of date now.
_________________
Richard

TT 910 V7.903: Europe Map v1045
TT Via 135 App 12.075: Western Europe Map v1110
Back to top
View user's profile Send private message Visit poster's website
Eddiejen
Occasional Visitor


Joined: Jul 02, 2006
Posts: 33

PostPosted: Fri Sep 18, 2009 7:52 am    Post subject: Reply with quote

If you want to play around with the file try CSVed it loaded the file without any problems and allows you to split and manipulate the file easier
http://csved.sjfrancke.nl/index.html
Back to top
View user's profile Send private message
NickG
Frequent Visitor


Joined: Nov 09, 2003
Posts: 357
Location: UK

PostPosted: Fri Sep 18, 2009 9:59 am    Post subject: Reply with quote

Oldboy wrote:
I don't know why users would want to pay for them. They're still available (free) in 12 files


No you're totally wrong. You always have to pay for postcode data and there are no free copies available. If you don't believe me, ring Royal Mail data licensing. If you think it's "free" then it's probably an illegal copy, in which case it's no more free to you than the car sitting on my drive. Yes you could steal it, but it doesn't mean it's free!

I wish the moderators of this site wouldn't so freely condone software theft when many of the site users are actually software developers ourselves, and we have to listen to others (including moderators) how to best make use of our stolen software/data! I pay around 3000 per year for the same data half the people on this thread have now pirated (that's what it costs to license this dataset for use by a website).

If I came and stole something from you, then started bragging about it on these forums, I think you'd get pretty angry.
Back to top
View user's profile Send private message Visit poster's website
Darren
Pocket GPS Staff
Pocket GPS Staff


Joined: 11/07/2002 14:36:40
Posts: 23848
Location: Hampshire, UK

PostPosted: Fri Sep 18, 2009 10:23 am    Post subject: Reply with quote

Jeepers Nick, before you click send you should think about what you are posting!

Oldboy was referring to PostCode POIs, not the Royal Mail PAF database.

With regard to this thread, you yourself said you didn't think it was the RM database that Wikileaks have posted, and the Guardian, Times, The Register and many others have posted the story so if you think our post makes any difference I'm flattered but you're wrong.

Without the names and addresses it's of little use to anyone who subscribes to the PAF anyway. I strongly object to your accusation that we condone software theft, we do not and have always taken a strong line on that issue but in this case there is no suggestion that this has been stolen from anyone.
_________________
Darren Griffin - Editor
Back to top
View user's profile Send private message Send e-mail Visit poster's website
NickG
Frequent Visitor


Joined: Nov 09, 2003
Posts: 357
Location: UK

PostPosted: Fri Sep 18, 2009 10:33 am    Post subject: Reply with quote

I know what he was talking about... a 7 digit postcode database is not free!

The 3000 dataset I was talking about is also not PAF data - its purely a list of 1.7M geocoded postcodes (or postcode POIs as you call them), it's in an almost identical format to the data you've linked to from this site (on wikileaks).

If postcode coordinates were free, we wouldn't have sites like http://www.freethepostcode.org/

If you don't think it's stolen and illegal, why have you described it as a "leak" then?
Back to top
View user's profile Send private message Visit poster's website
Darren
Pocket GPS Staff
Pocket GPS Staff


Joined: 11/07/2002 14:36:40
Posts: 23848
Location: Hampshire, UK

PostPosted: Fri Sep 18, 2009 10:36 am    Post subject: Reply with quote

NickG wrote:
If you don't think it's stolen and illegal, why have you described it as a "leak" then?

Since when did 'leak' indicate stolen or illegal subject matter?
_________________
Darren Griffin - Editor
Back to top
View user's profile Send private message Send e-mail Visit poster's website
NickG
Frequent Visitor


Joined: Nov 09, 2003
Posts: 357
Location: UK

PostPosted: Fri Sep 18, 2009 10:41 am    Post subject: Reply with quote

You wouldn't ever use the word "leak" to describe something you can legitimately download from the original owner. You'd call it a "release" wouldn't you?
Back to top
View user's profile Send private message Visit poster's website
Darren
Pocket GPS Staff
Pocket GPS Staff


Joined: 11/07/2002 14:36:40
Posts: 23848
Location: Hampshire, UK

PostPosted: Fri Sep 18, 2009 10:43 am    Post subject: Reply with quote

True but that doesn't mean a leaked file is stolen or illegal. Whatever, the data source is not known and this story has been widely reported across the tech press.
_________________
Darren Griffin - Editor
Back to top
View user's profile Send private message Send e-mail Visit poster's website







Posted: Today    Post subject: Pocket GPS Advertising

Back to top
Display posts from previous:   
Post new topic   Reply to topic    Pocket GPS World Forum Index -> News And Latest Information All times are GMT + 1 Hour
Goto page Previous  1, 2, 3, 4  Next
Page 2 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Make a Donation



CamerAlert Database

Click here for the PocketGPSWorld.com Speed Camera Database

Download Speed Camera Database
21.093 (20 Sep 23)



WORLDWIDE SPEED CAMERA SPOTTERS WANTED!

Click here to submit camera positions to the PocketGPSWorld.com Speed Camera Database


12mth Subscriber memberships awarded every week for verified new camera reports!

Submit Speed Camera Locations Now


CamerAlert Apps



iOS QR Code






Android QR Code







Terms & Privacy


GPS Shopping