Home PageFacebookRSS News Feed
PocketGPS
Web
Read the current newsletter! Weekly
Newsletter
SatNav,GPS,Navigation
Get Speed Camera Warnings For SatNavs Get Speed Camera Warnings For Android Get Speed Camera Warnings For iPhone
Pocket GPS World - SatNavs | GPS | Speed Cameras: Forums

Pocket GPS World :: View topic - Camper 770LMT-D: Diacritical mark POI problem
 Forum FAQForum FAQ   SearchSearch   UsergroupsUsergroups   ProfileProfile   Log in for private messagesLog in for private messages   Log inLog in 

Camper 770LMT-D: Diacritical mark POI problem
Goto page Previous  1, 2, 3  Next
 
Post new topic   Reply to topic    Pocket GPS World Forum Index -> Garmin Portable Navigation Devices
View previous topic :: View next topic  
Author Message
M8TJT
The Other Tired Old Man
The Other Tired Old Man


Joined: Apr 04, 2006
Posts: 9168
Location: Bexhill, East Sussex, UK

PostPosted: Tue Jan 02, 2018 4:08 pm    Post subject: Reply with quote

Privateer wrote:
However the Garmin search routine does not appear to be language-neutral so using the English keyboard you can't type "cafe" to find "Café", Which detracts from trying to force diacritical marks into the Garmin.
Does this fact not make the whole exercise pointless for Garmins then?
Back to top
View user's profile Send private message
sussamb
Lifetime Member


Joined: Mar 18, 2011
Posts: 3763
Location: West Sussex

PostPosted: Tue Jan 02, 2018 6:03 pm    Post subject: Reply with quote

I'm often in France and have had no issues searching for French addresses etc. Never tried cafe though so will do so later Smile

Edit: Just checked and searching for cafe in France works fine ;)
_________________
Where there's a will ... there's a way.
Back to top
View user's profile Send private message
Privateer
Pocket GPS Moderator
Pocket GPS Moderator


Joined: 30/12/2002 17:36:20
Posts: 4480
Location: Oxfordshire, England, UK

PostPosted: Wed Jan 03, 2018 3:23 pm    Post subject: Reply with quote

M8TJT wrote:
Does this fact not make the whole exercise pointless for Garmins then?

Yes you're right, which is why I'd like Garmin to change it so that diacritical marks can be used in custom POIs and that you can search for strings regardless of letters having diacritical marks.

sussamb wrote:
I'm often in France and have had no issues searching for French addresses etc. Never tried cafe though so will do so later Smile

Edit: Just checked and searching for cafe in France works fine ;)

Now that's very interesting. I've just tried on my Garmin 770 and searched for the attraction Hôtel del la Monnaie in Paris with the search string on "Hot" i.e. no diacritical marks and the hotel came up.

So, it would appear that built in POIs not only have diacritical marks but you can search for POIs without using diacritical marks.

I have retried my test favourite and have found that whilst you can add diacritical marks in a favourite, it seems that you can only search using the exact string i.e. use diacritical marks for the search if there are diacritical marks in the favourite name.

Now the question this poses is: Can diacritical marks be used in custom POIs like they are used in built-in POIs?

Looks like I need to re-read PhilHornby's post on using .GPX to see if I can get a test custom POI to a) show diacritical marks and b) search for text with/without using diacritical marks.

I've had a look at the Garmin built-in POIs for Jersey (Channel Islands) and they don't often use diacritical marks. Here are a few examples of my Jersey custom POIS that do use diacritical marks:

49.168060,-2.081460 Samarès Coast Hotel (Morvan) car park
49.181380,-2.241170 Corbière Lighthouse car park
49.173480,-2.078990 Samarès Manor
49.255740,-2.226860 Plémont Beach car park (top)
49.246220,-2.199640 Grève de Lecq Barracks
49.235990,-2.198900 Judith Quérée's Garden
49.180446,-2.230703 La Rosière Desalination Plant
49.251346,-2.226038 Café Ouen
49.246513,-2.199016 Le Câtel Fort
49.220129,-2.145919 Le Rât Cottage

Regards,
_________________
Robert.
iPhone 6s Plus, iOS 10.3.3: iOS CamerAlert v2.0.7
Garmin Camper 770 LMT-D - because TomTom GO Mobile iOS is so awful
TomTom (UK & ROI and Europe) iOS apps v1.29; TomTom GO Mobile iOS 1.9.2
Back to top
View user's profile Send private message Send e-mail
PhilHornby
Frequent Visitor


Joined: Dec 07, 2006
Posts: 511
Location: Stockport

PostPosted: Wed Jan 03, 2018 6:10 pm    Post subject: Experiments... Reply with quote

I have done some more experiments, using POILoader 2.7.3.0 on Windows 10. Test unit was a BMW-branded Nüvi 715.

This is what I believe to be to state of play, with regards to character sets available.

  1. As previously stated, UTF-8 encoded CSV files are not accepted as input by POILoader.
  2. GPX files that contain an "encoding=UTF-8" statement, must start with the ByteOrderMark /(BOM)/ 0xEF,0xBB,0xBF sequence to be accepted by POILoader.
  3. GPX files read directly by the unit (i.e. as Favourites), must contain the "encoding=UTF-8" statement. The BOM is not necessary in this case!
  4. Neither the unit, nor the .GPI file format support UTF-8!

    Data in the .GPI file is stored as 8-bit ANSI (aka Windows-1252) data. If POILoader reads a UTF-8 encoded file containing a character that cannot be mapped into ANSI, it is replaced by a "?". If the unit itself reads such a GPX file, it silently ignores the entry.

    So, if for example, a (properly) UTF-8 encoded .GPX file contains the sequence 0xE2,0x82,0xAC it is mapped to (and stored as) 0x80 in the .GPI file. (This is a Euro sign and exists in both character sets.)

    But the sequence 0xE2,0x84,0xA6 will be mapped to a "?" (0x3F0), as it is not present in the ANSI character set. (This is an Ohm symbol )


As an aside - what character do you see on screen, in your POI generating software (Excel?), for the binary character 0x80? Is it the Euro sign from the ANSI character set, or the Ä from the MacRoman set?


Back to top
View user's profile Send private message
Privateer
Pocket GPS Moderator
Pocket GPS Moderator


Joined: 30/12/2002 17:36:20
Posts: 4480
Location: Oxfordshire, England, UK

PostPosted: Wed Jan 03, 2018 8:34 pm    Post subject: Re: Experiments... Reply with quote

PhilHornby wrote:
As an aside - what character do you see on screen, in your POI generating software (Excel?), for the binary character 0x80? Is it the Euro sign from the ANSI character set, or the Ä from the MacRoman set?

Macs are frustratingly not like Windows, the Windows Character Map has no equivalent on a Mac, there are expensive apps that allegedly do a similar job but I've been had before so I don't really trust utility apps for the Mac. Rolling Eyes

Can I just confirm what you mean by "0x80", does the "x" represent a character A to F, or does it represent something else?

Sorry, it's been years since I've done any HEX and I just can't find the how to enter the "0x80" in the Mac or even how to Google to find the answer.

I'm using a Unicode Hex input virtual keyboard on the Mac and when I got to enter a character code in Mac Excel, I have the options of 0-9 and A-F, but no "x". Using A-F in 0x80 gives rubbish all the way through with neither nor Ä.

Regards,
_________________
Robert.
iPhone 6s Plus, iOS 10.3.3: iOS CamerAlert v2.0.7
Garmin Camper 770 LMT-D - because TomTom GO Mobile iOS is so awful
TomTom (UK & ROI and Europe) iOS apps v1.29; TomTom GO Mobile iOS 1.9.2
Back to top
View user's profile Send private message Send e-mail
PhilHornby
Frequent Visitor


Joined: Dec 07, 2006
Posts: 511
Location: Stockport

PostPosted: Thu Jan 04, 2018 2:39 am    Post subject: Re: Experiments... Reply with quote

Privateer wrote:
Can I just confirm what you mean by "0x80", does the "x" represent a character A to F, or does it represent something else?


By 0x80, I mean 80hex i.e. the equivalent of 128 decimal. It's used because of the immense difficulty of adding a subscript of ₍₁₆₎ ... 80₍₁₆₎ Wink

What I'm trying to ascertain, is if you "see" the same interpretation of the data in your 'POI Editing' software, that the Nüvi does. (i.e. the 8 bit ANSI character set).

Then we need to see what is being stored in the file, when you hit save. For example, if you have a "€" Euro sign on screen and then store it in a file, does it write the UTF-8 3 byte sequence 0xE2,0x82,0xAC, or does it write a single 0x80 byte?

If it does the former - and you can't stop it - then you can't use a CSV file as input to POILoader - it doesn't understand UTF-8 CSV files. You would have to pass it via a GPX file, with the "encoding" denoted as "UTF-8" and (probably) with the BOM present.

(This would be a travesty, as the next thing POIloader would do is convert it back to ANSI (if possible), for inclusion in the .GPI file.)

Do you see where I'm coming from? Rolling Eyes

I've uploaded a CSV file to Dropbox, containing most of the 'special' characters in the 8 bit ANSI character set. (It was supposed to be all of them, but I went a bit cross-eyed, half way through!).

Here is the file: https://www.dropbox.com/s/d2mnf0e18sw3t8x/HighBit.csv?dl=0

If you transfer it to the Nüvi via POILoader, it should look something like this:


These are examples of how two Windows components handle it. Notepad gets it right, the 'DOS' Command Processor doesn't.

Back to top
View user's profile Send private message
Privateer
Pocket GPS Moderator
Pocket GPS Moderator


Joined: 30/12/2002 17:36:20
Posts: 4480
Location: Oxfordshire, England, UK

PostPosted: Fri Jan 05, 2018 7:36 pm    Post subject: Reply with quote

Hi Phil,

I don't know how to insert a hex character 0x80 straight into Excel.

I use Excel worksheets (.xlsx) as my master file with a lot of columns:

ID; Latitude; Longitude; Name; Region; Categories; Phone; Created; Updated; Confirmed; Device; Name TomTom; Name Garmin.

Most columns are self-explanatory, the master list is a reference and basic change control:
  • ID is a simple Excel function "=ROW()-1" to give each POI a reference number for quick cross-checking.
  • Region, is not used at the moment but for my Jersey POIs it holds the name of the parish.
  • Categories: none; one; or more categories may be listed. They are a few characters such as Att (attraction); Food; JHT (Jersey Heritage Trust); etc to allow quick search for a type of POI in a single POI file.
  • Device: over the years I've used different SatNavs to capture POIs, more recently I use an app on my iPhone or even Google Earth.
  • Name TomTom: This contains a formula take the name, categories, ID, and telephone number and turn into a single string for use of TomTom.
  • Name Garmin: Similar to Name TomTom but doesn't use phone as Garmin handles that differently.

I manually take the Longitude, Latitude, Name (TomTom/Garmin), phone and create a .csv file which is Western (Mac OS Roman) encoded and has Windows (CRLF) line breaks. I can't remember whether BOM is used or not. The .csv is then passed through the Mac POI Loader.

I've downloaded your file and I will pass it through the the Mac POI Loader and then load it onto my Garmin Camper 770 LMT-D to see what the result is.

Regards,
_________________
Robert.
iPhone 6s Plus, iOS 10.3.3: iOS CamerAlert v2.0.7
Garmin Camper 770 LMT-D - because TomTom GO Mobile iOS is so awful
TomTom (UK & ROI and Europe) iOS apps v1.29; TomTom GO Mobile iOS 1.9.2
Back to top
View user's profile Send private message Send e-mail
Oldboy
Pocket GPS Moderator
Pocket GPS Moderator


Joined: Dec 08, 2004
Posts: 10566
Location: Suffolk, UK

PostPosted: Fri Jan 05, 2018 9:57 pm    Post subject: Reply with quote

Privateer wrote:
I don't know how to insert a hex character 0x80 straight into Excel.
On a PC you would press, and hold, the Alt key while typing 080 on the numeric keypad.

It might be different on a Mac. Confused
_________________
Richard

TT 910 V7.903 (9183/081205) OS:2344
Europe version 1005.8767
Back to top
View user's profile Send private message Visit poster's website
PhilHornby
Frequent Visitor


Joined: Dec 07, 2006
Posts: 511
Location: Stockport

PostPosted: Sat Jan 06, 2018 12:43 am    Post subject: Reply with quote

Privateer wrote:
I don't know how to insert a hex character 0x80 straight into Excel.


You can use the "=CHAR" function...

i.e. =CHAR(HEX2DEC(80)) or just =CHAR(128)

If I do that (on Windows 10, using ancient Excel 2007), a euro sign ( ) immediately appears in the cell. In other words, in the absence of any other instruction, Windows treats the 8 bit data as a "Windows-1252" character. My guess is, that on the Mac, you will get a Ä instead (this being 0x80 in the Mac Roman set). Various web sites are critical of Microsoft for sticking with the proprietary Apple stuff, when other software hasn't Rolling Eyes

If I save that file as a default "CSV", the file contains a "0x80" byte i.e. unchanged.

However, if I save that file as "CSV (Macintosh)", Excel writes the Mac Roman code for Euro sign to the file, which is"0xDB".

If I look at this "Macintosh" file in Notepad, or send it to a Nüvi, I now get a Û character instead of a - because neither is expecting Mac Roman data - and has no way to detect it.



I found a cryptic reference to
Code:
the trial version of Apple Numbers from the Apple website
Apparently, this can
Code:
 solve problematic issues encountered on Excel sometimes.


The suggestion being that you can import the CSV file and export in a different flavour.
Back to top
View user's profile Send private message
Privateer
Pocket GPS Moderator
Pocket GPS Moderator


Joined: 30/12/2002 17:36:20
Posts: 4480
Location: Oxfordshire, England, UK

PostPosted: Sun Jan 07, 2018 8:37 pm    Post subject: Reply with quote

Oldboy wrote:
On a PC you would press, and hold, the Alt key while typing 080 on the numeric keypad.

It might be different on a Mac. Confused

Thanks for that info. Yes, that sounds right as I’ve not used Windows for over a year now. You’re right, it is (very) different on a Mac.

PhilHornby wrote:
You can use the "=CHAR" function...

i.e. =CHAR(HEX2DEC(80)) or just =CHAR(128)

Thanks, that’s very useful to know. In Mac Excel 2016, I got the following results:

=CHAR(HEX2DEC(80)): Ä
=CHAR(128): Ä

This is all in a standard Mac Excel .xlsx workbook. Here are all of the .csv file types that you can save from Mac Excel. I have saved each .csv type and examined them with TextWrangler to show the details (i.e. encoding, line feed, and what the "Ä" is displayed as:

  1. CSV UTF-8 (comma delimited) (.csv) --- Unicode (UTF-8, with BOM), Legacy MAC OS (CR) "Ä"
  2. Comma Separated Values (.csv) --- Western (Mac OS Roman), Windows (CRLF) "Ä"
  3. Windows Comma Separated (.csv) --- Western (Mac OS Roman), Windows (CRLF) "ƒ"
  4. MS-DOS Comma Separated (.csv) --- Western (Mac OS Roman), Windows (CRLF) "é"

https://www.dropbox.com/s/d2mnf0e18sw3t8x/HighBit.csv?dl=0

I took the above file and processed it via Mac POI Loader and then loaded it onto my Garmin Camper 770 LMT-D. Here’s the result:



Regards,
_________________
Robert.
iPhone 6s Plus, iOS 10.3.3: iOS CamerAlert v2.0.7
Garmin Camper 770 LMT-D - because TomTom GO Mobile iOS is so awful
TomTom (UK & ROI and Europe) iOS apps v1.29; TomTom GO Mobile iOS 1.9.2
Back to top
View user's profile Send private message Send e-mail
PhilHornby
Frequent Visitor


Joined: Dec 07, 2006
Posts: 511
Location: Stockport

PostPosted: Sun Jan 07, 2018 10:03 pm    Post subject: Reply with quote

Oldboy wrote:
On a PC you would press, and hold, the Alt key while typing 080 on the numeric keypad.

Actually ... it's Alt 0128, if we're being strictly accurate..Wink

Privateer wrote:
In Mac Excel 2016, I got the following results:

=CHAR(HEX2DEC(80)): Ä
=CHAR(128): Ä


So here's the first issue. By default, MAC Excel is working in a different character set encoding to the Nüvi; it's using Mac Roman. You've got to stop it doing that (somehow), or use a different 'editor' that you can control.

Code:

CSV UTF-8 (comma delimited) (.csv) --- Unicode (UTF-8, with BOM), Legacy MAC OS (CR) "Ä"


Probably expected behaviour - but academic as POI Loader doesn't support CSV files encoded with a BOM.
Code:
Windows Comma Separated (.csv) --- Western (Mac OS Roman), Windows (CRLF) "ƒ]"

If it had left the value alone, it would be a start - it seems to have remapped it to something else. Exactly what and why, I can't figure out!

Privateer wrote:

I took the above file and processed it via Mac POI Loader and then loaded it onto my Garmin Camper 770 LMT-D. Here’s the result:

image link seems dead


This is more worrying...
it seems quite well documented that Mac Excel has its foibles, but Mac POI Loader seems to be getting in on the act too. It's almost as though it's expecting CSV files to be encoded using Mac Roman and has (sort of) translated their contents to Windows-1252. (Except it's mapped accented versions of the characters (as defined in Mac Roman), to their un-accented versions (as defined in Windows-1252).

aaaaaarrrrrrrrggggggghhhhhh Shocked

Since you may have to switch to GPX, to make this work, here are some more files for you to test with:

https://www.dropbox.com/s/dninznspanxllrb/Encodings.zip?dl=0

This contains a pre-made GPI file (so you can compare Mac and Windows POILoader output).

ANSI.GPX is encoded using Windows-1252 and UTF-eight.GPX uses UTF-8 (and a BOM).

On the Nüvi 715, they display like this:

ANSI


UTF-8


Last edited by PhilHornby on Mon Jan 08, 2018 4:05 pm; edited 2 times in total
Back to top
View user's profile Send private message
Privateer
Pocket GPS Moderator
Pocket GPS Moderator


Joined: 30/12/2002 17:36:20
Posts: 4480
Location: Oxfordshire, England, UK

PostPosted: Mon Jan 08, 2018 11:42 am    Post subject: Reply with quote

Hi Phil,

OK. I first loaded the V3.gpi file straight onto the Garmin, with the following results:


ANSI: Windows_1252 Encoding (V3.gpi file)




utf-eight: UTF-8 Encoding ¿ (V3.gpi file)



I then placed the ANSI.gpx and utf-eight.gpx into a folder for processing by the Mac POI Loader, with the following results:


ANSI: Windows_1252 Encoding (ANSI.gpx file)



utf-eight: UTF-8 Encoding (utf-eight.gpx file)



My observations

I'm presuming that your v3.gpi file was processed by Windows POI Loader.

  1. .gpx files (ANSI or UTF-8) that are processed by Windows POI Loader, appear to show characters correctly.
  2. The Mac POI Loader doesn't seem to be able to process diacritical marks or extended character sets either by ANSI or UTF-8 .gpx files.


It really does look like the Mac version of POI Loader is flawed.

One other thing Phil, I noticed that the name of the POI itself was either UTF-8 Encoding or Windows_1252 Encoding, with the diacritical marks wishing the POI description or address. Would it be possible to try another test with several POIs within one file, each with a diacritical mark in the actual name? As this would allow us to test whether the Garmin can search for a POI name that has an accent by using the English keyboard that doesn't have didactical marks, i.e. search for "Café" by using the search string "cafe".

Many thanks,
_________________
Robert.
iPhone 6s Plus, iOS 10.3.3: iOS CamerAlert v2.0.7
Garmin Camper 770 LMT-D - because TomTom GO Mobile iOS is so awful
TomTom (UK & ROI and Europe) iOS apps v1.29; TomTom GO Mobile iOS 1.9.2
Back to top
View user's profile Send private message Send e-mail
pcaouolte
Frequent Visitor


Joined: Dec 27, 2006
Posts: 931
Location: South Lincs, UK.

PostPosted: Mon Jan 08, 2018 1:52 pm    Post subject: Reply with quote

Privateer wrote:
...this would allow us to test whether the Garmin can search for a POI name that has an accent by using the English keyboard that doesn't have didactical marks, i.e. search for "Café" by using the search string "cafe".

I have tried this on Windows by converting the list of POIs that you posted earlier in this thread to a GPI file using the Windows version of POILoader. The Garmin doesn't find Café Ouen when searching for Cafe. I also tried translating the Diacritical marks into their basic equivalents and putting this into the fourth column of the CSV file e.g.
Code:
49.251346,-2.226038,"Café Ouen","Cafe Ouen"
This results in "Cafe Ouen" appearing in the POI description of the POI named "Café Ouen" but the Garmin still doesn't find it when searching for Cafe.
_________________
Paul
Back to top
View user's profile Send private message
Privateer
Pocket GPS Moderator
Pocket GPS Moderator


Joined: 30/12/2002 17:36:20
Posts: 4480
Location: Oxfordshire, England, UK

PostPosted: Mon Jan 08, 2018 2:51 pm    Post subject: Reply with quote

Hi Paul,

Thanks for looking into this,

I think that .csv files may be a problem. However it may be possible with .gpx files. Just a hunch or hope really. Very Happy

Regards,
_________________
Robert.
iPhone 6s Plus, iOS 10.3.3: iOS CamerAlert v2.0.7
Garmin Camper 770 LMT-D - because TomTom GO Mobile iOS is so awful
TomTom (UK & ROI and Europe) iOS apps v1.29; TomTom GO Mobile iOS 1.9.2
Back to top
View user's profile Send private message Send e-mail
PhilHornby
Frequent Visitor


Joined: Dec 07, 2006
Posts: 511
Location: Stockport

PostPosted: Mon Jan 08, 2018 3:59 pm    Post subject: Reply with quote

Privateer wrote:
My observations

I'm presuming that your v3.gpi file was processed by Windows POI Loader.Yes

  1. .gpx files (ANSI or UTF-8) that are processed by Windows POI Loader, appear to show characters correctly.
    Yes
  2. The Mac POI Loader doesn't seem to be able to process diacritical marks or extended character sets either by ANSI or UTF-8 .gpx files.

It really does look like the Mac version of POI Loader is flawed.

Sadly, yes that appears to be the case. While you could make the argument that with CSV input, it should default to expecting Mac Roman encoding (on a Mac), no such get-out exists for GPX input. The GPX files are properly formed and encoded. They are quite specific as to the encoding of the data. They use the same character set that the Nüvi uses, so there is really no excuse for mangling them into something else. Crying or Very sad

Would it be possible to try another test with several POIs within one file, each with a diacritical mark in the actual name?
I'll knock something together shortly

pcaouolte wrote:
Privateer wrote:
...this would allow us to test whether the Garmin can search for a POI name that has an accent by using the English keyboard that doesn't have didactical marks, i.e. search for "Café" by using the search string "cafe".

I have tried this on Windows by converting the list of POIs that you posted earlier in this thread to a GPI file using the Windows version of POILoader. The Garmin doesn't find Café Ouen when searching for Cafe. I also tried translating the Diacritical marks into their basic equivalents and putting this into the fourth column of the CSV file e.g.
Code:
49.251346,-2.226038,"Café Ouen","Cafe Ouen"
This results in "Cafe Ouen" appearing in the POI description of the POI named "Café Ouen" but the Garmin still doesn't find it when searching for Cafe.


I think that is in keeping with what I've seen so far, as well.
Back to top
View user's profile Send private message







Posted: Today    Post subject: Pocket GPS Advertising

Back to top
Display posts from previous:   
Post new topic   Reply to topic    Pocket GPS World Forum Index -> Garmin Portable Navigation Devices All times are GMT + 1 Hour
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB 2.0.11 © 2001 phpBB Group
phpBB port v2.1 based on Tom Nitzschner's phpbb2.0.6 upgraded to phpBB 2.0.4 standalone was developed and tested by:
ArtificialIntel, ChatServ, mikem,
sixonetonoffun and Paul Laudanski (aka Zhen-Xjell).

Version 2.1 by Nuke Cops © 2003 http://www.nukecops.com

Make a Donation



CamerAlert Database

Click here for the PocketGPSWorld.com Speed Camera Database

Download Speed Camera Database
16.072 (11 Jul 18)



WORLDWIDE SPEED CAMERA SPOTTERS WANTED!

Click here to submit camera positions to the PocketGPSWorld.com Speed Camera Database


12mth Subscriber memberships awarded every week for verified new camera reports!

Submit Speed Camera Locations Now


CamerAlert Apps



iOS QR Code






Android QR Code







© Terms & Privacy


GPS Shopping