GEO DATA GEODTA Question, HELP, WTF

[zerodays]

Hi all.

first I am here to learn and get this piece of gps device working.
You can probably assume that I am a completely Nubs, but please do try to read my problems carefully, and if you still find me nubs because I did not read the tutorials/forums carefully, then please do post the link here.

I am not discussing anything about Mapthis! softwares, here yet.

Just a pure GDML 0.5.02B (the latest)
I live in Auckland, New Zealand.

It seems to me that, a lot of GDML and Mapthis! are optimized for USA and Canada... somehow, I feel that way....

I've been trying to do various of stuffs. All I want is to get it working. as smooth as possible.

First, Google Map. Somewhat somehow I got the map downloaded working. I followed the guideline posted somewhere here. :)

Next, Geodata. WTF is this to be honest. I've been searching for ages on entire google search, and yet no one is explaining any of this. I know nothing about Geography, I barely know Lat/Log are, at least thats how GPS works. and maps are just 'illusions' to show on screen. at least thats how GPS works) :)

=Question 1.
I thought I downloaded the Map that includes street names, etc into JPG. Ain't I correct?
From the previews of my downloaded map. It looks I am right. (i.e., st names, etc are 'embedded' as JPG picture)

=Question 1.5.
I know what POIs are. such as Patrol station, macdonalds, etc. Google obviously knows big names such as 'schools' etc,
Then is there any ways to simply RETIREVE ALL of them (e.g., retrieve all POIs from Auckland area) ?
I can not find this as this is more to do with Google disallowing abusive behaviors?
As well, Google bans IPs (for Sat image) whenever I download more than thousands (out of 10s to 100 thounds), which means, it is impossible to download. and the Pop-up seems to be very buggy, there are no automatic re-trieve, i need to click OK button to proceed to next step,. thus I gave up on Hybrid images.

=Question2.
Then WHAT is Geodata for? hmm. maybe for searching St names? maybe showing names into 'TXT' into (already embedded) JPG maps??
I am lost here.
I tried to open the geodata.dat, it's unreadable in normal text editor.
I also tried the 'working' geodata.dat from other countries (i.e., the one from Netherland, i think), still, unreadable.

I know that all USA and Canada and Netherland dudes have noproblems of 'Geodata' because GDML said so. (which probably means the authors have retrieved nearly all, and linked on the website)

In my case, I need to get Geodata manually, using Google (semi-manual)

=Question3.
I got the map size of 200MB of Auckland, NZ. Not bad. I still don't know what is Geodata for.
Still, I now go to Geodata button.
Step 1/3 (Load the map) I did, good, loaded.
Step 2/3 (Select) shows 64 segments.

Here, Why can't I simply select all and download all ?
I can not find the way to select all, or any ways to automatically download Geodata.

Further serious problems. whenever there are some oceans/seas in one of the segment (out of 64), The download simply fails after it sees 'merely' 20 !!
Why is this? there are 5776 (75 by 75) to be requested, and I why is it simply failing just because of 20 ??

Step 3/3 You can not overwrite/save geodata.dat. I need to either delete or rename the geodata.data before I can refresh the new.

Yet. I still don't have any clue why I need geodata. But one thing for sure I need it, is because people seemed to be wanting it !! damn it!:eek:


Additional Question........

Route/directions/etc:

To do so, I need internet to retireve and Manually select TO/FROM from google-map, then download into PSP.
Erm... I am a bit lost here... Why would anyone wants to 'manually' download the route and finally fake it on PSP-GPS...?? I think this sounds silly.
But to me, this looks like,

"while I am driving, and I want to go somewhere, and need to know the directions and locations,
I must to go back home, go to google-map and search the direction, and download that info to your psp, get back to my car, and drive using the updated PSP-route direction ?????????????? "

Shouldnt it be opposite way around...? Again, I'm lost here. To my knowledge, once the maps and geodata(whatever it is) are ALL stored into PSP, the psp 'should' be able to perform the 'basic' ShortestPath Algorithms to find the route/directinos?
Actually, does Mapthis allows users to enter Street names and find some path?


I need some serious help here please.
thank you very much in advance!!!!

[Nieko]

Wow...



It seems to me that, a lot of GDML and Mapthis! are optimized for USA and Canada... somehow, I feel that way....


You're right, it is :). When I was constructing Dutch geodata, I saw 5 bytes were reserved for postal codes, as in the US 5 digits are used.



Next, Geodata. WTF is this to be honest.


Geodata is a database containing a number of streets with their respective location (city, zip code, state/province and latitude/longitude). You can use this to look up where a specific street is (e.g., you enter "Main street" and "Some town" in the geodata search), and MapThis'll show you where exactly that is on the map. Reverse lookup means MapThis'll show you what the nearest street is called.



=Question 1.
I thought I downloaded the Map that includes street names, etc into JPG. Ain't I correct?


Yes, the maps you download consist only of pictures. You can not use the street names in here, because they can't be read by MapThis.



=Question 1.5.
I know what POIs are. such as Patrol station, macdonalds, etc. Google obviously knows big names such as 'schools' etc,
Then is there any ways to simply RETIREVE ALL of them (e.g., retrieve all POIs from Auckland area) ?


Not as far as I know. You can Google a bit for POI download sites for New Zealand and see what they have.



As well, Google bans IPs (for Sat image) whenever I download more than thousands (out of 10s to 100 thounds), which means, it is impossible to download.


GMDL has a feature where it allows you to wait a number of seconds between downloading tiles. If you turn this on, it might work. However, this'll take really a lot of time for big maps. I've never really bothered :P



I tried to open the geodata.dat, it's unreadable in normal text editor.
I also tried the 'working' geodata.dat from other countries (i.e., the one from Netherland, i think), still, unreadable.


To keep the file small, zlib compression has been used. Also, latitude/longitude etc. are stored as 4 byte integers, not as strings.



Route/directions/etc:
To do so, I need internet to retireve and Manually select TO/FROM from google-map, then download into PSP.
Erm... I am a bit lost here... Why would anyone wants to 'manually' download the route and finally fake it on PSP-GPS...?? I think this sounds silly.
But to me, this looks like,
"while I am driving, and I want to go somewhere, and need to know the directions and locations,
I must to go back home, go to google-map and search the direction, and download that info to your psp, get back to my car, and drive using the updated PSP-route direction ?????????????? "


Unfortunately, MapThis cannot calculate routes from point A to point B itself, so yes, you will need to download routes beforehand. Using http://getroute.nieko.net/ you can also do that on the PSP itself, providing you have access to a WiFi point.



Shouldnt it be opposite way around...? Again, I'm lost here. To my knowledge, once the maps and geodata(whatever it is) are ALL stored into PSP, the psp 'should' be able to perform the 'basic' ShortestPath Algorithms to find the route/directinos?


Geodata doesn't contain edges, only nodes. Deniska is working on vector data implementation, perhaps then a shortest path algorithm can indeed be used.



Actually, does Mapthis allows users to enter Street names and find some path?


Yes, a straight path :). Other than that, no, at least, not yet.

[deniska]

Geodata doesn't contain edges, only nodes. Deniska is working on vector data implementation, perhaps then a shortest path algorithm can indeed be used.

would you be interested in writing the routing algorithm code? if so, we can discuss the details.. the latest "geoline.dat" format, that I am working on should have all required data for route calculation..

[Nieko]

Well... Recently I haven't had that much free time anymore, and the upcoming months it'll probably be even less (I'll be gone for one month soon as well).

If you have questions though, or want to brainstorm, I'd be interested :).

[deniska]

Well... Recently I haven't had that much free time anymore, and the upcoming months it'll probably be even less (I'll be gone for one month soon as well).

If you have questions though, or want to brainstorm, I'd be interested :).
I don't really have any big issues other than free time to actually implement all those things I want to implement... this is why I was hopping to unload some of these tasks on someone else

[Nieko]

I know, and had I had the time, I would've seriously considered it. Hopefully someone else can step in :).

[zerodays]

Nieko, I must say that I am very grateful to you, and sorry for my late post.. :(

I've been mad on trying to get GPS working, etc, etc

Good News is that,
My PSP-Slim 3.71M33-3 with latest Mapthis5.02 is working. I.e., I travel around, things work, I can see where I'm going, etc.

Sadly, my Geodata.dat is still not working.

I know that Geodata has been retrived correctly. I can confirm this vy simply going to the temp folder (which GDML) created.

All of these temp folders are only 340kB. And Geodata.dat is 135kB.
(This file is stored in the folder of _MyMap)

I can read these temp files, and i can read the names of the ST with their long/lat,

I am just not sure Why mapthis (which should do it automatically) does not 'seemed' to load it up.

I can confirm this,
- by attempting to search for a word, --> not found
- by reversing one of the very crowded place (which I am very sure I've retrieved the geodata), --> not found.

thus, geodata.dat is either not working, or mapthis is not loading is properly,
or..... geodata.dat is currpted..??

Please help.. Cheers.

[zerodays]

http://www.dcemu.co.uk/vbulletin/showthread.php?t=76428&page=2

Slightly discussed in the above thread. However, I see no solutions...

Auckland Geodata (temp text) I get is very simple.

E.g.,

......
ABEL RD,1,+174461960-36645600
ALPINE RD,1,+174475970-36615790
......

Cheers.

[Nieko]

Could you post the geodata.dat you have? Then I can take a look at it.

[zerodays]

ok, i've attached, this..

cheers...

[aberz]

http://getroute.nieko.net/

Thnx for the link nieko, you make my day. Now I can get route on the fly, well, with wifi of course. Still better than nothign :D

[Nieko]

zerodays: this geodata.dat is corrupt. The city names (Sparta, Wayland, etc.) are padded with spaces instead of null bytes, which means you have to enter them in the search too.

Also, though the size fields are correct, the offsets are wrong (first one is 50Fh but should be 383h).

I think this is simply a bug in GMDL. in7ane, can you take a look at this? zerodays: can you tell what exact steps you have taken to get this geodata.dat?

[zerodays]

Neiko, here is the actual TEMP folders zipped

I used the latest 'B' version og GDML

As you can see, once you put them into GDML temp folder (e.g., C:\GDML\...),

Start Program ->
Click Geodata -> Click Geodta (Semi-Auto)
Step 1/3 Load map (I did that)
Step 2/3 Click (automatically finds 30/64, done) I only need 30, as the rest 34 segments are mountains, etc.

Step 3/3 Save ! (I also made sure, I 'deleted' any geodata.dat to make sure to create a new geodata.dat)

Cheers

Neiko, I do not think you need the actual map, as the file I've attached here are the temps of 'readable' formats... I just need to covert to .dat
But if possible, I'd like mapthis to load these .txt file..
It would be great (e..g, possibly a bit faster loading)

Cheers

[Nieko]

I think this is more something for in7ane to take a look at, because I think it's a GMDL bug.

Btw., my name is Nieko :).

[suzukici]

Hi Neiko,
Is it possible to adding ''http://maps.live.com'' to your http://getroute.nieko.net/?

[zerodays]

I think this is more something for in7ane to take a look at, because I think it's a GMDL bug.

Btw., my name is Nieko :).

Ok, sorry about that, Neiko sounds more Japanese, I have a eurobeat friends from Netherland, and somehow their names seem like that. :)

In that case, I guess I will have to try 'a' version.

Cheers.

[zerodays]

ok, sadly, 'a' still do not work..

Nieko, can u tell me the strutures of the geodata.dat ?

As long as I know how things work, I am pretty sure I can simply convert this txt file to .dat (using whatever zlib, as well)

Cheers.

[Nieko]

suzukici: see PM, I'll post when I'm done (if possible), but don't expect it too soon. Also, this isn't the right topic for that, http://www.dcemu.co.uk/vbulletin/showthread.php?t=66572 is.

zerodays: you can get all this information from the scripts in http://stuff.nieko.net/psp/mapthis/geodata/process.zip. See the comments in header.pl and split-up.pl. The header and data are divided by "#####".

You could of course also use these scripts to make your geodata.dat :).

[in7ane]

Nieko, yes, I see the bug - thanks for that. I will try and get that fixed as soon as I get a chance (it's in at least two places that I can see so far).

[in7ane]

That should be fixed in 0.5.03b get it from http://www.in7ane.com/psp/gmdl.html and rerun the geodata.

[zerodays]

Oh, thanks, I will try that asap. I've been away for a few days :)

[zerodays]

Hi in7ane, the geodata.dat is still not loaded.

Could you please check what went wrong? I've posted my 15_9 folder zipped... It would be good if you can assist me on this.

Cheers

[zerodays]

In7ane, I also realized that GDML will *not* download anything closer than 100m Scale..

According to google map I'm viewing (through GDML), I can zoom up to 20M (i.e., I can zoom 2 more from 100m scale) so that I can actually see and read out the 'numbers' attached around the blocks.

I've tried various ways to tell GDML '03b' to download those 2 close zooms, but it doesnt seem to work out...!

Some assitance would be great.

***************************************
EDIT: it seems that, in Advance setup, there 4 choices

zoom = null
x = null
y = null
min zoom = 0

as soon as GDML starts, all null and min-zoom is set to 0.
I've tried to change this value to -2 (so that I can further zoom-in)
Also, I am also setting zoom to 8...

Does this make sense???
It seems to me that (after reading lots of forum threds),
you've decided to 'disable' these 2 closest zoom, because people considered that they are a waste of spaces??

I am not sure if I'm doing it right or wrong, because im now downloading one tile zooming all the way to 8.
(As far as I can see from 'preview' I can now see even more closer zoom (than before)
***************************************
Thanks.

[Nieko]

zerodays: did you check the Advanced options? If you fill in 0 or 1 for min zoom, things work fine for me, with 0.5.03b also.

I did also notice that in some places min zoom can be negative in Google Maps. Do check what level the highest position on the scale gives you if you want really a lot of detail :).

[DaLex]

Hi all, I´m new here, and i dont really speak good english.
Can anybody tell me where or how I can get/create a geodata.dat file for german-cities?

[Nieko]

AFAIK there's no official German geodata, so your only bet is the Google tab in the geodata part of GMDL. It's rather slow and may block you after a while I think, so not sure it'll work for larger areas.

[zerodays]

Nieko, it seems that, In7ane put default not to zoom more than 6 times.

From my experience fiddling around with Advance setup, I need to set min.zoom to -2
. Then GDML will download up to street-level where I can read the 'numbers' on the road. :p

[zerodays]

Neiko, one more thing, sorry. I've briefly looked at your scripts.
Regardless, it seems that mapthis only reads the compressed geodata.dat,
Any 'working' structures of TXT files ? All I want is to covert my txt file to zlib, but I do not know the structures, and there are several folders.... (refering to my previously uploaded zip file)

cheers.

[Nieko]

I've briefly looked at your scripts.
Regardless, it seems that mapthis only reads the compressed geodata.dat,
Any 'working' structures of TXT files ? All I want is to covert my txt file to zlib, but I do not know the structures, and there are several folders....


geodata.dat consists of several parts.

First there's the header part, which contains info on the blocks. Each block usually consists of data for one zip code. In the header you need 43 bytes per block: zip(5) + lat(4) + lon(4) + state(2) + city(20) + size(4) + offset(4). Note that lat, lon, size and offset are stored as little-endian 4 byte integers. The other fields are strings, null byte padded.

After the header there are 5 pound signs: #####.

Then you get the blocks, which are encoded using zlib. I've put an unzipped block in the attachment, containing all the information for post code 1011 in the Netherlands (the house numbers are fake though). The format is: street,lowest house number;highest house number,(+ or -)longitude*1000000,latitude*1000000.

[Toxicwaste79]

hi.
i tried to download geodata from germany for address lookup. for testing, i only downloaded 1 segment of 256. for this segment the street lookup works, but there is something wrong with the city names and the postal zip codes. it seems that gmdl has download US zip codes and city names. someone know what i've done wrong? i've read something about "regional and language options" but i can't find the panel to change it. i am using gmdl 0.5.04b. is there also a possibility to change "ß" to "ss" and "ö" to "oe" ... in geodata.dat?

[fragreaver]

Hey, first post here :)

seems like there's no hope getting geodata for europe *sigh* :(

I was so looking forward having a PSPNavi, but since the missing geodatas...argh :D

[derek1212]

hi everyone ...i am new and got questions abt routes getting on below http://getroute.nieko.net/

When i press '''Get routes'' , there are lots of codes showing out inn HTML....
So....what should i do to put them in psp ?
I meant which file in psp and how to show the routes in MAPthis ??

sorry for nub's questions and poor english...

[Nieko]

See http://en.wikibooks.org/wiki/Map_This! (also mentioned in one of the stickies).

[derek1212]

See http://en.wikibooks.org/wiki/Map_This! (also mentioned in one of the stickies).


thank you so much ...man :thumbup: