Openstreetmap Geodata Coming...

[Codaz]

Programming Genius Nieko is known for his extra features onto the Mapthis program, but he has more upon his sleeve !

I pointed Nieko on the openstreetmap.org geodata which can be extracted and converted to CSV.

Since it's a very very large file it will take a while before it's converted.

Nieko is working his hands of trying to make the Netherlands geodata work with GMDL now.

England, Belgium, Luxemburg and even Germany and Denmark are there too, but aren't converted since they aren't complete (only the Netherlands is complete). When finished it will surely have a very very nice addon to Mapthis in general.

Stay tuned for more updates :)

[fak3]

congrats, nice work dudes

[Nieko]

A short update:

The original planet.osm file was 15.5GB;
I've extracted the Netherlands from this, netherlands.osm is now 1.0GB;
The osm2csv.pl script from OpenStreetMap stagnates after it's produced 2MB of CSV;
I'll try to write my own script that produces the CSV file;
There may be several versions, because the AND data also contains supermarkets, parking lots, etc.
I have no idea yet when this will be done, but hopefully in the upcoming days :).

[PDG]

A short update:

The original planet.osm file was 15.5GB;
I've extracted the Netherlands from this, netherlands.osm is now 1.0GB;
The osm2csv.pl script from OpenStreetMap stagnates after it's produced 2MB of CSV;
I'll try to write my own script that produces the CSV file;
There may be several versions, because the AND data also contains supermarkets, parking lots, etc.
I have no idea yet when this will be done, but hopefully in the upcoming days :).


Nice to Hear ;)
Sad thing was me living in Sweden and not getting any Geos but that hopefully comes after a while :)
:thumbup: to you

[Airwave]

England, Belgium, Luxemburg and even Germany and Denmark are there too, but aren't converted since they aren't complete (only the Netherlands is complete). When finished it will surely have a very very nice addon to Mapthis in general.

Stay tuned for more updates :)


Oh thats nice news,but i hope the germanydata are complete in future.


I wait for next news from you;) good luck.........

[johoo]

this is great ,i hope you can get it to work

i see on openstreetmap.org that you can download the nl AND file
only 84,4 mb

http://www.opengeodata.org/images/2007/AND/AND-NL.tar.gz

johoo

[Nieko]

I saw that file, but it contains mainly binary data, which didn't work well yet with MapThis. I prefer textual data (XML from OpenStreetMap in this case) :).

Also an update: I've parsed the nodes, segments and ways from the OSM file and am now writing a first CSV version containing street names. There is however one big problem, the OSM data doesn't indicate what city a street belongs to. On the OSM interface, it simply says that a street is closest to a particular city, so I'll have to make a script that does the same.

Problem is, I have ~900.000 streets and ~4.000 places+zip codes. That'd mean 900.000*4.000 = a lot of comparisons. Does anyone have a good idea on how to bring this number down?

[jakekoh]

guys,

how do i make my geodata work? i converted it frm csv to geodata.dat and 2 other files together now i placed it in the main folder of mapthis in my psp. how do i know if its working or not then?

[Natanael]

Do i need the geodata.dat to locate me on the map, when i have the psp290?

[Nieko]

jakekoh: first make sure the entire CSV file is uppercase, otherwise it won't work (took me a while before I figured that out). You can then check out the Address lookup and Reverse address functions in MapThis. It should work if you only fill in a streetname.

Natanael: no, with the PSP-290 working it'll just show you where you are. You can use geodata to lookup which street is closest or where a particular street is.

[Nieko]

As for my progress, I've got everything figured out, and it's now just a matter of time :).

I now have a CSV file with all the streets with their coordinates in it, and another CSV file with all the zipcodes with city and coordinates in it.

Today I'll run the script that will merge these two files, so then I'll have one CSV file containing coordinates, street, city, zip code and province. (Turns out Perl has a nice module that figures out the closest zip code for a street.)

After that, I need to split and sort the streets by zip code, compress them and write them to a geodata.dat.

The scripts for this entire process I will put on my server.

(Btw., I'm not using GMDL's CSV import cause I couldn't find out (and didn't want to spend too much time on finding out) how to include zip code and province in the geodata.)

So for all the Dutch users, be patient, in a couple of days I'll be done :).

[Codaz]

Nice Nieko keep it coming :)

[Natanael]

When i pointed a street on and want to get lead there, is that able without geodata?

[Nieko]

Natanael: I don't understand your question, please rephrase.

[Natanael]

I have a mark set on the map.
Map!This shows me the distance to the mark.
I`m going to get to this target.
Will Map!This lead me to my target, or do i need geodata for this?

[Nieko]

If you have a GPS dongle, MapThis will show you where you are on the map. If you have geodata, you can look up where a specific street is on the map. If you want to have street-to-street directions on how to drive, then you can use either GMDL (click the tab 'Directions') or my script, which can be found at http://getroute.nieko.net/. This will generate a POI file, which you can load in MapThis.

[Nieko]

And I'm done! :) For those who can't wait, Dutch geodata can be found here:
http://stuff.nieko.net/psp/mapthis/geodata/geodata.dat

Put it in the same folder where MapThis' EBOOT.PBP is, or in a map folder.

On http://stuff.nieko.net/psp/mapthis/geodata/ you can also find the scripts I've made to convert the OSM data to MapThis compatible geodata (process.zip). Try it out if you want :)

There, you can also find a _ZIPCODE POI file, which you can use to lookup where a specific zipcode is (POI Lookup => circle button), or just to show the zipcodes around the current location on the map.

Be sure to also check out the readme.txt!

in7ane: with these scripts, do you think you could take a look at geodata for entire Europe? My version heavily relies on Dutch zip codes, so it won't work directly.

[Codaz]

HURRAY for Nieko :)

England is also available on openstreetmap geodata so anyone can import that too on Mapthis.

Get Nieko's perl scripts, rebuild them for England and convert the planet.osm to geodata.dat :)

Easier said that done but ok :)

[Codaz]

Nieko needs to check the poi file hahaha :)

[Nieko]

There are indeed some anomalies in the data... If anybody has better data for zip codes, cities, provinces and their coordinates, feel free to contact me ;).

[Airwave]

Hey Nieko is it possible for germany? can i use your tut. to?

[Codaz]

Yes you can use Niekos script, BUT:

-Germany isn't complete (See Dortmund).
-You have to rebuild the perl scripts (thus you have to know how to code perl).
-You have to extract planet.osm which is 15,5 gb extracted. Germany is 1 gig ripped from that i think.

[johoo]

hoi nieko
works great man ,a big thnx

i noticed that i only can search for the streetnames and not the city or zipcode
it says incorrect values entered
but i see the city and zipcode when i find the street so it must be possible to search for city ,state or zipcode

i don't know how to correct that
i've tryed it before with the city csv file and got it working as geo but also only under street search
on the internet i couldn't find the answer for it
also on wiki etc.

anybody how knows the answer
haha if you understand what i wrote

how to search under other topics ?street,city,state,zipcode

[Nieko]

If you're using geodata, you always have to fill in the street name. If you want to search for a zipcode or a specific town, you can do a POI Lookup + search (circle button) on the _ZIPCODE POI file.

[Natanael]

Can anyone upload a Geodata for Germany or Western Europe, because downloading the whole World takes so a long time!
That will help me and some others alot.

[Airwave]

Yes you can use Niekos script, BUT:

-Germany isn't complete (See Dortmund).
-You have to rebuild the perl scripts (thus you have to know how to code perl).
-You have to extract planet.osm which is 15,5 gb extracted. Germany is 1 gig ripped from that i think.



Thanks Codaz;) but i see that I`m not able to rebuild perl scripts,i`m not a pro *grins*

I wait for userfriendly things;).I hope.


greets

[Nieko]

With the new geodata, it seems we've gone from 734512 streets to 886259. That's over 20% more! :) Enjoy the new geodata at: http://stuff.nieko.net/psp/mapthis/geodata/geodata.dat.

[v1ck]

Hi.
Can anyone upload geodata for poland? 15gb its really huge for me and downloading that would be hard.. Thx

v1ck

[Codaz]

It's only 900 mb, extracted it's 15.5 gb :) Very good compression :)

[v1ck]

It's only 900 mb, extracted it's 15.5 gb :) Very good compression :)

i have HDD 20gb :> ~5gb left ;)

[PDG]

Hi.
Can anyone upload geodata for poland? 15gb its really huge for me and downloading that would be hard.. Thx

v1ck

Whats the diffrence of generating it yourselve and downloading it ?
Its still the same size, but if you download a zipped file you must have twice the space than if you generate it yourselve ;)

+ If you generate it yourselve you can check off some zooms and stuff ;)

[v1ck]

Whats the diffrence of generating it yourselve and downloading it ?
Its still the same size, but if you download a zipped file you must have twice the space than if you generate it yourselve ;)

+ If you generate it yourselve you can check off some zooms and stuff ;)

15gb is for planet.osm, i just need Poland extracted and that would be much lower, right?

[PDG]

15gb is for planet.osm, i just need Poland extracted and that would be much lower, right?

Makes sense :)

[Nieko]

On openstreetmap.org, Poland looks pretty empty to me. Warschaw has some detail, but I think "geodata for Poland" isn't too useful, because most streets won't be in it yet anyway.

[v1ck]

On openstreetmap.org, Poland looks pretty empty to me. Warschaw has some detail, but I think "geodata for Poland" isn't too useful, because most streets won't be in it yet anyway.

ehh.. too bad ;/ Btw, what about south Ireland? Is it empty as well?

[Nieko]

You can check that for yourself :).

[v1ck]

Nieko:

would you be able to make some map-editor that whe could edit streetnames on generated googlemaps ? And then save it to geodata.dat? you can edit maps on
http://openstreetmap.org/ but after that you still have to download planet.osm and export what you need.. its usless..

I think that you could add some options to GMDL where you can put names for streets and generate geoadata..

its just an idea :) What you think?

[Nieko]

I don't think that's a good idea, or I'm not quite getting it. The idea of geodata is that you can look up streets in a particular area, and that's not much use if that data isn't anywhere near complete (you might as well make a POI file with those few streets you need).

I only started working on Dutch geodata when the AND import was done, and basically the entire Netherlands was available.

[Codaz]

AND provided Dutch geodata and is a provider of various map and navigation solutions, like Map24 and mapquest.com

Maybe you can email them on the status of geodata and/or maps of Poland.

Donation to Openstreetmap won't happen that often since that material is licensed and very very expensive (think in amounts of 200-500 dollars).

[LouNGeR]

woohoo, awesome!
Works quite well, it's actually able to find my street, the postal-code is reported incorrectly though :P

[Nieko]

LouNGeR: that's probably because you live too close to another postal code. Unfortunately, I can't do much about this problem...

[LouNGeR]

No that's not it, its actually a postal code that is 2 KM away, funny though, the postal code it reports is where I lived last year :)

It's not that big of a deal, at least it can find streets.
Any way to grab all POI's like shops and stuff?
GMDL Tool doesn't support it, however, Google Maps does have the information.

[Nieko]

Is your new postal code listed in _ZIPCODE? It might just be old data I used.

About POIs, AND also has a lot of them, I'll see what I can do.

[LouNGeR]

I'm sorry where do I find _ZIPCODE ? (I just downloaded the GEODATA.DAT from here and put it next to the mapthis! eboot)
And I think that you are right, that it just picks the closest zipcodes available..

[Nieko]

It's a POI file with all the Dutch postal codes and their cities. It's in the same directory as the geodata.dat: http://stuff.nieko.net/psp/mapthis/geodata/

[LouNGeR]

Oh, I just clicked the direct link to geodata.dat
The ZIPCODE file is a bit strange, here's the deal.
In MapThis, when I search for my street, it takes me to the right location. Though it reports that my street is at 1353, while it should be 1359. In the ZIPCODE file, it shows that both of these codes are in "Noord Holland", while they are both actually in "Flevoland". So the coords in Geodata are correct, but the Zipcodes are wrong. (i did not have the ZIPCODE file in my MapThis folder, do I have to put it there?)

Btw, today I did a good test with my GPS for the first time (from Almere to Amsterdam). My Nederland map was fine, but I had some issues with starting the whole GPS process, is it normal that it takes like 10 minutes before it actually starts the tracking system?

[johoo]

is it normal that it takes like 10 minutes before it actually starts the tracking system?

yes it is,but it depents day by day ,normal it takes about 5 to 10 minutes
i noticed also that the angle from the receiver is important too
when i change it i got more or less sat's and fixes

[LouNGeR]

Hm, then how come this VDO Dayton system in the car that we were driving this afternoon, starts to work like, immediatly. Is it on standby or something, or is MapThis just not that fast?

BTW, I also had like 12 visible and 5 fixed. Also, when moving the PSP-290 horizontal it seemed to work better.

[johoo]

maybe different receivers
there is also a mod for an holux receiver on the psp and from what i know is that one faster

i don't have a holux but i'm thinking about it

[LouNGeR]

I doubt that the upcomming official mapping software takes that long to startup, guess we'll have to wait and see.

[Nieko]

In MapThis, when I search for my street, it takes me to the right location. Though it reports that my street is at 1353, while it should be 1359.


When you load the _ZIPCODE file, you'll see on the map that 1359 is in the bottom left of Flevoland (near A6 to Noord-Holland) and that 1353 is the first zipcode to the right (in Almere-Haven). I expect your street to simply be closer to the 1359 dot than to the 1353 dot, is that true?




In the ZIPCODE file, it shows that both of these codes are in "Noord Holland", while they are both actually in "Flevoland". So the coords in Geodata are correct, but the Zipcodes are wrong. (i did not have the ZIPCODE file in my MapThis folder, do I have to put it there?)


Regarding this, you might want to read http://stuff.nieko.net/psp/mapthis/geodata/readme.txt. It doesn't matter where you place the POI file, as long as you can load it (i.e., in /PSP/COMMON or in a map folder).




Btw, today I did a good test with my GPS for the first time (from Almere to Amsterdam). My Nederland map was fine, but I had some issues with starting the whole GPS process, is it normal that it takes like 10 minutes before it actually starts the tracking system?


LouNGeR: the PSP-290 isn't that great a GPS receiver, so that's why it takes longer to get a fix. It does really help to have an extension cable btw., I sometimes got fixes within 40 seconds.

Its reception is indeed better when the flat part (the part that says 'GPS') is facing the sky.

[LouNGeR]

Yes, 1353 is closer to my home because 1359 is way off.
According to the source that was used for the data, Flevoland is only included in the 8000 - 8387 range which is probably correct, and a part of Flevoland, is included in Noord Holland.
http://nl.wikipedia.org/wiki/Lijst_van_postcodes_in_Nederland#Postcodes_en_prov incies

However, the paragraph below that, is correct :
1343-1349: Almere-Hout
1350-1361: Almere Haven
1359: De Velden
1361: Marina
1380-1383: Weesp

It's all very strange :p

[Nieko]

About POIs, AND also has a lot of them, I'll see what I can do.

There are some categories that are almost empty, but a few others that are quite valuable. I'll put them online soon.

[Nieko]

I've extracted some of the more useful POI collections. They can be found here (together with the script needed for extracting them from OSM data): http://stuff.nieko.net/psp/mapthis/geodata/poi.zip

[Nieko]

I've rebuild the Dutch geodata, which can be found here: http://stuff.nieko.net/psp/mapthis/geodata/geodata.dat.

This file is a lot bigger than before, because it includes all the nodes of a street (corners etc.). This allows for more accurate reverse geodata lookup.

Also, a few new neighborhoods are present in this file.

As before, the scripts and instructions to build this geodata.dat yourself can be found here: http://stuff.nieko.net/psp/mapthis/geodata/. You can also find a README.txt here for people who merely want to use Dutch geodata.

[Codaz]

GREAT :)

So does this also work with the new format Deniska is setting up for Mapthis Vector Data ?

And on which OSM data is this one build, the 5-12 ?

[Nieko]

MapThis vector data is a completely different format, and the file would then be called geoline.dat.

Since vector data isn't supported yet and because the format may change, I'm not creating a Dutch geoline.dat yet. When this becomes useful, of course I will :).

It's indeed built using 5-12 build.

[Codaz]

Can geoline.dat and geodata.dat cooperate ?

So that you are driving onto line 4302392 and that road is Leidseweg for example ?

I suppose it's lat/lon based but ok :)

When geoline.dat appears, the GPSFS data becomes obsolete :)

[Nieko]

I think they very well can :).

I don't think GPSFS data will become obsolete though. OSM data doesn't contain cycle lanes e.g., while Google Maps does. Also, GPSFS will remain useful for locations where no vector data is available.

[spychacz84]

i am crying, dieing, and am simply jalous about geodatas! openstreet do not support polis map as well as we want to :( maybe you know some other way to import some map to csv or any oter file type. I ve read about tomtom, etc but i just symply can't find the way :( and i am not sure how legal is that :( maybe google, i' send a ask for help my cartography teachers - cuz i am a student of geology. Maybe some of u may help me as well in that?

[felipeou]

Hi friends, could you please tell me if I can follow the same procedures that you guys are following to Europe Geodatas for me here in Brazil using OpenStreet Maps?

If so, what should I do?
Thanks a lot

[Nieko]

If www.openstreetmap.org has data for Brazil (or a particular city there), then you can use these scripts (you need to modify them of course) to get something going: http://stuff.nieko.net/psp/mapthis/geodata/.

For this to work you'll need quite some programming knowledge though.

[felipeou]

THanks for the answer
But how do I know IF openstreet has data from my Country / City?
It finds streets and places when I search over there, that show that it has data about this place? thanks

[Nieko]

The bunch of scripts and particularly process.txt in process.zip explain exactly how to run everything. Also, the file you downloaded, when unzipped, contains all the data in XML format, so you can search that for a particular street too.

If this doesn't get you going, I'm afraid you won't get it working, because quite some programming knowledge is required.