GPSFS Utility Program

[MIB.42]

Here is the first beta (v0.1) release of GPSFS_util.

As I was hinting it for some time now, it can analyze and recompress as well as omit tiles/levels etc...
( Like you have a 2GB map, but for sure you are not using some parts of it as they are things like
ocean or mountain areas with no roads and you are not a hiking-kind-a-guy or shady parts of town
which you will never ever visit etc... ), so you would love to save some memorystick space by removing
all those "never to be used" tiles, individually. Or, say, you downloaded a map of Paris and turns out
that your map is 9.2GB and you don't have a memorystick which can hold all of it and you are looking
for ways to reduce the size to 300MB even if it involves removing entire levels and sacrificing quality.
GPSFS_util can do all of those...


Double click on the window title bar for detailed how-tos...

Cheers,
MIB.42

PS: You may need to copy the jpeg62.dll to your C:\WINDOWS directory or where you put the GPSFS_util progam...
Let me know if you have trouble running it...

The below GPSFS_util.zip file is outdated and is here only for archival reasons...

[DogRed9]

After it gets to about 36000 files it slows down... and keep slowing down and at this rate, will never finish, might as well re-download the whole map?

[DogRed9]

Ok, well it finished and I got rid of the biggest zoom, but the new map wont load now, any clue why?

[MIB.42]

Ok, well it finished and I got rid of the biggest zoom, but the new map wont load now, any clue why?

Won't load where? in MapThis!?

I assume you renamed the resulting files from N_GPSFS... to GPSFS... ?

Try loading it in GPSFS_util... What does it say?

[MIB.42]

After it gets to about 36000 files it slows down... and keep slowing down and at this rate, will never finish, might as well re-download the whole map?

It processes faster, when the tiles are empty or simple. If the tiles have a lot of graphics, it "slows down" of course, as it needs to process those.

I checked for memory leaks ( which usually leads for slowing down ) and it seems to be fairly robust.

It processed one of my maps with 10 levels and 1398100 tiles in about 6 hours without leaking a single byte, but - being a software - one can never know what weird combinations someone may throw at you...

That's why it's beta... But it also seems that it was OK for you eventually, right? As in it did not crash...

[DogRed9]

I assume you renamed the resulting files from N_GPSFS... to GPSFS... ?
I feel like such an idiot lol... I even read that in the help file and... just forgot to rename them, now it loads fine.
I dont have my GPS cable yet so I coudlnt test this, but when you do use it, will it try to zoom in all the way and end up at the no tile images?
And I know this would be alot more complicated, but a suggestion for the program is to allow for previewing, with those checkboxes. So, you could download all the maps for the area, then open them in this and make more refined ones of just what you need each time. Oh and the preview in the program of the tiles never worked it was just solid colors of one form or another.

[MIB.42]

I feel like such an idiot lol... I even read that in the help file and... just forgot to rename them, now it loads fine.
I dont have my GPS cable yet so I coudlnt test this, but when you do use it, will it try to zoom in all the way and end up at the no tile images?
And I know this would be alot more complicated, but a suggestion for the program is to allow for previewing, with those checkboxes. So, you could download all the maps for the area, then open them in this and make more refined ones of just what you need each time. Oh and the preview in the program of the tiles never worked it was just solid colors of one form or another.

v0.2 will have a little preview... I am working on it...

solid colors mean that the tile is either missing or have problems. Double click on the tile and it will pop-up a status report...

[Codaz]

Nice, GREAT work MIB ! Is there any GZLIB or other LIB compression possible in the future ?

Because maps can be MUCH smaller. I don't care if map loading takes half a minute, if maps are significant smaller (150-200% compression) than i prefer compression.

[Codaz]

I noticed some bugs.

1 If you open a GPSFS* file, eg GPSFS 1 2 3 4 etc it says TOO MANY tiles. That is strange.

2 If you analyse a file, and then accidentally select OPEN FILE, you can't go back to the program because it freezes.

I had to CTRL ALT DEL to exit it.

Error:
LOCALS~1\Temp\WERfbc6.dir00\GPSFS_util.exe.mdmp

3 Also why can't i delete level 0 ? I don't like em. Because it's a picture from half of the city and i can't see anything usefull on it.

4 And there are level 1 tiles in my map (so they fill up space !) but they are empty ? So can you delete them ?

5. The CTRL + A function doesn't work for selecting all tiles at once, can that be fixed ?

6. My _MAP directory is 7 mb large, but the Util says: No Recompress, map size is 9.44 mb. Why is that ?

7. I'm working on a laptop. For heat issues i downclock my CPU to 800 mhz. If i let it run at normal speed (1.73 GHZ) the ETA isn't correctly updated. It goes down yes, but my CPU is working twice as fast so it must be done sooner. The analysing is done faster so why isn't the ETA updated ?

VERY USEFULL PROGRAM, the faster your pc is the faster the program goes !

If you implement the feature from GMDL that checks if the tiles arent broken, and pops up to redownload them, it would sure be a real all-in-one map solution.

Or maybe you can give the code from your program to In7ane so he can implement this in his tool.

Last thing, i created a simple BAT file for you guys that can rename the N_GPSFS to normal files, have fun !

[sl1pkn07]

if possible porting this progran for running in linux?

[MIB.42]

Thanks Codaz, this is exactly the feedback I was looking for! Let's see :



1 If you open a GPSFS* file, eg GPSFS 1 2 3 4 etc it says TOO MANY tiles. That is strange.


Only the GPSFS ( and NOT! the GPSFS1/2/2/3/4 ) files have the information on the tiles. GPSFS1/2/3/4 files simply contain the image dumps.

At the moment I am not doing any super extra error checkings to see if you "obeyed" the basic rule of "Select the GPSFS file only"...



2 If you analyse a file, and then accidentally select OPEN FILE, you can't go back to the program because it freezes.


I tried this several times and saw no freezing. Altough it takes quite some time for windows to delete/reinit the list box, especially if there were throusands of entries. Be patient...



3 Also why can't i delete level 0 ? I don't like em. Because it's a picture from half of the city and i can't see anything usefull on it.


Level 0 is the root map. It does not make sense deleting it. It also is the smallest, only 4 tiles.



4 And there are level 1 tiles in my map (so they fill up space !) but they are empty ? So can you delete them ?


Empty maps do not occupy any image space, but are important for map logistics. They only occupy a mere 8 bytes each in GPSFS.



5. The CTRL + A function doesn't work for selecting all tiles at once, can that be fixed ?


I'll look into that. Why would you want to select all tiles?



6. My _MAP directory is 7 mb large, but the Util says: No Recompress, map size is 9.44 mb. Why is that ?


It is very likely a PNG based map. v0.1 does not have PNG-recompression capability. It will recompress PNG based maps in JPG. To have roughly the same image quality, it would require that many bytes. I will fix this in probably v0.3 or so. Meanwhile, I found that if you go around 70~80% of your original PNG based files, it still produces a pretty decent quality files...



7. I'm working on a laptop. For heat issues i downclock my CPU to 800 mhz. If i let it run at normal speed (1.73 GHZ) the ETA isn't correctly updated. It goes down yes, but my CPU is working twice as fast so it must be done sooner. The analysing is done faster so why isn't the ETA updated ?


ETA is almost impossible to compute precisely. I wanted to omit it altogether, but decided to leave it. Now I am thinking it was a mistake. The reason is, that if the maps have a lot of "holes" as in empty tiles or missing levels, there is no recompression necessary, thus it speeds up. At the moment the ETA is a very dumb; "How many tiles have I done in the past x seconds, thus I'd need y seconds more to do all of them"... I might just do a pre-run phase to do a pre-analysis on ETA as well, but that might take a few seconds, which will come from the analysis time... I wasn't sure if it was worth it... Based on your feedback, it surely is now.



VERY USEFULL PROGRAM, the faster your pc is the faster the program goes !

If you implement the feature from GMDL that checks if the tiles arent broken, and pops up to redownload them, it would sure be a real all-in-one map solution.

Or maybe you can give the code from your program to In7ane so he can implement this in his tool.

I hear you on the missing tile download! As far as I know GMDL already has that feature...
GPSFS_util is a simple post-processing program with some extra features. The real hard-work is in GMDL!

[MIB.42]

Nice, GREAT work MIB ! Is there any GZLIB or other LIB compression possible in the future ?

Because maps can be MUCH smaller. I don't care if map loading takes half a minute, if maps are significant smaller (150-200% compression) than i prefer compression.

I've looked at this in the past. As you probably know, both JPG and PNG files already have heavy compressions. After I applied aggressive zlib compression, I got 16% compression on PNG based maps and 3% for JPG based maps. And this was for compressing the whole GPSFS file and NOT individual tiles. If I did individual tile compression, the result was less than half of this due to per-tile compression table overheads.

The CPU load on the PSP for decompressing (inflating) was significant, thus battery life could be reduced as much as 30~40 percent! I don't think it's worth it...

Cheers,
MIB.42

[MIB.42]

if possible porting this progran for running in linux?

Oh man, it took me 2 weeks to port my initial code from Linux to Windows... While the core still works under Linux, the UI won't. I'd have to go back and rewrite the .RC based resouce management... I'd rather spend my time on feature development... *sigh*
( Does GMDL run under Linux? )

[Codaz]

Ok great ! What about grey-scaling the images or going to 16-bit color ?

O before i forgot, the feedback on your program took me a whole half our to type and think about :)

And why did you choose JPG or PNG ? Maybe GIF/TIFF/ JAS could be smaller.

JAS seems to be VERY usefull.

http://www.why-not.com/articles/formats.htm#JAS

Only problem is the psp can't read JAS files.

O and about the Linux problem, you CAN run Windows programs on linux running WINE.

[MIB.42]

Last thing, i created a simple BAT file for you guys that can rename the N_GPSFS to normal files, have fun !

Suggestion on naming the resulted files are welcome!
- I thought of renaming the original GPSFS files to O_GPSFS, but I thought if you want to try several recompressions, this might be awkward...
- I thought of creating a subdirectory like RECOMPRESSED_date and put the new files in there, but had some possible directory creation problems due to access control sometimes... I might just revisit this one...

Any ideas, suggestions?

[MIB.42]

Ok great ! What about grey-scaling the images or going to 16-bit color ?


For v0.3. I plan on adding color-reduction to 16 color for PNGs. Haven't thought of the 16-bit reduction for JPGs. Good idea! I'll look into it. Thanks!


O before i forgot, the feedback on your program took me a whole half our to type and think about :)

Much obliged! Keep'em comin'!

Cheers,
MIB.42

[MIB.42]

And why did you choose JPG or PNG ? Maybe GIF/TIFF/ JAS could be smaller.

JAS seems to be VERY usefull.

http://www.why-not.com/articles/formats.htm#JAS

Only problem is the psp can't read JAS files.


PNG - generally - has better compression than any of the above. It most certainly is better than TIFF and/or GIF. Not sure about JAS, but I'd be really surprised if it was otherwise... JPGs have excellent compression - better even than PNG - on "real" imagery, like satellite photos etc. due to it's lossy approach...


O and about the Linux problem, you CAN run Windows programs on linux running WINE.

You're right. Also see Lindows / Linspire...

[Codaz]

At what map size is your tool able to recompress it. I have now 5 test maps which are 10 mb of size.

On ALL of those 5 your util wasn't able to recompress to a smaller size.

Only on my large map of 300 mb, but it takes until tomorrow morning to analyze that :S

Your util should be able to remove levels BEFORE it analyzes them :)

Last addition for 2night: maybe you can add image ripple to the jpegs. If they are loaded they appear as "blocky" but if they are fully loaded (eg if you drive near them) they are displayed fine.

Now i have the idea that Mapthis is loading my 300 mb map in total and that's not good.

[MIB.42]

At what map size is your tool able to recompress it. I have now 5 test maps which are 10 mb of size.

On ALL of those 5 your util wasn't able to recompress to a smaller size.

Only on my large map of 300 mb, but it takes until tomorrow morning to analyze that :S

I am a bit surprised on those 10MB ones. I have several ones as well and one - which is 6MB - got recompressed to 3MB!

Are your files PNG based ones?

[Codaz]

Yes PNG. So waiting for 0.2 or 0.3 then :)

BUT befor i forget:
those are old maps, created with GMDL .011 or something. I can recreate JPEG maps with GMDL.

Is JPEG better or PNG ?

I want smalles images possible.

[Codaz]

Last error:

GMDL_util.exe:

NTDLL.DLL error, application is hung.

And why isn't level 6 present in my small maps.

E.G:
I have a large map, with levels 1 to 4. I remove level 4 with your tool.
If i reopen the NEWLY generated, level 4 is present as EMPTY tile. It's for the filestructure you say.

But level 6 ? doesn't that need a file structure ?

LOL I forgot to enter the compression rate, WHAT A difference.

But improve the slider bar. It's now VERY unclear that you can control the amount of compression with that. YES it's in the readme, but your prog is so small and the readme so big that even I forgot to apply compression.

Now that i've done that:

Map that was 7 mb to 1 mb. Still readable.

But i wonder how long the 300 mb map takes to do. Maybe 2 days, LOL

[MIB.42]

Last error:

GMDL_util.exe:

NTDLL.DLL error, application is hung.


I'd love to have that map!




And why isn't level 6 present in my small maps.

E.G:
I have a large map, with levels 1 to 4. I remove level 4 with your tool.
If i reopen the NEWLY generated, level 4 is present as EMPTY tile. It's for the filestructure you say.

But level 6 ? doesn't that need a file structure ?



I don't quite understand this. Do you have a level 6 in that original map? GPSFS_util can only remove tiles from levels, but not add.




LOL I forgot to enter the compression rate, WHAT A difference.

But improve the slider bar. It's now VERY unclear that you can control the amount of compression with that. YES it's in the readme, but your prog is so small and the readme so big that even I forgot to apply compression.

Now that i've done that:

Map that was 7 mb to 1 mb. Still readable.

Excellent. Yeah, I crammed quite a few things in there... I wanted to keep the program small and have the controls adopt to situations only available at the given moment...

FYI, After the analysis is done, you can select with the scroll bar on the right the recompression ratio. Way left is no-recompression ( as in it will store the original jpg files without touching them or in PNG's case it recompresses them at 78% ratio to JPG ), as you start moving the silder to the right, it shows recompression result between 20% and 80% JPG quality range.



But i wonder how long the 300 mb map takes to do. Maybe 2 days, LOL


One of my 410MB GPSFS took 8 hours for analysis and 2 hours to save on my core-solo...

[Codaz]

No it only shows file size that the map will be, not compression rate :)

And with the NTDLL.DLL error, it seems that your program is eating a LOT of CPU. 100% occurs.

If cpu power is reserved for e.g Torrent downloading, your program is hungry for power, if it can't get that it will stop and produce an error.

But it works now.

What programming language is it developped in ? Because i downloaded Notepad++ (very wide reading and developing program) but it can't really read your program.

[in7ane]

MIB.42, first of all excellent app! Thanks for writing this, it's exactly how (I feel) post processing of maps should be done (for various reasons, post processing downloaded tiles too much in GMDL is not a good idea).

Anyway, some suggestions, feel free to ignore them though:

Placing the dll into the same directory as the exe seems to work as well.

Select the folder with the GPSFS files rather than the file itself (i.e. just the parent dir of whatever file was selected) then you just get GPSFS inside that, avoiding the problem of people selecting GPSFS1,2,3,4.

Considering the above, or even as is, I would vote for placing the compressed GPSFS files into a separate folder like _map_name_date

ETA, it may be possible to read in the index quickly to just count of used tiles - if so, count the total/skipped number of tiles, then after encoding the first 100 or so actual tiles do the calculation based on time taken for that and the number of actual tiles from the index (but like you said, reading in the index will take a few seconds).

Compression settings, and stats, maybe have an option which compresses a random set of tiles (for example the first 100 used) using various settings (PNG 16, JPEG 25%, 50%, etc.) and gives an estimate of the resulting compressed GPSFS (and a sample pre/post tile).

[Codaz]

Now that's a good feedback you got on your first day program release MIB.42 :)

Also because i can't stop reading these forums:

Add a little sound (the windows bling/beep/buzz) that beeps when analysing or recompression is done. Because i don't think you will be staring at the recompression window for 4-5 hours :)

Last thing is recompression ETA is presented in minutes not in hours left. Fix that plz.

Hope to see the next version this week/month :)

[MIB.42]

No it only shows file size that the map will be, not compression rate :)

And with the NTDLL.DLL error, it seems that your program is eating a LOT of CPU. 100% occurs.

If cpu power is reserved for e.g Torrent downloading, your program is hungry for power, if it can't get that it will stop and produce an error.

But it works now.

What programming language is it developped in ? Because i downloaded Notepad++ (very wide reading and developing program) but it can't really read your program.

I most certainly is a CPU hungry application!
That's an interesting comment on when producing an error. I don't quite know what to do with that, as this seems to be an NT issue...

I develop in C/C++, but I only publish the executable for now...

Cheers,
MIB.42

[MIB.42]

MIB.42, first of all excellent app! Thanks for writing this, it's exactly how (I feel) post processing of maps should be done (for various reasons, post processing downloaded tiles too much in GMDL is not a good idea).


I agree, GMDL is already quite complex and does the most important work! That's why instead of bugging you with adding new features, I developed this code. I plan on expanding it a bit more, mostly towards having a better tile browser and PNG recompress functionality.

I hope to see this code as part of the GMDL suite... :)


Placing the dll into the same directory as the exe seems to work as well.


Ok, I will try that, thanks. I had other dll's as well, but managed to rewrite some parts for zlib/png, so no dll's are necessary any more. I doubt I reimplement JPG de/comp. I might actually go back to the png/zlib dll's as well, as decompress of png is quite simple, but compression is a different kinda animal...


Select the folder with the GPSFS files rather than the file itself (i.e. just the parent dir of whatever file was selected) then you just get GPSFS inside that, avoiding the problem of people selecting GPSFS1,2,3,4.


Yeah, I know... That's lack of my Windows programming skills... I use the OpenFileDialog box... Help is welcome... :)


Considering the above, or even as is, I would vote for placing the compressed GPSFS files into a separate folder like _map_name_date


Yeah, you're right, I will do that in the next release.


ETA, it may be possible to read in the index quickly to just count of used tiles - if so, count the total/skipped number of tiles, then after encoding the first 100 or so actual tiles do the calculation based on time taken for that and the number of actual tiles from the index (but like you said, reading in the index will take a few seconds).


That's exactly what I plan on doing. I will split the ETA computation into "empty" and "normal" and compute the result accordingly...


Compression settings, and stats, maybe have an option which compresses a random set of tiles (for example the first 100 used) using various settings (PNG 16, JPEG 25%, 50%, etc.) and gives an estimate of the resulting compressed GPSFS (and a sample pre/post tile).

At the moment during the analysis phase I compute 20%, 30%, 40%, 50%, 60%, 70% and 80% JPEG quality compression sizes and interpolate between when you move the slider. The slide basically a simple quality setting slider in the 0~80% range, where 0~19% is interpreted as "No recompression".

I will do the png 16 color recompress at v0.3 and look at the 16bit JPG as well, probably even for v0.2.

Thanks for your suggestions!

MIB.42

[MIB.42]

Now that's a good feedback you got on your first day program release MIB.42 :)

Also because i can't stop reading these forums:

Add a little sound (the windows bling/beep/buzz) that beeps when analysing or recompression is done. Because i don't think you will be staring at the recompression window for 4-5 hours :)

Last thing is recompression ETA is presented in minutes not in hours left. Fix that plz.

Hope to see the next version this week/month :)

Cool, those are great ideas. Thanks!

[sl1pkn07]

tanks

(sorry my very poor english)

[Codaz]

Ok MIB.42 this is for your investigation.

This map was originally PNG based and 9.44 mb big.

Now it's compressed to JPG maximum compression (78% ?)

It doesn't work on Mapthis .99 original.

If i load it into mapthis it says please wait while loading and then freezes up the psp and turns it off.

I have 3.10OE-A and 2.71 and 3.03 PRX.

[sl1pkn07]

( Does GMDL run under Linux? )

it does not work as it must (red screen)

http://sl1pkn07.no-ip.com/imagenes/gpsfs.png

I prefer that he is native

sorry my bad english)

[MIB.42]

it does not work as it must (red screen)

I prefer that he is native

sorry my bad english)

Red screen means problem with reading...
Does it do the same with non-empty tiles?
( You were looking at an empty tile ( it said EMP )...

[MIB.42]

Info...

GPSFS_util already compresses the JPG images at it's highest compression rate ( besides the quality setting ). I use the 420 encoding. When I change to 422, I get 32% size growth!
Also, jfyi, there is no "16bit jpeg"...

So, it would seem, that GPSFS_util is already compressing the jpg images at the maximum setting... No improvement can be done on this part.

Moving on from the 16bit jpg topic...

[sl1pkn07]

Red screen means problem with reading...
Does it do the same with non-empty tiles?
( You were looking at an empty tile ( it said EMP )...

yes, all, EMP and ON tiles

[Codaz]

MIB did you investigate the map i provided that didn't work ?

[MIB.42]

MIB did you investigate the map i provided that didn't work ?

Oh yes, it works for me. Try deleting the map_pref.txt file from the directory...
Mine starts with these parameters withouth any problem : 7488 7388 5

Again, I tried it with my MapThis (v0.4.99) on the PSP...

[in7ane]

MIB.42, about integration with GMDL - I've done a mock up that calls GPSFS_util and passes the full path to the map directory after the map is downloaded: http://www.in7ane.com/psp/gmdl0.4.17a.zip

rename
\Debug\GPSFS_util REMOVE THIS FOR GMDL TO CALL
to
\Debug\GPSFS_util
then \Debug\GPSFS_util\GPSFS_util.exe is called

[FatesWebb]

I am running .1 and it always errors out, is it because my files are too big? it is a 2gb hybrid map I am trying to load. I get the prompt to send the dump to microsoft usually torward the end of the initial loading of the files.

[MIB.42]

I am running .1 and it always errors out, is it because my files are too big? it is a 2gb hybrid map I am trying to load. I get the prompt to send the dump to microsoft usually torward the end of the initial loading of the files.

Hmmm, I only have tried this with sub-1G, I am going to test it with my 3GB now... Will let you know... The new version is coming in a few days...

[FatesWebb]

FYI I have 512 mb on this machine so it isnt a workhorse... I can check out the memory dump if you need. I do have access to full blown racked servers if I need to test it on a workhorse.

FW

[MIB.42]

FYI I have 512 mb on this machine so it isnt a workhorse... I can check out the memory dump if you need. I do have access to full blown racked servers if I need to test it on a workhorse.

FW

I just tried a 3GB map with 349524 tiles, it loaded it quite nicely and is currently analysing, but I see no memory footprint increase. I spent extra efforts in making sure, that there is no memory leak...

Anyways, for this 3GB map, my memory footprint was around 30MB only... Can you check how much free memory you have? My favourite tool for it is from http://www.prcview.com

Anyways, can you upload your map somewhere, so I can have a look?

[FatesWebb]

I just tried a 3GB map with 349524 tiles, it loaded it quite nicely and is currently analysing, but I see no memory footprint increase. I spent extra efforts in making sure, that there is no memory leak...

Anyways, for this 3GB map, my memory footprint was around 30MB only... Can you check how much free memory you have? My favourite tool for it is from http://www.prcview.com

Anyways, can you upload your map somewhere, so I can have a look?

OK, so I am going to copy it to a racked server with a Xeon 2.8GHZ and 4gb ram, and test it there, here is a copy of it if you want to test it yourself. (sending a private message with the location) I will take it down after you get it, because this is not a permanent place to host this file.

[FatesWebb]

ok so I have loaded it on the racked server, and it does not crash where it did on this machine. It is analyzing. I am not sure about the steps to take to use the program.

Do I have it correct that it will attempt to re-compress the images to make the map smaller? is there a good tutorial on what to do once you have loaded the map? I clicked analyze but I am not sure if there is more to it than that.

Do you want me to continue chasing why it would not run on this machine/reboot and retry or just use a beefier box. it is a 2.6GHZ running winxp and 500mb ram...

FW

[MIB.42]

ok so I have loaded it on the racked server, and it does not crash where it did on this machine. It is analyzing. I am not sure about the steps to take to use the program.

Do I have it correct that it will attempt to re-compress the images to make the map smaller? is there a good tutorial on what to do once you have loaded the map? I clicked analyze but I am not sure if there is more to it than that.

Do you want me to continue chasing why it would not run on this machine/reboot and retry or just use a beefier box. it is a 2.6GHZ running winxp and 500mb ram...

FW

Double Click the window title bar for instructions. It even says so! ;-)

I have made lots of modifications already, so it wouldn't help if we chase this with the v0.1...

Thanks.

[FatesWebb]

cool, trying it now..

what is your experience? that max compression is the way to go? or does it cause too many artifacts?

Also when it says emp, that is empty, so seemingly useless....

but on/off does off mean it removes it from the compression that the util does? or that it removes it from the archive altogether? .....(EDIT) ok wait I think I figured that one out by reading this thread, it means disable it from the processing? because I see in this thread where it says that EMPs only take up 8 bytes anyways(/EDIT)

testing with max compression to see if I like it that way...

This is fun, thanks!

FW

[FatesWebb]

Hmm, I think something went wrong, after running it a full compression here is what I was left with.... It looks like it started to work, but somehow made 4 too big, and the files I ended up with are just as big as the source files...

http://img299.imageshack.us/img299/3864/screen1sf6.jpg


http://img299.imageshack.us/img299/3593/screen2ma4.jpg

[MIB.42]

cool, trying it now..

what is your experience? that max compression is the way to go? or does it cause too many artifacts?


I usually go with compressions at around 40~50% of the slider ( translates to ~65% JPG )


Also when it says emp, that is empty, so seemingly useless....


It simply shows, that there is no title assosicated with a given position. Quite an important feature for analysis, but useless for recompression. GPSFS_util is not a recompression tool only, but an analysis one as well...


but on/off does off mean it removes it from the compression that the util does? or that it removes it from the archive altogether? .....(EDIT) ok wait I think I figured that one out by reading this thread, it means disable it from the processing? because I see in this thread where it says that EMPs only take up 8 bytes anyways(/EDIT)


Yes, when you turn off a tile, that tile will be omitted from the resulting conversion. This in turn will produce and EMPTY tile reference in the newly generated GPSFS files.

Also, if your original GPSFS was a PNG based file ( it will show after you load it in the information bar on the bottom left, recompression is not always good, as conversion from PNG to JPG can vary quite much...
v0.3 will have PNG recompression to PNG...

I am almost done with v0.2, I am only running tests now. It is a major improvement... I plan on releasing it within 24hours.

Cheers.

[Codaz]

When does version 3 come. I hope PNG compression is better.

Within some days i have 1 gb memory, no more 512 mb so i hope analyzing goes a lot faster.

[MIB.42]

v0.2 is attached.

v0.2 changes :
- By popular demand; Evaluated 16bit JPG compression, no benefit...
- Implemented 'FastPass' for jpg compressions.
Improvement on analysis time is ~20~30%, but reduces size prediction. Still worth it...
- Select map FOLDER instead of the GPSFS file
- Multi-tile preview window with navigations ( control buttons and mouse )
- Better logistics on task interruptions
- Play sound after analysis / convert is done
- Split the ETA computation into "empty" and "normal" tiles and compute the result accordingly
to provide more accurate ETA. This is still not 100% accurate, as it would require more system resources
which I prefer using on speeding up the conversion process instead...
- Added hour on top of the existing minutes:seconds in ETA


Again,

The following text can also be accessed by clicking on the title bar of the window.

Usage :

1. Click on [Open] and select the map directory where the GPSFS files are.
2. Enable/disable tiles, levels...
[Levels] for omitting zoom-levels.
[On/Off] to enable/disable (turn on/off) individually selected tiles.
( multi selection is available in the listbox )
3. Click on [Analyze] - May take quite some time... Enabling [No Visual] can speed it up...
4. Adjust compression settings ( after completed analysis only )
Scrollbar : choose approximate target size (compression ratio setting).
Enabling/disabling selected tiles/levels will reduce size approximations. Re-Analyze helps...
5. Click on [Create] and choose/create a folder where the new map will be stored.
- Again, might take a while, but not as long as the analysis...

Others:
'Re-Analyze' is there if you enabled/disabled several tiles ( changed lots of settings ) and already
had an analysis done before. ( This will guarantee a more precise size approximation on the files to
be created... )

Navigation :
Besides the obvious buttons in the 'Navigation' area ( I/O = Zoom In/Out, U/D/R/L = Move Up/Down/Right/Left )
there are mouse controls as well :
- Clicking with the left button on the tiles around the central tile will result in repositioning the tiles
( move to the selected direction )
- Clicking with the left button on the central tile will toggle its status (enabled/disabled).
- Clicking with the right button on the central tile will pop up a detailed information on the tile.
- Clicking with the right button on the upper tiles and tile to the right from the central one will zoom in.
- Clicking with the right button on the lower tiles and tile to the left from the central one will zoom out.

Limitations :
- Only JPG based GPSFS will be created, PNG based ones will be converted to JPGs. (v0.3 will have PNG recompression )
- Be patient, it may take a long time to do the proper analysis...
- If there are insane amount of tiles, the listbox will be disabled.
( Individual tile selection can be done with the scrollbar / navigations )
- After the analysis is done, you can elect to create several outputs by creating maps into different directories...
The analysis has to be done once only, while creation with different settings can be done multiple times...
- Completion time predictions are very rough and can be inaccurate if a map has lots of missing tiles.
( A more precise prediction would consume a lot more system resources... )

Cheers,
MIB.42

[Codaz]

Great ! Many of my additions have been added :)

But the 16 bit addition, why doesn't it have benefit ?
Also i like the Fastpass !!

Is PNG to JPG conversion bad for file size ? Because i think JPG is larger than PNG.

I don't like the levels 1 to 10 placed under eachother, make that like version 0.1 it saves space on your desktop !

But there's a BIG bug in 0.2

I analysed the same old map i did in 0.1 and it was there 9.44 mb. If i analyse it in 0.2 it becomes 10.21 mb !

Second BIG bug:

What is this after Analyze ?

http://img522.imageshack.us/img522/5919/analyzeerror02ad5.png

Not displayed correctly !

Third:

If i have recursion in map dirs EG:

C:\Maps\_blabla

and

C:\Maps\_blabla\_newrecompressed

the software takes em both OR can't analyze it.

[MIB.42]

Great ! Many of my additions have been added :)


I am trying to listen to you guys... :)


But the 16 bit addition, why doesn't it have benefit ?


Apparently there is no such thing as a 16bit JPG. :) All channels are encoded on 8bits. So, even if I convert the pics to 16bit, they get expanded out to 24 during the conversion. There is a way to rewrite the encoder to support 16bit, but that we'd need to also rewrite decoder on the PSP/GMDL etc... It's just not worth the hassle...


Also i like the Fastpass !!


Yeah, that was a major addition. Unfortunately the approximation on the final filesize ( which is done with the slow pass to create better quality and smaller images ) can be different up to 20%, which results sometimes in an uneven filesizes between the GPSFS files... It still does work, but not as pretty... :p


Is PNG to JPG conversion bad for file size ? Because i think JPG is larger than PNG.

Actually, it's not that bad, but you loose a bit of quality. I will run some test on the 4bit/8bit PNG conversions later...


I don't like the levels 1 to 10 placed under eachother, make that like version 0.1 it saves space on your desktop !


Since the preview window is already big, it actually makes a lot of sense to put them as such... I am open to redesign the look&feel of the app... :)



But there's a BIG bug in 0.2

I analysed the same old map i did in 0.1 and it was there 9.44 mb. If i analyse it in 0.2 it becomes 10.21 mb !


This is not a bug. v0.2 provides a different result for analysis due to the fastpass phase. What matters is that your final filesize is the same. See above and README.txt


Second BIG bug:

What is this after Analyze ?

[Image removed to save space, see above]

Not displayed correctly !


Well, I think it's actually a problem with your map! Check the stats on the individual tiles on it's right and left and compare them if they are next to each other ( as in click right mouse button on the central tile, jot down the filesystem and offset values, than move to the next one...) See README.txt for more info on how to do this...




Third:

If i have recursion in map dirs EG:

C:\Maps\_blabla

and

C:\Maps\_blabla\_newrecompressed

the software takes em both OR can't analyze it.

Now this is news to me. I have several folders with subfolders in them and it works just perfectly. Please re-check and describe your problem more precisely.

Thanks,
MIB.42

[Codaz]

Hmm that doesn't solve the problem.

And how to disable/enable the fastpass ?

[MIB.42]

Hmm that doesn't solve the problem.

What problem?


And how to disable/enable the fastpass ?

Can't at the moment. Why would you want it?

[Codaz]

Because it may cause the error of displaying that ugly tile at the wrong place.

The tile IS enabled, in right and left screen it is correct.

Also i got the re-encoded one now working on Mapthis !

-Removed the prefs file.
-Added more POI icons in that _map dir

It works now but it's a bit ugly on the small psp screen.

[MIB.42]

Because it may cause the error of displaying that ugly tile at the wrong place.


FastPass does NOT touch the decompression part. This "bug" is due to wrong offset/filesystem, which can be either in the original map file, or in the processing of GPSFS_util.


The tile IS enabled, in right and left screen it is correct.


If the tile is shown (and not red), that means it is enabled. That wasn't the question. I can also see, that the tiles on the right and left are correct, that wasn't the question either. The question was if they represent an uninterrupted sequence in the GPSFS file.

Please give me the infos ( as I described earlier... right mouse button ) on the left, center and right tiles and I will let you know if they are indeed correct.


It works now but it's a bit ugly on the small psp screen.

Not sure what this means... You are probably referring to the recompressed file being lower in quality... Yes, it is. There is no "free lunch"... :)

[Codaz]

It's better that i give you the map, it's only 10 mb so i will upload it tomorrow.

I'm not a genius so that's maybe the best option.

It's an original map from GMDL.

EDIT: HERE IS IT:

[meier3283]

Hey MIB.42, I've got .2 running as we speak, and so far so good. I was wondering... For the SMP (dual core) users out there, how hard is it to make the app SMP aware? It's always painful to watch a process take up only one core :)

[MIB.42]

Hey MIB.42, I've got .2 running as we speak, and so far so good. I was wondering... For the SMP (dual core) users out there, how hard is it to make the app SMP aware? It's always painful to watch a process take up only one core :)

Hah, I hear you... Ok, I'll have a look at it...

[MIB.42]

It's better that i give you the map, it's only 10 mb so i will upload it tomorrow.

I'm not a genius so that's maybe the best option.

It's an original map from GMDL.

EDIT: HERE IS IT:

It works for me, no problem. Tile #27 is perfectly fitting in. #26 is 0:100813:7168, #27 is 0:107981:4859 and #28 is 0:112840:8879 ( FileSystem:Offset:Size )

[Codaz]

MIB.42 why do i get the weird thing i posted in the screenshot then ? The map is correct you say !

And is 0.2 already capable of working on pentium 4 HT technology ? Because HT is somewhat also 2 cores.

[FatesWebb]

ok, so I did this, it shrunk my 2gb map to 1gb, and I placed it on the mestick, and it just hangs with the memory card flashing, and says "please wait while map is loading. I used v02

[Codaz]

Same as mine. Fix the icons in the directory and it works !

Some poi icons are missing so that's why it's chrasing.

[MIB.42]

MIB.42 why do i get the weird thing i posted in the screenshot then ? The map is correct you say !

And is 0.2 already capable of working on pentium 4 HT technology ? Because HT is somewhat also 2 cores.

Sorry, you were right, I found the bug... It was only happening on PNGs after Analyze... It's fixed now.

Other improvements in v0.2b (attached) is that you can select to reconvert PNG based ones to JPGs or not. This is really useful when you are only removing tiles from a bigger map and want to keep the map in PNGs.

There is also a new functionality, you can disable more than one tile per click ( if you use SHIFT ). It will disable all underlying tiles as well on different zoom levels recursively...

Cheers,
MIB.42

[Bolemichel]

Hi MIB,

I've read this thread since beginning but until now I don't understand for what your program is good for.

May be you could edit your post #1 with a little explanation what this tool is doing. You could also edit it with version 0.2.

This would help very much and would prevent from silly questions like mine :)

Thank you!
michel

[MIB.42]

Hi MIB,

I've read this thread since beginning but until now I don't understand for what your program is good for.

May be you could edit your post #1 with a little explanation what this tool is doing. You could also edit it with version 0.2.

This would help very much and would prevent from silly questions like mine :)

Thank you!
michel

Ok, I updated post#1 with "examples"... ;)

[Bolemichel]

Thanks a lot! Just finished connecting my holux 236 with the psp and may be this little tool could something good for me ;)
For my understanding it could validate and compress your files on the memstick.

[MIB.42]

Just a blog :

I did some tests, moving from the default 256 color pngs to 16 will result in a ~42% size reduction... Altough converting the pngs down to 16 color can take quite some time, in some test cases it did take about 3 seconds per tile on my machine... Imagine my 1.4 million tiles map... That's over a month continuous work! I gotta do something to speed this up...

[Codaz]

Yup, as Fateswebb says: He shrunk a map from 2gb to 1gb. That's taking now more than a day.

[Bolemichel]

Sounds good! I'm searching for a way to delete the 1x zoom levels. When I drive by car I don't need such detailed graphics.

Is it possible to delete this 1x zoom level with your tool or is there a way to deactivate it in GDML download tool? The second would be probably better to speed up the download...

[FatesWebb]

ok, I am testing out the 2b version, I analyze, but then when I slide the slider bar over to the right to choose a compression level, it goes from No Recompress, Size = 2236.09 MB to Recompress, Size = 2236.09 MB

It is telling me it is not possible to compress this any smaller? even though it is full size?

Thanks...

FW

[Codaz]

Sounds good! I'm searching for a way to delete the 1x zoom levels. When I drive by car I don't need such detailed graphics.

Is it possible to delete this 1x zoom level with your tool or is there a way to deactivate it in GDML download tool? The second would be probably better to speed up the download...

The highest levels must be enabled to form a map in total. The lowest levels (eg 1 ) can be disabled in GMDL under advanced settings.

[MIB.42]

ok, I am testing out the 2b version, I analyze, but then when I slide the slider bar over to the right to choose a compression level, it goes from No Recompress, Size = 2236.09 MB to Recompress, Size = 2236.09 MB

It is telling me it is not possible to compress this any smaller? even though it is full size?

Thanks...

FW

That means that the map is PNG based and 16 color recompression is not done yet... Click on the "Force JPG" to convert the PNGs to JPGs which will give you compression...

[Codaz]

When is version 0.3 coming :) ? Can't wait for it !

Hope to see some improvements and bugfixes !

[MIB.42]

When is version 0.3 coming :) ? Can't wait for it !

Hope to see some improvements and bugfixes !

Me too, but can't really work on it at the moment, thus progress is a wee bit slow... Anyways, what kind of bugfixes and improvements do you have in mind?

[Codaz]

Remove the black edges around the map, it makes it somewhat unhandy (don't know the english word voor onhandig lol )

And can you make analyzing disable able ? So that i can process maps without analyzing them.

And make that you don't have to analyze to resize :) that would be great. A sort of on the fly encoding.

And i still miss the missing or broken tiles redownload option ;)

See my wishlist posted earlier:
-Beep when analyzing or saving is complete.
-CTRL+A function
-Implement the BAT file i made ? It's renaming the files from N_ to normal. Maybe you can use this with an extra option that automatically creates a new folder named _CORRECTED-$ and then the $ for mapname offcourse.

You said you look after this...
-Make the slider more visible, so put a name next or onto the slider, SLIDE HERE TO RESIZE FOLKS.

[MIB.42]

Remove the black edges around the map, it makes it somewhat unhandy (don't know the english word voor onhandig lol )

This is in the program or what's being generated? Can you take a screenshot and show me what you mean? Thanks!


And can you make analyzing disable able ? So that i can process maps without analyzing them.

And make that you don't have to analyze to resize :) that would be great. A sort of on the fly encoding.

Yeah, I hear you. The analyzing is important for knowing the final size of the map. This is not only for generating the even GPSFS[1234] files, but a way for you to choose the target size. No analysis would mean that you are firing in the dark about what the size of the newly created map is going to be... If that's OK, I can add this feature...


And i still miss the missing or broken tiles redownload option ;)

That's a GMDL feature and is already available!

[Codaz]

Hurray new bug found. If i select almost 22000 tiles, and click analyze, it completes.
If i then select them all, and click on/off the listbox goes flickering and scrolls down. But if i then click in the tile view window, it stops and all the tiles are enabled/disabled in 2 seconds ?

Weird bug, happens on all maps.

Also weird, if i have a map full of PNG tiles, why would it analyze ? It can't be recompressed ?
The util should check if any tiles are changed and THEN decide to analyze or not. Now it always has to analyze before output can be done.

[MIB.42]

Hurray new bug found. If i select almost 22000 tiles, and click analyze, it completes.
If i then select them all, and click on/off the listbox goes flickering and scrolls down. But if i then click in the tile view window, it stops and all the tiles are enabled/disabled in 2 seconds ?

Weird bug, happens on all maps.

You mean that their "selection" status in the listbox loses the status and goes to "not selected"?

I tried replicating what you described, but I might not have understood well what the sequence was... Can you please be a bit more specific? Thanks!

[Codaz]

Ok i have 20 tiles in the listbox.

I select them. with click on the 1st and then press shift and click on the last now they are selected. I press on/off button. The listbox does flicker a lot because there are many tiles to turn off. The scrollbar goes down and down and down. I have to wait before it finishes.

BUT if i don't want to wait for that, it click ONCE in the right tile view window. The scrollbar stops scrolling and the flickering of the listbox stops, and PING in 1 second al my tiles are disabled.

To do the same trick for enabling works also. It's a bug, but also a trick ?

Normally i have to wait for disabling is complete (the scrollbar goes to the bottom and all files are disabled)

[Codaz]

Ok extra, on maps smaller than 15, maybe 17500 tiles this doesn't happen.

I have a REALLY large map of France, it's 330 mb and contains a million tiles or something and it chrashes the program if i do this trick.

Says: program does not respond.

To repeat this trick, if the program is busy disabling or enabling ALL the tiles, click like a maniac on the right tile window. Soon or later it stops, gives an hourglass, and PING the tiles are enabled in 1 second. Also disabled if you do the "trick" again.

[FatesWebb]

just wanted to say thanks!! this tool made my 2gb map half the size, (it could have even made it 600 mb but I stuck with 1gb... Thanks!!!! it looks perfectly fine... I am going to use this on all my maps!!

FW

[Codaz]

How long did that take to process Fateswebb ?

[FatesWebb]

To be honest I don't know, I put it on the server, which has 4gb of ram, and a 2.8ghz zeon processor, and just let it run... didn't time it. but I would guess about 20 minutes to load, about 2 hours to analyze, and about 1.5 hour to compress...

[Codaz]

4gb of ram, wtf that's amazing.

Hope MIB.42 can make his software faster for normal earth pc users :) Not everybody has a server in his home with that specs :)

[MIB.42]

4gb of ram, wtf that's amazing.

Hope MIB.42 can make his software faster for normal earth pc users :) Not everybody has a server in his home with that specs :)

GPSFS_util seldom uses more than 64MB ( as in megabyte ) memory!
It is simply CPU bound due to the compression works...

[FatesWebb]

I work in a data center, actually 4 data centers, so I have access to lots of hardware. I loaded it on one of the server.

[Codaz]

GPSFS_util seldom uses more than 64MB ( as in megabyte ) memory!
It is simply CPU bound due to the compression works...

In September buys my brother a Quad Core system.
Will this speed up the compression or does it use only a single core ?

[MIB.42]

In September buys my brother a Quad Core system.
Will this speed up the compression or does it use only a single core ?

Since I only have a single core machine, I have no idea - just yet - on how to code for multi-cores... I'll have a quick look later...

[Codaz]

Since I only have a single core machine, I have no idea - just yet - on how to code for multi-cores... I'll have a quick look later...

Maybe you can use code from Wprime, a dualcore testing tool.

See www.wprime.net

I googled some things, i need to know which coding language you are using. I Guessed Python but found out that it was probably C++

See these links:

http://mail.python.org/pipermail/python-list/2006-September/405775.html

And this:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2094.html

But remember this:

Why Doesn’t C++ Contain Built-In Support for Multithreading?

C++ does not contain any built-in support for multithreaded applications. Instead, it relies entirely upon the operating system to provide this feature. Given that both Java and C# provide built-in support for multithreading, it is natural to ask why this isn’t also the case for C++. The answers are efficiency, control, and the range of applications to which C++ is applied. Let’s examine each.

Source:
http://www.devarticles.com/c/a/Cplusplus/Multithreading-in-C/

Simple script:
http://www.thescripts.com/forum/thread631573.html

If you are coding in Visual Studio (2005) then read this:

http://msdn2.microsoft.com/en-us/library/tt15eb9t(VS.80).aspx

and this (don't mind the German):

http://www.microsoft.com/germany/msdn/webcasts/library.aspx?id=118757738

Hope that helps a bit, maybe not :P

[Codaz]

MIB.42 are you checking the bikini babes on vacation or ?

We thought you left this project !

When does the .3 version pop-up, with PNG support ?

Thanks !

[deniska]

@MIB.42, I finally got some time to try out this program, and I must say, it's pretty cool.

I used to do similar thing with cygwin scripts and image-magic, but, of course, it was not as user friendly...

[MIB.42]

MIB.42 are you checking the bikini babes on vacation or ?

We thought you left this project !

When does the .3 version pop-up, with PNG support ?

Thanks !

Ok, bikini babes are all accounted for. I am back on the project. I still have to think of an effective way of color reducing those png files to 16 though... My present routines either are memory hogs or CPU ones... must... think... more...

[Codaz]

Good luck, i'm still converting my 550 mb map on a spare celeron computer i found.

Way too slow but ok.

[MIB.42]

Just status update... Getting closer to a decent - and relatively fast - color reduction... It's not trivial... Here is a snapshot on the analysis code to fine-tune the algorithm :

[deniska]

nice!
Looks almost as complex as photoshop.. :-)

[MIB.42]

nice!
Looks almost as complex as photoshop.. :-)

Thanks, actually, it does a lot more than photoshop... ;-)
Anyways, I thought I go after the tools, as you guys seem to have the PSP code quite well covered... ;-)

Cheers, MIB.42

[deniska]

I was hopping you'd do some psp coding too...
I may need help in following areas:
- psp-290 -> nmea converter
- stick to road functionality (check current pixel and see if you are on a road, if not try to look around for road colors..
- route calculator (based on google maps imagery)

anyway.. it's up to you!
great work with the tool!

[Codaz]

Great MIB.42 !

Could you make this simple function into 0.3 ?

Comparison table of sizes. If i resize a picture in Photoshop it says:

At quality 100 the size will be ... At quality 50 the size will be ....

Now in your program could you make a comparison (with radio buttons) to choose if you want to have JPEG conversion, or PNG conversion)

So for example:

Your map with 60% compression will be 500 mb with JPEG and 400 mb with PNG. So we can choose what's best for us. Not only the file quality but also file size comparison :)

This is calculated on the fly after conversion.

[Ixar]

Is there any up to date GPSFS documentation available ? I have read mapthis parsing code, but it is a bit ennoying... So if you have just some clues they are welcome :)

I would like to understand the "basezoom", is it to tell the zoom used for the map first level of tiles (ie the first 4 tiles if I understood properly) ?

Regarding the tile index in the early documentation I found it is said it is an offset, but as there are several files I guess there must be a file index as well.

Regarding the tiles size is it constraint ? (ie 256x256) or is it free ?

Thanks for your help ;)

[Ixar]

Is there any up to date GPSFS documentation available ? I have read mapthis parsing code, but it is a bit ennoying... So if you have just some clues they are welcome :)

I would like to understand the "basezoom", is it to tell the zoom used for the map first level of tiles (ie the first 4 tiles if I understood properly) ?

Regarding the tile index in the early documentation I found it is said it is an offset, but as there are several files I guess there must be a file index as well.

Regarding the tiles size is it constraint ? (ie 256x256) or is it free ?

Thanks for your help ;)

Finally I found the missing informations in gmdl source code ;) (GPSFS.cs FYI)

What I was missing is the 2 formats for GPSFS, I was only aware of the 1st one. Now things are more clear.

Best Regards,
Ixar

[deniska]

Some documentation is available here:
http://en.wikibooks.org/wiki/Map_This!

in case you still have questions..

[Ixar]

Some documentation is available here:
http://en.wikibooks.org/wiki/Map_This!

in case you still have questions..

Thanks I have already read the GPSFS part in that wikibook.

I still have one question regarding the GPSFS file split: what is the motivation ? Is there any rule behind that split or is it arbitrary ?

[in7ane]

Ixar, as I recall seeking within a single file was a bit slow - so that's the reason for the 5 file setup (this is hard coded, so a 4 or 6 file setup will not work). The split is on size to try to get the files to be of comparable size (the last file ends up a bit larger due to rounding/left over tiles, but since the discrepancy is minor there hasn't really been a reason to fix that).

Also, keep in mind Int64 vs Int32s being used in different fields.

[Ixar]

Ixar, as I recall seeking within a single file was a bit slow - so that's the reason for the 5 file setup (this is hard coded, so a 4 or 6 file setup will not work). The split is on size to try to get the files to be of comparable size (the last file ends up a bit larger due to rounding/left over tiles, but since the discrepancy is minor there hasn't really been a reason to fix that).

Also, keep in mind Int64 vs Int32s being used in different fields.

Thanks for the explanation ;)

I think I have now understood the format, with the IDXFile and IDXTile on 32 bit. One thing that is not specified is the endianess, but I guess it is little endian as is the PSP.

I will soon try my own maps.

Ideally I would like to implement a new feature in MapThis to be able to merge maps "on the fly". This way if a tile is not available in a map, it take the one in another map.

But let's see first if I managed to make something with GPSFS :P

[Codaz]

Any progress MIB.42 or is it too difficult to make ?

[MIB.42]

Any progress MIB.42 or is it too difficult to make ?

Oh man, I'm so swamp'd at work and school also started for the kiddos... I can write like 10 lines a code a day...

Nevertheless, I am getting there. The HSV based base-color reduction seems to work ( I can already reduce to 32 with minor bugs ) and I will only need to write the final reduction code. There is still some "tweaking" involved in choosing the human perception baselines, but it's getting really good. I stop'd making predictions on when this will be ready, but I am at the moment now that I can see the light at the end of the tunnel... I was actually thinking of taking a few days of vacation ( from work ) to finish this... *sigh*

[Codaz]

Lol, is it CPU intensive like u said ?

Good luck with the kiddos and work.

[MIB.42]

Oh man, I'm so swamp'd at work and school also started for the kiddos... I can write like 10 lines a code a day...

Nevertheless, I am getting there. The HSV based base-color reduction seems to work ( I can already reduce to 32 with minor bugs ) and I will only need to write the final reduction code. There is still some "tweaking" involved in choosing the human perception baselines, but it's getting really good. I stop'd making predictions on when this will be ready, but I am at the moment now that I can see the light at the end of the tunnel... I was actually thinking of taking a few days of vacation ( from work ) to finish this... *sigh*

Just a quick update, the PNG color reduction finally works now and it's pretty darn fast and only uses about 1/2 MB..., I have started integrating it into the Utility Program... I'll keep you posted...

[MIB.42]

Just a quick update, the PNG color reduction finally works now and it's pretty darn fast and only uses about 1/2 MB..., I have started integrating it into the Utility Program... I'll keep you posted...

Sorry for the delay, libpng keeps crashing after a thousand or so writes... I've checked non-png version, it works, and isolated the problem into a separate app and I am now debugging the png ( actually the zlib )... I love how things we take for granted don't work... :(

[Codaz]

How long do you need to fix it do you think ?

[MIB.42]

How long do you need to fix it do you think ?

I've isolated and separated the png read part and decoupled from write. It still is crashing in zlib... After I manage to fix it, I am rewriting the PNG handler, so that there is no "Analysis" phase, as there is no need for it... When a map is PNG based, it will "simply" generate a 16 color reduced map... I really want to finish this as I'd love to work on something else as well, so the motivation is really high on finishing it asap....

I'll keep you posted...

[MIB.42]

All right, I finally had 4 hours of uninterrupted coding time... Found the bug ( needless to say, it was in my code, but it was so elusive, that it crashed zlib and by debugging libpng which led to debugging zlib gave me the hints for fixing it... *sigh* )

Anyways, if you open a map which is PNG based, it will offer a "Q-Create" button. This bypasses the analyze part and starts creating the 16 color reduced map. Since this uses only a guesstimate on the final size, GPSFS4 might be a bit out of sync with the rest, but it should still work...
If you want really nice evenly spaced GPSFS files, run the "Analyze" before the "Create".

zlib and libpng is not needed as they are linked in statically...

The 16 color PNGs reduce the maps to 45%~65% of the original sizes...

Cheers,
MIB.42

PS: I haven't tested it extensively, so let me know if you find "bugs"... Also, there is a minor memory leak ( ~20KB/1000 tiles) I will track down as soon as I have time...

PS2: I have found a minor bug in the size computations, thus the rev1 attachment update

[deniska]

cool!
I'll give it a try and let you know!

[deniska]

I tried it on a few, relatively small maps and I must say - I am impressed!
IMHO, the quality is much better than via ->jpeg conversion. The compression artifacts are hardly visible on the most detailed zoom levels..
Good job!

[johoo]

hi MIB.42
it's working great for me
but there is 1 thing i'm on a old monitor 800/600
and when i use it it doesn't fit to the screen
can u make it fitting the screen by pressing the square button???

thnx for the progr.johoo

[MIB.42]

I tried it on a few, relatively small maps and I must say - I am impressed!
IMHO, the quality is much better than via ->jpeg conversion. The compression artifacts are hardly visible on the most detailed zoom levels..
Good job!

Glad it works. Any "burning" issues I can help with?

[MIB.42]

hi MIB.42
it's working great for me
but there is 1 thing i'm on a old monitor 800/600
and when i use it it doesn't fit to the screen
can u make it fitting the screen by pressing the square button???

thnx for the progr.johoo

That's a bit tough, since 3 tiles are already 768 pixels and there is no menu or anything just yet... Hmmm, I may put together a single-tile version...

[deniska]

Glad it works. Any "burning" issues I can help with?
I may need some help with Vector maps implementation.. and gu routines..
I'll post the latest code snapshot once I clean it up a bit...
basically, I already re-wrote most of that stuff,but it has some minor issues...

[Codaz]

Beautiful that coders ask eachothers help.

MIB i will try the program out in a few seconds !

I'll report ASAP.

What's on stake for version 0.4 ?

========

Report:

Program is working VERY well, BUT i can't set the image quality of PNG maps with a slider or something.
On JPEG maps i can set the end size with the slider.

Also the Qcreate doesn't give an estimate of the size it will be.

A bit confusing is it now. It would be better if there was a build in comparison chart that says:

Process the map now and you get:

JPEG normal = 50 mb.
JPEG max = 20 mb
PNG normal (current) = 50 mb.
PNG max (16bit) = 18 mb.

Or something like that.

=====

I analyzed a 54,1 mb PNG map which gives with no compression a 82 mb JPEG map. WOW ! How does that come ?

If i say compress it to jpeg max compression it will look much worse (64 mb) than png 16 bit color which is only 28,1 mb.

So great job done, map is from 54 mb to 28. With png > jpeg on max it will only go from 54 to 64 and it looks blocky.

Why is the estimate size on jpeg conversion not right ? It says with max jpeg compression (slider set) it will be 46 mb. But i viewed the folder and it was 64,6 !

[Codaz]

I tried some things out with photoshop on several tiles and other images. Wouldn't it be good to do a 8 color GIF addon because it will give good viewable results.

8 bit diffused with restriction (216 web colors).

[Codaz]

Also, i'm trying to run your program on a spare win98 machine.

Version 0.3b isn't well prepared for win98. Too bad.
Screen resolution is 1024x768 so it should be good.

The status bar is gone ? So i can't see how far the analyzing/creation is...

Hope you can fix it.

[MIB.42]

Also, i'm trying to run your program on a spare win98 machine.

Version 0.3b isn't well prepared for win98. Too bad.
Screen resolution is 1024x768 so it should be good.

The status bar is gone ? So i can't see how far the analyzing/creation is...

Hope you can fix it.

Sorry, but I have no access to Win98, thus have no idea what might be wrong... I am using "standard" Windows services for the GUI...

[MIB.42]

I tried some things out with photoshop on several tiles and other images. Wouldn't it be good to do a 8 color GIF addon because it will give good viewable results.

8 bit diffused with restriction (216 web colors).

PNG 4bit I believe has definitely better compression for the PNG type maps than 8 bit diffused GIF.
I guess you're talking about instead of JPGs, right? That would require adding an other decoder into MapThis!, which I am not sure would buy a lot of extra features...

[Codaz]

Ok clear on that GIF addon. PNG-4 would be nice.

For the win98 testing, you could use the compatibility options of winXP.

So run this program in win95/win98 mode (right mouse click).

Or is there any way that i can use your tool within DOS ?
A command line prompt ? Would be very efficient, i don't want to see those tiles :)

[Codaz]

Beautiful that coders ask eachothers help.

MIB i will try the program out in a few seconds !

I'll report ASAP.

What's on stake for version 0.4 ?

========

Report:

Program is working VERY well, BUT i can't set the image quality of PNG maps with a slider or something.
On JPEG maps i can set the end size with the slider.

Also the Qcreate doesn't give an estimate of the size it will be.

A bit confusing is it now. It would be better if there was a build in comparison chart that says:

Process the map now and you get:

JPEG normal = 50 mb.
JPEG max = 20 mb
PNG normal (current) = 50 mb.
PNG max (16bit) = 18 mb.

Or something like that.

=====

I analyzed a 54,1 mb PNG map which gives with no compression a 82 mb JPEG map. WOW ! How does that come ?

If i say compress it to jpeg max compression it will look much worse (64 mb) than png 16 bit color which is only 28,1 mb.

So great job done, map is from 54 mb to 28. With png > jpeg on max it will only go from 54 to 64 and it looks blocky.

Why is the estimate size on jpeg conversion not right ? It says with max jpeg compression (slider set) it will be 46 mb. But i viewed the folder and it was 64,6 !

What do you say about this MIB ?

[MIB.42]

What's on stake for version 0.4 ?


I don't have anything planned... I have a thousand other projects waiting and this seems to work for now...



Program is working VERY well, BUT i can't set the image quality of PNG maps with a slider or something.
On JPEG maps i can set the end size with the slider.


Of course not. PNG is non-tunable ( ...well, as in I already set all the z-pmeters to max compression etc. ). There are simply 2 formats; 8bit and 4bit. These are included in 0.3b.



Also the Qcreate doesn't give an estimate of the size it will be.


Correct. The size is approx. half of the original one.



A bit confusing is it now. It would be better if there was a build in comparison chart that says:

Process the map now and you get:

JPEG normal = 50 mb.
JPEG max = 20 mb
PNG normal (current) = 50 mb.
PNG max (16bit) = 18 mb.

Or something like that.


I see. Well, I discourage making JPGs from PNGs. The 4bit version pretty much represents the best compression/quality ratio. You can still do it of course by clicking on the "Force JPG" if you really, really need to, but with the 4bit PNG version available now, there is no real need for it...



I analyzed a 54,1 mb PNG map which gives with no compression a 82 mb JPEG map. WOW ! How does that come ?


JPEG is really bad compressing lines etc. It was designed to work on "organic" images ( like satellite pictures )... PNG does a better job on geometric images, like vector type maps ( fonts included )...



If i say compress it to jpeg max compression it will look much worse (64 mb) than png 16 bit color which is only 28,1 mb.


Indeed. That's the nature of PNG vs. JPG.



So great job done, map is from 54 mb to 28. With png > jpeg on max it will only go from 54 to 64 and it looks blocky.


Indeed. That's the nature of JPG.



Why is the estimate size on jpeg conversion not right ? It says with max jpeg compression (slider set) it will be 46 mb. But i viewed the folder and it was 64,6 !

Hmmm, that could be an artifact of the quick JPG estimation, which accumulative produces this difference. I was debating for a while to slow down the compression in the analyze state to produce more evenly distributed GPSFS files, or do it faster with less even results... I voted for the latter, as there is no significant seek time impact on the PSP due this "minor" distribution difference.

Cheers,

MIB.42

[MIB.42]

Ok clear on that GIF addon. PNG-4 would be nice.

It is done in v0.3b.


For the win98 testing, you could use the compatibility options of winXP.

So run this program in win95/win98 mode (right mouse click).

Or is there any way that i can use your tool within DOS ?
A command line prompt ? Would be very efficient, i don't want to see those tiles :)

Oh, Lordy, lordy... You want a bit too much now, don't you? :D
( Would work under Linux a lot faster than on any other winblows versions, but hey, had to do it for the "masses"... :eek: )

[Codaz]

Ok ok :D i'll cool down.

You say there is 8bit and 4bit PNG present in your 0.3b release.

How to choose between it to see the differences ?

[MIB.42]

Ok ok :D i'll cool down.

You say there is 8bit and 4bit PNG present in your 0.3b release.

How to choose between it to see the differences ?

GPSFS_util only outputs 4bit PNGs if PNG maps are optimized. ( There is no reason to output 8bit ones, as that's what the original map was in )

What I usually do is to load 2 GPSGS_utils and browse the original (8bit) and the newly created (4bit) maps. If all went well, they are "almost" identical...

[MIB.42]

Bug fixes...

[Codaz]

Ok thanks. Do you have a changelog on that .3c version ?

Too bad the win98 problem still isn't fixed.

I thought of a solution, can you make the progress appear in the window bar (aside the X _ and square button (to maximize the window).

So where it says GPSFS Util click here...

Should be a small change and it would help me a LOT :)

[johoo]

Ok thanks. Do you have a changelog on that .3c version ?

Too bad the win98 problem still isn't fixed.

I thought of a solution, can you make the progress appear in the window bar (aside the X _ and square button (to maximize the window).

So where it says GPSFS Util click here...

Should be a small change and it would help me a LOT :)


it would help me too
because of my 800/600 monitor
that way i could see the progress

[MIB.42]

it would help me too
because of my 800/600 monitor
that way i could see the progress

Ok guys... On it...

[MIB.42]

Mini version done... Please see attached...

[Codaz]

What is different ?
I still can't see a status indicator.

Or is it only on creating a map, not analyzing ?

If you have a changelog from 0.3b to 0.3c to mini that would be great.

Thanks MIB.42

[johoo]

hi mib.42

thnx man works great now

it now opens in a much smaller window so that everything fits on the screen :thumbup:

[MIB.42]

What is different ?

You mean in the 0.3c-Mini or 0.3c ?!


I still can't see a status indicator.

Or is it only on creating a map, not analyzing ?


Status indicator? It's the scroll bar and the status bar on the left at the same height as the scroll bar.
I tested it on Win98 and even on Win95 compatibility mode with 256clrs and 640x480 screen resolution, worked well.




If you have a changelog from 0.3b to 0.3c to mini that would be great.

Thanks MIB.42

0.3b : 4bit PNG support with special color reduction.
0.3c : Bug fix on the JPG compression selection ( the slider now correctly controls the predicted file output size)
0.3c-Mini : Reorganized the controls to fit on even 640x480 screens.

[Codaz]

I mean that the status bar is gone when the tool is running on Win98.

Where it says analyzing tile 3/1000

[johoo]

hi mib.42

today i finished a map from 3.9gb to 1.9gb
now i have 2gb left on my memorystick
i was driving with it the whole day long and it worked very well
no crashes and fast loading also

mapthis is going to be better and better
i love it
the latest version runs very smooth

[deniska]

nice to hear that.. were you testing with v0.505?

[johoo]

yes the 505 version
edit: psp fat ,ta-81 ,371m33-3 ,gps-290

and it's going great
it works with the bigmap very well and is fast
u did a great job again and also nieko and mib etc.etc.

the develpment is going to fast for me :D so i wait for the next off. release

i was thinking ..when searching a streetname maybe it's possible to auto fill in the name while writing it ,like u see in other nav's
if you know what i mean

[vblanche]

Hi,

I'm trying 0.3c Mini.

open my downloaded map, click "analyze", then it crashed when I click "create".

also,
when the software is analysing, if I double-click for help bar, boom...crash

thanks

[MIB.42]

Hi,

I'm trying 0.3c Mini.

open my downloaded map, click "analyze", then it crashed when I click "create".

also,
when the software is analysing, if I double-click for help bar, boom...crash

thanks

I just tried it running on XP and under Win95/256clr compatibility mode. Didn't get what you described, it worked fine.

Tell me more about your system and the maps...

[vblanche]

I just tried it running on XP and under Win95/256clr compatibility mode. Didn't get what you described, it worked fine.

Tell me more about your system and the maps... oops...didn't know I have to run in non-xp mode...will try later. Thanks

[Adabow]

Hey can anyone teel me how to get rid of empty levels because when I open the file on MapThis I get 2 or 3 empty zoom levels and I want to get rid of them.

[Codaz]

You have to analyze the map first before you can remove levels.

[Adabow]

I have done this but it only lets me remove the levels I want it dosnt let me remove the empty levels.

[aberz]

Empty level is well ... empty . I think.

[Nieko]

Exactly, how would you go about removing something that isn't there in the first place? :)

If you want all levels to contain imagery, you'll have to download the entire map again through GMDL.

Btw., it's not really possible to just cut out a certain zoom level, i.e., when zooming in in MapThis, you can't go from level 5 to 3 in one step.

[Adabow]

The picture shows what I have and what I want. I want my map so it uses the first three levels instead of skipping the first three and using 4,5, and 6.

[zerodays]

adabow: I think you will need to download maps again?
It looks to me that, your map do not have level 1/2/3 ? (i.e., the most magnified zooms)

[Adabow]

thats it every time I download a map i get rid of all the big zooms and just keep the 2x 4x and 8x and thats all i keep. And I still end up without the 1,2,3 levels.

[zerodays]

adabow, i am not too sure
so from what I know from your questions,

- you do not like the close zooms?
in GDML I think x64 + is one of the far way ones, and x1 is the most-close view

and in that util, Level 1 takes the biggest MBs, as it is the most-close view
So I think x64 = Level 1, x32 = Level 2, so on...

so yeah, it makes sense that to me now :p ?

[aberz]

Adabow :

Before you dl the map, in zoom lv, uncheck the box of the smallers numbers . Try to uncheck x1 x2 x4 x8 x16 And see if you get what you want.

[Adabow]

Hey thanks a lot to all who helped me out very appreciated. I will try this out.

[finalmagic2]

hey hey ^^

great work! your tool works fine, MIB... but i have some questions.

i have downloaded the whole map of germany and the size is 5,29 GB.

when i press the analizebutton it say that the job will be finished in 65h oO is this normal?

[fischauge]

1st of all sry for my english and bad grammar :S


I`m using this Version GPSFS_util_v03c_Mini and I have a Problem with it. During the compress of a map my pc lags, this is understandable, but after the compress and the program is closed, my machine lags like hell, it lags like he`s doing the compression, but he`s not. I don`t know what it is. In the Taskmanager the process is closed and my machine is not that bad.

[MIB.42]

hey hey ^^

great work! your tool works fine, MIB... but i have some questions.

i have downloaded the whole map of germany and the size is 5,29 GB.

when i press the analizebutton it say that the job will be finished in 65h oO is this normal?

Yes, analyzing can take a long time... Check the "No Visual", it can speed the analysis process up in some cases tremendously...

[MIB.42]

1st of all sry for my english and bad grammar :S

I`m using this Version GPSFS_util_v03c_Mini and I have a Problem with it. During the compress of a map my pc lags, this is understandable, but after the compress and the program is closed, my machine lags like hell, it lags like he`s doing the compression, but he`s not. I don`t know what it is. In the Taskmanager the process is closed and my machine is not that bad.

Well, that's a tough one. I have made extra efforts not to have any memory leaks or so, and even if there is, the result would be a crash.

Start TaskManager and then run GPSF_util and watch it's memory allocation. If it stays constant after a while, than there is something else on your computer... This is Windows after all...

[linux2512]

Oh man, it took me 2 weeks to port my initial code from Linux to Windows... While the core still works under Linux, the UI won't. I'd have to go back and rewrite the .RC based resouce management... I'd rather spend my time on feature development... *sigh*
( Does GMDL run under Linux? )

If no one has started work on porting this to Linux yet, I'll take a crack at it, just point me to the source.

[vistaboy]

great work! great program.
Just for information, I ran GPSFS_util_v03c_Mini successfully on Window Vista.
Probably in next version can allows us to add/update new map. Environment keep changing as will as map have to update frequently.