Printable View
I just tried a 3GB map with 349524 tiles, it loaded it quite nicely and is currently analysing, but I see no memory footprint increase. I spent extra efforts in making sure, that there is no memory leak...Quote:
Originally Posted by FatesWebb
Anyways, for this 3GB map, my memory footprint was around 30MB only... Can you check how much free memory you have? My favourite tool for it is from http://www.prcview.com
Anyways, can you upload your map somewhere, so I can have a look?
OK, so I am going to copy it to a racked server with a Xeon 2.8GHZ and 4gb ram, and test it there, here is a copy of it if you want to test it yourself. (sending a private message with the location) I will take it down after you get it, because this is not a permanent place to host this file.Quote:
Originally Posted by MIB.42
ok so I have loaded it on the racked server, and it does not crash where it did on this machine. It is analyzing. I am not sure about the steps to take to use the program.
Do I have it correct that it will attempt to re-compress the images to make the map smaller? is there a good tutorial on what to do once you have loaded the map? I clicked analyze but I am not sure if there is more to it than that.
Do you want me to continue chasing why it would not run on this machine/reboot and retry or just use a beefier box. it is a 2.6GHZ running winxp and 500mb ram...
FW
Double Click the window title bar for instructions. It even says so! ;-)Quote:
Originally Posted by FatesWebb
I have made lots of modifications already, so it wouldn't help if we chase this with the v0.1...
Thanks.
cool, trying it now..
what is your experience? that max compression is the way to go? or does it cause too many artifacts?
Also when it says emp, that is empty, so seemingly useless....
but on/off does off mean it removes it from the compression that the util does? or that it removes it from the archive altogether? .....(EDIT) ok wait I think I figured that one out by reading this thread, it means disable it from the processing? because I see in this thread where it says that EMPs only take up 8 bytes anyways(/EDIT)
testing with max compression to see if I like it that way...
This is fun, thanks!
FW
Hmm, I think something went wrong, after running it a full compression here is what I was left with.... It looks like it started to work, but somehow made 4 too big, and the files I ended up with are just as big as the source files...
http://img299.imageshack.us/img299/3864/screen1sf6.jpg
http://img299.imageshack.us/img299/3593/screen2ma4.jpg
I usually go with compressions at around 40~50% of the slider ( translates to ~65% JPG )Quote:
Originally Posted by FatesWebb
It simply shows, that there is no title assosicated with a given position. Quite an important feature for analysis, but useless for recompression. GPSFS_util is not a recompression tool only, but an analysis one as well...Quote:
Originally Posted by FatesWebb
Yes, when you turn off a tile, that tile will be omitted from the resulting conversion. This in turn will produce and EMPTY tile reference in the newly generated GPSFS files.Quote:
Originally Posted by FatesWebb
Also, if your original GPSFS was a PNG based file ( it will show after you load it in the information bar on the bottom left, recompression is not always good, as conversion from PNG to JPG can vary quite much...
v0.3 will have PNG recompression to PNG...
I am almost done with v0.2, I am only running tests now. It is a major improvement... I plan on releasing it within 24hours.
Cheers.
When does version 3 come. I hope PNG compression is better.
Within some days i have 1 gb memory, no more 512 mb so i hope analyzing goes a lot faster.
v0.2 is attached.
v0.2 changes :
- By popular demand; Evaluated 16bit JPG compression, no benefit...
- Implemented 'FastPass' for jpg compressions.
Improvement on analysis time is ~20~30%, but reduces size prediction. Still worth it...
- Select map FOLDER instead of the GPSFS file
- Multi-tile preview window with navigations ( control buttons and mouse )
- Better logistics on task interruptions
- Play sound after analysis / convert is done
- Split the ETA computation into "empty" and "normal" tiles and compute the result accordingly
to provide more accurate ETA. This is still not 100% accurate, as it would require more system resources
which I prefer using on speeding up the conversion process instead...
- Added hour on top of the existing minutes:seconds in ETA
Again,
The following text can also be accessed by clicking on the title bar of the window.
Usage :
1. Click on [Open] and select the map directory where the GPSFS files are.
2. Enable/disable tiles, levels...
[Levels] for omitting zoom-levels.
[On/Off] to enable/disable (turn on/off) individually selected tiles.
( multi selection is available in the listbox )
3. Click on [Analyze] - May take quite some time... Enabling [No Visual] can speed it up...
4. Adjust compression settings ( after completed analysis only )
Scrollbar : choose approximate target size (compression ratio setting).
Enabling/disabling selected tiles/levels will reduce size approximations. Re-Analyze helps...
5. Click on [Create] and choose/create a folder where the new map will be stored.
- Again, might take a while, but not as long as the analysis...
Others:
'Re-Analyze' is there if you enabled/disabled several tiles ( changed lots of settings ) and already
had an analysis done before. ( This will guarantee a more precise size approximation on the files to
be created... )
Navigation :
Besides the obvious buttons in the 'Navigation' area ( I/O = Zoom In/Out, U/D/R/L = Move Up/Down/Right/Left )
there are mouse controls as well :
- Clicking with the left button on the tiles around the central tile will result in repositioning the tiles
( move to the selected direction )
- Clicking with the left button on the central tile will toggle its status (enabled/disabled).
- Clicking with the right button on the central tile will pop up a detailed information on the tile.
- Clicking with the right button on the upper tiles and tile to the right from the central one will zoom in.
- Clicking with the right button on the lower tiles and tile to the left from the central one will zoom out.
Limitations :
- Only JPG based GPSFS will be created, PNG based ones will be converted to JPGs. (v0.3 will have PNG recompression )
- Be patient, it may take a long time to do the proper analysis...
- If there are insane amount of tiles, the listbox will be disabled.
( Individual tile selection can be done with the scrollbar / navigations )
- After the analysis is done, you can elect to create several outputs by creating maps into different directories...
The analysis has to be done once only, while creation with different settings can be done multiple times...
- Completion time predictions are very rough and can be inaccurate if a map has lots of missing tiles.
( A more precise prediction would consume a lot more system resources... )
Cheers,
MIB.42
Great ! Many of my additions have been added :)
But the 16 bit addition, why doesn't it have benefit ?
Also i like the Fastpass !!
Is PNG to JPG conversion bad for file size ? Because i think JPG is larger than PNG.
I don't like the levels 1 to 10 placed under eachother, make that like version 0.1 it saves space on your desktop !
But there's a BIG bug in 0.2
I analysed the same old map i did in 0.1 and it was there 9.44 mb. If i analyse it in 0.2 it becomes 10.21 mb !
Second BIG bug:
What is this after Analyze ?
http://img522.imageshack.us/img522/5...error02ad5.png
Not displayed correctly !
Third:
If i have recursion in map dirs EG:
C:\Maps\_blabla
and
C:\Maps\_blabla\_newrecompressed
the software takes em both OR can't analyze it.
I am trying to listen to you guys... :)Quote:
Originally Posted by Codaz
Apparently there is no such thing as a 16bit JPG. :) All channels are encoded on 8bits. So, even if I convert the pics to 16bit, they get expanded out to 24 during the conversion. There is a way to rewrite the encoder to support 16bit, but that we'd need to also rewrite decoder on the PSP/GMDL etc... It's just not worth the hassle...Quote:
Originally Posted by Codaz
Yeah, that was a major addition. Unfortunately the approximation on the final filesize ( which is done with the slow pass to create better quality and smaller images ) can be different up to 20%, which results sometimes in an uneven filesizes between the GPSFS files... It still does work, but not as pretty... :pQuote:
Originally Posted by Codaz
Actually, it's not that bad, but you loose a bit of quality. I will run some test on the 4bit/8bit PNG conversions later...Quote:
Originally Posted by Codaz
Since the preview window is already big, it actually makes a lot of sense to put them as such... I am open to redesign the look&feel of the app... :)Quote:
Originally Posted by Codaz
This is not a bug. v0.2 provides a different result for analysis due to the fastpass phase. What matters is that your final filesize is the same. See above and README.txtQuote:
Originally Posted by Codaz
Well, I think it's actually a problem with your map! Check the stats on the individual tiles on it's right and left and compare them if they are next to each other ( as in click right mouse button on the central tile, jot down the filesystem and offset values, than move to the next one...) See README.txt for more info on how to do this...Quote:
Originally Posted by Codaz
Now this is news to me. I have several folders with subfolders in them and it works just perfectly. Please re-check and describe your problem more precisely.Quote:
Originally Posted by Codaz
Thanks,
MIB.42
Hmm that doesn't solve the problem.
And how to disable/enable the fastpass ?
What problem?Quote:
Originally Posted by Codaz
Can't at the moment. Why would you want it?Quote:
Originally Posted by Codaz
Because it may cause the error of displaying that ugly tile at the wrong place.
The tile IS enabled, in right and left screen it is correct.
Also i got the re-encoded one now working on Mapthis !
-Removed the prefs file.
-Added more POI icons in that _map dir
It works now but it's a bit ugly on the small psp screen.
FastPass does NOT touch the decompression part. This "bug" is due to wrong offset/filesystem, which can be either in the original map file, or in the processing of GPSFS_util.Quote:
Originally Posted by Codaz
If the tile is shown (and not red), that means it is enabled. That wasn't the question. I can also see, that the tiles on the right and left are correct, that wasn't the question either. The question was if they represent an uninterrupted sequence in the GPSFS file.Quote:
Originally Posted by Codaz
Please give me the infos ( as I described earlier... right mouse button ) on the left, center and right tiles and I will let you know if they are indeed correct.
Not sure what this means... You are probably referring to the recompressed file being lower in quality... Yes, it is. There is no "free lunch"... :)Quote:
Originally Posted by Codaz
It's better that i give you the map, it's only 10 mb so i will upload it tomorrow.
I'm not a genius so that's maybe the best option.
It's an original map from GMDL.
EDIT: HERE IS IT:
Hey MIB.42, I've got .2 running as we speak, and so far so good. I was wondering... For the SMP (dual core) users out there, how hard is it to make the app SMP aware? It's always painful to watch a process take up only one core :)
Hah, I hear you... Ok, I'll have a look at it...Quote:
Originally Posted by meier3283
It works for me, no problem. Tile #27 is perfectly fitting in. #26 is 0:100813:7168, #27 is 0:107981:4859 and #28 is 0:112840:8879 ( FileSystem:Offset:Size )Quote:
Originally Posted by Codaz
MIB.42 why do i get the weird thing i posted in the screenshot then ? The map is correct you say !
And is 0.2 already capable of working on pentium 4 HT technology ? Because HT is somewhat also 2 cores.
ok, so I did this, it shrunk my 2gb map to 1gb, and I placed it on the mestick, and it just hangs with the memory card flashing, and says "please wait while map is loading. I used v02
Same as mine. Fix the icons in the directory and it works !
Some poi icons are missing so that's why it's chrasing.
Sorry, you were right, I found the bug... It was only happening on PNGs after Analyze... It's fixed now.Quote:
Originally Posted by Codaz
Other improvements in v0.2b (attached) is that you can select to reconvert PNG based ones to JPGs or not. This is really useful when you are only removing tiles from a bigger map and want to keep the map in PNGs.
There is also a new functionality, you can disable more than one tile per click ( if you use SHIFT ). It will disable all underlying tiles as well on different zoom levels recursively...
Cheers,
MIB.42
Hi MIB,
I've read this thread since beginning but until now I don't understand for what your program is good for.
May be you could edit your post #1 with a little explanation what this tool is doing. You could also edit it with version 0.2.
This would help very much and would prevent from silly questions like mine :)
Thank you!
michel
Ok, I updated post#1 with "examples"... ;)Quote:
Originally Posted by Bolemichel
Thanks a lot! Just finished connecting my holux 236 with the psp and may be this little tool could something good for me ;)
For my understanding it could validate and compress your files on the memstick.
Just a blog :
I did some tests, moving from the default 256 color pngs to 16 will result in a ~42% size reduction... Altough converting the pngs down to 16 color can take quite some time, in some test cases it did take about 3 seconds per tile on my machine... Imagine my 1.4 million tiles map... That's over a month continuous work! I gotta do something to speed this up...
Yup, as Fateswebb says: He shrunk a map from 2gb to 1gb. That's taking now more than a day.
Sounds good! I'm searching for a way to delete the 1x zoom levels. When I drive by car I don't need such detailed graphics.
Is it possible to delete this 1x zoom level with your tool or is there a way to deactivate it in GDML download tool? The second would be probably better to speed up the download...
ok, I am testing out the 2b version, I analyze, but then when I slide the slider bar over to the right to choose a compression level, it goes from No Recompress, Size = 2236.09 MB to Recompress, Size = 2236.09 MB
It is telling me it is not possible to compress this any smaller? even though it is full size?
Thanks...
FW
The highest levels must be enabled to form a map in total. The lowest levels (eg 1 ) can be disabled in GMDL under advanced settings.Quote:
Originally Posted by Bolemichel
That means that the map is PNG based and 16 color recompression is not done yet... Click on the "Force JPG" to convert the PNGs to JPGs which will give you compression...Quote:
Originally Posted by FatesWebb
When is version 0.3 coming :) ? Can't wait for it !
Hope to see some improvements and bugfixes !
Me too, but can't really work on it at the moment, thus progress is a wee bit slow... Anyways, what kind of bugfixes and improvements do you have in mind?Quote:
Originally Posted by Codaz
Remove the black edges around the map, it makes it somewhat unhandy (don't know the english word voor onhandig lol )
And can you make analyzing disable able ? So that i can process maps without analyzing them.
And make that you don't have to analyze to resize :) that would be great. A sort of on the fly encoding.
And i still miss the missing or broken tiles redownload option ;)
See my wishlist posted earlier:
-Beep when analyzing or saving is complete.
-CTRL+A function
-Implement the BAT file i made ? It's renaming the files from N_ to normal. Maybe you can use this with an extra option that automatically creates a new folder named _CORRECTED-$ and then the $ for mapname offcourse.
You said you look after this...
-Make the slider more visible, so put a name next or onto the slider, SLIDE HERE TO RESIZE FOLKS.
This is in the program or what's being generated? Can you take a screenshot and show me what you mean? Thanks!Quote:
Originally Posted by Codaz
Yeah, I hear you. The analyzing is important for knowing the final size of the map. This is not only for generating the even GPSFS[1234] files, but a way for you to choose the target size. No analysis would mean that you are firing in the dark about what the size of the newly created map is going to be... If that's OK, I can add this feature...Quote:
Originally Posted by Codaz
That's a GMDL feature and is already available!Quote:
Originally Posted by Codaz
Hurray new bug found. If i select almost 22000 tiles, and click analyze, it completes.
If i then select them all, and click on/off the listbox goes flickering and scrolls down. But if i then click in the tile view window, it stops and all the tiles are enabled/disabled in 2 seconds ?
Weird bug, happens on all maps.
Also weird, if i have a map full of PNG tiles, why would it analyze ? It can't be recompressed ?
The util should check if any tiles are changed and THEN decide to analyze or not. Now it always has to analyze before output can be done.
You mean that their "selection" status in the listbox loses the status and goes to "not selected"?Quote:
Originally Posted by Codaz
I tried replicating what you described, but I might not have understood well what the sequence was... Can you please be a bit more specific? Thanks!
Ok i have 20 tiles in the listbox.
I select them. with click on the 1st and then press shift and click on the last now they are selected. I press on/off button. The listbox does flicker a lot because there are many tiles to turn off. The scrollbar goes down and down and down. I have to wait before it finishes.
BUT if i don't want to wait for that, it click ONCE in the right tile view window. The scrollbar stops scrolling and the flickering of the listbox stops, and PING in 1 second al my tiles are disabled.
To do the same trick for enabling works also. It's a bug, but also a trick ?
Normally i have to wait for disabling is complete (the scrollbar goes to the bottom and all files are disabled)
Ok extra, on maps smaller than 15, maybe 17500 tiles this doesn't happen.
I have a REALLY large map of France, it's 330 mb and contains a million tiles or something and it chrashes the program if i do this trick.
Says: program does not respond.
To repeat this trick, if the program is busy disabling or enabling ALL the tiles, click like a maniac on the right tile window. Soon or later it stops, gives an hourglass, and PING the tiles are enabled in 1 second. Also disabled if you do the "trick" again.