RomRaider

I've considered a way to save maps that would be more easily portable, use less disk (and more importantly, web server) space, store notes and allow maps to be applied to different ECU revisions. I've given it a bit more thought lately, and wanted some input.

Basically, I'm imagining an XML file that'll store only the tables changed (the app will need to an unmodified ROM as a reference). It should be simple enough to implement, and I think it would simplify sharing maps.

Doing this will help out with a couple of ideas I've had for the site (a good basemaps section, a "my rom archive" type of thing for remotely storing and receiving input from people and such).

Anyway, just looking for input. Let me know what ya think.

160kB up to 1MB is large?

I imagine an XML for a tune that has had about 2 dozens maps touched up could easily surpass that since text isn't very efficient for file size. Unless you ZIP or RAR it. You can ZIP or RAR the whole HEX/ROM file and get good compression as well.

What are the differences in the data size of the tables and the typical .hex raw file? If it is of any significance, then the app sounds like a good idea.

I thought about your idea some more and it seems like an elegant solution to a huge storage nightmare. We have what ~80 different Subaru Roms? Using a changeset idea on a known ROM upload (you would have to ID the rom upon upload), your web server could do a compare and parse out the data that is different, saving only the changes. Then upon download, it would "rebuild" the entire image. I would think you would want a service that actually compares the binary data after ID-ing the known image and not use any RomRaider definitions. It would be much like a code control changeset idea.

NOOOooo!

This would make sharing far more difficult for any who prefer to use EcuFlash or EcuEdit or any program other than RomRaider to modify their ROMs.

What about ROMs with modifications outside of RomRaider's defined maps? These would include ROMs modified with EcuEdit and ROMs which have their executable code changed for extra functionality (e.g. faster logging, realtime, etc.).

As Freon noted earlier, ROM files compress fairly well. I suggest that an RomRaider archive of "good" maps be based on .ZIP files (please, for simplicity's sake, no .RAR) each of which contains at least two compressed files. There would be a .HEX compressed file containing the ROM image and a sstandardized format .TXT compressed file of the same name containing a description of the modified ROM, expected engine mods, author, special instructions, etc.. If tables not defined in RomRaider were modified, a compressed .XML file might also be included. It would be wise to develop a naming standard/convention for the .ZIP file and the compressed files within it.

i have to say that i have never said to myself "damn, these rom files are taking up too much storage space."

Considering Ecuedit comes with no definitions, that would be hard for it to accomplish. I'm guessing you mean roms modified with xmlwrite defs.

You could add the comments to the rom itself. The ecu defs could designate a small number of bytes for each rom that could be used for comments and changed through RomRaider.

If it is a server space issue, you could disallow .hex and .bin files and require people to submit zip archives only for roms.

Most EcuEdit users create XML definitions automatically, either with XMLWRITE or with EcuEdit's builtin RomRaider XML converter. Those using EcuEdit with XMLWRITE might modify tables which aren't defined in RomRaider. Other EcuEdit users (Freon springs to mind) have the ability to create and understand map definitions beyond even those created by XMLWRITE.

It seemed like the original proposal was to store changes based on RomRaider's map definitions. I may have misunderstood.

gabedude's proposal is the best one if server storage is an issue: treat ROM images as if they were source files and only store the the differences. You'd start by collecting all the unmodified FHI ROM images you can find, store them in the database, and index each one by its calibration ID (CID). When a modifed ROM is submitted the server pulls its calibration ID from either location x'200' for a 16-bit ROM or x'2000' for a 32-bit ROM.

However, these 'diff' files aren't really 'portable.'

Umm, this is a proposal for sharing roms among RomRaider users on the RomRaider site. We don't have a need to support Ecuedit in any way. And certainly shouldn't scrap an idea just because a few users that don't even use RomRaider might not be able to take advantage of it. There's an Ecuedit site, isn't there? Maybe Epifan can do something similar, but that is up to him. As far as those creating their own Subaru definitions, I can count the number of guys on one hand.

I'm very disappointed in the direction you guys are going with this.

I'm not buying this "save disk space" routine. It doesn't even make sense. Not only are the files already comically small to begin with, but your proposal would create files that are probably going to be just as large, if not larger. There is going to be a point at which once you've touched enough of a map that the XML text spelling out all the changes is going to be larger than just the whole HEX file was to begin with.

Enable file quotas for users. Maybe 5-10MB per user. Ask for donations to upgrade your web host for more space. Just 5GB would hold 5000 worst-case 1MB 7058 ROMs, or over 30,000 160kB HC16 roms.

This is really pissing me off. A map archive system, based on saving either full ROM files or ROM binary difference files, is far simpler and easier to implement than any system based on RomRaider map XMLs. And it has the added benefit of being ROM-editor-neutral. Why would you want to exclude people who use Colby's EcuFlash as their ROM-editor from being able to use these archives? What do you have against Colby or his program?

What are talking about? Regardless of what editor they are using, it would support any tables that are supported by RomRaider. That is sufficient for user's viewing another user's tune. The idea is not just to save space. You would be able to compare/copy easily between different ecu revisions (assuming the same table size). Remember, this is merely to make it easier for users to compare/copy tunes from one revision to another and save some storage space. Unless I am misunderstanding Jared, RomRaider would still save/open a hex file (it would have to in order to flash with Ecuflash), but would have the option of creating an additional xml file that could be uploaded and shared by the community.

Hmmm. Well, using only RomRaider definitions has its own advantages as well. It also keeps us seperated from the other open projects in a way. Either way, an easier way to share maps is a great idea for this site. Many people have loads of basemaps they start to tune a particular car with and have no way to store the 50+ 1 MB images, etc on the web.

I was thinking in the form of flexibility and supporting definitions/modifications to ROMs that RomRaider does not have definitions for. This would be up to Jared really on whether or not to only support RomRaider defs or to take a binary changeset style code control approach supporting any modified ROM. This may be a bad idea. What if someone posts a modification to a ROM that kills the boot code and bricks the ECU? If we use only RomRaider defs, this would minimize that type of risk.

-Gabe

My vote is to focus development on the RomRaider overhaul and RamTune functionality. Adding RamTune is going to make a huge leap in open source tuning for the user. All the other stuff can come later.

It would be smaller to just save out chunks of raw HEX in the XML file. Every time there is a difference, add a tag with the start location of the difference, and the actual hex bytes of the changed file. i.e.

I see no reason for basing this on the Enguinity XML besides trying to lock other editors out.

I also question encouraging people to allow their work in your database in your proprietary data format.