RomRaider

For the application the XML files is the database. But that doesn't mean we have to work in that environmental. XML is just a language for describing data for translation between two non-compatible systems, that's why is was invented. In the open-source spirit, we are all free to work in the environment and with the tools we are most comfortable with (Fred uses a Mac, I use Linux and Win) as long as the end result is still compatible with what the user or application needs to operate.

Sure it can always use more optimization, no argument here...

That had been mentioned a while ago. Java is quite at home working with databases. XML would allow advanced user customization much more easily than a DB though.
The current RR Editor XML parsing code needs to be re-factored/rewritten anyway (not sure which term is correct here, Fred will let me know) as it is currently based on a bunch of deprecated classes. We should probably try and model it on the code the RR Logger uses to parse XML.

Thus far I've been unsuccessful in getting an answer out of you about additional practical benefits and information (aside from the two I already discussed). I recognize there is some analytical shortcuts for recognizing trends across different roms, etc.. but I don't see much value in this, and the information exists in the xml.

So, I still don't see any benefit to using two different database formats for development and runtime, other than temporary convenience (yuck).



After looking at the logger xml classes a bit, I agree.

Regarding advanced user stuff, I would think it is trivial to add a small gui to edit the database and a standardized import format (IDA map, etc) once the heavy lifting is done. From there, a similar concept to the three levels of definitions could be implemented.

And of course, it can always spit out a definition for ECUFlash.

Anyway, the important things to figure out before proceeding with any changes:
1. SQL vs XML re-factoring of RomRaider, which is more difficult or expensive (time). My knowledge of SQL is quite limited.
2. What kind of speed increase would SQL provide over a good xml design.
3. What other functionality benefits.

I've been using SQL for 20+ years so there's no learning curve for me. But, I would stick to XML for the apps and use the EcuFlash format with the three meta data dirs as discussed. I believe there would be less to do in the code to accommodate this and advance users can still edit XML without needing a DB support interface. At least for now.

I'm not sure to be honest. With a DB there's less to do initially as you don't need to parse the XML into a DOM and manipulate it until you have all the tables variables set. In a DB all that info would exist through related normalized tables, issue query, get the results and set the table up. The DB would also be smaller in size. My logger DB is about 450kB but the logger.xml generated from it is 2.5Mb


That's hard to answer when an ECU version doesn't exist. But I'm sure they would appear soon after its creation. I know that's vague but I wasn't pushing to use a DB directly with RR, but more for the support of the XML file creation.

Fixed! I'm a Linux guy through and through. 4 physical machines, 1 VPS, 3 VMs. 1 VM is XP, 1 ubuntu 12, 1 debian. 1 physical is OSX, the other 3 are deb sid. The VPS is Debian Stable. Deb Sid > MacOSX, for me.


Replaced! It's hideous.


No, it's not. It's based on a bunch of Sun internal classes instead of the same code from an external dep. This is easy to fix. I already did the fix here. I just didn't want to add bloat to the repo, so it can wait until Maven, from my perspective.


If that involves serialisation and direct file > object auto-parsing by a package, then yes, otherwise probably no. But I didn't look at what you have.


Dale's point here is that the source format should be the richest format. If the richest format is not the XML, then that should be an end user format. I have a similar problem to solve for my project. I'm still trying to figure out what the source format should be. Output formats could be XML, C code, YAML, JSON, documentation, and more. If I used C for the source, it'd be awkward to get it into the other formats.

I think using a database for the end user app is an absolutely terrible idea.

What needs to be done is that the defs need to be properly indexed, and then the required def loaded when needed. Then you're dealing with in-memory in-app access, and it doesn't get faster than that. Right now with the one file setup you load ALL of them in one HUGE go, using bad/inefficient/manual/breakage-prone code.


Likely negligible or worse. You'd have an entire DB process-set on the box, using, in most cases, a sh|tty low-performance NTFS file system to back it, hogging more ram, plus inter-process communication overheads. Terrible idea for this application IMO.


Is that single-line XML or indented? Are the tags all optiomised single-char or <bigLongCamelCaseWordsWithTinyDataInBetween>1</bigLongCamelCaseWordsWithTinyDataInBetween> point made? :-)

Future input to this thread via skype and someone else's keyboard.

Fred.

I stand corrected. The MacBook is to lull you to sleep...

Yes you are right, still bad and thanks for the fix.

That's where we are trying to get to by moving away from the single huge XML file and onto the EcuFlash format which is based on loading two or maybe three files worst case. This will speed up file access and reduce what's loading into memory.

Agreed, that's why I didn't pursue this option. Something like HSQLDB would make the app extremely bloated and a memory hog to run at it's best. And there's really no need. All we want is to read the XML an create our table objects, then unload the XML as we don't need it anymore.

I was comparing the normalized tables as they are stored on disk to the pretty XML file that's exported from the DB. There's no need to have pretty XML, the application doesn't need it, only the humans that want to hack away at it.

You've heard of speech to text apps right?

I have, but I'm finding good success with the "lay on the beach and avoid the internet" technique too. :-)

+1

I've switched everything over to a triple branch schema of Stable<-Beta<-Alpha and added all of the latest definitions from the experimental forum to their appropriate branches.

Again, if anyone has any questions on how to use git for this purpose, or any other comments/concerns/threats, join #RomRaider on freenode irc. You can also send me a PM to set up a skype call.

I've refactored all of the alpha definitions into a base inheritance schema. Templates now only contain table name/address/elements(if needed), and also have a link to their discussion threads here.

So, any changes to descriptions, scalings, etc... will occur in the 32BITBASE.xml file so they can be inherited for easy fixes/changes.

You added the basics that I needed to create the definition in the ecuflash tool into the definition, but now at what point do you want me to resubmit the actual code back into the definition?


Do you want us contributing or would you rather us just keep posting in threads and you doing the leg work?


Edit: ohhhhhhhhhh, I am slow. Now I see what you did here. So essentially we'll have to re-pull the 32bit base every time we want to use some of these new defs. Cool!

Yep, the idea is to keep everything together and organized. I know first hand how much of a pita it can be to manage a bunch of custom definitions, and I'm sure it's worse for tuners working with a bunch of cars.

Once I get everything entered into this repo and finish up some speed density patches I'll be adding functionality to RomRaider to automatically download all the repo with the ability to switch between stable/beta/alpha as well as an update feature.

Feel free to contribute using git as per my OP, it's much easier than it sounds and there are a few people around that would be glad to help.

Lots of cool stuff going on here!

I really like the idea of using a database as the the canonical store, and generating XML from it. I just think of it as a big step toward creating better tools for creating and editing definitions. XML is perfectly fine as a way to transfer information from one process to another (e.g. out of the DB and into RomRaider/EcuFlash) but it's a cumbersome format for manual editing. It works, certainly, but it's just adequate.

Imagine if we wanted to rename a table, for example. With XML that would mean opening every file and doing a search-and-replace. With a database that would only require editing the one row in the database that contains the name of the table, and then you run a utility (or just click a button in the SuperAwesomeRomRaiderDatabaseEditor that we'll have one day) which re-generates all of the XML. I'm sure this would come in handy for adding support for more languages, since it seems optimistic to think that translating every possible table name will get them all correct on the first try.

For starters the database could be essentially the same data as the XML, just normalized.

But we could go further... for 2D and 3D tables the DB could contain the addresses of the table headers in the ROM - not the addresses of the actual rows, columns, and data cells, but the address of the structure in the ROM that contains those addresses. Adding a new table for a particular ECU should be as easy as opening the DB editor, picking the cal-id from a list, picking the table name from a list, and entering the table header address into a form. And click the "generate xml" button.

It would not be hard to write a tool that extracts the information that the XML needs, based on table header addresses in a DB and a copies of the ROMs.

Since you never actually touch XML yourself, there's no chance of mismatched angle brackets or quotes.

Perhaps the DB should contain both the table header addresses and the table details (that stuff we have in the XML now - row address, row count, column address, column count, data address) so that they can be cross-checked against ROM images. If they don't align, that should raise a red flag. We could extract those from a collection of ROM files and the information in the current XML.

For 1d tables we could tell the DB what the expected ranges are, and the ROM cross-referencing utility could raise a red flag if it finds a ROM with an injector scaling that turns out to be 0.01231 cc/min, or 2,432,976 cc/min, because the address was entered incorrectly.

If you're feeling really motivated one day, you could ask the database how many ROMs are missing some new table you're interested in, and the DB would give you a list of cal-ids, and you could work through the list, alternating between IDC and the DB editor, and ever needing to touch XML. You just click the generate button when you're done.

Or when someone adds a new ROM you could have the database tell you if any tables were left out.

There's a ton of useful things that we could do if everything we know about ROMs got stored in a normalized database, with XML being used for just the subset of that's needed by RomRaider and EcuFlash.

Of course those things only happen if someone writes the tools to do those things, but the odds of those tools ever being written will be better if the database gets created first. And since dschultz seems motivated to make the DB happen, I think y'all should take advantage of the opportunity.

Thanks for pointing out some of the advantages of the database.

I have been working on the database for the ECU defs over the last few weeks. I've already found some inconsistencies in the Editor defs between standard and metric versions. As pointed out this type of thing will be corrected once exported from a common db source. I like the idea of having the ROM table structure in the db rather than the axis/data ROM address (I use the structure already in my IDC scripts and it works well). The advantage here, as you say is the accuracy of the final product. The disadvantage is that we need a copy of every ROM, and some we don't have as well as it's Subaru specific. Another PLUS you can work with the ECU base units of measure as you can get that from the structure too. Once you know that then you can easily expand on how to convert that base unit measure to the Standard, Imperial and Metric units, however you wish to work. This could be in the database or we can move that function to the RR and EcuFlash app for user selection on the fly.

I've always liked the idea of using the table vector table / lookup table, at least for verification purposes.

I still do not see any benefit from using a database other than the ability to output different definition sets, which I do not care for.

It may only require a one line edit for a table change, but that's only telling half the story. It still has to do a search of sorts when it dumps the individual files, and still does plenty of file accessing. Once definitions are in an object mapped format, they can easily be mapped to objects without clunky parsing code, then a search and replace can be performed. Same goes for populating a list of undefined roms, which would also require a search in the db format. If speed is a concern the entire set can be binary serialized after editing.

Sure there's lots more that needs to happen after the one-line-edit, but the rest of that work can be automated.

Edit one line, click "generate," done.

It's not about saving CPU time, it's about saving people time.

It'll take work to write the DB-to-XML dumper, but I expect that it'll be well worth the time that it saves.