RomRaider

Scope creep is real... in my efforts to work on live tuning, I was getting sick of the very scattered and painfully organized ECUFlash/RomRaider definition database, and have decided to do something about it. Progress has slowed considerably now that the weather is better here in the Northeast USA, and regular life is starting to return slowly as vaccines roll out. In any case, at the moment I'm working on developing a combined definition format that should be much easier to maintain, extend, and collaborate with compared to the current ECUFlash/RR definitions. The major goals are as follows:

• Combined editor/logger definitions are stored in deterministic JSON for easy parsing (one-liner in Python, I'm sure similar in other languages as well). Definitions are split into individual files (a. la ECUFlash repository), one file for editor-related definitions, one for logger-related definitions, and a "general" file that points to editor/logger definitions. This directory/file structure is deterministic and auto-generated based off the contents of the definitions. File formatting also must be deterministic.... meaning if two people add a table to a definition, it will appear in the same portion and formatted identically to make it easy to diff and follow merge history on a repository. Easily handled by forcing sorting of all data, and auto-formatting the output. This leads to the second bullet-point...
  
• A usable GUI to facilitate definition creation, editing, etc. In my eyes, the ONLY way to make this effort worthwhile is to completely disallow hand-editing of definitions. This determinism will be generated by the GUI, and can likely be enforced by pull request hooks on a GitHub repository. Edit the definitions as needed in the GUI, and the repository is automatically generated.
  
• Sensible table/parameter definitions regarding scaling. Totally eliminate the "standard" vs "metric" repositories and get everything in one repo instead. Similarly, it should be straightforward to make language localization modular as well. I'm also storing datatypes and byte orders in the table/parameter definition, NOT in the scaling definition. Scalings should act as a black-box that take in one number and spit out another, with no regard for whether that input is a uint8 or a float.
  
• Multiple inheritance. Classic example for me is making an easy way to maintain MerpMod definitions. For example, let's take a definition for my '07 spec.B ECU, A2UI001L. Currently, I need to modify 32BITBASE with all the MerpMod tables, and then generate a new ECUFlash definition for the ROM, copying all the stock tables from the stock definition. Not to mention creating an ENTIRE new RomRaider logger definition, since they're all contained in one file. What I envision instead is making a MerpModBASE definition file (which only contains definitions and scalings relevant to MerpMod), and then my new definition simply inherits from A2UI001L (to get all the stock tables) as well as MerpModBASE, and then defines the addresses for the MerpMod tables only. I've already gotten a rough implementation of this working, so I know how I want to approach this and it's just a matter of implementing it.
  
• Forward/backwards compatibility with existing definitions. I'm currently writing the code to parse td-d's SubaruDefs GitHub repository, and store all of the definitions in my own format. I plan to implement writing this new combined definition format into legacy XML files that can be used in ECUFlash/RomRaider as is. To make my life easy, I have no intention of making these "diff-compatible" with the legacy definition repositories, but they should be functionally identical. Chances are the XML files that I generate will be completely self-contained, which is not an issue if they can be auto-generated from the new "master" repository (again... no hand editing of text files!)

Been working on this for a while now, albeit at a snail's pace. Starting out with just handling the SubaruDefs/SSM side of things, but I believe my approach will easily extend to handle all the other platforms RomRaider supports. Happy to hear any input from anyone on extra things to think about.

Just a couple of quick points:
- Measurement systems within one definition is already supported by both the RR Editor and Logger.
In the Editor, add the name attribute to a scale element:
<scaling name="Metric" ... />
<scaling name="Standard" ... />
etc.
In the Logger these are called conversions and the units attribute dictates the measurement system.

- Datatype is a table attribute in the Editor and is not defined in a scale element. In the Logger the conversion has a datatype, as that is part of the definition of the parameter as dictated by the ECU.

I agree a better definition management system is required. Most definitions are created from automated tools from IDA outputs now. The only hand editing that I'm aware of is that of people adding unique or custom items. Or creating a complete new definition for a new manufacturer.