RomRaider

Hi Everyone,

I've set up a git repository on GitHub to keep track of experimental definitions. I initially set this up just to keep track of definitions for Mods/Patches, and still plan to. I added the existing definitions after some definition requests, and defining some new standard parameters for use with Mods.

You can find the repo here: https://github.com/Merp/SubaruDefs

Currently, the master branch contains the latest, up to date, NON-experimental definitions. The experimental branch contains all of the definitions posted in this forum into the romraider directory (including the unified logger def), and the "/ECUFlash/subaru standard" directory (including 32BITBASE).

Benefits of using Git:
Better tracking of definition history/changes.
Easier to implement standardization of new tables, scalings, descriptions, formats, etc.
Ease of use for end users, no more scouring a forum to find definitions or get an updated def.
Easier to collaborate and stay organized.
By pointing RomRaider or ECUFlash at the local repo, you can easily switch between different branches for quick tests/comparisons.
Ability to directly link to the latest commit: https://github.com/Merp/SubaruDefs/zipball/Stable or https://github.com/Merp/SubaruDefs/zipball/Beta or https://github.com/Merp/SubaruDefs/zipball/Alpha
-this will allow the RomRaider installer to download the latest definitions automatically at install time.

Learn how to use Git:
The basics:
http://gitimmersion.com/
http://rogerdudler.github.com/git-guide/

Useful References:
http://gitref.org/
http://learn.github.com/p/intro.html
http://zrusin.blogspot.com/2007/09/git-cheat-sheet.html
http://tech.fredcooke.com/2012/01/19/git-succinctly/

QUICK START GUIDE
If you havy any questions about Git, join #romraider on irc.freenode.net or PM me to set up a skype call.

INSTRUCTIONS FOR USERS:

1a. Windows Users: Download and install the latest mysysgit from here: http://code.google.com/p/msysgit/downlo ... ficial+git
1b. Linux Users: Why are you reading this? .. apt-get update && apt-get install git, then pick a GUI if you require one.
1c. Mac Users: Format your hard drive and proceed to step 1a/1b

2. Clone the github repo locally.
-Open Git Gui and select Clone Existing. Use the Git URL: https://github.com/Merp/SubaruDefs.git and select a save location.

3. Create a tracking branch.
-Branch -> Create. Select 'Match tracking branch name'
-Select appropriate branch (typically Alpha for experimental stuff)

4. Point ECUFlash and RomRaider to your new definitions.

5. To update your definitions: Repeat step 3.

INSTRUCTIONS FOR DEFINITION DEVELOPERS:

1. Visit http://github.com and create an account

2. Visit https://github.com/Merp/SubaruDefs and click 'Fork' in the upper right. This creates a 'clone' of the repo on your github account.

3a. Windows Users: Download and install the latest mysysgit from here: http://code.google.com/p/msysgit/downlo ... ficial+git
3b. Linux Users: Why are you reading this? .. apt-get update && apt-get install git, then pick a GUI if you require one.
3c. Mac Users: Format your hard drive and proceed to step 3a/3b.

At this point, you can proceed using Git GUI installed by mysysgit or use Git Bash (Command Line)

4. Clone your remote github repo locally.
-Open Git Gui and select Clone Existing. Use your Git URL: https://github.com/*username*/SubaruDefs.git and select a save location.

5. Create a tracking branch.
-Branch -> Create. Select 'Match tracking branch name'
-Select appropriate branch (typically Alpha for new defs)

6. Before doing additional work, first make sure you're up to date with my repo.
-Remote -> Add Remote, Enter my git url and a name. You may need to restart Git GUI after this!!
-Repository -> Visualize All Branch History
-If your local branch ie: "Alpha" is not on the same commit (dot) as "Merp/Alpha", you need to update your branch. You can either rebase, or use step 5 (only works if you haven't made changes before realizing we're out of sync).

7. Make changes to definitions.
-Remember, commit often! Preferably commit once for each table you're working on.

8. Push to your github repo "origin", and send a pull request or message on IRC.


Rebase using Git GUI:
Add the following to your .gitconfig:

Good work! :-)

I bet this must be useful: "By pointing RomRaider or ECUFlash at the local repo, you can easily switch between different branches for quick tests/comparisons."

Fred.

You've been busy
Excellent work... STICKY!

Damn... that must have taken some time. Will need to do some reading - this looks like a much neater solution.

I'll be posting up some basic instructions for those that want to use it to develop.

I don't know what units everyone is using for experimental stuff (ecuflash format) so I just put everything in the standard folder with the newest 32BITBASE. I don't see a point in separate sets (other than ECUFlash support), as the work would be better spent adding functionality to RomRaider to switch units at runtime, using a single def set.

Another thing that needs to get ironed out:

Criteria to have a definition moved from Experimental to Stable.

I'll just throw this out there..

1. 2 Definition developers must perform runtime sanity checks on all tables.
2a. 5 End users give a positive report
--OR-
2b. Minimum waiting period (from receiving first positive report?) 1 month?

Feel free to chime in!

Agreed, but with so many un-defined parameters being uncovered, when do we exit "experimental"? Maybe for the new ROMs with current parameters defined. (2a)

I'll quote the guy who managed all this before (BTW: Git won't do any of this for you).



The IDA scripts and a couple tools I wrote did much of this for the 32 bit stuff. So it would make very accurate defs if the developer defined it correctly in IDA first.

Backporting new finds to existing defs is a whole new ball of wax though. That's why I have a database for the logger info and was going to expand it for the editor data.

I would recommend moving the definition into stable after:
1. All currently uncovered tables are defined
2. It passes whatever checks we decide to implement

After that, the workflow for adding new tables to an existing rom is as follows:
1. Branch the rom from experimental
2. Use the public templates from experimental to define the new table using whatever tools you have at your disposal
3. Merge back to experimental
4. Vetting process again -> move to stable

Of course, Git is for managing history, workflow, collaboration, etc.. please see the first post if the intention is unclear. Whatever tools/methods are used for the vetting process work on top of Git.



I take it merch's tools are all unreleased and closed source?

Is the reason you have not moved many definitions into "stable" a lack of these checks? That is, you feel they are mandatory checks before going to stable?

Could you elaborate on the database and how it is used for backporting?

What I had in mind for backporting is a directory containing XML templates so that everything gets standardized instead of perpetuating the huge mess it has become. Another alternative to this is to create a branch from experimental and add the templates directly into 32BITBASE and merge back when they are finalized. Using 32BITBASE would be a cleaner solution, although harder to work with for those using text editors to define.

I think we need to make a distinction between the newly discovered, truly 'experimental' parameters versus definitions defined for new roms that are post 2009 (which are now classified 'experimental' by default). For the more vanilla new definitions, I think a minimum number of verified reports and time as suggested above at the very least are needed, whereas the newly discovered parameters need much more scrutiny.

I would be very uncomfortable with some of the material I've put up recently, making it into any stable release without a lot more testing...

I agree, there should be some distinction made between those two.

Perhaps we should have three parallel branches: Stable, Beta, and Alpha.

Where:

Stable = Table structures and actual rom data have been vetted/finalized.

Beta = Table structures have been vetted/finalized, actual rom data only has basic sanity checks.

Alpha = Table structures are in development, actual rom data only has sanity checks.

Then if you want to define existing tables for a new rom, branch out of Beta and do your work.

If you want to define new/alpha tables for the bases or an individual rom, branch out of Alpha.

Had a chat with Fred (Fearless) about this and he agrees with making the Stable/Beta/Alpha distinctions.

He recommends 2 users with 2 week use period before the Beta -> Stable move.

And a publicly reported peer review with script checks (like Dale quoted from Merch) before the Alpha -> Beta move.

No he did not release them.

I mentioned my checks in my previous post. I believe the defs that I release are 99.9% accurate in the first version.

The reason for a lack of experimental to official is two fold. First was to be a move from RR style Editor def to EchFlash and Jerod had started to develop this last year. Check his branch and forum posts. The second part is the ability to take the information discovered about a new table and back port them into the existing defs. So for instance, back port the "Per Gear Max Requested Torque" addresses of each ROM and add it to the existing defs and release it. For this you are not going to go and manually edit 10, 20, 100 CALIDs with the new info in a Git repo. The smart approach is to use a database and import/add the info based on relationships. Then export the new def out to XML format and store it in your repo. As I do with the logger def. I can add 50+ entries in a fraction of a second and publish a new logger.xml from the database. The only other option is to do it by hand and no one has the time for that.


If you are referring to the RR Editor def as a huge mess, then I agree. That's the reason to move away from that style. As for the EcuFlash def, there is no mess. They are all in the Experimental forum and they should all be based on the 32BITBASE the I've kept up to date (except for the metric one, sorry). Gathering them together to add them to an EcuFlash release is not difficult. Right now people just grab what they need.

I think you're really missing the point here, and I don't know how to make it more clear. How you do your work has absolutely nothing to do with Git. You could do your work through a text editor, script, database output, telepathy, or whatever you please. It is totally irrelevant to using Git!

Have you published the apps you're using to manage this database? Could you please elaborate on it a bit? Describe one of these relationships? Does it store any data/information that is not in or could not be interpreted from the definition set alone?

This database could work external of git (in the shadows so to speak, only as a tool to manipulate xml), or possibly within git depending on the encoding of the database.



I'm talking about the development and standardization of new tables. I also consider a forum filled with definitions, xml snips, and param lists a mess, but that's just me.

Agreed. Git is nothing more than a place that stores a bunch of files. So not very useful for the process of actually updating defs and such.

Yes. You must have missed my post.

Since XML is just a representation of data for the ease of exchange of information between systems. In our case the ECU ROMs is the source of the data. A relational database provides a method to relate all the ROM data and describe it in various measuring systems and languages. The XML defs we create are a way to describe the source data in a way an app and human can understand it. The Editor and Logger are the app users of the XML to understand how to read the data of the source.

Sure it does. You have to remember that XML is a representation of the database but does not describe relationships of data in a normalized fashion. A single data base would include the different measurement systems (imperial/metric) or languages for descriptions/labels. If you wish a def in German you ask your export tool to dump the data to XML using the German language description table and the measurement system of choice, metric for instance. No need to manage a set of defs in each language and measurement system. You just ask for an export of XML with the desired values. Currently we have:
EcuFlash - imperial
EcuFlash - metric
RR Editor - imperial
RR Editor - metric
RR Logger - English (metric/imperial)
RR Logger - German (metric/imperial)
That's six repositories of XML files that need to be managed. Easily done from a single relational database.

Yes external.

That we do need consensus on. That's why I had kept the 32BITBASE up to date for people to base their work on.

I do to. That's why I had created the Trac page, so people could see what is required and what is done. I don't consider a ROM defined until we have a RR def (both Editor and Logger) and an EcuFlash def. That's what I always post and mark it complete in Trac. Then it's easy to pull up all the "completed" tasks and move them to an official release. It's all tidy and neat in one place. We can't do much about people "defining" 1/3 of a ROM in one format and not posting updates or completing the work with a Logger def to.
Then we circle back to the addition of new tables to previously defined ROMs. Adding those tables and such to the existing repository of XML defs is the work of a database not with a text editor and 100s of XML files to edit.


And all of this is for the benefit of the end user how knows nothing of Git or databases or XML an just wants to download a def or set of defs to work on tuning a car.