RomRaider

Here are some scripts I crafted up for bulk loading of CSV log files. OpenPort's CAN RAM logging is awesome, but you need a more powerful tool like this one if you don't want to sift through huge log files. I data log my entire commute and end up with logs that are typically in the 4-6MB range. Yeah. Excel is not going to cut it. Excel's 65000 row limit and my slowish computer made interpreting my logs a headache.

With this approach, my 7 year old Pentium 4 can crunch through 16MB of CSV data in about 20 seconds. It's awesome.

Disclaimer: I have neither the time nor the patience to teach you Matlab and I offer these scripts to the community without any implied support. I'm just too busy. But if you know Matlab, or can get these to work on freeware tool like Octave (possibility), more power to you.

el_comp.m: Similar code/same goal as Airboy's spreadsheet:

viewtopic.php?f=32&t=5483

data_bin.m: Takes data structs and bins the data into specified number of bins.
data_cat.m: Concatenates data structs.
data_filter.m: Filters data based on some criteria (usually from a find() expression)
data_load.m: load a CSV file into a data struct.
data_write.m: Write data to a CSV file. Can be useful for extracting filtered data to another CSV file.
knock_analysis.m: Load in huge log files and quickly identify knock events (FBKC > 0).
load_excel_table.m: Copy a table from RomRaider or Excel, paste it into your text editor, and quickly load it into matlab.
ol_analyzer.m: Quickly filter and load data for which throttle is wide open.
timing_scale.m: THis is mildly relevant but shows an approach to interpolate RomRaider's 3d tables, i.e. re-scale X and Y axes.
write_excel_table.m: Write output back to text that can be copied and pasted back into Excel or RomRaider.


Cheers

I know, I know, these came with a proviso about not teaching Matlab - but could someone at least give a basic idea of how to use these - e.g. which scripts to use first, and the syntax? I'm getting so frustrated crunching huge CAN logs using Excel...

You would, for example, run el_comp.m first.

Matlab is the ideal environment to process these logs due to the simplicity of csv i/o and straightforward method of processing large matrices of data

WARNING: These are not for the average "no computer skills" user. Much reading and experimentation on your part is required to get meaningful results.

For these scripts to operate on your CSV files the OpenPort2.0 column headers must have the following simplified case sensitive names:
anything else and the scripts will break.
You should make your OP2 config file with these in mind if you plan on using these scripts. Or change your CSV files after the fact to use these column names.

Here are a set of the original scripts tweaked to work with GNU Octave. When I say tweaked, I mean they at least load the data and display some graphs but I have not verified the accuracy of the computed results. You will need to complete that task on your own.

Happy Computing!!!

Thanks, will give this a go Btw, suggest to just use mrp corrected instead.

I installed gnu octave using the package found here. http://downloads.sourceforge.net/project/octave/Octave_Windows%20-%20MinGW/Octave%203.2.4%20for%20Windows%20MinGW32%20Installer/Octave-3.2.4_i686-pc-mingw32_gcc-4.4.0_setup.exe?r=http%3A%2F%2Foctave.sourceforge.net%2F&ts=1313190576&use_mirror=nchc

Installation is as is only. I then put the scripts in the same folder where Octave is. Launching Octave, I type run el_comp.m I used the sample file included in the zip.

This is one figure 2, one of several generated by that script.

Where can I put the x and y values of my map for el comp? Also I get a warning message in the command prompt, division by zero.

What dschultz said is correct about the column names. Those scripts expect certain parameters to be named accordingly. ANd you must of course log all those parameters!

MFB: What you need to do is, in the directory where your log files are, is create a new text file. Copy from RomRaider your Cruise EL-COMP table and paste it into the text file. Save that text file as 'maf_comp_table.map' (the script looks for this file)

Okay, once the el_Comp.m script has run, it will save the new comp table to a file called maf_comp_table_new.map. You can copy and paste this back into RR.

Please, be on the lookout for any bugs you might see. For example, there is no compensation for O2 sensor latency--I'm not sure if that's a problem or not.

Enjoy


---edit
I Forgot to mention. I developed this script based on the el-comp table from an 08 sti. I imagine the table has different number of rows and columns on other ECU's, so you might have to tweak the 'bins' for rpm and MRP in the script itself.

Hi,

how do I tweak the bins for rpm and mrp? Better yet to make another script that will accomodate custom x and y values for the el comp. TIA

Just change the following lines:

mrp_bins = [-10.64,-9.67,-8.7,-7.73,-6.77,-5.8,-3.87,-1.93,0,1.93,3.87];
rpm_bins_cruise =[ 650, 800,1000,1200,1400,1600,1800,2000,2200,...
2400,2600,2700,2750,2800,3000,3200,3400,3600,...
3800,4000,4400,4800,5000];

to be the manifold pressure and rpm bins that you need.

Or you can grab these from the file that is loaded later on:

[rpm,mrp,old_comp_matrix] = load_el_comp_table('maf_comp_table.map');

which I should really just change to load_excel_table() since they do the same thing, really...

Version of el_comp.m that will automatically load the proper RPM and MRP bins from the user's maf_comp_table.map file.

I had to add the .txt extension to the filename to get around RR forum attachment rules (wouldn't allow .m), so rename it to el_comp.m

This Matlab file will not work in Octave so the changes made to this file need to be copied over to the Octave specific version of el_comp.

EDIT: I update the Octave scripts in my post above, hopefully it works I did not test it.

woops sorry about that. I don't have any experience with Octave. I'll have to try that out.

I got this to work - problem was that my headings were not renamed as above (thanks for the tip Dale!).

All I can say is wow... 16Mb log with 82,000 plus lines in 0.72 seconds, astonishing!

Thanks so much for this, Nil5.

Great! Tips on how you used the data to come up with results flashed back to car? Octave? Matlab aint freeware and costs more than usual.
What is the full parameter name for fi1pw ? Is that injector pulse width?

yes; Fuel Injector r1 Pulse Width

It is not used for anything really meaningful at all in the engine load compensation calculations, except possibly filtering out samples where the injectors are shut off (this ought to be filtered out anyway by other logical tests). I think I was (at one time) trying to look for correlations between fuel trims and other parameters, such as Injector pulse width, temperature, MAF, etc.

Anyway... if I weren't so distracted with life right now (and if hacking matlab scripts weren't so similar to real work) I might devote some free time to improving these. Maybe later.

The best thing somebody could do is double check my algorithm and data filters for the input data. I.e. is the algorithm correct? Is the idea of comparing AFR sensor directly to closed-loop fueling target correct? Do I need to compensate for the O2 sensor latency (not currently done)? I couldn't come up with a good way of filtering out big throttle transitions. That is another area for improvement.