Ok... I have a not so small task... and am looking for some suggestions.

Basically we have a group of 40 complicated 'units' that we consulted on. We had the builder include fault logging systems. Each one generates about 2000 faults a month. Each fault has a snapshot of 200 signals attached (what was the voltage/current/etc when the fault occured).

The problem is that this is just TOO MUCH data. Our client isn't using the system because it takes too much time to go through all the data. Most of the faults are fine, but if you have a combination of fault A, B, and C within a cetain time frame then you can suspect a bigger problem. Some faults you need to look at the attached signals to see if it's an urgent problem.

So the plan is to take the all the data and somehow simplify it to a report that says what is suspected to be wrong, and how to fix it. Does anyone have ANY idea how you take this much data and have a program go through and pick out the bad apples?

(To make things even worse we don't have any 'real' programmers... so I have me an 2 interns working on it)

Anyone have any references to similar Expert Systems and possible hints on what kind of parsing they do?

I suppose if I was in your predicament I would try to decide if I want a system that does a Cronological search of a database of entries for a particular pattern filter, or to have the system apply any results that are sent to log through the filter at the time of being logged.

Doing the first would mean having all the data logged then having to find the needle in the haystack so to speak, which sounds like it's being done currently. It's good to have that method as a "Secondary" method of approaching the task should their be a failure in the other system I mentioned.

As long as you as a programmer can take the data output and parse it through a filter before being "logged" then you should have no problems. However with such a project colleges and universities would teach that Documentation of the philosophy of how your small project works and implimented should be created so as to both aid a team understanding how to create and implimentate but also can be of use if the system in the future needs altering and expanding.

(Of course it's up to you how professionally you want to conduct it)

The first method may be the only method. The data is stored on the device, and downloaded about once a week using WLAN or CDPD.

What I'm more worried about is methods of taking signal/fault data and extracting patterns from them. I could very easily hard code certain patterns in.. but this doesn't at all seem robust.

Perhaps what you could do then is create a Table within the database that contains your "Filter" entries. You can then match each entry within that Table with that of the main database through SQL (I guess your using SQL).

This way if you create any new filters, you just add them to that table.

It's up to you to decide how the Filter is matched, either you have to loop the Filters with each position on the main table, or you have to loop the main Table with each position in the filter Table. I would suggest looping the Filter table guessing it will be small than the main table.