I need an equation for some data

Discussion in 'Physics & Math' started by Pi-Sudoku, May 5, 2007.

  1. Pi-Sudoku Slightly extreme Registered Senior Member

    Messages:
    526
    I have a large amount of data and need to calculate an equation of the line of best fit.

    This may sound simple but....

    I need it of a very high order due to the fact that each data set i have has 3000 data points that dont correlate particularly closely. The line will have to be a complex polynomial with over 1000 roots

    I am prepared to have a computer do the actual number crunching but need some kind of idea about how to calculate the equation.

    Thanks in advance

    Pi
     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. S.A.M. uniquely dreadful Valued Senior Member

    Messages:
    72,825
    Is that the same as a linear regression line of best fit?

    If so, you can plot a scatter plot using Microsoft Excel and ask for the equation.
     
  4. Google AdSense Guest Advertisement



    to hide all adverts.
  5. Pi-Sudoku Slightly extreme Registered Senior Member

    Messages:
    526
    the linear regression line is a linear (y = mx+c) equation, i need a curved line, i tried the statistics program "Autograph" but that only works to order 6

    I have consolidated my data now to around 200 key points if that makes it easier
     
  6. Google AdSense Guest Advertisement



    to hide all adverts.
  7. S.A.M. uniquely dreadful Valued Senior Member

    Messages:
    72,825
    If you need a curved line, then don't tick on linear, and you can still get it in excel.

    I use SAS but that involves writing code.
     
  8. przyk squishy Valued Senior Member

    Messages:
    3,203
    What in the world could you want a 200[sup]th[/sup] degree polynomial for?
     
  9. Pi-Sudoku Slightly extreme Registered Senior Member

    Messages:
    526
    I did it in excel but excel only goes up to order 6

    Please Register or Log in to view the hidden image!



    I am interested in this SAS thing....
     
  10. S.A.M. uniquely dreadful Valued Senior Member

    Messages:
    72,825
  11. Zephyr Humans are ONE Registered Senior Member

    Messages:
    3,371
    A polynomial with such high degree sounds like major overfitting. It will be memorising the data rather than giving you a few parameters to interpret.

    If you're sure that's what you want, though, look at polynomial regression. And if you don't have access to SAS, try R. It's open source.

    http://www.r-project.org/
     
  12. D H Some other guy Valued Senior Member

    Messages:
    2,257
    Zephyr is spot-on here. It is very rare to need a model of thousands of terms, and when such a model is needed, millions of data points are needed rather than just a few thousand. The problem with overfitting is that an overfit model has very limited extrapolative capabilities. With even more overfitting, the resultant model has very little interpolative capabilities.
     
  13. S.A.M. uniquely dreadful Valued Senior Member

    Messages:
    72,825
    Oh yeah, I forgot R, mostly cos I don't use it.

    Please Register or Log in to view the hidden image!

     
  14. Roman Banned Banned

    Messages:
    11,560
    I would natural log it, then run it through SPSS.
     
  15. temur man of no words Registered Senior Member

    Messages:
    1,330
    Polynomial fitting is the same as linear fitting. You just have to solve a bigger least squares problem. Fitting by a degree 1000 polynomial would give you 1001x1001 normal equation, which is nothing for computer.

    I would suggest you to use spline fitting or a low degree piecewise polynomial fitting, which are much more flexible and efficient.
     
  16. Pete It's not rocket surgery Registered Senior Member

    Messages:
    10,167
    Hi Pi-Sudoku,
    What is the problem that you're addressing?
    Perhaps a polynomial isn't the best model?
     
  17. iceaura Valued Senior Member

    Messages:
    30,994
    Depending on what you are doing, it's possible that specifying too high an order in your fitting equation will actually reduce your accuracy.

    Very few real situations in my admittedly limited experience (ecology/biology) yield an improved fit past order 5.
     
  18. Dinosaur Rational Skeptic Valued Senior Member

    Messages:
    4,885
    If a polynomial fit of order ten or less does not do the job, you should not try a polynomial fit at all. Are you sure that your data is not random?

    Have you plotted it? If so, what does it look like?

    If you have not plotted it, you have skipped a critical step in developing an approximating function.
     
  19. DieDaily Registered Member

    Messages:
    4
    No form of polynomial fitting is appropriate, probably. You have a wave more than a function. Why not just use a Fourier Transform?
     

Share This Page