Yet correlation analysis is argueably the single most important thing that one does with a data set. Such an analysis can help define trends, make predictions and uncover root causes for certain phenomena. But in order to do this properly, one must examine and test how good the correlation actually is.

For our purposes, the most critical parameter is the scatter or dispersion around the fit. We will deal with this issue explicilty later below.

An example of a "False Correleation"

An Example of a correlation that has a lot of "scatter" around it:

An Example of outliers at the data extremes

An Example of uneven data sampling in X:

An Example of uneven scatter as a function of X:

However, one has to be extremely careful about the form of the data
and whether or not a * linear * function is the best approximation.
As an example, consider the following data set which is the time
evolution of the world record in the 100 meter dash:

While there are standard tools for performing correlation analyses (this will be provided to you later) it is often done poorly. As a result, lots of erroneous analysis gets published, in virtually all fields.

Often times, there is simply not enough data to adequately define a correlation. This allows one to make ridiculous predictions which, although they can be supported by the data, make no sense.

A favorite example:

Here is a prediction that I made in the year 1839 (that was in the pre-internet era):

All presidents that are elected in a year that ends with a zero will die in office:

- 1840: William Henry Harrison dies in April of 1841

- 1860: Abraham Lincoln Assassinated

- 1880: Garfield Assassinated

- 1900: McKinley Assassinated

- 1920: Harding Died of illness

- 1940: Roosevelt Died of illness

- 1960: Kennedy Assassinated

Gee 7 events in a row, pretty good prediction, huh!

- 1980: Reagan Almost Assassinated
Conclusion: Its probably safe to run for office in the year 2000

Basics of Correlation:

- You have two sampling variables (X and Y). Does the
value of one variable depend on the other or are the variables
random.
- Correlation then determines the probability that the two
variables are randomly correlated.
- Correlations can be strong or weak.
- Strong correlations are extremely useful in identifying root
causes and/or what the most important variables are
- Weak correlations open the way for tremendous ambiguity
- If two variables are correlated, it means that one variable
can be written as a
*function*of the other. - Math Anxiety Alert While functions
can be complicated, here we will only be concerned with
*linear*correlation:Y = AX + B - A = the slope of the correlation
- B = a calibration constant (often called a zeropoint)
- if A is negative, the variables are anti-correlated

Correlation can be used to summarise the amount of association between 2 continuous variables. Plotting a "scatter" yields a "cloud"of points :

In general, we measure correlation by a parameter known as the correlation
coefficient, * r *.

* r * is between - 1 and 1

Mathematically, r is defined as

But we don't really care about this - we only care about using the value
of *r* as a rough guide to how well two variables are correlated.
Usually your eye is a good estimator of *r*.

Regression is now built into the tool

What we care about most is is the amount of diserpsion (scatter) that exists around the fitted linear relation

Final points about regression:

- Sometimes correlations can be defined by a single point which significantly extends the range of the data.
- Sometimes a few deviant points (devaint due to measurement error or poor sampling) can degrade the correlation. One cycle of rejection should occur for most data sets.