A little more about Degrees of Freedom

From my posts, it might be argued that the correct choice of the number of degrees of freedom (df) is fundamental, as it determines the significance of the results. It is crucial when we calculate the weighted mean. In my previous post, when we have applied the Bland and Kerry procedure, I’ve said that: “The most immediate thought is k-1 df, which is 2 in our case. However, we should take into account that each experiment has n samples (n-1 df). Therefore, in my opinion the correct way to take into account of df is: df=(k-1)*(n-1) if n is the same in all the experiments”.

It is logically correct, but with an important limitation: what about df if n varies along experiments?? Can we find df in a more general way starting from the formulae we used?

Suppose this simple case: we have an experiment with three replicates (1,2,4) and another with other three replicates (2,3,4). Exp1: Mean=2.33 (SD: 1.53) – Exp2: Mean=3 (SD: 1).

Let’s calculate the weighted mean.

W1=3/2.34=1.28; W2=3/1=3 =>

Weighted mean = (1.28*2.33 + 3*3)/(1.28+3) = 2.80

How can I change my data without varying the weighted mean? How many constraints do we have?

I can change the single means in n-1 ways (two per experiment), but this change influences the SD we use in the formula! To do an example:

To maintain 2.33 as mean in Exp1, I have to change at least two replicates, e.g. 0.5, 2.5, 4. However, SD=1.76 and also the Weighted mean is varying:

(0.99*2.33 + 3*3)/(0.99+3)=2.83 =>

To maintain the weighted mean unaltered, I have to maintain also the same SD in each experiment, and I can do it only varying all the three replicates => IN EACH EXPERIMENT, we have two constraints: mean and SD => each experiment has n-2 df.

To generalize, If I have k experiments and the experiment i has n_i replicates, the total number of df is:

Df=S_{i=1 to k} (n_i-2) (MATHEMATIC WAY)

In the example of the previous post (k=3 experiments each with 5 replicates), DF=(5-2)+(5-2)+(5-2) = 9 despite of 8 calculated with the “LOGIC METHOD”. The significance of the difference would be 0.0001 despite of 0.0002.

It should be noted that the LOGIC METHOD has more df than the MATHEMATIC one when k is high and n low, the contrary when k is low and n is high, although with several df (e.g. >20-30 per condition) the significances are slowly influenced by this choice.

WHAT is the BEST? The MATHEMATIC method starts from the used formula to calculate weighted mean and is more general, and therefore it is statistically more rigorous.

Weighted Means: Part 2

Let’s look at the Bland and Kerry procedure, giving that we have calculated the weighted mean as reported in the previous post. We will see the direct application of the procedure, without details about theory that can be read directly at the bibliographic reference.

Suppose that we have the same data recalled in the previous post.  We use only the difference between Controls and C1, which has the lowest significance.

The weighted mean of controls is 69.8, that of C1 is 89.6.

Firstly, we calculate a term called “Weighted sum of observations squared” (s²)  which is dependent on the number of experiments (k):

s² = k*(S_{i=1 to k} X_i²*W_i)/ (S_{i=1 to k} W_i)

In our cases:

s² (C)=3*(62²*0.0682+72.6²*0.0693+83.6²*0.0243)/(0.0682+0.0693+0.0243)=14782.25

s² (C1)=3*(85.6²*0.0266+86.8²*0.109+99.6²*0.0425)/(0.0266+0.109+0.0425)=24218.09

Then, we have to calculate the correction term, Corr, simply calculated as:

Corr=k*X²_weighted

Corr(C)=3*69.8²=14616.12 and Corr(C1)=3*89.6²=24084.48

Then, we define “Sum of squares about the mean” (S²) as:

S²=s²-Corr =>

S²(C)=166.13 and S²(C1)=133.61

The weighted estimation of SD of each condition is simply:

SD_best=sqrt[S²/(K-1)] =>

SD_best (C) = 9.11 and SD_best(C1) = 8.17

Note that in this case we have some comments to do:

1)  The values are independent on the number of replicates, but depend on the SD values of the single experiments, being it a SD weighted mean;

2) The relationship between SD calculated here and those calculated with the previously presented method was approximately sqrt(5) [some rounding differences];

3)  How many degrees of freedoms we have? The most immediate thought is k-1 df, which is 2 in our case. However, we should take into account that each experiment has n samples (n-1 df). Therefore, in my opinion the correct way to take into account of df is: df=(k-1)*(n-1) if n is the same in all the experiments. In our case, df=8. Why the multiplication? Because we want that each experiment has a constraint, and therefore we introduce an AND among the assumptions on our experiments, and therefore the probabilities are multiplicative. It solves also the point 1.

Therefore,  we have to compare C=69.8 (SD: 9.11) and C1=89.6 (SD: 8.17) with 8 df per group, which implies a significance of p=0.0002, lower than that obtained with the calculations reported in my previous post. On the contrary, if we retain that n-1 should not be considered as df, we may use only k-1 df (2 in this case) with a very lower significance (p=0.049). In this case, the number of experiments should be well > 3 to control the beta error.

In the next post, we will summarize all the results.

Weighted Means: Part 1

The general idea is that each experiment , with n replicates that can vary experiment by experiment, is exactly performed in the same way, but, as already said, random factors may influence it (the researcher’s state of mind, a different hand in performing it, atmospheric conditions, etc). However, differently from the case in which we have used the number of experiment as random factor, I consider the experiment as statistical unit and not the replicate. Therefore, its statistical power increases with the number of performed experiments (I suggest at least 5, although we will apply our method with our data from three experiments).

Suppose we have k experiments, and each experiment has n replicates that may vary (n₁,n₂,….n_k). Each experiments has its own mean (X₁,….,X_K ) and its own standard deviation (s₁,…,s_k) and therefore its own standard error (SE), calculated as SE₁=s₁/sqrt(n₁),…, SE_K=s_K/sqrt(n_K). When we calculate weighted mean, we want to “weight” the general mean of means for the SE of each experiments, giving more weight to those experiments with lowest SE (lower SD, higher n, or both). Therefore, we can calculate a weight for each experiment:

W_i=1/(SE_i)^2, i=1,…,k

And use this weight in the calculation of mean of means:

X_best=(S_{i=1 to k} X_i*W_i)/ (S_{i=1 to k} W_i)

We can also calculate the general SE starting from the standard errors of the experiments:

SE=1/sqrt (S_{i=1 to k} W_i)

Let’s look at our example (not normalized data, with n=5 replicates for each experiment):

 

Controls:

SE1=8.57/sqrt(5)=3.83; SE2=8.50/sqrt(5)=3.80; SE3=14.33/sqrt(5)=6.41 =>

W1=1/SE1^2=0.0682; W2=1/SE2^2=0.0693; W3=1/SE3^2=0.0243 =>

Xbest=(62*0.0682+72.6*0.0693+83.6*0.0243)/(0.0682+0.0693+0.0243)=11.29/0.1618=69.8

The value is different to the crude mean of three means (72.7), as the third experiment has higher variability and therefore less weight on the weighted mean.

Let’s calculate SE:

1/sqrt(0.0682+0.0693+0.0243)=1/0.402=2.488

Note that the statistic unit is the number of experiments, and therefore SD=SE*sqrt(k)=2.488*sqrt(3)=4.309

Again a value completely different from the mean of the SDs of the experiments: 10.47 =>

We conclude that controls have a mean of 69.8 (SD: 4.31) with 2 (k-1) degrees of freedom (df) for each condition.

Other conditions:

C1=89.6 (SD: 4.08)

C2=108.8 (SD: 3.77)

C3=131.8 (SD: 4.10)

C4=139.8 (SD: 4.20)

On this data, we may apply ANOVA and post hoc tests.

IMPORTANT: one may think that the method only depends on the number of experiments, but not on replicates. Not True. The number of replicates determines the SD of the weighted means (high n values highly reduce SD) and therefore the model TAKES INTO ACCOUNT BOTH OF THEM.

To see the comparisons (t-student with Bonferroni’s correction, significance at p=0.0125):

C1 vs C p=0.0045

C2 vs C p<0.001

C3 vs C p<0.001

C4 vs C p<0.001

Substantially in line with what found when we have used the experiment as random factor of we have put all the experiments together, indicating that the method is anyway efficient also with few experiments.

Another possible calculation of SD: It has not solid statistical basis but may be logically reasonable. As we calculate weighted mean, we can calculate weighted SD as:

SD_best=(S_{i=1 to k} SD_i*W_i)/ (S_{i=1 to k} W_i) =(S_{i=1 to k} n_i/SD_i)/ (S_{i=1 to k} W_i)

SD of controls=(5/8.57 + 5/8.50 + 5/14.33)/0.1618 = 9.39

SD of C1 =(5/13.07+5/6.76+5/10.85)/0.181 = 8.75

Therefore, we have to compare C=69.8 (SD: 9.39) and C1=89.6 (SD:8.75), but in this case, being the weighted mean (SD) a sort of “gold standard” experiment (see approximated method), the number of df=n-1=5-1=4 for each condition

Making the comparison: p=0.0087, again significant but less than the previous case despite n>k

NOTE: in this last case, each experiment should have the same n to univocally define df.

Next time we will see the method by Bland and Kerry to have a weighted estimation of SD, comparing the results with those found today.