Showing posts with label Statistics. Show all posts
Showing posts with label Statistics. Show all posts

Monday, February 14, 2011

Correlation Test using R software

There is no reason why anyone interested in statistical computing will not choose R software, unless they are already wedded to existing software from their school days, or the choice was already made for them by their companies or their professors!. R boasts the most number of libraries and is open source and best of all FREE!
It takes time to master the syntax and we suggest that readers should try to download and test the R software in their machine if they had not done so.

In this post, we shall describe our online solver for testing correlation between two variables X and Y.
The solver is mainly a web interface to the cor.test function available in R. This function has the following spec for input arguments:


cor.test(x, y,
alternative = c("two.sided", "less", "greater"),
method = c("pearson", "kendall", "spearman"),
exact = NULL, conf.level = 0.95, continuity = FALSE, ...)


Here is how our solver for correlation test looks like at the moment.



When the user clicks on the submit button, the solver will call the R software, grabs its output, and print the results:

0001 > x <- c(44.4,45.9,41.9,53.3,44.7,44.1,50.7,45.2,60.1)

0002 > y <- c(2.6,3.1,2.5,5.0,3.6,4.0,5.2,2.8,3.8)

0003 >

0004 > cor.test(x,y, alternative="two.sided", method="pearson",conf.level=0.9,exact=TRUE, continuity=TRUE)

0005 

0006 Pearson's product-moment correlation

0007 

0008 data:  x and y

0009 t = 1.8411, df = 7, p-value = 0.1082

0010 alternative hypothesis: true correlation is not equal to 0

0011 90 percent confidence interval:

0012 -0.02223023  0.86697863

0013 sample estimates:

0014 cor

0015 0.5711816

0016 

0017 >

The printed p-value .1082 tells us not to reject the Null hypothesis that the correlation between
x and y is zero.

We will a scattergraph to the solver in the future.


Here is our interface code for the correlation test.

__version__ = "0.0.2"  
__catalog__ = "TEST-STAT-0036"
__date__    = "2011.02.14"
__author__  = "E.P. Adorio"


from   qp.fill.form     import Form, SingleSelectWidget, RadiobuttonsWidget, TextWidget, StringWidget, CheckboxWidget
from   qp.sites.extreme.lib.uicommon  import renderheader, renderfooter, processheader, processfooter
from   math             import sqrt
from   qp.sites.extreme.lib.webutils import vecRead
from   qp.sites.extreme.lib   import webutils
from   qp.fill.directory import Directory
from   qp.pub.common     import header, footer, page, redirect
from   qp.fill.css  import BASIC_FORM_CSS
from   qp.sites.extreme.lib.webutils import vecRead, matRead, matReadByColumn, runRcode
from   qp.sites.extreme.lib.qpyutils        import printRlines, showLogo


from   scipy          import stats

import time
from   scipy          import stats

import copy
import math


def getGrFile(grType):
    grfile   = GraphicsFile("%s" % grType)
    barefile = grfile.split(str("/"))[-1]
    return grfile, barefile

def getformDict(form):
    D={}
    for field in form.get_all_widgets():
        D[field.name] = form.get(field.name)
    return D    

def asColumns(S, ncolumns):
    """
    Args:
      ncolumns is number of columns.
      S is a string containing a space delimited columns of values.
    Return value
      a list consisting of the columns of S.
    """
    columns = [[] for i in range(ncolumns)]
    for i, v in enumerate(S.split()):
        columns[i % ncolumns].append( v)
    return columns

def asRvector(x):
    """
    Arg
      x - Python string array.
    Return value
      an R vector as a string c(v1,....vn)
    """
    S = "c(" 
    for v in x:
        S += str(v) + ","
    S += ")"
    return S.replace(",)", ")")

 


def SolveProblem(form):
    D    = getformDict(form)
    x, y = asColumns(D["pairedSample"], 2)
    alpha       = float(D["alpha"])
    method      = D["method"]
    alternative = D["alternative"]
    cont        = D["continuity_correction"]
    exact       = D["exact"]

    exact = "TRUE" if exact else "FALSE"
    cont  = "TRUE" if cont else  "FALSE"

    xstr = "x <- " + asRvector(x)+"\n"
    ystr = "y <- " + asRvector(y)+"\n"
    Rcode =xstr + ystr

    conflevel = 1-alpha

    Rcode += """
cor.test(x,y, alternative="%s", method="%s",conf.level=%s,exact=%s, continuity=%s)               
""" %(alternative, method,conflevel, exact, cont)
     
    # create the R command.
    Rcode = Rcode.replace(",)", ")")
    return runRcode(Rcode)


class CorrTestPage(Directory):
    def get_exports(self):
        yield ('',       'index',   'Correlation test for Paired Data', "R stat software corr.test for Correlation")

    def index[html](self):
        form = Form(enctype = "multipart/form-data")

        form.add(CheckboxWidget, name = "continuity_correction", \
                 value = True,
                )
        form.add(CheckboxWidget, name = "exact", \
                 value = True,
                )

        form.add(TextWidget,  name = "pairedSample",  \
                 rows = "10", cols = "30", value=""" 
          44.4 2.6
          45.9 3.1
          41.9 2.5
          53.3 5.0
          44.7 3.6
          44.1 4.0
          50.7 5.2
          45.2 2.8
          60.1 3.8
                """
                )

        form.add(StringWidget, name = "alpha", size = 5, value = "0.10")
        form.add(SingleSelectWidget, name="alternative", options=[
                 ("two.sided", "two.sided"),
                 ("less", "less"),
                 ("greater", "greater")])

        form.add(SingleSelectWidget, name="method", options=[
                 ("pearson", "Pearson"),
                 ("kendall", "Kendall"),
                 ("spearman", "Spearman")])


        form.add_hidden("time",   value = time.time())
        form.add_submit("submit", "submit")


        def render [html] ():
            renderheader("Correlation test using R-software")

            pairedSample = form.get_widget("pairedSample")
            alpha = form.get_widget("alpha")
            exact = form.get_widget("exact")
            method   = form.get_widget("method")
            alternative = form.get_widget("alternative")
            cont = form.get_widget("continuity_correction")
            """
            







MethodAlternativeAlphaContinuity CorrectionExact p value?
%s %s%s%s%s
X Y data
%s
""" %( method.render(), alternative.render(), alpha.render(),cont.render(), exact.render(), pairedSample.render())

renderfooter(form, __version__, __catalog__, __author__)

if not form.is_submitted():
return page('CorrTestPage', render(), style= BASIC_FORM_CSS)

def process [html] ():
processheader("Correlation test using R software corr.test() function")
calctime_start = time.time()

if True:
flag, output = SolveProblem(form)
printRlines(output)

if 0:
for widget in form.get_all_widgets():
"
%s %s
" % (widget.name, widget.value)
processfooter(form, calctime_start, "./", "./stest_t1samp")
process()

Friday, August 13, 2010

Finite sampling demonstrator

I have to intall a Latex processor for my blog. Look for the mirrored article in
Sampling Demonstator


Finite Sampling Deomonstrator


Given a finite population S with N elements which may not be unique (some elements are repeated),
we extract a finite sample X with n elements where n < N. The way we extract the n elements may be done in the following manner:
  1. Permutions without replacement.
  2. The ordering of the sample elements is important and a sample element once chosen in the sample may NOT be chosen again.
  3. Permutations with replacement.
  4. The ordering of the sample elements is important and a sample element once chosen in the sample may be chosen again.
  5. Combinations without replacement
  6. The ordering of the sample elements is NOT important and a sample element once chosen to be in the sample is not available again.
  7. Combinations with replacement
  8. The ordering of the sample elements is NOT important and an element chosen to be in the sample may still be chosen again.

The total number of samples for each type of finite sampling above is given in the following table:


Number of Samples
sampling CombinationsPermutations
without replacement $$\frac{N!}{n! (N-n)!}$$ $$\frac{N!}{(N-n)!}$$
with replacement $$\frac{(N+n-1)}{n!(N-1)!}$$ $$N^n$$



Let $$S = [s_0, s_1, s_2, ....,s_{N-1}]$$. Our generated sample X is actually X = $$[s_{i_0}, s_{i_1}, ...., s_{i_{n-1}}]$$
where the indices $$i_0 to i_{n-1}$$ is sequentially generated by a combinatorial algorithm.

We may be interested in the following statistic which is a random variable for the totality of all samples:


  1. sample mean
  2. sample sum or total
  3. sample s.d.(standard deviation) (divisor is n-1)
  4. populaton s.d(divisor is n)
  5. sample var (sample variance)(divisor is n-1)
  6. population variance(population variance) (divisor is n)
  7. sample vaiance(sample variance) (divisor is n-1)
  8. sample max (maximum value)
  9. sample min (minimum value)
  10. range (max - min )


We wish to gather data on ALL possible finite samples for the desire statistic,
the mean and standard deviation and of course the distribution table for the statistic which contain the
Columns for statistic, frequency, (rf) relative frequency, (crf) cumulative relative frequency. x rf

Here is a complete example for the sum of the numbers which show up in a throw of three dice:
The population consists of [1,2,3,4,5,6].
The population size is 6.
The sample size is 3.
The ordering is considered important, for example [1,3,2] will be considered different from [3,1,2]. Thus it
is a permutation with replacement.
The "first" sample is [1,1,1] with a total of 3 and the "last" sample is [6,6,6] with a total of 18.
To help with our computations, we use our solvers hosted at www.extreme.adorio-research.org to do it for us!

We will only show the generated frequency distribution table as the 216 generated samples is too long for this page.




Sampling Statistic Frequency Distribution Table
xfrf crfx rf(x-mu)^2 rf
3.010.004629629629630.004629629629630.01388888888890.260416666667
4.030.01388888888890.01851851851850.05555555555560.586805555556
5.060.02777777777780.04629629629630.1388888888890.840277777778
6.0100.04629629629630.09259259259260.2777777777780.9375
7.0150.06944444444440.1620370370370.4861111111110.850694444444
8.0210.09722222222220.2592592592590.7777777777780.607638888889
9.0250.1157407407410.3751.041666666670.260416666667
10.0270.1250.51.250.03125
11.0270.1250.6251.3750.03125
12.0250.1157407407410.7407407407411.388888888890.260416666667
13.0210.09722222222220.8379629629631.263888888890.607638888889
14.0150.06944444444440.9074074074070.9722222222220.850694444444
15.0100.04629629629630.9537037037040.6944444444440.9375
16.060.02777777777780.9814814814810.4444444444440.840277777778
17.030.01388888888890.995370370370.2361111111110.586805555556
18.010.004629629629631.00.08333333333330.260416666667
Sum2161.0mean=10.5variance=8.75
std.dev=2.95803989155
Finite Population Parameters, Correction factor=0.774596669241
(N,n)MeanPvarPstdSvarSstd
(6, 3) 3.5 2.91666666667 1.70782512766 3.5 1.87082869339

Visit the solver Stats sampling

Be sure to specify the right parameters for the solver for the above example, see the screen below:


We will try to complete this solver with a graph of the relative frequency and the cumulative relative frequency vs. the statistic or random variable x.

Hope you will find this finite sampling demonstator a quick tool for mastering statistical concepts!