An article on - Bayes Theorem application and usage

 

Bayes Theorem application and usage

 

Instance and example of usage of Bayes Theorem in Maths and Statistics :

 

P(B|E) = P(E|B)*P(B) / P(E)

 

If one reads the formula , then one will come across the following terms within the instance which can be elaborated with the help of an instance in the following manner :

 

* P( B | E ) - The probability of a belief(B) given a set of evidence(E) is called over here as Posterior Probability . Here , this statement tries to convey the underlying first condition that would be evaluated for going forth over to the next condition for sequential execution . In the given case , the hypothesis that is presented to the reader is whether a person is a female and given the length of her hair is sufficiently long , the subject in concern must be a girl

 

* P( E | B ) - In this conditional form of probability expression , it is expressed that one could be a female given the condition that the subject has sufficiently long hair . In this case , the equation translates to a form of conditional probability .

 

* P ( B ) - Here , the case B stands for the general probaility of being a female with a priori probability of the belief . In the given case , the probability is around 50 percent which could be also translated to a likelihood of occurrence of around 0.5 likelihood

 

 

* P(E) - This is the case of calculating the general probability of having long hair . As per general belief , in a conditional probability equation this term should be also treated as a case of priori probability which means the value for its probability estimate is available well in advance and therefore , the value is pivotal for formulation of the posterior probability

If one would be able to solve the previous problem using the Bayes Formula , then all the constituent values would be put in the given equation which would fill in the given values of the equation .

The same type of analogy is also required for estimation of a certain disease among a certain set of population where one would very likely take to calculate the presence of any particular disease within a given population . For this one needs to undergo a certain type of test which would result in producing a viable or a positive result .

 

Generally , it is perceived that most of the medical tests are not completely accurate and the laboratory would tell for the presence of a certain malignancy within a test which would convey a condensed result about the condition of within a test which would convey a condensed result about the condition of illness of the concerned case .

 

For the case , when one would like to see the number of people showing a positive response from a test is as follows :

1) Case -1 : Who is ill and who gets the correct answer from the test .

This is normally used for the case of estimation of true positives which amounts to 99 percent of the 1 percent of the population who get the illness

2) Case-2 : Who is not ill and who gets the wrong diagnosis result from the test . This group consists of 1 percent of the 99 percent of the population who would get a positive response , even though the illness hasn't been completely discovered or ascertained in the given cases . Again , this is a multiplication of 99 percent and 1 percent ; this group would correspond to the discovery of false positive cases among the given sample . In simple words , this category of grouping takes into its ambit , those patients who are actually not ill (may be fit and fine ) , but due to some aberrations or mistakes in the report which might be under the case of mis-diagnosis of a patient that , the patient is discovered

as a ill person . Under such circumstances, untoward cases of administration of wrong medicines might happen , which rather than curing the person of the given illness might inflict aggravations over the person rendering him more vulnerable to hazards , catastrophies and probably untimely death

 

* So going through the given cases of estimation of correct cases of Classification for a certain disease or illness could help in proper medicine administration which could help in recovery of the patient owing to right Classification of the case ; and if not then the patient would be wrongly classified in a wrong category and probably wrong medicines could get administered to the patient seeking medical assistance for his illness .

 

( I hope , there is some understanding clarity in the cases where the role of Bayesian Probability estimations could be put to use . As mentioned , the usage of this algorithm takes place in a wide-manner for the case of proper treatment and classification of illnesses and patients ; classification of fraudulent cases or credit card / debt card utilisation , productivity of employees at a given organisation by the management after evaluation of certain metrices :P ...... I shall try to extend the use case and applications of this theorem in later blogs and articles )

 

Exploring the World of Probability Theory in ML .. derived article with own interpretations

               Exploring the World of Probability Theory in ML

 

* What is Probability and how can it be used? Probability is the likelihood of an event which means that Probability can help someone to determine the possibility of something to happen or not using the mathematical (Gannita Gyaana) where one can establish the possibility or likelihood of occurrence of an event in terms with the total number of possible events that could likely occur .

 

* The probability of an event is measured in the range from 0 (no probability that an event occurs) to the value of 1 ( a certainty that an event occurs ) which in relative terms says about the extent of any value towards the any of the extremes from the left most to the right most values .

 

* The probability of picking a certain suit from a deck of Cards (generally referred to as "Taash" in many Asian countries) is one of the most classic example on explanation of probabilities.

 

* The deck of cards contains 52 cards (joker cards excluded) which can be divided into four suits as clubs and spades which are black , and diamonds and hearts which are red in colour .

 

* Therefore , if one wants to determine whether the probability of picking the card is an ace , then one must consider that there are four aces of different suits .The probability of such an event can be calculated as p = 4/52 which is again evaluated to 0.077.

 

* Probabilities are between the values of 0 and 1 ; no probability can exceed such boundaries as everything's possibility of occurrence lies between nothing to everything and probability of not occurrence of something is always zero and the probability of occurrence of everything is always equal to 1 .

 

* If someone tries to do a Probability Possibility prediction for a given case of fraud detection in which one would like to see and find out the number of times a bank transaction related fraud has occurred over a given set of bank accounts or how many times fraud happens while conducting a banking transaction or how many times people get a certain disease in a particular country . So , after associating all the events , one can estimate the probability of occurrence of associating all the events , one can estimate the probability of occurrence of such forthcoming event with regards to the frequency of occurrence , mode of occurrence , time of occurrence , as well as the likely accounts which could be affected by the fraud and the conditions which are likely to affect the accounts .

The calculation for the estimation would take into consideration of counting the number of times a particular event occured and dividing the total number of events that could possibly occur for a set of operations and calculations.

 

* One can count the number of times the fraud happens using recorded data ( which are mostly taken from databases ) and then one would divide that figure by the total number of generic events or observations available

 

* Therefore , one should divide the total number of frauds by the number of transactions within a year or one can count the total number of people who fell ill during the year with respect to the population of a certain area . The result of this is a number ranging from 0 to 1 which one can use as baseline probability for a certain event under certain type of circumstances

 

* Counting all the occurrences of an event is not always possible for which one needs to know about the concept of sampling. Sampling is an act which is based on certain probability of expectations , which one can observe as a small part of a larger set of events or objects , yet one may not be able to infer correct probabilities for an event , as well as exact measures such as quantitative measurements or qualitative classes related to a set of objects

 

* Example - If one wants to track the sales of cars in a certain country , then one doesn't need to track all the sales that occur in that particular geography ... rather using a sample comprising of all the sales from new car sellers around the country , one can determine the quantitative measures such as average price of a car sold or qualitative measures such as the car model which were sold most often

 

Some Operating cases on Probabilities

                   Some Operating cases on Probabilities

 * It is suggested that operations on probabilities are a bit different from numeric operations because the range of occurrence of such probability values generally lies between the range of 0 & 1

 

* One must rely on some set or rules in order for the operation to make sense to the user who is conducting the experiment on probabilities. For example , if someone is conducting an experiment of tossing a coin then he/she must strictly define the rules according to which the game of tossing a coin would be played out . The instructor would declare which outcomes should be taken as valid outcomes and which should not be taken in as valid outcomes , rather must be negated the moment the norms of the game are violated .

 * For example , suppose say a case happens over where a coin does not fall over any of the sides rather falls over the floor standing erect , then the outcome is neither a heads and nor a tails , and neither a 50-50 heads-tails can be taken as consideration for the throw of the dice . Rather what would happen in such a circumstance is that the throw of the dice for this case would be nullified , the entire event of throw of such a dice would be struck off from the probable set of outputs that should happen as a result of the throw of the dice . Thats why one should also keep adhering to the rules of the experiment before conducting such an experiment which would require to know what set of events should be taken in as considerable outcomes and which should not be considered .

 

* Again another property of Probabilities that one needs to be aware is summations between probabilities which states that summations of probabilities is possible only when all the constituting events of the sample space are mutually exclusive to each other . For example lets consider an experiment of rolling a dice over a game of ludo , in this all the possible events that could turn up as a result of throw of the dice are 1 , 2 , 3 , 4 , 5 , 6 . The probability of occurrence of each of the events is 1/6 or 1 by 6 . And here , each of the events within the given sample space are disjoint and mutually exclusive to each other which makes the individual events probability of occurrence as equal to each of the given event divided by the total number of events over the entire sample space . And in case one would like to know the probability of occurrence of all the events together in unison , then one may have to add up the probabilities of each of the individual events as a summation of each of the individual events .. which would yield an output of 1 . So in retrospect, all individual elements of an experiment of probability are disjoint and mutually exclusive and in unison lead to a summed up value of 1 .

 

* We can take another simple example to demonstrate to demonstrate the case of understanding of probability calculation ; in this case one can consider for example the case of picking a spade or a diamond from a set of cards can be calculated in the following manner . Total number of cards in the entire deck = 52 . Number of cards in the house of clubs = 13 , number of cards in the house of clubs = 13 , number of cards in the house of hearts = 13 , number of cards in the house of diamonds = 13 . If a person takes out a card from the house of diamond then the probability if picking up one of the cards is 13/52 ; the same goes for the case of picking up a random card from a house of clubs is 13/52 . So , total probability of finding a card from both the houses is 26/52 which is equals to 0.5

 

* One can take the help of subtraction operation to determine the probability of some events where probability of an event is different from the probability of an event that one would want to compare . For instance , if someone wants to determine the probability of drawing a card that does not belong to some house of card for example , say I want to draw a card which is not a diamond from the overall deck of cards , then one will approach the problem in the given manner . He will first find out the overall probability of finding any card and then he will subtract the chance of occurrence of a particular card from the total , 1 - 0.25 which happens to be as 0.75. One could get a complement of the occurrence of the card in this manner , which could be used for finding the probability of not occurrence of a particular event .

 

* Multiplication of a set of events can be helpful for finding the intersection of a set of independent events . Independent Events are those which do not influence each other . For instance , if one is playing a game of dice and one would like to throw two dices together , then the probability of getting two sixes is 1/ 36 . This can be obtained by multiplication of dices over both the cards , where first the probability of obtaining a 6 is found out to be as 1/6 and then the subsequent independent event would also produce an probability of obtaining another 6 is found out to be as 1/6 , here both the values are multiplied with each other and found that product of both the probabilities of independent events would yield a value output as 1/36 or 0.28 .

 

* Using the concepts of summation , difference and multiplication , one can obtain the probability of most of the calculations which deal with events . For instance , if one would want to compare the probability of getting atleast a six from two throws of dice which is a summation of mutually exclusive events . Probability of obtaining two sixes of dice , p = 1/6* 1/6 = 1/36

 

* In a similar manner if one would like to calculate the probability of having a six on the first dice and then something other than a six on the second throw of the dice is p = (1/6)*(1- 1/6) = 5/36 ,

 

* Probability of getting a six from two thrown dice is p = 1/6* 1/6 +2*1/6*(1- 1/6) = 11/36

 

"Apply()" Command for finding Summaries on Rows / Columns of a Matrix or Dataframe Object

 

"Apply()" Command for finding Summaries on Rows / Cols

=======================================================

* The "ColMeans()" and "RowSums()" command are designed as quick alternatives to a more generalised command "Apply()"

 

* The "apply()" command enables one to apply a function to rows or columns of a matrix or a dataframe

 

* The general form of the "Apply()" command is given in the following manner :

apply(X,margin,FUN,....)

 

In this command , the applicable MARGIN within the parameter is either 1 or 2 where 1 is for the rows and 2 is for the columns applicable for the dataframe

 

* One can replace the "FUN" part within the parameter of the apply() function and one can also add additional instructions which might be appropriate to the command / function that one is applying

 

* Example :

One might add the parameter "na.rm = TRUE " as an instruction to the apply function .

> mat01

     [,1] [,2] [,3] [,4]

[1,]    1    2    3    4

[2,]    5    6    7    8

[3,]    9   10   11   12

[4,]   13   14   15   16

 

> apply (mat01 , 1 , mean , na.rm = TRUE )

[1]  2.5  6.5 10.5 14.5

 

* In such a case , one can see that the row names of the original dataframe are displayed as output .

 

* If the dataframe has no set row names , then one will see the result as a vector of values .

 

> apply (fw , 1 , median , na.rm = TRUE )

2.5  6.5 10.5 14.5

 

Rotating Data Tables / Matrices in R

 

        Rotating Data Tables / Matrices in R

 

* One can do a rotation operation over a T table / dataframe / matrix object so that the rows become the columns and the columns become the rows .

 

* For doing the rotation operation over any data table object / data frame object or a matrix object in R , one can use the t() command .

 

* One can think of such a program as short for transpose operation

 

* The given example begins with a dataframe that contains two columns of numeric data with its rows also named .

  

* The final object is transposed ( means - the rows are changed to columns and the columns are changed to rows / reversal of attributes )

 

* Also , the new object is in fact a matrix rather than a dataframe .

  

* One can see this much more clearly if one tries the same t() command over a simple vector .

* Vectors are treated like columns , but when they are displayed they look like rows

* In any event , when you apply a t() command one would get a matrix object as a result .

* One can easily convert the objects to a dataframe using the "as.dataframe()" command .

 

Python Program to Convert Strings into Dictionary Object ( with sample code )

 

Extended example on Rule of Cross Multiplication in Ratio and Algebra ( Notes with Example )

 

Rule of Cross Multiplication in Ratios / Algebra ( Theorem and Example)

 

Example Question on Ratios ( Example - 04 )

 

Example Question on Ratios ( Example - 02 )

 

Example Question on Ratios in Algebra ( Example - 01 )

 

Introduction to Hypothesis Testing In Statistics with sample problems and explanations

 

       Introduction to Hypothesis Testing 

In Statistics with problems and examples

 

 

* This article will try to breakdown the concept of Hypothesis testing into smaller chunks, and as someone goes through the concept of Hypothesis Testing as covered in the article, one would be able to have a very good idea and hold over the concept of Hypothesis testing and how and when these testing types could be put to use .

 

* The best part of learning this from this articulated version of Hypothesis Testing is that one can get to have a good understanding of the concept of Hypothesis Testing is by going through the article segment by segment as one may get to notice each of the discussed concepts .

 

* One of the main reasons why many of the students consider the topic of Hypothesis Testing to be difficult is that many of the students consider the case studies of Hypothesis Testing to be very much different from the various other concepts discussed in within the topic of Statistics as the testing methods of the various kinds of tests within Hypothesis Testing is much different from one another and many a students face a lot of hard time in understanding the concepts and also differentiating each of the concepts from one another on a wide scale where there would be cases where there are smaller sample sizes and there are also going to be samples of higher sample sizes and there would be cases discussed with usage of different to different sets of parameters for each of the underlying test types constituting the concept of Hypothesis Testing .

 

 

* And in each of the various distributions, one is going to use the case of different types of distributions and different techniques for arriving at a result which puts to use the various methods used for finding the results of each of the hypothesis tests .

 

* So we will begin to dig deeper into the concept of Hypothesis Testing by starting with the concept of Hypothesis Testing .

 

One can state that Hypothesis Testing is a premise or claim that we want to test so basically it is not a type of test where one would go to some laboratory to do some tests, rather one may need to formulate some statistical sample surveys from which the task of the statistician or the sample analyst would be to collect the associated results out of the data and then collect meaningful results from

the sample by testing the hypothesis.

 

* Next let us get to know what is Null Hypothesis. But first let’s understand what the Null value within the hypothesis means. Null stands for something which is zero or empty which is a widely used jargon in the world of computer science and programming language literature .

 

 

* So, when someone is talking about the Null Hypothesis then someone is talking about the Default hypothesis which talks about something which is already established . The Null Hypothesis is generally denoted by the letter capital H with a small zero as the sub-script . H(0) is the most widely accepted value for a parameter which is accepted by almost all the statisticians and analysts .

 

* And whenever someone wants to challenge the Null and wants to read and analyse the What Else portion of the Hypothesis then it could be said that one is interested in the Alternate or Alternative form of Hypothesis. This Alternative Hypothesis can be represented by the symbol (H1) or (Ha) where the sub-script for a means that it is an alternative hypothesis form. In some statistics books, the alternative hypothesis is also called as the Research Hypothesis and it involves the claim to be tested .

 

* Lets take the principle of Gravity into consideration, Many centuries back, Newton discovered the essence of Gravity when an apple suddenly comes and falls over his head and did some research and gave the postulates supporting his hypothesis upon Gravity But later Einstein comes into the picture and gives another set of hypothesis which is an alternative from the views that Einstein

Provided and thus they formed as the postulates for an Alternative View of Hypothesis formulated by Einstein. And his view of gravity was much different and complicated for generalisation as what had been provided by Newton and thus his view of alternative hypothesis was a lot more different than that of Newton and thus the view was eventually accepted and gained stature in the Science journals of that time .

 

========================================================

 

* Ok now lets take a question to get a better perspective into the idea of Hypothesis Testing with an example .

 

It is believed that a candy machine makes chocolate bars that are on average 5g in weight. A worker claims that the machine after maintenance no longer produces 5g bars. Formulate the expression for H0 that is Null Hypothesis and H1 which stands for alternative Hypothesis.

 

Ho = Null Hypothesis

H1 = Alternative Hypothesis

 

Ho :  mean weight of each chocolate bar is 5 grams which is stated as the hypothesis question

H1 :  mean weight of each of the chocolate bars is not equal to 5 grams as produced by the chocolate factory

 

So , one can sense from both the above given statements that the null hypothesis and the alternative hypothesis are mathematically opposite to each other .In all of the hypothesis tests that anyone can perform , the null hypothesis and the alternative hypothesis are always mathematically opposite to each other .

 

* So now we are interested in testing the outcomes of the test as have been provided in the null hypothesis and the alternative hypothesis when we need to consider that the null hypothesis must be true .

 

========================================================

 

* So what are the possible outcomes of the test for Null Hypothesis that one can do over the hypothesis :

1) Reject the Null Hypothesis

2) Accept the Null Hypothesis

 

* If we reject the null hypothesis then we mean that whatever has been provided against the data is held to be be false which means that mean of all the weights of all the chocolate bars in the sample is not equals to 5 grams and hence the hypothesis is rejected

 

* If we fail to reject the null hypothesis (the statement - fail to reject the null hypothesis doesn't absolutely mean that we are accepting the null hypothesis ) then one would mean that whatever statement has been provided in the Alternative Hypothesis is false and as such the weights of all the chocolate bars in the given sample is not equal to 5 gram

 

* The possible outcomes as pointed within the hypothesis  statements is just like the opinion formulated in the court of law and then one can either reject the hypothesis ( H0 ) or else one can fail to reject the null hypothesis ( H1 ) .

 

 

* So now after conducting the test and then finding out the possible outcomes of the test , one would try to ascertain the Test Statistic in the following manner .

 

* Test Statistic :

 

The test statistic is calculated from sample data and can be used to decide whether to reject the null hypothesis or fail to reject the null hypothesis. As an example in the case of the Candy-Bar factory , may be one may start sampling 50 chocolate bars in the factory and from the factory we would be doing a statistical descriptive analysis of the data and get average value of the amount of chocolate bars present within the sample and get the value of the test statistic for the data .

 

Then one can determine Statistically, the significant value for the data which means how to arrive at a decision whether to reject the null hypothesis or fail to reject the null hypothesis.

 

Suppose say one guy draws a sample on Monday consisting of 50 bars and finds an average of 5.12 grams which is technically not equivalent to 5 grams ,  similarly another guy draws a sample of 50 bars which is again not equivalent to exactly 5 grams but is roughly around 5.72 grams and similarly on Friday the average count is noticed to be 6.53 grams then one can deduce from the above three calculations that the values are so much statistically different from each other and as such values are very much different from each other and also very much distant from the null hypothesis set at average price of 5 grams per candy bar .

 

Therefore, comparing the obtained prices as given for Monday is 5.12 which is pretty much closer to the value of the average weight of 5 gms , then the next average weight is that of 5.72 grams which is a bit distant from the accepted null hypothesis accepted value of 5 grams and then the third reading taken for the 50 bars is found out to be 6.53 grams which is very different and distant from the hypothetical mean value of 5 grams . Therefore , one can come to a general conclusion about the data that it is okay to reject the null hypothesis as the values are nowhere very near to the hypothesis accepted mean value of 5 grams per chocolate bar .

 

 

So, in statistical terms by looking at the hypothesis value and the actual values, an analyst should be readily able to make a concrete decision when to reject the null hypothesis and when not to reject the null hypothesis.

 

This is what in general the purpose of a hypothesis test is that is a hypothesis test needs to collect the data , generalise the data and obtain a test statistic which would enable to make a decision when to reject the null hypothesis and when not to reject the null hypothesis by having a proper look at the data obtained and then ascertain the case where the value obtained is too high and when the value obtained is too low and when to accept the condition for null hypothesis and when to reject the hypothesis which means that one needs to concretely decide the boundaries for the null hypothesis and conditions where to reject or not reject from the statistically significant data available .

 

========================================================

 

Level of Confidence

Level of confidence is also alternatively referred to as level of significance in many terms where it is graphically determined where to reject the null hypothesis and where to not reject the null hypothesis. Thus it enables one to know how confident someone is in their decision and what level of confidence expressed in percentage values.

 

For example if the level of confidence or value is 99% for rejection of a null hypothesis then, everyone would accept the decision for rejection of the hypothesis but in case if the level of confidence is around 50% then it means that it is not a right decision to reject the null hypothesis under the given circumstance as the statistically obtained value for level of confidence is very less .

 

========================================================

 

Level of Significance

Basically this is called as alpha which is numerically expressed as

 ( 1 - C) that is : level of significance is equals to 1 minus the level of confidence for some sample of data .

 

Mathematically , if the level of confidence is found out to be as 95% with the level of significance (alpha) is found out to be as alpha = 1 - 0.95 , then alpha = 0.05

 

========================================================

 

From both the terms "level of confidence" and "level of significance" one can make an appropriate decision whether to reject or not to reject the formulated hypothesis

 

The analogy that needs to be drawn from this test of hypothesis that could be also thought to be analogous to another is when someone is accused of a crime, the first assumption to be made in favour of the convict is that the convict is innocent and it is upto the lawyers and the evidence that the convict is guilty and if the lawyers and the evidences don’t prove the alternate assumption that the convict

is guilty then automatically the assumption that the convict is innocent would be held true .

 

Same is the case of our example. We assumed that the mean / average weight of one chocolate bar is around 5 grams but also formulated an alternative assumption that the average / mean weights of the chocolates are not the same but different from each other . To support / test our assumption we took the case of testing / experimentation and found out contradictory results for the data

values obtained and after the results were analysed we came to a conclusion whether to reject the null hypothesis or not to reject the null hypothesis .