What is the Concept of Confidence Interval in Statistics?

Consider a process that is generating a certain output (like cycle time) that can be characterized by a certain statistical parameter (like average cycle time).

Suppose you are interested in determining the value of this parameter for the entire population. So how would you go about it?

You start with selecting a sample and measuring the value of the parameter using the sample data.

The question that arises then is how to estimate the population parameter from the sample parameter value.

That’s where confidence interval can come to your rescue.

Confidence interval is a statistical interval estimate of the population parameter that is derived from the sample parameter value.

Type I / Alpha Error or Producer’s Risk and Type II / Beta Error or Consumer’s Risk

Think of a manufacturer that supplies certain electronics component to its consumers. The components, before they are shipped, undergo final inspection by the team assigned for the purpose.

This team needs to make the call whether to ship the component or not by doing an inspection.

This situation can be viewed as being similar to a test of hypothesis. The null hypothesis (Ho) in this case is that the component is not defective.

Ho: Component is not defective (and hence good)

H1: Component is defective

Suppose the team can make error in their judgment in determining whether the component is defective or otherwise.

Statistical Concepts Underlying Control Charts

Control charts are part of the repertoire of statistical tools collectively named as SPC (Statistical Process Control) or SQC (Statistical Quality Control). They were first applied to an industrial context by Walter Shewhart who is regarded as the father of SQC or SPC.

Control charts provide a set of powerful statistical tools that can be effectively used for quantitative monitoring and control of processes. Clear understanding of the statistical concepts underlying  control charts is essential for their correct use.

Control Chart as a Graphical Test of Hypothesis

Conceptually speaking a control chart is a graphical test of hypothesis which involves testing the following null and alternative hypothesis (the test is a two-tailed test).

X, X bar Chart

• Ho : Mu = Mu 0 (there is no shift in process mean)
• H1 : Mu is not equal to M 0 (there is shift in process mean)

MR, R bar Chart

• Ho : Sigma ^ 2 = Sigma 0 ^2 (there is no shift in process variance)
• H1 : Sigma ^ 2 is not equal to Sigma 0 ^2 (there is shift in process variance)

Control charts graphically represent the time sequence behavior of a certain process characteristic against the set of following three lines collectively to  monitor and detect any shift in its mean or variance.

• Upper Control Limit (UCL)
• Center Line (CL)
• Lower Control Limit (LCL)

Level of Significance of Test Performed by Control Chart

Control chart sets the two controls limits UCL and LCL at 3 sigma on right side and 3 sigma on left side of the mean respectively. In other words,

• UCL = CL + 3 sigma
• LCL = CL - 3 sigma

Sigma can be obtained using the appropriate method. For example, for Individual and Moving Range control chart (I - MR or X - MR), sigma is expressed in terms of MR bar as:

• sigma = MR bar  / d2
• where d2 is a statistical constant that depends on the sample size 'n'
• and for Individual and Moving Range chart, n =2

Given that it is assumed that the distribution followed is a Normal distribution, the area between UCL and LCL will be 99.73%. This means the area outside will be 100% - 99.73% = 0.27%. This is the level of significance of the test or the alpha.

In test of hypothesis, alpha is generally kept at 5% level. Hence it is obvious that the alpha value used in control charts is very less compared to what is generally used in the test of hypothesis. It is also to be noted that alpha is fixed for control charts.

It is useful to recollect that alpha is also known as the Type I error (which is the probability of detecting a shift when there is none).

Keeping a low alpha ensures that the control chart doesn't throw up too many false alarms. False alarms need investigation and have a cost associated hence in that sense control charts reduce the cost associated with false alarms.

Power of Control Chart

The power of a control chart is expressed in the form of 1 - beta, which is the probability of detecting a shift when there is one.

It is useful to recollect that beta is also known as the Type II error (which is the probability of not detecting a shift where this is one).

The power of a control chart can be represented using an OC Curve (operating characteristic curve). For a control chart alpha is fixed (in the manner explained above) and beta varies depending on certain factors.

One of the most important factor is the sample size. Higher the sample size higher the power of the control chart. This explains the reason why the X bar - R is more powerful than an I - MR chart.

Outliers and Non-random Patterns in Control Charts

The rules of SPC related to outliers and non-random patterns are derived from the fact that such events have a very very low probability of occurrence given that only common causes are present. So if they occur then something must have gone wrong and there is a strong indication against only common causes being present. Here are some considerations to take note of:

• Any point outside the 3 sigma limits on either side is considered as an outlier. The reason - probability of such an event occurring is less than 0.27%
• Any trend or pattern is unlikely to form if the process is a stable normal distribution. In a stable normal distribution the mean and sigma remain constant and individual observations fall randomly as governed by the probability density function of the normal distribution

Stability and Capability

Control charts can be used to obtain a clear indication about the statistical stability of a process. A process is said to be in a state of "statistical stability" if it operates only under a set of common causes. Common causes are also called as non-assignable or inherent causes. As compared to that, a process is not stable if there are certain special or assignable causes present.

Control charts can also be used to assess the capability of the process. Plotting lines for the Upper Specification Limit (USL) and Lower Specification Limit (LSL) along with the UCL and LCL lines can provide good idea regarding the capability of the process. If the lines of UCL and LCL are well within the lines of USL and LSL, the process can be called as a capable process.

In respect of stability and capability following points need special mention:

• A process which is not stable may still be capable. This can happen in following cases:
• when the specification limits are set very wide apart and the inherent variation due to the design of the existing process is in a much narrower zone (this could be case of over-design and may mean high cost was incurred in setting the process)
• when the process is centered close to the mean, has an extremely low variation and constantly experiences minor shifts in the process mean.
• A process which is stable may not be capable. This can happen in following cases:
• when the process is centered close to the mean, has a large variation and constantly experiences shifts, minor or major, in the process mean
• when the process is centered away from the mean, has some variation, either small or large, and constantly experiences shifts, minor or major, in the process mean.
• A process may be stable around a certain value of the mean and then undergo a shift and become stable around another value of the mean. It might appear that the process has large variation in this case but in reality the variation around a certain mean is not that large.

ANOVA (Analysis of Variance)

Many people carry the impression that ANOVA is meant for testing differences in variances. However, contrary to what its name indicates ANOVA is used for testing differences in means and not in variances.

The basic concept behind ANOVA is that if the means of the treatments or factors are different the variances between treatments will be different that variance within treatments. The ratio of two variances follows an F distribution. So ANOVA essentially involves testing the ratio of variance between (the influence of the treatment) against the variance within (the error component) using F test.

Variances can be expressed in terms of sum of squares (SS). Consider an ANOVA for three factors A, B and C. In this case for ANOVA the total sum of squares is partitioned into SS due to the main effects of the various factors, the interaction effects and the error.

SS Total = SS A + SS B + SS C + SS AB + SS AC + SS BC + SS ABC + SS Error

For ANOVA it is assumed that:

• Variances are homogeneous
• Data is normal

Conceptually speaking ANOVA is a test of hypothesis where the null hypothesis (Ho) is that the means of the treatments are equal. This means the alternative hypothesis (H1) is that not all means are equal.

Ho : Mu 1 = Mu 2 = ... = Mu k (k factors)

H1 : Not all Mu are equal

If the ratio of variances between to variance within is small then it can be concluded that the differences between the means of the treatments is not significant or there is not enough evidence that the means of the treatments are significantly different.

F ratio is compared with the critical value of F from the F distribution table. F ratio being more than the critical value means Ho has to be rejected, hence H1 is accepted. The F value depends on the level of significance (alpha value, generally kept at 0.05 or 5%).

Statistical Tools for Building Process Performance Baselines (PPBs) and Models (PPMs) for Level 5

Here are some commonly used statistical tools for  building Process Performance Baselines (PPBs) and Models (PPMs) for CMMI Level 5:

For Sub Group Identification

• F-test for equality of variances (sigma square)
• ANOVA for equality of means (mu)
• Additionally Tukey's test can be applied to group the subgroups in case difference in means come out as significant from ANOVA

Process Performance Baselines (PPBs)

•  X MR Control Charts

Process Performance Models (PPMs)

• Multiple Linear Regression
• Following model adequacy tests also need to be performed
• Normality of residuals or errors using Normal Probability Plot
• p-value of Normality check for Y, Xs and residuals (more than 0.05 is desirable - this follows from the fact that null hypothesis in this case is that data follows normal distribution hence null hypothesis being accepted means distribution is normal)
• R square adjusted (more than 70% to 80% is desirable)
• VIF for multi-collinearity or dependency amongst the Xs (less than 10 is desirable)
• p value of the regression coefficients of Xs (less than 0.05 is desirable)
• p value of the regression equation (less than 0.05 is desirable)
• Additionally following can be applied
• Lack-of-fit test
• Variable transformation for satisfying Normality assumption (if nothing works Box Cox Transformation can be tried)

Building PPMs Using Support Vector Machine (SVM) and Artificial Neural Network (ANN)

Support Vector Machine (SVM) and Artificial Neural Network (ANN) are concepts/methods employed in the field of Artificial Intelligence and can be useful in building Process Performance Models (PPMs) that in some sense learn and evolve as more data points are fed into them.

What are the key characteristics and advantages of PPMs built using SVM and ANN? Here are some:

• The PPM can dynamically adjust to underlying changes in the process in the range of Ys and Xs that was used while building the PPM
• Many of the subjective and fuzzy factors that are important in the software development process can be more effectively modeled. Software development process is influenced by following fuzzy factors, whether or not they can be easily modeled:
• Competency level of team members on various activities like architecture, design, coding, reviews, testing and in the domain/technology
• Complexity level of the work being performed
• Mental fitness level of the team members (remember human brain is the machine used for manufacturing software so if a software engineer is overworked or had a fight at home in the morning it might affect the quality of work performed)
• Clarity level of requirements provided by the customers and also as understood by the project team (customers themselves may not know what they want until late in the project when they start seeing the first few screens, demos, etc.)
• Clarity level of the technical solution envisaged by the project team (this might be a big factor in case of projects in new areas or yet-to-be-stable technologies.)
• SVM and ANN can be used to build a PPM that handles the Y and X relations in an evolutionary manner where a series of Xs influence a Y which along with another set of Xs and maybe some Ys also influence another Y. The PPM in this case is an inter-connected network model (not the typical regression equation that one thinks of at the mention of the term PPM). By being a network model the PPM in case of SVM and ANN has, in effect, n-levels.
• SVM and ANN based models represent the process to a high degree of fidelity. The various sub-processes, sub-sub-processes, sign linkages, input and control factors can be represented in a SVM or ANN with all its richness and variants to make it almost at par with the real process.

Having understood the advantages of PPMs built using SVM and ANN the question that naturally arises is why are they not being used that widely for PPM building and for building models in areas other than CMMI high maturity? Here are some challenges and constraints associated with PPMs built using SVM and ANN:

• Building meaningful PPMs based on SVM and ANN assumes the PPM builder has a fine and granular understanding of the sub-processes, sub-sub-processes, "probable" sign linkages based on physical reality, input and control factors
• Building meaningful PPMs based on SVM and ANN takes long with one of the primary reason being that extensive data are required to be collected.
• Data used for building PPMs based on SVM and ANN needs to be highly accurate for correctly mapping the relations between a set of Xs and Ys
• Due to significant, sudden change in the process (which leads to changes to several sub-processes, sub-sub-processes,  input and control factor) the SVM and ANN may be rendered totally ineffective and building the "revised" PPM may take long.
• SVM and ANN are concepts that are not easy to learn and the PPM builders are required to build statistical expertise of a very high order.

In the end, however, it can be said that PPMs built using SVM and ANN can be much more powerful and hence more useful as compared to PPMs built using Regression and other techniques.

Even if PPMs built using SVM and ANN are not used the exercise of trying to build PPMs using SVM and ANN would allow a profound understanding of the overall process being modeled in terms of the constituent sub-processes, sub-sub-processes, "probable" sign linkages based on physical reality, input and control factor. This deeper understanding will surely help make PPMs built using Regression for example much better than otherwise.

Why Linear Regression-based PPMs May Fail to Work?

For CMMI high maturity, building and using PPMs (Process Performance Models) is a mandatory practice. This follows from the fact that the goal for OPP (Organizational Process Performance) includes the statement "provide process performance data, baselines, and models to quantitatively manage the organization’s projects" and goals are required components in the CMMI model. PPMs along with PPBs (Process Performance Baselines) are at the heart of high maturity practices and do pose their fair share of challenges for the organization. This is also summarized in CMMI L5 Nemesis - PPBs and PPM.

Linear Regression-based PPMs seem to be in vogue and probably because of the fact that most of the implementers and consultants find regression easy to learn. This may in turn be a result of the fact that many of the implementers and consultants have minimal or no theoretical and applied background in statistics

Coming back to PPM building, before deciding on the modeling approach (regression is just one of the many) it is a good idea for one to probe the following fundamental questions:

• How much time and energy does one have for building a model? 
• How much prediction power one is aiming for? 
• How much does one already know about statistical modeling?
• How much does one understand the underlying physical process?
• How reliable is the data to be used for building a statistical model?

The last question is the most critical as it directly impacts the goodness of the PPM regardless of what statistical method one chooses. It is but obvious that good PPMs need good data. And in respect of data the following points are important to note:

• The only data available are historical - collecting experimental data or using DoE is not a feasible option due to cost reasons or the data generation frequency imposing natural constraints
• Data are not reliable - historical data generally has high noise factor and tends not to be too reliable

Linear Regression is a good starting point while building PPMs. However, it may fail to deliver PPMs which work well when deployed (despite showing great statistical promise - p-value, R Square Adjusted, VIF, etc. all showing good statistical significance). This may happen due to many reasons like:

• Y and/or X (one ore more) is not normally distributed - this can be handled statistically through transformations and despite the Central Limit Theorem and Chebyshev's Inequality requires careful evaluation of the underlying statistical characteristics.
• One or more Y-X combinations are non-linear - fitting the equation of a line to a curve is like trying to put a cylinder in a square hole, if the diameter of the cylinder is equal or less than the length of the side of the square one can still do that but one would have a good amount of gap left around the cylinder.
• Interaction effects exist between one or more X - this will require modeling X in combinations like X1/X2 or X1*X2 so that the interaction effects are damped or amplified as is appropriate.
• Regression equation shows unrealistic sign relation between Y and X - though a certain relation might be statistically valid it might be physically invalid as understanding of the process should prevail over the sign relation one gets from statistical analysis (it's worth remembering that all sign relations in respect of earth's movement around sun being a circle were supposed to be valid until the sharp insight by Kepler revealed that it is actually an ellipse).

So what does one do beyond the regression-based PPMs. There are many more and probably better options available. Some examples include General Regression, Bayesian Belief Networks (BBNs), Simulation-based PPMs based on underlying empirical distribution, Queuing models, etc.

No matter what statistical modeling technique is used in the end what becomes important is that the PPM works well when deployed. Constant refinement and calibration thus becomes important so that the changes and improvements in the physical process, if any, get reflected in the PPM with a minimal time lag.

Characteristics of Prediction Models

Prediction models are quite useful in many fields like weather forecasting, demand and supply studies, production planning and control, project management, etc.

Here are some of the essential characteristics of a prediction model:

• Prediction models should be able to provide the value of a a parameter before it occurs. Example: the model should be able to tell which team will win the Cricket world cup before the world cup starts till the final match reaches a conclusive stage.
• Prediction models shouldn't just provide the %times the actual value will be within a certain range (or outside as the case may be) but what the range of values in a particular case could eventually be. This is like telling a patient that 20% of patients who get admitted to the hospital don't leave alive. For a particular patient though it will be important to know what is %chance of survival for him or her given the health condition at the time of admission. Example: the model instead of telling that India will win since it has won 2 world cups in the past should instead tell that given India has won last five ODI, series which is more than all the other teams, India has 80% chance of winning the world cup.
• Prediction models should provide the prediction either in an ordered manner with %likelihood of occurrence, or a set of outcomes in a narrow range or a set of few values or at best a single value (in the case of a single value the prediction model is no less than God if the prediction comes to be true in 100% cases). Example: the model should tell that India will win with 80% probability and Australia will win with 40% probability, or one of India or Australia will win it, or India will win it.
• Prediction models should be able to dynamically adjust the prediction made in a progressive manner continuously or at least at multiple interim points till the actual value occurs. This point hides a weakness of prediction models based on historical data - if something happens that has never happened in the past the prediction model may fail to adjust the prediction. Example: In case India looses first 3 matches the model may change the prediction to Australia or maybe some other team winning?
• Prediction models should have some predictors that can be controlled or manipulated to get a desired outcome in case the original prediction is not a desired one. If control or manipulation is not possible then the prediction should be available early enough to plan for appropriate contingency actions. Example: the model should be able to tell that while on the way to airport to catch a flight one would get delayed if one goes by road in which case one can go by train instead. in the case where control or manipulation is not possible like hurricane in an area (which cannot be controlled) the prediction should be available early enough to inform everyone staying in that area to move to another area.

Statistics

Statistics means both the following:
1. Data and analysis results
2. Data analysis methods

Statistics provides a scientific approach to understand, analyze and change the world around us. It is a fact that we can't change anything for better unless we know what it is and statistics helps us in this!

Statistical Tools - Simple

Pareto Analysis
Determine relative priority or importance of available alternatives

Run Chart or Trend Chart
Understand the behavior of a process over time

Mean or Average
Most likely value of a process parameter

Median
Middle-most value

Mode
Value that occurs most of the time

Statistical Tools - Advanced

Statistical Process Control (SPC)
Adoption of Control Charts theory to monitoring and control of business processes

Statistical Data Analysis

Use of Statistics in the form of various Statistical Tools and Techniques have found a useful place in almost all business organizations for different types of analysis.

Statistical Data Analysis techniques can be broadly categorized as:

Statistical Tools - Simple
Statistical Tools - Advanced This categorization is based on the relative difficulty one would face in applying a tool and technique and doesn't imply anything about the underlying theoretical concepts.

How to Perform Statistical Data Analysis

Statistical Data Analysis can be performed using various Software Applications such as MS-Excel, MINITAB, SPSS, SAS, etc.

From a general purpose angle, MS-Excel offers enough feature to perform basic statistical analysis easily. MS-Excel can also be used in a limited way to perform certain types of advanced statistical analysis. In addition, many MS-Excel Add-ins are available which when plugged to MS-Excel can be used to perform most of the advanced statistical analysis.

Useful Utilities which provide Sample Examples or can be used as Templates for performing certain types of basic and advanced statistical analysis are also available on this site.

Making Data Speak

'Making Data Speak' or analyzing data to extract information contained within the data is an important competency for business organizations. Some of the key questions which emerge when 'Making Data Speak' are: 

What data to collect?

This also includes related questions like how to collect, when to collect, who will collect, how to store, etc.

Data collection is the foundation for identifying meaningful and effective actions from data analysis. Inappropriate and inaccurate data will speak but not rightly and would generally be misleading. Data collection must also account for subjective factors like people's attitude towards reporting data correctly and on time, estimate of the error in measurement due to the measuring devices, etc.

What analysis to perform?

This also includes related questions like how to clean data before subjecting it to analysis, which analysis tool to apply, when to perform analysis, who will perform analysis, how to store analysis results, etc.

Data analysis serves the purpose of subjecting the data to speak up through the application of the right kind of analysis method. In data analysis the most crucial aspect is selecting the appropriate analysis method. Employing the selected analysis method is generally easy due to the plethora of analysis software tools available in the market. At times, data may need to be collated or organized differently than how they are stored before analysis can be performed.

How to interpret the analysis results for decision making?

This also includes related questions like what other non-data factors to consider, how to check validity of the inference drawn, how to identify what actions to take going forward, etc.

Data interpretation is the real skill. Inferring what the data speak rightly is important to guide right actions. Inferring from data analysis results requires good, solid understanding of the actual underlying physical processes. This is where domain understanding becomes absolutely important. There are times when an expert in data analysis can provide useful insight - this is because the underlying processes across the various domains follow certain statistical distributions (most common is the Normal Distribution, there are others also like Poisson, Exponential, Geometric, Weibull, Hypergeometric, etc.).