Comparative Analysis of Frameworks and Models for Improvement, Certification and Excellence

It is quite interesting and informative to compare and contrast the various frameworks and models for improvement, certification and excellence. Comparative analysis provides deep insights into how to take an integrated view of business excellence.

Comparative analysis looks at the similarities and differences between the models with respect to the following dimensions:

• Framework Assumptions
• Framework Structure
• Framework Content

Click here to download the comparative analysis of CMMI and ISO 9001

Process Improvement Versus Process Certification Focus

Companies can be differentiated on the basis of the predominant behavior they demonstrate in their operations as far as focus on process is concerned. Viewed in this manner, there are broadly speaking two types of companies:

• Those that focus on process improvement (Cost Reduction, Cycle Time Reduction, etc.)
• And those that focus on process certification (ISO 9001, etc.)

It is always the case that companies focus on both the above dimensions. If one were to rank the focus level in terms of  process improvement versus process certification the picture would be as depicted below:

Startup Product Companies:
The key driver is to stay afloat till the first shipment of the product. If that goes on well lot of cash inflow will happen. Systems and processes are not stressed upon since the sole aim to to get the product out once, the first time and there is no point thinking about second and subsequent time.

Stable Product Companies
The key driver is to milk the cash cows consistently and cost-effectively. There might be new products in the pipeline but some failures will not necessarily mean business closing down. Systems and processes are needed to ensure consistent delivery and after-sales service. Process certifications however may have limited or minimal business value.

Startup Service Companies
The key driver is to stay afloat till the first customer project. If that goes on well lot of cash inflow will happen. Improvement is not stressed upon initially since the sole aim to to get the first customer. Need of process certifications is very high though right from the start to give confidence to the customers on the company's capability to execute the awarded contract.

Stable Service Companies
The key driver is to service customers cost-effectively. This makes process improvement important to gain efficiency in operations. Process certifications remain important and at the same time process improvement also becomes quite important.

Building PPMs Using Support Vector Machine (SVM) and Artificial Neural Network (ANN)

Support Vector Machine (SVM) and Artificial Neural Network (ANN) are concepts/methods employed in the field of Artificial Intelligence and can be useful in building Process Performance Models (PPMs) that in some sense learn and evolve as more data points are fed into them.

What are the key characteristics and advantages of PPMs built using SVM and ANN? Here are some:

• The PPM can dynamically adjust to underlying changes in the process in the range of Ys and Xs that was used while building the PPM
• Many of the subjective and fuzzy factors that are important in the software development process can be more effectively modeled. Software development process is influenced by following fuzzy factors, whether or not they can be easily modeled:
• Competency level of team members on various activities like architecture, design, coding, reviews, testing and in the domain/technology
• Complexity level of the work being performed
• Mental fitness level of the team members (remember human brain is the machine used for manufacturing software so if a software engineer is overworked or had a fight at home in the morning it might affect the quality of work performed)
• Clarity level of requirements provided by the customers and also as understood by the project team (customers themselves may not know what they want until late in the project when they start seeing the first few screens, demos, etc.)
• Clarity level of the technical solution envisaged by the project team (this might be a big factor in case of projects in new areas or yet-to-be-stable technologies.)
• SVM and ANN can be used to build a PPM that handles the Y and X relations in an evolutionary manner where a series of Xs influence a Y which along with another set of Xs and maybe some Ys also influence another Y. The PPM in this case is an inter-connected network model (not the typical regression equation that one thinks of at the mention of the term PPM). By being a network model the PPM in case of SVM and ANN has, in effect, n-levels.
• SVM and ANN based models represent the process to a high degree of fidelity. The various sub-processes, sub-sub-processes, sign linkages, input and control factors can be represented in a SVM or ANN with all its richness and variants to make it almost at par with the real process.

Having understood the advantages of PPMs built using SVM and ANN the question that naturally arises is why are they not being used that widely for PPM building and for building models in areas other than CMMI high maturity? Here are some challenges and constraints associated with PPMs built using SVM and ANN:

• Building meaningful PPMs based on SVM and ANN assumes the PPM builder has a fine and granular understanding of the sub-processes, sub-sub-processes, "probable" sign linkages based on physical reality, input and control factors
• Building meaningful PPMs based on SVM and ANN takes long with one of the primary reason being that extensive data are required to be collected.
• Data used for building PPMs based on SVM and ANN needs to be highly accurate for correctly mapping the relations between a set of Xs and Ys
• Due to significant, sudden change in the process (which leads to changes to several sub-processes, sub-sub-processes,  input and control factor) the SVM and ANN may be rendered totally ineffective and building the "revised" PPM may take long.
• SVM and ANN are concepts that are not easy to learn and the PPM builders are required to build statistical expertise of a very high order.

In the end, however, it can be said that PPMs built using SVM and ANN can be much more powerful and hence more useful as compared to PPMs built using Regression and other techniques.

Even if PPMs built using SVM and ANN are not used the exercise of trying to build PPMs using SVM and ANN would allow a profound understanding of the overall process being modeled in terms of the constituent sub-processes, sub-sub-processes, "probable" sign linkages based on physical reality, input and control factor. This deeper understanding will surely help make PPMs built using Regression for example much better than otherwise.

Why Linear Regression-based PPMs May Fail to Work?

For CMMI high maturity, building and using PPMs (Process Performance Models) is a mandatory practice. This follows from the fact that the goal for OPP (Organizational Process Performance) includes the statement "provide process performance data, baselines, and models to quantitatively manage the organization’s projects" and goals are required components in the CMMI model. PPMs along with PPBs (Process Performance Baselines) are at the heart of high maturity practices and do pose their fair share of challenges for the organization. This is also summarized in CMMI L5 Nemesis - PPBs and PPM.

Linear Regression-based PPMs seem to be in vogue and probably because of the fact that most of the implementers and consultants find regression easy to learn. This may in turn be a result of the fact that many of the implementers and consultants have minimal or no theoretical and applied background in statistics

Coming back to PPM building, before deciding on the modeling approach (regression is just one of the many) it is a good idea for one to probe the following fundamental questions:

• How much time and energy does one have for building a model? 
• How much prediction power one is aiming for? 
• How much does one already know about statistical modeling?
• How much does one understand the underlying physical process?
• How reliable is the data to be used for building a statistical model?

The last question is the most critical as it directly impacts the goodness of the PPM regardless of what statistical method one chooses. It is but obvious that good PPMs need good data. And in respect of data the following points are important to note:

• The only data available are historical - collecting experimental data or using DoE is not a feasible option due to cost reasons or the data generation frequency imposing natural constraints
• Data are not reliable - historical data generally has high noise factor and tends not to be too reliable

Linear Regression is a good starting point while building PPMs. However, it may fail to deliver PPMs which work well when deployed (despite showing great statistical promise - p-value, R Square Adjusted, VIF, etc. all showing good statistical significance). This may happen due to many reasons like:

• Y and/or X (one ore more) is not normally distributed - this can be handled statistically through transformations and despite the Central Limit Theorem and Chebyshev's Inequality requires careful evaluation of the underlying statistical characteristics.
• One or more Y-X combinations are non-linear - fitting the equation of a line to a curve is like trying to put a cylinder in a square hole, if the diameter of the cylinder is equal or less than the length of the side of the square one can still do that but one would have a good amount of gap left around the cylinder.
• Interaction effects exist between one or more X - this will require modeling X in combinations like X1/X2 or X1*X2 so that the interaction effects are damped or amplified as is appropriate.
• Regression equation shows unrealistic sign relation between Y and X - though a certain relation might be statistically valid it might be physically invalid as understanding of the process should prevail over the sign relation one gets from statistical analysis (it's worth remembering that all sign relations in respect of earth's movement around sun being a circle were supposed to be valid until the sharp insight by Kepler revealed that it is actually an ellipse).

So what does one do beyond the regression-based PPMs. There are many more and probably better options available. Some examples include General Regression, Bayesian Belief Networks (BBNs), Simulation-based PPMs based on underlying empirical distribution, Queuing models, etc.

No matter what statistical modeling technique is used in the end what becomes important is that the PPM works well when deployed. Constant refinement and calibration thus becomes important so that the changes and improvements in the physical process, if any, get reflected in the PPM with a minimal time lag.