Characteristics of Prediction Models

Prediction models are quite useful in many fields like weather forecasting, demand and supply studies, production planning and control, project management, etc.

Here are some of the essential characteristics of a prediction model:

• Prediction models should be able to provide the value of a a parameter before it occurs. Example: the model should be able to tell which team will win the Cricket world cup before the world cup starts till the final match reaches a conclusive stage.
• Prediction models shouldn't just provide the %times the actual value will be within a certain range (or outside as the case may be) but what the range of values in a particular case could eventually be. This is like telling a patient that 20% of patients who get admitted to the hospital don't leave alive. For a particular patient though it will be important to know what is %chance of survival for him or her given the health condition at the time of admission. Example: the model instead of telling that India will win since it has won 2 world cups in the past should instead tell that given India has won last five ODI, series which is more than all the other teams, India has 80% chance of winning the world cup.
• Prediction models should provide the prediction either in an ordered manner with %likelihood of occurrence, or a set of outcomes in a narrow range or a set of few values or at best a single value (in the case of a single value the prediction model is no less than God if the prediction comes to be true in 100% cases). Example: the model should tell that India will win with 80% probability and Australia will win with 40% probability, or one of India or Australia will win it, or India will win it.
• Prediction models should be able to dynamically adjust the prediction made in a progressive manner continuously or at least at multiple interim points till the actual value occurs. This point hides a weakness of prediction models based on historical data - if something happens that has never happened in the past the prediction model may fail to adjust the prediction. Example: In case India looses first 3 matches the model may change the prediction to Australia or maybe some other team winning?
• Prediction models should have some predictors that can be controlled or manipulated to get a desired outcome in case the original prediction is not a desired one. If control or manipulation is not possible then the prediction should be available early enough to plan for appropriate contingency actions. Example: the model should be able to tell that while on the way to airport to catch a flight one would get delayed if one goes by road in which case one can go by train instead. in the case where control or manipulation is not possible like hurricane in an area (which cannot be controlled) the prediction should be available early enough to inform everyone staying in that area to move to another area.

Quantitative Process Composition and Process Tailoring in CMMI Model

The concept of quantitative process composition (QPC) is integral to the implementation of the QPM/QWM (Quantitative Project/Work Management) process area at level 4 of CMMI. 

For understanding quantitative process composition (QPC) it is important to first understand qualitative process tailoring (QPT) at level 3.

So how is QPC at level 4 different from QPT at level 3 and what exactly is it?

Qualitative Process Tailoring (QPT)

QPT is the definition of project-level processes by selecting from the organization's set of standard processes (OSSP or the organization's QMS in a simple language) based on subjective factors and qualitative considerations. 

The following example illustrates the concept underlying QPT.

Consider a project that needs to work on developing a complex application.

The project may identify technical complexity as a risk that can impact the quality of application delivered to the customer.

For overcoming the above risk the project may decide to perform review of the architecture/design in a manner more rigorous than the usual.

It is assumed that the customizations done to the process in this case will have positive impact in achieving the desired outcome.

The above is an example of qualitative process tailoring and might mean one or more of the following in this specific case:

• The review checklist might be 'tailored up' by injecting it with additional questions related to design aspects like fan-in, fan-out, modularity, coupling, cohesiveness, etc. (tool/template level tailoring)
• The method selected from review might be perspective-based review by a group of experts rather than desk check by a single peer reviewer (procedure/method level tailoring)

In general, usual approaches to QPT include above variants along with some more as listed below:

• Use of customer or client provided processes and templates over the organization's standard processes and templates
• Use of organization's standard processes and templates with certain sections turned into "not applicable"
• Use of the option of 'tailoring out' certain of the organization's standard processes and templates in entirety (case of waiver)

Quantitative Process Composition (QPC)

QPC is the definition of project-level processes by selecting from the organization's set of standard processes based on objective factors and quantitative considerations.

The following example illustrates the concept underlying QPC.

Consider the same example as above of a project that needs to work on developing a complex application.

The project may identify technical complexity as a risk that can impact the quality of application delivered to the customer.

For overcoming the above risk the project may decide to perform review of the architecture/design in a manner more rigorous than the usual.

And here comes the twist.

The quality of application delivered to the customer is first expressed in quantitative terms such as number of post release defects in six months will not be more than 'n' per SLOC of code in the delivered application (the Big Y).

It is assumed that the customizations done to the process in this case will not only have positive impact in achieving the desired outcome but also the same can be expressed in quantitative terms (typically mean and SD assuming normal distribution).

The above is an example of quantitative process composition and might mean one or more of the following in this specific case:

• The capability of review process using the normal and 'tailored up' checklists for design review (categorical X variable) is known and its influence on Y is known.
• Obviously, one would expect that the 'tailored up' checklist leads to a better Y (if not, the assumption that 'tailored up' checklist is better doesn't hold true)
• The impact on Y of the extent and intensity of X is known.
• So, by varying the levels of X (similar to the concept of varying the levels of treatment in DOE/ANOVA to determine the optimal combination) Y can be forced to behave at the desired level.
• Another possibility is to determine some other Xs that influence Y (like number of weighted requirements, modules, code size, etc.).

In general, usual approaches to QPC include above variants along with some more as listed below:

• Defining alternative sub-processes for performing an activity and selecting the set of sub-processes for performing various activities that result in achieving Y at the desired level
• Determining Xs that influence Y and establishing the relation between Y and Xs (typically expressed in the form of statistical/mathematical model or based on simulation of Y by varying the Xs in the range observed based on historical data).

 QPC versus QPT

So the primary difference between QPC and OPT is that in QPC process selection is done based on statistically validated hypothesis whereas in QPT process selection is done based on experentially assumed behaviors.

In QPC defining project processes is called as process composition whereas in QPT defining project processes is called as process tailoring.

QPC being a level 4 high maturity practice is expected to make use of heavy duty statistical data analysis techniques like Sampling Distributions, SPC, ANOVA, Statistical Modelling.

Whereas QPT being a level 3 or 'low maturity' practice is expected to make use of light duty statistical data analysis techniques like Averages, Run Charts, Box Plots, etc.

P.S. Some Interesting Observations

• It might appear that in the CMMI model process 'composition' is higher up in the pecking order compared to process 'tailoring'. 
• The term process 'composition' makes one imagine Arts, Music, Mozart, etc. whereas the term process 'tailoring' makes one think of Stitching, Cloth, Scissors, etc. 
• This is not a sound view of the fundamental differences between these two because 'tailoring' could also make one imagine about Versace, Chanel, etc.
• The other point is that use of statistical analysis doesn't imply high maturity. 
• It might surely mean high visibility and probably high control but that doesn't always translate into high maturity. 
• Also, if one were to look at industries other than software, the practices which make level 4 and 5 high maturity will not probably quality as high maturity. 
• Usage of DOE, ANOVA, SPC, etc. is so commonplace in manufacturing companies that high maturity is business as usual.
• The use of statistical and/or simulation-based models for quantitative planning and monitoring & control may not be very effective if the behavior of the Big Y or the small y(s) is influenced by factors which are essentially not controllable or controllable in a limited way. 
• A good example of this is that the maintainability of a product is heavily influenced by the complexity of the business logic, but the complexity factor is controllable in a limited way.
• Another example is that the product quality is heavily influenced by the deep domain expertise of the personnel working on it which is not controllable in a short span of time (over a long span of time this is definitely controllable but due to personnel churn at times this may not happen).