Architecture Evaluation

From Wiki
Jump to navigationJump to search


  • Design decisions, patterns are used because of the effects they have on the system.
  • Hence, these choices are analysable.
  • Design must be evaluated continuously during the life cycle, especially early on.
  • There are many benefits to evaluation:
    • Financial.
    • Early detection of problems.
    • Captured rationale - Documented design rationale is important so that implications of modifications can be assessed.
    • Validation of requirements - opens up the requirements themselves for discussion. e.g. conflicting requirements.
    • Improved architectures.
  • Use scenarios as vehicles for asking probing questions about how the arch. responds to various situations.
  • Propose scenarios against a quality attribute.
  • For e.g. against performance - give usage profiles that stretch the system.


  • Software Architecture Analysis Method.
  • Steps:
  1. Develop scenarios
  2. Describe candidate architecture
  3. Classify scenarios - direct (arch directly support) or indirect.
  4. Perform scenario evaluations - For each indirect scenario, changes to the architecture that are necessary for it to support the scenario.
  5. Reveal scenario interaction - When 2 or more indirect scenarios require change to a single component - they are said to interact in that component. Areas of high scenario interaction

can indicate poor separation of concerns in a system component.

  1. Overall evaluation : Tabulation, weighting of scenarios, rating architecture support. To compare candidate architectures.


  • Architectural Tradeoff Analysis Method.
  • ATAM is a refinement of SAAM.
  • ATAM reveals how well an architecture satisfies particular quality goals.
  • Recognises that an arch. decision tends to affect more than one quality attribute and provides insight into how quality goals trade-off against each other.

Participants in ATAM

  • Evaluation team - Group is external to the project whose arch. is being evaluated. Should be competent, unbiased outsiders.
  • Project decision makers - Speak on behalf of the project or have authority to make changes to it. The architect must always be included.
  • Architecture stakeholders - Developers, Testers, Users, Maintainers. They are needed to articulate the specific quality attribute goals.

Goals of ATAM

  • Elicit and refine a precise statement of the architecture’s driving quality attribute requirements
  • Elicit and refine a precise statement of the architectural design decisions
  • Evaluate the architectural design decisions to determine if they satisfactorily address the quality requirements

Outputs of ATAM

  • A concise presentation of the architecture. Requirement is to be concise - so that it is understandable.
  • Articulation of business goals.
  • Quality requirements in terms of a collection of scenarios.
  • Mapping of architectural decisions to quality requirements.
  • A set of identified sensitivity and trade-off points.
    • Senstivity : Where small changes have big impact on QA's.
    • Tradeoff : Where more than 1 QA is affected.
  • A set of risks and non-risks.
  • A set of risk themes.

Phases of ATAM

  • Partnership and preparation phase - Eval team and project decision makers.
  • Evaluation phase 1 - Eval team and project decision makers.
  • Evaluation phase 2 - Eval team and project decision makers and stakeholders.
  • Follow up - Eval team and eval client.

Evaluation phase steps

1. Present the ATAM

  • Eval leader presents the ATAM to the project team.

2. Present Business Drivers

  • Project decision maker presents a system overview from a business perspective.
  • Business goals, major stakeholders.
  • Architectural Drivers.

3. Present Architecture

  • Lead architect makes a presentation describing the arch.
  • Describes patterns used, tactics, styles. Describes constraints.
  • Uses views to describe the architecture.

4. Identify Architectural Approaches

  • Evaluation team lists and catalogues all the patterns and approaches that are evident from the architects presentation.

5. Generate Quality Attribute Tree

  • Identify, prioritize and refine the system's most important quality attributes.
    • Level 0 - Root node - utility, overall expression of how good the system is.
    • Level 1 - Quality attributes e.g. performance, modifiability, security.
    • Level 2 - Specific quality attribute refinements e.g. - Performance is divided into 1) latency and 2) transaction throughput
    • Level 3 - Scenarios. Each scenario is given a priority - Importance, Difficulty of achieving e.g. (High, High), (Medium, Low ), (Low, Low)
  • The tree tells the ATAM team where to spend its time.

6. Analyse Architectural Approaches

  • Examines the highest ranked scenarios one at a time.
  • Probe the architectural approaches in order to identify risks, sensitivity points, and tradeoffs. Document these.
  • e.g. no of DB clients affects TPS. Assignment of clients is thus a sensitivity point.

Hiatus and start of Phase 2

  • Evaluation team summarizes the learning and communicates with architect.
  • Questions and clarifications answered.

7. Brainstorm and prioritise scenarios

  • Create and analyse scenarios that represent the various stakeholders’ interests to understand quality attribute requirements and their relative importance.
  • Eval team asks stakeholders to brainstorm scenarios.
  • Scenarios must be prioritized. Stakeholders vote.
  • Compare scenario list from this exercise with those from the utility tree exercise.

8. Analyse Architectural Approaches

  • Analyse highly ranked brainstormed scenarios from previous steps in same manner as step 6.
  • i.e. identify sensitivity points, tradeoffs, risks.

9. Present Results

  • All collected info from ATAM needs to be summarized and presented to stakeholders.
  • Arch approaches, utility tree, discovered risks, non risks, sensitivity points and trade-offs.
  • Value addition by grouping risks into risk themes and relate them to business drivers.


  • Cost Benefit Analysis Method
  • ATAM misses an important consideration - the economic cost of tradeoffs.
  • CBAM builds on ATAM to model the costs and benefits of architectural design decisions and is a means of optimizing such decisions.
  • CBAM's goal is to maximize the difference between the benefit derived from the system and the cost of implementing the design.
  • Each arch decision has a cost. For e.g. redundant HW for high avail has a cost. This can result in a benefit for the org.
  • CBAM elicits these costs and benefits - so that stakeholders can decide whether to use the tactic to achieve availability. Choose arch. strategies based on their ROI - ratio of benefit to cost.


Variations of scenarios

  • CBAM uses a set of scenarios rather than a single scenario in ATAM. The set of scenarios are generated by varying the values of the responses.
  • Every stimulus - response pair in a scenario provides some utility to the stakeholder. Utility of different possible values for the response can be compared.

Utility response curve

  • The utility-response curve depicts how the utility derived from a particular response varies as the response varies.
  • Best case qa level is that above which the stakeholders foresee no utility. for e.g. 0.1 sec is instantaneous, so improving it to 0.03 sec has no utility. Best case utility is 100.
  • Worst case qa level is a threshold above which a system must perform. Worst case utility is 0.
  • Then determine current and desired levels using this scale.

Priorities of scenarios

  • 1st, stakeholders vote on scenario to establish an ordering among them. Based on expected response rate.
  • Then assign a weight of 1 to highest ranked and fractional to others.

Arch strategies

  • For each strategy, get the expected utility value of response through the curve.
  • Effect on other attributes.
  • Cost estimate for implementing the strategy.
  • Determine benefit from normalization using formula.

Calculating ROI

  • Ratio of Benefit to the Cost for a particular strategy.

CBAM Steps

1. Collate scenarios

  • Collect scenarios from ATAM.
  • Prioritize and choose top 1/3rd.

2. Refine scenarios

  • Elicit Worst case, best case, current and desired response levels

3. Prioritize scenarios

  • Allocate votes to stakeholder.
  • Voting based on desired response value.
  • Choose top 50% of scenarios.

4. Assign Utility

  • Determine utility for each QA level from step 3.

5. Develop arch strategies and determine QA response levels

  • Develop/capture strategies and determine "expected" QA levels.

6. Determine expected utility value of arch strategy using interpolation

  • From the curve.

7. Calculate total benefit from arch strategy

  • Formula.

8. Choose arch strategies based on ROI subject to cost constraints

  • Calc ROI. Rank the strategies according to ROI.

9. Confirm results with intuition

  • Consider if they align with org's business goals.