Agile and DevOps

Two popular Project Management Models are:

• Waterfall
  • Defined, Linear
  • Each phase of the project is completed before handing over to the next phase
  • Phases include Requirements, Design, Development, Testing & Deployment
  • DisAdv: late feedback, costly to fix errors discovered late, high cost for changes
• Agile
  • Empirical, iterative development of features
  • Each phase may be revisited multiple times
  • Each element of the project goes through each of the usual phases
  • Elements may be at various stages of development at any given time
  • Focus on delivering highest priority items first

Breaking a project into it’s elements (features) enables faster change and greater feedback loops.

Agile Values

• Primary
  • Individuals and interactions
  • Collaboration with stakeholders
  • Rapid response to change
  • Working Software
• Secondary
  • Processes (&tools)
  • Contracts
  • Plans
  • Documentation

5 Agile Phases

• Envisioning
  • Where the project is initiated
• Speculating
  • A planning phase
• Exploring
  • Development
• Adapting
  • Review and improvement
• Closing

Measure of Progress.

• Working software is the primary measure of progress
• Satisfied customers is a secondary measure

Questions

What is the agile manifesto? Write it down.
What are the Phases?
What are the twelve principles?
Why is Agile considered a culture shift?

Agile Roles and Responsibilities

Whole Team
Each team should have a sufficient skill set to complete the task. This is referred to as the Whole Team. These skills include database, UI, testing and other skills as required. Team members should be ‘generalizing specialist’. This refers to the idea of having a particular area of expertise but knowledge in other areas and the willingness to continuously learn new areas as necessary. Any team member can be given one or more roles as necessary. This helps remove the idea of a specialism. This results in a team where all team members are equal. Where possible working teams should remain together.

There are many Agile techniques but the one used for this module is SCRUM. As such the roles and responsibilities listed here refer to the SCRUM methodology. The roles in SCRUM include:

1. ScrumMaster
2. Team
3. Product Owner
4. Stakeholders.

ScrumMaster
The ScrumMaster is a facilitator of the Scrum Process. The ScrumMaster focuses on facilitating sprint planning, Daily Scrums, Sprint Reviews and Retrospectives. In essence their job is to aid the team to recognise the business value of the task at hand. They attend every scrum meeting to help the team to stay focused and aid with the removal of any impediments that may be discovered. To this respect the ScrumMaster assists the Team, Product Owner and Stakeholders to honour their commitments.

The Team
The Team is made up of a number of members with the goal of working with the Product Owner to decompose Backlog items. They maintain the Sprint Backlog, Burndown Chart and Task board. The demonstration of the final product is also carried out by the team. They work together as a team to help each other improve via collaboration and sharing experience. The team members take on the role of the ScrumMaster if he/she is unable to do so.

Product Owner
The product owner is responsible for defining user stories. They help define the priorities for each of the user stories. The product owner also confirms if the backlog item has met the acceptance criteria so that they are defined as done. In other development methodologies the product owner remains active only in the user requirements phase of the SDLC. In Agile development the product owner continues to attend meetings and provide feedback throughout the SDLC.

Stakeholders
The stakeholders include anyone who has an interest in a product. The end user is an obvious stakeholder but other stakeholders include people who are impacted by the project, domain experts, policy enforcers and others. Stakeholders provide advice on the product and may further clarify user requirements.

Agile and DevOps

Culture Shift
The Agile Manifesto is a mantra for modern software developers. It requires developers to consider more carefully how and why they develop software. To this end a number of prominent and well published software developers got together to define what they felt was important with regard to software development. The following image has been taken directly from the agilemanifesto.org.

The reason the entire image is included as opposed to simply typing the words is that the manifesto speaks of individuals over interactions. The image portrays Kent, Ward and Martin Fowler with others as they discuss this as a group. The notion is that following rigid requirements has resulted in less elegant software.

Note that the items listed to the right ‘over processes and tools’ are not discarded. The manifesto simply states that the items to the left should be considered more valuable. The manifesto, developed in 2001 is a vastly different approach to that taken using traditional methodologies such as Waterfall and others. Traditional methodologies are often considered heavyweight as they contain many artefacts, significant documentation and lack flexibility.

In order to truly buy into the manifesto new methodologies had to be created. More importantly it required a significant culture shift in development houses. Some would argue that the manifesto argues for undisciplined software development. Many new methodologies which conform to the agile manifesto favour structure but also flexibility to the changing needs of the customer. This increased flexibility should result in greater success in software development projects.

Four Values
The Four Values of the manifesto are simply the four main statements.

1. People are required to deliver software. Users or clients (people) must explain their needs to the team. People are generally overworked and undervalued. By focusing on the people first then the appropriate tools and techniques can be selected to ‘support’ them. This then focuses on the people not the tools.

2. Working software over documentation. Requirements documents are unwieldy and yet necessary. Initial planning dictates that the requirements must be outlined in detail prior to commencement of the project. Regardless customer requirements change. Documentation required at the end of each development phase may be significant as is the case with the waterfall methodology. Are items documented in detail which could simply be covered by adequate commenting? The agile manifesto requires the team to focus on doing no more than absolutely necessary.

3. Customer Collaboration over contract negotiation. Contracts are necessary and will not go away. However if the development methodology included a greater level of input from the system user or client then the final delivered system would have a greater chance of matching the customer requirements.

4. Respond to change over following a plan. No matter how well a project is planned something can go wrong or needs will change. Without flexibility the final released system may not meet the customer’s needs. Plans, charts and other techniques to measure the success of a project is still a requirement but the need to dynamically update to the changing environment means that the measure of success will prove more realistic.

Twelve Principles

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Continuously informing the customer and enabling them to be involved in the decision making empowers them to take ownership early in the process. Continuous delivery improves trust.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage. 
Sprints are often two weeks. Change can easily be adopted in any new sprint.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. 
Continuous Delivery in sprints of 2 to 4 weeks.

4. Business people and developers must work together daily throughout the project. 
Daily meetings or in the Scrum methodology they are referred to as stand-ups enable enhanced communications between the stakeholders.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. 
People perform best when they are motivated about what they have been asked to do. They should be passionate about their job or task. The team should trust and support each other. This can only be developed over time. The retrospective allows teams to look at how they could improve. By admitting where they went wrong each person can make changes that will enhance the work of the entire team.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. 
Teams gel better when the know each other.

7. Working software is the primary measure of progress.
Something is declared ‘done’ when it is shippable and agreed upon by the team.

8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
The focus on people over process enables the team to grow which means that no-one should become overburdened. Development could continue on very large projects for as long as necessary.

9. Continuous attention to technical excellence and good design enhances agility.
Getting the balance correct between ‘Building the right thing’ and ‘Building the thing right’.

10. Simplicity — the art of maximizing the amount of work not done — is essential.
Refactoring. Do no more coding than necessary.

11. The best architectures, requirements, and designs emerge from self-organizing teams. 
The team knows best but a servant leader can help remove obstructions from the team and encourage the team when necessary.

12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.
Self-reflection through retrospectives help the team to become more effective. The scrum-master helps the team to work on improving.

Why Agile?

There are many reasons for adopting agile some of which are outlined below.

© https://www.slideshare.net/NigelFds/agile-software-development-monash-university-guest-lecture-series

However it is worth noting that agile will not increase the rate of delivery of software products nor will it cut costs. It does enable development to focus on the most needed areas and enable increased feedback from customers to ensure that the final product more accurately reflects the user requirements.

Agile Culture Shift

Left Shift
•This is where Software teams focus on Quality instead of bug detection.
•This is aided by Continuous Integration, Continuous Testing and Continuous Deployment.

How to make change
•The Reengineering Alternative by William Schneider describes 4 cultures:
•Collaboration
•Control
•Cultivation
•Competence

LalouxCulture Model

• Reinventing Organizations by Lalouxindicates how society evolves.
• Each section is defined by colour.
  • Red is Power.
  • Amber is achievement.
  • Green is (empowering) people .
  • Teal is shared power. Autonomous teams.

Agile : Scrum Key Features

Roles

• The team members level of investment in the process is important
• Each member can have 1 or more roles
• In Scrum the main roles are:
  • Product Owner
  • Scrum Master
  • Team Member

Product Owner
•This person serves as the business liaison –he is not the actual customer
•He may be a product manager or a business analyst
•He makes decisions and clarifies issues
•He identifies and maintains the priority of each task in the product backlog
•There is only one product owner

1.Creates and maintains the product backlog
2.Prioritize and sequences the backlog according to business value
3.Assists with elaboration of Epics, themes and features into user stories which are achievable in a single sprint.
4.Conveys business goals at the beginning of every sprint
5.The Product Owner is the interface to the customer
6.The PO participates in the scrums, sprint reviews and retrospectives
7.The PO accepts or rejects the work at the end of the sprint
8.The PO can introduce change at the end of a sprint
9.The PO communicates with customers, management and others as necessary to support the team
10.The PO can terminate a project if necessary in extreme circumstances

Scrum Master
•It is a full time job
•The Scrum Master supports the Product Owner particularly in arranging the backlog to maximise value
•He facilitates (coaches) and motivates the team
•He ensures Scrum Processes are adhered to including facilitating events
•He removes obstructions
•Facilitates (not attends) the stand-ups
•Facilitates retrospectives, sprint reviews and sprint planning
•Maintains burn-down charts
•Removes obstructions and interruptions to the work of the team
•Works with the PO to aid understanding of the technical user stories
•Encourages collaboration with the team and the PO

Scrum Events

There are four main events in Agile Scrums
•Sprint
•Daily Scrum
•Sprint Review
•Sprint Retrospective

Sprint
•A sprint is a development cycle. It can be 30 days or is now often 2 weeks.
•A planning meeting happens first where tasks to add to the sprint are taken from the backlog.
•Items in the sprint are referred to as Features.
•The sprint has it’s own backlog
•Items not completed are returned to the Product Backlog
•A time estimation to complete the sprint is made by the team
•A Sprint goal defines the success criterion for the sprint to keep the team focused.
•After the sprint the review or retrospective meetings occur

Daily Scrum
•Ashort meeting of less than 30 minutes.
•Daily Stand-up where the team shares information about
•What they did
•What stopped them completing the task (if true)
•What they will do next

Sprint Review
•This is an informal meeting
•There should be no power point and no formal preparation
•The team demonstrates any work completed during a sprint but it is not a demonstration of the working product to completion
•Stakeholders can ask questions

Sprint Retrospective
•Three key questions are asked:
•What went well during the sprint
•What went wrong
•What could be improved
•It should be an open and honest meeting
•The focus should be on proactive improvement
•The retrospective should aid the teams sense of ownership

Backlog Grooming
•The Team and PO discuss items on the backlog to refine them if they are unclear.
•A backlog grooming session is not one of the key meetings but it can be required particularly where teams are new.
•They are used to break down Epics and improve stories that are poorly written
•They provide an opportunity to do long-range planning.

Artifacts
•Product Backlog
•List of requirements for the product
•Items are ordered by the product owner based on business value, date needed, dependencies, risk.
•PO is responsible for the backlog and the business value of the items
•The team is responsible for defining the estimated effort to complete the item in Story-points or estimated hours
•Features
•Items added to the backlog written in story format

• Sprint Backlog
  • List of work to be carried out as part of that sprint.
  • Items are selected from the Product backlog
• Stories/Features
  • Should be broken into tasks
  • Should be between 4 and 16 hours work. Preferably 8 hours or less.
• Tasks
  • During the daily scrum team members choose items from the sprint backlog based on priority and the skills of the team member
• Increment
  • All items completed during a sprint and all previous sprints
  • Team members define if something is considered ‘done’
  • An increment must be ‘working’ or in ‘usable’ condition but the PO does not have to release it.
• Burn Down Chart
  • Sprint Burndown Chart is a graphical chart showing how much work remains in the sprint backlog. Viewed daily.
  • Release Burndown Chart shows the work remaining to the committed Product Release which may span multiple iterations.
  • Alternative Release Burndown Chart shows the scope changes to the release content by resetting the baseline.

Other Terms

• User Story
  • It is a feature added to the backlog
  • Written in the form –As a (role) I want to (task) so that I can (Benefit)
  • It is a negotiable, estimable, small and testable requirement
  • Stories can be clustered into epics
• Tasks
  • Each task should be <= 8hours
• Defined as Done
  • The exit-criteria to determine item completion
  • Regression tests are normally required first. All tests previously passed must still pass.
• Velocity
  • The effort the team is able to carry out in a sprint
  • It is made from adding all of the story points from the sprint
• Impediment
  • Anything that stops the team from working efficiently

Sprint Planning Meeting

The Sprint Planning Meeting is attended by the Scrum Master, the Product Owner and the other members of the team. The first decision that has to be made in the sprint planning meeting is the duration. Whilst the Scrum methodology would suggest a 30 cycle most teams do two to three week cycles.

The first four hours of a given sprint is usually taken up with the Sprint Planning meeting. The Backlog refinement may occur two days before the end of a two week sprint. The Review meeting and retrospectives happen on the last day of the sprint and should be scheduled into the time plan.

A number of items should be considered during the Sprint Planning Meeting including:

• Groomed and prioritised Backlog
• Estimates for the highest-priority stories
• The Velocity of the team
• Definition of Done
• The Schedule of the Sprint – staff holidays
• Other inputs – are there any meeting that all team members must attend, new equipment or anything that will disrupt the team?

Mature teams can measure team velocity easily but new teams find it difficult. Team members also change in their ability over time. There may be periods where they are more stressed and their productivity reduces. It is best to estimate velocity based on the average of the total velocity over past sprints.

Once the sprint duration is decided the goal for the Sprint must be decided upon. An objective for the sprint should be decided upon. Features should be selected from the Product Backlog based on the objective and that can be achieved in the sprint duration. It is best practice to add a little more to the sprint backlog than the team feel they may achieve in the time allocated. These items are then marked as stretch tasks.

Each item in the Product Backlog should be reviewed carefully. The PO explains how the item functions. The team then asks question to refine the details. The User Stories are often backed up by a sketch of the UI or other visuals. The sketches are annotated with functionality. Identifying scenarios from the outset helps the team identify how the product will be used.

A Definition of Done (DoD) is agreed upon for the story. This may include:

• Test conditions which are beneficial to the story
• Unit tests
• Acceptance tests
• Functional tests
• System tests
• Regression tests
• Documentation – keeping in mind the Agile manifesto!
• Code completed
• Code committed
• Code pushed
• Code merged to master branch
• Code reviewed

There are many issues that may arise during sprint planning. Teams may continuously miss their deadlines and push features back into the product backlog for later resolving. This may be due to poor time estimation. A given feature may be continuously pushed to a later sprint. This may be due to some impeding factor that has not been discussed. There may be a lack of knowledge in the team.

Estimation
During the initial meetings the PO needs to decide about priorities. He must decide whether or not a feature is worthwhile. In order to do this a loose estimate of time needed to complete a feature is required. As the details of the feature is not yet know this estimate can vary from reality by a large amount. The longer a team is in place the more accurate the estimation becomes. The estimate of the backlog is provided in points. Points are not a unit of time. Team members of different capabilities will take different amounts of time to complete the same task. This is why measurement in points is important. There are many techniques to evaluate the features in the product backlog.

Relative Mass Valuation
This is useful when going through a large group of stories as it enables the team to evaluate tasks relative to each other. A card is written up for each story. Stories should be placed on a table. As each card is picked up it should be evaluated relative to the previous cards. Sizes are given as large, medium and small. After evaluating all cards point values are assigned based on the positions the card have on the table (1-small, 2-medium and 3 – large).

Planning Poker
Using planning poker every team member must participate in the planning phase. Each team members is given a set of cards with numbers on them. They are 0,1,2,3,5,8,13 and 21. Each story is read aloud. Each team member is asked to show the card referring to the level of effort they believe the story requires. Once all ‘votes’ are in team members with values at either extreme must explain why they chose those numbers. They may have ascertained something that others had missed. Stories with estimates of 13 or greater are usually Epics and should be further broken down.

To make estimations efficient teams may be asked what the easiest item is in the backlog and the most difficult item in the backlog. In this way, whichever technique is selected for estimation, it is easier to define items relative to each other.

Agile, Lean & Kanban

Agile
Agile development is now the most common technique1 as it aids with communications. Agile facilitates change, refinements and reprioritizing in short cycles. This is particularly useful where the final product is continuously evolving and improving. Communications are prioritized where team members are equal and customer input is continuous throughout the delivery.

Some of the difficulties in Agile must also be considered. Delivery dates may move due to changes made during sprints. Small agile teams mean that all aspects of knowledge must be encapsulated within the small team. Whilst documentation is not prioritized it does not mean that documentation is not required. Documentation may be poorly carried out or may be neglected. Changes to business goals during the SDLC means that the final product may vary greatly in Agile from the original target.

There are a number of methodologies that can be used to implement Agile including:

Extreme Programming (XP)
XP focuses on improvement in quality and adaptability to user requirements. Simplicity and continuous change are features in XP.

Feature Driven Development (FDD)
FDD focuses on building features iteratively.

Lean Software Development (LSD)
LSD has seven key principles, namely: eliminate waste, amplify learning, late decision making, fast delivery, team empowerment, in-built integrity and viewing the whole system. Items in the process that do not add value are considered wasteful.

Kanban
Kanban is a visual way of implementing Agile. It includes continuous change and improvement. Focus is placed on ‘seeing’ the workflow, limiting work in progress and making policies explicit.

Scrum
Scrum defines roles and responsibilities, meetings and outputs. Sprints are development cycles that are usually two weeks lo

Kanban
Kanban is Japanese for ‘visual sign’ or ‘card’. It is a framework which shows what to create, how to create it and when to make it. It implements short cycles. Where scrum focuses on time constrained sprints, Kanban focuses on tasks completion. Kanban Boards may include magnets or sticky notes where items are moved from the To-Do list to Doing and finally to Done. Electronic versions are also available through Jira and other software. The number of swim lanes may vary from three (to-do, doing, done) to six (backlog, ready, coding, testing, approval/release and done). The colour of the card on which the work item is detailed may represent levels of work such as Features and Tasks. There are a maximum of items that will be allowed in each lane in the board at any time.

Kanban is a very fluid model as it does not have the time restrictions of Scrum Sprints. The Board makes it very easy for all team members to see the progress of the product development at any given time. Continuous delivery of Features and completion of tasks is referred to as just-in-time delivery where customers receive regular updates and releases. The cycle time is the length of time it takes the team to move items through the workflow.

As with Scrum there are a few issues to consider. As items are only shown in one of three phases (to do, in progress, done) it is difficult to gauge time to completion. In order to fix some of the disadvantages of the Kanban board some teams may add their own ideas which then focuses on the process rather than the product under development. If the board is not kept up to date it may be found that items under development may not have the current priority applied which could result in loss of focus on business goals.

Kanban boards are good for projects requiring continuous change but without the need for iterations. As Kanban does not require estimations and iterations releases to the customer can be made at any time. An alternative is to use Scrumban which has the advantages of defined roles, daily scrums, etc. and yet applies a Kanban board with continuous pull, continuous workflow and the ability to make changes as needed.

1http://techbeacon.com/survey-agile-new-norm

Lean
By using Lean Dropbox has grown from 100,000 registered users to over 4,000,000 in 15 months3. The main focus of the Lean philosophy is to eliminate waste. To reduce or eliminate any task that does not add value to the product under development. LSD has seven key principles, namely: eliminate waste, amplify learning, late decision making, fast delivery, team empowerment, in-built integrity and viewing/optimize the whole system

Within Agile there are a number of commonly held tasks including backlog grooming. As with scrum a backlog is created with stories. These stories should then be refined or groomed. Items with the highest priority should be added to the top of the backlog. Stories that are epics are refined into their appropriate stories. Interim releases are carried out every four sprints approximately so the grooming of epics usually happens once in that period.

There may be a drive towards merging Test Driven Development (TDD) with Acceptance Criteria. Acceptance criteria are generally written to capture possible scenarios, non functional requirements etc. Acceptance tests usually start with the phrase ‘Verify That’. Unit tests are then written based on acceptance criteria. Acceptance Test Driven Development is referred to as ACDD.