4Th key: Tol Dagor: The Fourth Key – Quest

Use Four Keys metrics like change failure rate to measure your DevOps performance

Editor’s note: A lot has changed since this post was originally published in 2020. In 2021, the DORA team added a fifth metric — reliability — to the list of things that can impact organizational performance. And for the 2022 State of DevOps Report, cluster analysis only detected three clusters: High, Medium, and Low. That means no more ‘Elite’ performers! Regardless, the Four Keys remains a valuable tool to help you assess your team’s DevOps performance, and we hope you’ll tell us about how DevOps has helped your organization, by applying to the 2022 DevOps Awards.

Through six years of research, the DevOps Research and Assessment (DORA) team has identified four key metrics that indicate the performance of a software development team: 

• Deployment Frequency—How often an organization successfully releases to production

• Lead Time for Changes—The amount of time it takes a commit to get into production

• Change Failure Rate—The percentage of deployments causing a failure in production

• Time to Restore Service—How long it takes an organization to recover from a failure in production

At a high level, Deployment Frequency and Lead Time for Changes measure velocity, while Change Failure Rate and Time to Restore Service measure stability. And by measuring these values, and continuously iterating to improve on them, a team can achieve significantly better business outcomes. DORA, for example, uses these metrics to identify Elite, High, Medium and Low performing teams, and finds that Elite teams are twice as likely to meet or exceed their organizational performance goals.¹

Baselining your organization’s performance on these metrics is a great way to improve the efficiency and effectiveness of your own operations. But how do you get started? The journey starts with gathering data. To help you generate these metrics for your team, we created the Four Keys open source project, which automatically sets up a data ingestion pipeline from your Github or Gitlab repos through Google Cloud services and into Google DataStudio. It then aggregates your data and compiles it into a dashboard with these key metrics, which you can use to track your progress over time. 

To use the Four Keys project, we’ve included a setup script in the repo to make it easy to collect data from the default sources and view your DORA metrics. For anyone interested in contributing to the project or customizing it to their own team’s use cases, we’ve outlined the three key components below: the pipeline, the metrics, and the dashboard. 

The Four Keys pipeline

The Four Keys pipeline is the ETL pipeline which collects your DevOps data and transforms it into DORA metrics.

One of the challenges of gathering these DORA metrics, however, is that, for any one team (let alone all the teams in an organization), deployment, change, and incident data are usually in different disparate systems. How do we develop an open-source tool that can capture data from these different sources—as well as from sources that you may want to use in the future? 

With Four Keys, our solution was to create a generalized pipeline that can be extended to process inputs from a wide variety of sources. Any tool or system that can output an HTTP request can be integrated into the Four Keys pipeline, which receives events via webhooks and ingests them into BigQuery.

Click to enlarge

In the Four Keys pipeline, known data sources are parsed properly into changes, incidents and deployments. For example, GitHub commits are picked up by the changes script, Cloud Build deployments fall under deployments, and GitHub issues with an ‘incident’ label are categorized as incidents. If a new data source is added and the existing queries do not categorize it properly, the developer can recategorize it by editing the SQL script.

Data extraction and transformation

Once the raw data is in the data warehouse, there are two challenges: extraction and transformation. To optimize for business flexibility, both of these processes are handled with SQL. Four Keys uses BigQuery scheduled queries to create the downstream tables from the raw events table.

Four Keys categorizes events into Changes, Deployments, and Incidents using `WHERE` statements, and normalizes and transforms the data with the `SELECT` statement. The precise definition of a change, deployment, or incident depends on a team’s business requirements, making it all the more important to have a flexible way to include or exclude additional events.

While the definition may different from team to team, the scripts do provide defaults to get you started. As an example, here’s the Deployments script:

Loading…

SELECT
…
CASE WHEN source = “cloud_build” then JSON_EXTRACT_SCALAR(metadata, ‘$.substitutions.COMMIT_SHA’)
WHEN source like “github%” then JSON_EXTRACT_SCALAR(metadata, ‘$.deployment.sha’)
WHEN source like “gitlab%” then JSON_EXTRACT_SCALAR(metadata, ‘$.commit.id’) end as main_commit
FROM four_keys.events_raw
WHERE ((source = “cloud_build”
AND JSON_EXTRACT_SCALAR(metadata, ‘$.status’) = “SUCCESS”)
OR (source LIKE “github%” and event_type = “deployment”)
OR (source LIKE “gitlab%” and event_type = “pipeline” and JSON_EXTRACT_SCALAR(metadata, ‘$.object_attributes.status’) = “success”)

Four Keys uses the WHERE filter to only pull relevant rows from the events_raw table, and the SELECT statement to map the corresponding fields in the JSON to the commit id. One of the benefits of doing data transformations in BigQuery is that you don’t need to re-run the pipeline to edit or recategorize the data. The JSON_EXTRACT_SCALAR function allows you to parse and manipulate the JSON data in the SQL itself. BigQuery even allows you to write custom javascript functions in SQL!

Calculating the metrics

This section discusses how to translate the DORA metrics to systems-level calculations. The original research done by the DORA team surveyed real people rather than gathering systems data and bucketed metric into a performance level, as follows:

Click to enlarge

However, it’s a lot easier to ask a person how frequently they deploy than it is to ask a computer! When asked if they deploy daily, weekly, monthly, etc., a DevOps manager usually has a gut feeling which bucket their organization falls into. However, when you demand the same information from a computer, you have to be very explicit about your definitions and make value judgments.  

Let’s look at some of the nuances in the metrics definitions and calculations.

Deployment frequency

`How often an organization successfully releases to production.`

Deployment Frequency is the easiest metric to collect, because it only needs one table.  However, the bucketing for frequency is also one of the trickier elements to calculate. It would be simple and straightforward to show daily deployment volume or to grab the average number of deployments per week, but the metric is deployment frequency, not volume.  

In the Four Keys scripts, Deployment Frequency falls into the Daily bucket when the median number of days per week with at least one successful deployment is equal to or greater than three. To put it more simply, to qualify for “deploy daily,” you must deploy on most working days. Similarly, if you deploy most weeks, it will be weekly, and then monthly and so forth.

Next you have to consider what constitutes a successful deployment to production. Do you  include deployments that are only to 5% traffic? 80%? Ultimately, this depends on your team’s individual business requirements. By default, the dashboard includes any successful deployment to any level of traffic, but this threshold can be adjusted by editing the SQL scripts in the project. 

Lead Time for Changes

`The amount of time it takes a commit to get into production`

Lead Time to Changes metric requires two important pieces of data: when the commit happened, and when the deployment happened. This means that for every deployment, you need to maintain a list of all the changes included in the deployment. This is easily done by using triggers with a SHA mapping back to the commits. With the list of changes in the deploy table, you can join back to the changes table to get the timestamps, and then calculate the median lead time. 

Change Failure Rate

`The percentage of deployments causing a failure in production`

The Change Failure Rate depends on two things: how many deployments were attempted, and how many resulted in failures in production? To get this number, Four Keys needs the total count of deployments—easily acquired from the deployment table—and then links it to incidents. An incident may come from bugs or labels on github incidents, a form to spreadsheet pipeline, an issue management system, etc. The only requirement is that it contain the ID of the deployment so we can join the two tables together. 

Time to Restore Services

`How long it takes an organization to recover from a failure in production`

To measure the Time to Restore Services, you need to know when the incident was created and when it was resolved. You also need to know when the incident was created and when a deployment resolved said incident. Similar to the last metric, this data could come from any incident management system. 

The dashboard

Click to enlarge

With all the data now aggregated and processed in BigQuery, you can visualize it in the Four Keys dashboard. The Four Keys setup script uses a DataStudio connector, which allows you to connect your data to the Four Keys dashboard template. The dashboard is designed to give you high-level categorizations based on the DORA research for the four key metrics, and also to show you a running log of your recent performance. This allows developer teams to get a sense of a dip in performance early on so they can mitigate it. Alternately, if performance is low, teams will see early signs of progress before the buckets are updated. 

Ready to get started?

Please head over to the Four Keys project to try it out. The setup scripts will get you started setting up the architecture and integrating with your projects. We welcome feedback and contributions! 

To learn more about how to apply DevOps practices to improve your software delivery performance, visit cloud.google.com/devops. And be on the lookout for a follow-up post on gathering DORA metrics for applications that are hosted entirely in Google Cloud.

^{1. The 2019 Accelerate State of DevOps: Elite performance, productivity, and scaling}Posted in

• DevOps & SRE
• Google Cloud
• Open Source

633 4th Key Dr, Fort Lauderdale, FL 33304 | MLS #F10357496

$5,250,000

4 bd3,388 sqft

For sale

: $4,833,300

Loading

• • TypeSingle family residence
  • Year BuiltBuilt in 1969
  • HeatingCentral, electric, fireplace(s)
  • CoolingCentral air, electric
  • Parking2 Attached garage spaces
  • Lot0. 27 Acres
  • Price/sqft$1,550 price/sqft
  • Buyers Agency Fee2.5% 

  Overview

  Master bathroomGrand master suiteProtected dockageGourmet kitchenSpacious rooms

  “HOME IS WHERE THE YACHT IS!” Do not miss this incredible opportunity to purchase this impeccably appointed & furnished 4 Bedroom /3 Bath waterfront estate in sought-after Sunrise Key. Boasting 120′ on the water with no fixed bridges to the Atlantic Ocean and just one in from the point with protected dockage for your motor yacht or sailboat. This move-in turn-key formal & elegant residence is a “must see” which is perfect for the buyer that wants it all now or an investor that wants a long/short term rental situation. Spacious rooms, light, bright and sliding windows galore open to outdoor entertaining areas. Grand Master Suite with 2 Master Baths (Her Master bathroom is absolutely stunning!), gourmet kitchen & open living areas await you! A delight to show & ready for the lucky new owner.

  73 days

  on Zillow

  |

  937

  |

  6

  |

  Travel times

• Take a tour with a buyer’s agent

• Facts and features

  Interior details

  Bedrooms and bathrooms

  • Bedrooms: 4
  • Bathrooms: 3
  • Full bathrooms: 3
  • Main level bathrooms: 3
  • Main level bedrooms: 4

  Bedroom

  • Features: Entry Level, Master Bedroom Ground Level, Sitting Area – Master Bedroom

  PrimaryBathroom

  • Features: 2 Master Bathrooms, Dual Sinks, Separate Tub & Shower

  DiningRoom

  • Features: Florida/Dining Combination, Formal Dining, Snack Bar/Counter

  Flooring

  • Flooring: Marble, Tile

  Heating

  • Heating features: Central, Electric, Fireplace(s)

  Cooling

  • Cooling features: Central Air, Electric

  Appliances

  • Appliances included: Dishwasher, Disposal, Dryer, Gas Range, Microwave, Refrigerator, Washer
  • Laundry features: Utility Room/Laundry

  Interior Features

  • Door features: High Impact Doors
  • Window features: Drapes & Rods, High Impact Windows, Impact Glass, Storm Protection Impact Glass (Complete)
  • Interior features: Kitchen Island, Split Bedroom, Walk-In Closet(s), Wet Bar

  Other interior features

  • Total structure area: 3,617
  • Total interior livable area: 3,388 sqft
  • Fireplace: Yes
  • Virtual tour: View virtual tour

  Property details

  Parking

  • Total spaces: 2
  • Parking features: Circular Driveway, Driveway, Paver Block, Attached
  • Garage spaces: 2
  • Covered spaces: 2
  • Has uncovered spaces: Yes

  Property

  • Levels: One Story
  • Stories: 1
  • Entry location: First Floor Entry
  • Pool features: In Ground, Heated, Pool
  • Exterior features: Barbecue, Exterior Lighting, Outdoor Shower, Private Dock
  • Patio and porch details: Open Porch, Patio
  • Fencing: Fenced
  • View description: Canal, Pool, Water
  • Waterfront features: Canal Width 81-120 Feet, No Fixed Bridges, Ocean Access, Seawall, Unrestricted Salt Water Access, WF/Pool/Ocean Access
  • Frontage length: Water Frontage: 120
  • Has waterview: Yes
  • Waterview: Canal,Water

  Lot

  • Lot size: 0. 27 Acres
  • Lot size dimensions: 23 x 26 x 20 x 120 x 120 x 70 x 62
  • Lot features: 1/4 To Less Than 1/2 Acre Lot, Cul-De-Sac, East Of Us 1, Auto Sprinkler

  Other property information

  • Parcel number: 504201350410
  • Zoning: RS-4.4

  Construction details

  Type and style

  • Home type: SingleFamily
  • Property subType: Single Family Residence

  Material information

  • Construction materials: Concrete Block Construction, Cbs Construction
  • Roof: Curved/S-Tile Roof

  Condition

  • Property condition: As Is,Substantially Remodeled
  • New construction: No
  • Year built: 1969

  Utilities / Green Energy Details

  Utility

  • Sewer information: Public Sewer
  • Water information: Municipal Water
  • Utilities for property: Bottled Gas, Cable Available

  Community and Neighborhood Details

  Security

  • Security features: Owned Burglar Alarm

  Community

  • Community features: Paved Road, Security Patrol, Voluntary Hoa

  Location

  • Region: Fort Lauderdale
  • Subdivision: Sunrise Key

  HOA and financial details

  Other financial information

  • Buyer agency compensation: 2. 5%%

  Other

  Other facts

  • Listing Terms: Cash,Conventional

  Services availability

Street Key 4th with house numbers on the map of Vladivostok panorama, photo, satellite, location map, location

2
4
5A
8
9
9A
10
10A
eleven
12
12B
13
14D
14G
14
15
16
17
18
18A
1921
22
23
24
25
26
27B
29D
29
31
33
44
45
47
50V
50
52A
53
68
70
72
75
76
77
78
7980
81
83B
83A
83
85
87
87A
87B
89
91
92
93
95
96
97
98
99
101A
102
105
107
109
111
113
+

Show all numbers ↓

Clicking on a house number will display it on the map

” title=”Remove item”>

Noticed an error on the map?
Specify a point on the map, describe the problem and cartographers will add this data to the map

Report a map error

The nearest streets

1st Zarechny street

Klyu

Street Klyu

South Street

Yankovsky Street

Briner Street

Zarechnaya Street

Severnaya Street

non-profit garden partnership Chaika

Okhotskaya street

Mezhgorodnyaya street

garden non-profit partnership Primorsky gardener

3rd Shosseynaya street

Lilac street

4th Klyuch street on the satellite map of Vladivostok

With the help of the satellite map you will be able to see familiar objects from a different angle and discover details that are inaccessible with conventional observation on the ground.

For a detailed study of the area, go to the page
detailed map of Vladivostok or to the list of all streets of Vladivostok

Kemerovo region – Kuzbass, City of Kemerovo, Street 4th Warm Key

Russian address classifier

Online service

Search by address

Search by codes

Electronic signature for legal entities and individuals in 1 day
More than 60 areas of application.

FIAS object card:
KLADR code:
4200000

64100

FIAS code:
831638d2-b510-48b9-98b9-79196808c31f
GAR code:
675957

Administrative address: Kemerovo region – Kuzbass, City of Kemerovo, Street 4th Teply Klyuch
Abbreviated address: obl. Kemerovo region – Kuzbass, Kemerovo, st. 4th Warm Key

Municipal address: Kemerovo Region – Kuzbass, Kemerovo City District, City of Kemerovo, Street 4th Teply Klyuch
Abbreviated address: obl. Kemerovo region – Kuzbass, city of Kemerovo, Kemerovo, st. 4th Warm Key

Region: 42 Kemerovo Region – Kuzbass Region
City/Locality:
Kemerovo
g

Houses

d. 1
D 2
d. 3
d. 4
d. 5
d. 6
D 7
d.