Using Objectives and Key Results (OKRs) and Slack: A Case Study of Coordination in Large-Scale Distributed Agile

Today, many large-scale software projects have members working from home, which has changed the way teams coordinate work. To better understand coordination in this setting, we conducted a case study through which we examined two teams in a large-scale agile project by observing meetings and conducting 17 interviews. Through the lens of Relational Coordination Theory (RCT), we analyzed the use of the goal-setting framework Objectives and Key Results (OKRs) and the collaboration tool Slack. Slack was used for frequent, timely, and problem-solving communication and, and its use decreased the number of planned meetings. However, discussions often started on Slack and continued in virtual ad-hoc meetings. The use of OKRs facilitated knowledge sharing, helped the teams align their goals, and provided inter-team insights. The main implication of our research is that projects using OKRs need to support project members, especially in formulating the key results that align and motivate the teams to work toward the same mission.


Introduction
There is an urgent need for more studies on how to coordinate work in agile projects for two main reasons. First, many projects are being scaled, which results in projects with many people, teams, and dependencies [1,2]. Second, agile projects are becoming more and more distributed, primarily because of the COVID-19 pandemic, during which project members have been required to work from home. The pandemic has changed the way we coordinate work [3], and several companies have implemented work-from-anywhere policies.
In large-scale agile projects, teams need to be aligned with other teams and the rest of the organization, which is challenging [4,2]. To improve understanding of how to coordinate effectively and achieve success in large-scale projects in a distributed setting, researchers need to consider theories from other disciplines, such as the Relational Coordination Theory (RCT) [5]. In her work on the airline and health industries, Gittell observed that in cross-functional work processes (across departments and hierarchies), the companies that performed best had higher levels of relational coordination between roles [5]. To ensure effective coordination, project members should be connected through shared goals, knowledge, and mutual respect [5,6]. Rather than trying to control behavior with rules and programs in such an uncertain environment, it is better to regulate the output by setting appropriate goals and targets.
Shared goals are crucial for agile teams, but goals are difficult to implement in large-scale agile settings. Challenges include goals that are not known to the team, goals set by the management without involving the teams, and the use of deliverables and deadlines as goals [7]. As large-scale agile projects struggle with setting and communicating goals as well as establishing a shared direction, many companies now try to implement goal-setting frameworks such as the Objectives and Key Results (OKRs) framework [8,9].
New methods and frameworks for goal setting like OKRs do not alleviate all coordination challenges in large-scale agile projects. Because teams need to manage dependencies with other teams, experts, managers, and stakeholders [10], they also need access to software tools for coordination and communication [11]. Examples of such software include Slack, Discord, Zoom, Whereby, and Microsoft Teams. During the pandemic, such tools have become a requisite for coordination and act as enablers for change in agile work practices and distributed work. This is in line with previous research that agile methods need to be adapted when the context in which the method is applied changes [12,13]. For example, stand-up meetings have been extended to include a focus on members' well-being [14] in the work-from-home context. Further, agile practices such as pair programming [15,16], if done "by the book," require physical collocation, which is impossible when everyone works from home.
Motivated by the need for more research on how to coordinate work in large-scale distributed settings [17,18], and to understand how coordination is affected by the use of goal-setting frameworks and communication tools, we ask the following research question: "How does the use of OKRs and Slack affect coordination in distributed large-scale agile?" To answer this question, we relied on the lens of RCT [5] and studied two agile teams in a Norwegian agency with 50,000 employees who were working distributed because of the COVID-19 pandemic. Our work builds on a preliminary analysis of the data collected in one of the teams, presented in [19].

Background
Initially designed for small single-team projects, agile methods have shown benefits such as a high adaptability to change, reduced risk, and continuous customer involvement, making them attractive for larger projects and companies [20]. After analyzing studies with regard to the number of people and teams in large-scale projects, Dikert et al. [20] proposed a definition of large-scale agile being software development organizations with at least 50 people or a minimum of six teams.

Coordination in Large-Scale Agile
Agile methods aim to achieve early delivery of quality software by focusing on collaborative teamwork and collaboration with customers. Like other project teams, agile development teams collaborate effectively using mechanisms to coordinate their interdependent work.
Unfortunately, this is quite difficult, and coordination breakdowns are a major problem in distributed software development [21,22].
When projects or programs grow, there will be a growth in the number of dependencies, amount of uncertainty, and complexity [23] that requires a continuous emergence of coordination mechanisms [24]. In organization studies, coordination is defined differently across organization-, project-, and team level in software development [25]. As organizations grow and progress into large-scale, coordination across teams is equally important as within individual teams. Coordination across teams is referred to as inter-team coordination and is important in situations where particular tasks must be performed by several teams under certain constraints caused by inter-dependencies [26].
Considering the challenges that arise as a result of the number of teams and individuals working simultaneously, working with large-scale agile requires a substantial amount of coordination. Challenges include managing a large number of stakeholders and coordinating the teams while keeping an informal approach to the communication [20,2]. Dependencies among people, tasks, synchronization, and schedules need to be managed [23]. Accordingly, it is important to study coordination practices in large-scale distributed agile development. RCT is a theoretical framework well-suited for understanding coordination in complex, uncertain, fast-paced, and interdependent work settings [6]. The theory was developed more than 30 years ago from field research in the airline industry, and today has been used to explain coordination in 73 industry contexts across 36 countries [6], including research on shared information systems [27], and large-scale agile [24].

Relational Coordination Theory
Relational coordination is defined as "a mutually reinforcing process of interaction between communication and relationships carried out for the purpose of task integration" [28], p.300. According to RCT, coordination is characterized by mutually reinforcing dimensions [5] as illustrated in Figure 1. RCT includes the following concepts [5,6]: • Shared goals direct participants towards a unified outcome or delivery, rather than focusing only on their specific tasks and priorities.
• Shared knowledge about what is important to accomplish, how and when, and by whom, informs participants about how their own and others' tasks contribute to the overall goal.
• Mutual respect motivates participants to consider their actions in relation to the work of others, further reinforcing the regard for a shared overall outcome.
These three concepts, in turn, reinforce and are reinforced by high-quality communication (i.e., frequent, accurate, timely, problem-solving communication) [5]. Finally, cross-cutting organizational structures-that is, coordination mechanisms going beyond single teams or departments-are theorized to support relational coordination, and successful coordination is theorized to depend on the design of such organizational structures [5,6]. In the digital work environment, relational coordination can be supported by digitally mediated work processes and by shared information systems. A recent study on relational coordination in digital settings found that coordination mechanisms allowing participants to select whom to coordinate with, allows for customizing content and enabling shared interpretations of meaning, should be more efficient in digitally mediated coordination [27].
As key characteristics of distributed large-scale agile settings include high levels of complexity, uncertainty, and interdependence, as well as time pressure from the need for fast-paced delivery [23,20], we believe RCT is well-suited for understanding coordination in large-scale agile.
A recent review on RCT shows that several mechanisms commonly used in large-scale agile are related to successful coordination [6].
Cross-cutting organizational structures, such as boundary spanner roles, shared meetings, shared spaces, shared protocols and routines, as well as shared information systems, were highlighted as mechanisms empirically demonstrated to support relational coordination [6]. In our case, we chose to investigate the organizational structures OKRs and Slack, described next.

Objectives and Key Results
OKRs is a goal-setting framework used by organizations to define a certain set of objectives and measure progress toward those goals [8,9]. The framework originated at Intel and is used by large companies such as Google and LinkedIn as well as an increasing number of smaller companies in various industries [29]. OKRs is designed to help organizations achieve their business goals quickly and in a structured manner. It is described as "a critical thinking framework and ongoing discipline that seeks to ensure employees work together, focusing their efforts to make measurable contributions that drive the company forward" [9] (p. 6). Using OKRs is suggested to provide benefits such as focus and frequent establishment of priorities, and that introducing OKRs enables transparency on an inter-team level, which allows teams to cross-functionally align, provide feedback, and collaborate [9].
The framework consists of a set of objectives and corresponding key results (see Figure 2). Objectives are set for a certain period, and key results are defined to progress toward these objectives continually through tasks. An objective describes what the team wants to achieve. A well-described objective should be obtainable within a quarter and represent the shared imagination of the team [30]. Objectives are phrased as concise statements that outline broad, qualitative goals, pushing the organization in the desired direction [9]. One objective from the case we studied was, "Other teams can easily get an overview of where they can find the data they need from us." Key results, on the other hand, are quantitative statements. They allow the team to follow up on their progress and are designed to measure whether a given objective has been reached [30,29]. An example of a key result targeted to the objective above was, "We have documented three data sources in the data overview." Niven and Lamorte [9] recommend the implementation of scores for key results, describing to what extent the organization is likely to achieve them. The scores can communicate expectations and provide clarity and context around what progress looks like for a particular key result.

Slack
Although agile methods put an emphasis on face-to-face communication, the current situation requires new ways of working. The distribution of members creates new challenges-for example, delays in feedback when team members need help [31]. In addition, the lack of awareness of what distributed members are doing and if they are available for communication increases barriers for initiating contact [21]. Instant messaging tools help alleviate some of the challenges that distributed agile team members face [32]. Slack is such a tool that provides agile project members with the capability of responding quickly to each other in written text.
Launched in 2014, Slack now has more than 12 million daily users, and the tool is used mainly for instant messaging communication but also for video calls and file sharing [33]. A workspace in Slack can be broken into separate channels. These channels can be created for different projects, teams, topics, specific company announcements, or different types of support (e.g., IT, HR). These channels can be set as private or public. That is, teams can designate if they want to have private conversations (people must be invited in) or if they want anyone from the company to be able to join and engage in the discussions. Slack also enables employees to send private messages to each other (have private one-to-one conversations). Messages in Slack can include emojis, and people can edit and delete their messages after the messages have been posted.
Tools such as Slack has been found to increase team awareness and facilitate almost synchronous communication [34,11]. Lin et al. [35] conducted an exploratory study to understand how developers used Slack and found that they used it for purposes that were personal (networking and social activities), team-wide (team collaboration and communication) and community-wide (participating in communities of practice).
Stoeckli et al. [36] used the theory of affordances to investigate Slack chatbots and integrations and explored their constraints within enterprises. Finally, in their study of agile distributed software development, Lous et al. [37], found that the adoption of Slack eliminated the use of e-mails for internal communication.

Research Methods
To investigate how the use of OKRs and Slack affect coordination in distributed large-scale agile, we decided to conduct a case study. A case study is suitable for investigating contemporary phenomena, or cases, in-depth within their real-world context, especially when the boundaries between context and phenomenons are ambiguous or not apparent [38]. The case organization was established as a project in 2017 and later organized as an agency in early 2020. The case had 11 permanent and three temporary teams and was structured with people from multiple departments. We chose two agile teams, Alpha and Bravo. Team Alpha worked iteratively in one-week sprints similar to Scrum. Team Bravo did not use sprints but a more flexible approach following the principles of Kanban. Both teams utilized the OKR framework which influenced how the teams prioritized their work tasks. They also used the collaboration tool Slack for communication and coordination in their everyday work.
The first part of the data collection involved observation of virtual meetings such as daily stand-up meetings, retrospective meetings and sprint planning. The observations allowed us to get to know team members and served to provide insight and context used to develop the interview guides. An overview of the timeline of our data collection is illustrated in Figure 3.
The observations lasted from November 9 to December 7, 2020, and consisted of 21 meetings throughout the period. During this period, the team worked remotely because of the global situation of the pandemic. A structured approach to the observations was chosen, following an observation protocol. This protocol included the number of participants, time stamps, content, and abnormalities if any.
We conducted semi-structured interviews. The interview guide is provided in Appendix A. All members in both teams were interviewed (see Table 1). Team Alpha had 10 members (three females and seven males), and the average time they had been working on the team was 1 year and six months. Team Bravo had 7 members (three females and four males) and the average time on the team was one year and two months.
All the interviews were transcribed, and we used the qualitative data analysis software NVivo to analyze them. First, we noted overall impressions, and then we reduced, organized, and coded the data, using a reflective analysis [39]. Further, we found themes and patterns, and we also coded following the categories described in [5], such as "shared goals", and "timely communication."

Results
In both teams Alpha and Bravo, coordination took place through a number of scheduled team and inter-team meetings. Table 2 presents a detailed description of the team and project meetings the two teams participated in. Next, we describe the use of OKRs and Slack and how these organizational structures relate to the relational coordination concepts of shared goals, shared knowledge, mutual respect, and high-quality communication.

Use of OKRs
At roughly one-year intervals, teams across the large-scale project were trained in the use of OKRs. The goal of the training was to help teams to identify and describe high-quality objectives and key results. The training sessions were led by experienced certified OKR facilitators and functioned as the key events for the teams to increase their understanding of the OKR process and to support the teams in working with OKRs.
As such, the OKR training facilitated shared knowledge in how to use OKRs across the large-scale project.
Moreover, the teams worked to define, evaluate, and improve the OKRs in OKR workshops, which were held every quarter (see Table 2). The workshop increased shared knowledge within the team about what was important to work on. However, the two teams struggled with defining the objective and identifying appropriate key results, and as a result, the OKR workshop became something negative that was seen as a tedious event. The need to conduct the workshop virtually increased the difficulties. Members of both teams explained that meetings requiring discussions, such as an OKR workshop, had been negatively affected by the change to remote work. Virtual meetings made it difficult to read body language, and network issues could often lead to participants talking simultaneously and canceling each other's voices out. Also, some people were less engaged in the discussions. One back-end developer stated, "We had one virtual OKR workshop where some people didn't write a single post-it note. That would not have happened in a physical workshop." Objectives in OKR are qualitative by nature and can, as a result, be vague. An example of an objective set by team Alpha was, "Our service is usable, robust, and easy to maintain." Two of the key results they chose for this objective were, "The time from when the code is merged to when it is in production is less than 15 minutes," and "We have automated seven processes." The teams progressed on key results continually throughout the quarter as they completed tasks related to the key results. However, not seeing progress on the key results was demotivating. A member from Alpha elaborated on this issue, "One of our biggest challenges with OKRs is that typically we cannot measure and progress a key result in a single week. But suddenly after three weeks, when tasks are finished, the key result is updated from 0 to 100%." Nevertheless, as it does with many new techniques, the situation improved over time. Both teams stated that they were continually improving, and after a year, they had made good progress in applying the framework. The team lead in Alpha stated, "In the beginning, we tried to do everything by the book, but then we found our own modified way. We found it was easier to agree on objectives and key results when people, in advance of the meeting, had to think about what we wanted to achieve in the next weeks, and then we had a democratic voting. Also, we went through the key results each week, which helped us stay focused." Further, a UX designer stated, "I think OKR is a good way to prioritize workload and manage our vision. Also, for me as a designer, it is a nice tool to help emphasize that we need insight, we Even though working on OKRs forced the teams to discuss their goals and to agree on how to measure them, it was not an easy task. One of the hardest parts about using OKRs, as stated by several team members from both teams, was quantifying the objectives through key results and the corresponding formulation of the specific key results. A team member from team Alpha stated, "OKRs are useful for knowing what to work on and aligning the team, but it is hard to create good, measurable key results which make sense." The teams used an OKR tracker, a digital dashboard available to all in the large-scale project, which provided transparency and inter-team insight. One team member explained, "I think the OKR tracker works very well. It's nice to see what the other teams are working on. One can easily measure progression. It's fun to update the tracker and see that we are at 60%; it's motivating. Further, I appreciate that it provides transparency across the project." While the OKR tracker provided the opportunity to increase shared goals and knowledge across teams, many team members did not care or have the time to be updated about the other teams' OKRs. One back-end developer in Alpha stated, "We have an OKR tracker, which is open for everyone in the project, so that we can see what goals other teams have set. And that's good. But, it's a bit up to each individual to actually bother to read it and look at what the others are working on." So while the tracker was meant to support alignment and transparency, in practice, most team members focused on their own team's goals and work.

Use of Slack
Team members in Team Alpha and Bravo conducted a substantial amount of communication digitally because they were distributed. During work hours, Slack was extensively utilized as a collaboration tool by both teams. Slack was used for ad-hoc communication, meetings, and social events.
Communication and coordination on Slack took place both within the individual teams and also in inter-team contexts. The project members discussed, for example, implementation of new technology, debugging, and when to do pair programming. Further, they used Slack to locate someone with a particular expertise or domain knowledge outside the team, and they also reached out to team members when they were stuck on a task. A developer in Bravo described how the threshold to ask for help had been lowered, "It is easier to ask for help on Slack than physically asking someone in the office. Additionally, we often ask in the channels rather than sending a direct message to one specific person. I think is better for the project that more people see the question and can help." Further, frequent communication on Slack and the lack of co-location enabled teams to reduce the amount of time spent in meetings. Moreover, Slack also offered a way to replace regular meetings. In contrast to the daily stand-up meeting on video utilized by Team Alpha, Team Bravo changed to a written stand-up every Tuesday to Friday. They had named it "Slackup," and it was essentially a regular stand-up but in a written format. As a result, team members in Bravo could share their status report when it was convenient for them. A UX-designer stated, "It works really well to just write our updates and have a low threshold to call each other on Slack. I feel the written stand-up meetings give me more time to do focused work, and I think it is much better to read what people are working on instead of hearing about it in a meeting." The team lead of team Bravo stated, "You can be much more productive during the day because there aren't as many meetings. The threshold to initiate a meeting is probably a little bit higher now." While Slack was appreciated as a tool for communication, it did not completely remove the need for meetings. One team member explained, "We have talked a lot about when to say stop in a Slack thread and say, 'Hey, this requires a meeting.' We have spent much time discussing these issues because it is difficult to find an exact rule. But, I think that especially the developers have a low threshold to call into dev meetings based on conversations that go on too long in Slack." While Slack was important for problem-solving communication, it could also be time-consuming and even overwhelming. The team lead in Bravo stated, "You can sit all day just chatting. There is so much information in all the different channels that you can end up doing nothing but just reading updates and questions and answering and communicating with people. So we spend a lot more time on digital communication than before." Several others also stated that the communication on Slack could be overwhelming. A developer in Bravo explained, "With Slack, there is a lot of context-switching. Also, if you are gone for half an hour, then it can be like 50 new comments. You don't have the time to start reading all those and you might miss out on important information." Further, the responses to questions were expected to be timely. A member of team Bravo explained, "You can ignore Slack for two hours if you want, but the expectations for a quick response is higher compared to, for example, a comment on the Kanban board." Additionally, the organic nature of conversations on Slack also had some consequences for who participated in the discussions. A team member stated, "Sometimes people, unintentionally, get excluded from discussions. If there had been a formal meeting, then they would probably have received an invitation." Another challenge mentioned was where to document decisions made in Slack when a discussion was finalized. However, the benefits outperformed the reported challenges. As one developer in Bravo stated, "We use different tools, but it is Slack that keeps us together."

Discussion
We conducted a case study to understand how organizational structures such as OKRs and Slack affect coordination in large-scale agile projects. To understand coordination in the case, we relied on RCT [5]. This theory suggests that coordination effectiveness is improved by tools and guidelines that support shared goals, shared knowledge, and mutual respect, as well as frequent, timely, and problem-solving communication between project members [6]. We now discuss our results in light of our research question: "How does the use of OKRs and Slack affecting the coordination in distributed large-scale agile?" We found that OKRs supported the teams in identifying goals through a structured process in which they agreed on objectives. Each objective had associated key results that made it possible to measure progress toward the objective. As these key results were visited frequently in team meetings, we argue that OKRs helped the teams to focus on overall outcomes and not only on specific tasks. In RCT [5], shared goals are believed to direct participants to focus on a unified outcome or delivery rather than on their specific tasks and priorities.
The results show that OKR was used to prioritize work in the teams and to get an overview of what was most important. Therefore, we argue that OKR strengthened shared knowledge. Shared knowledge about what is important to accomplish, how, when, and by whom, informs participants about how their own and others' tasks contribute to the overall goal [5].
Even so, while most project members received OKR training, many were not convinced that using OKRs substantially improved their work. Having to conduct the regular OKR workshops digitally further obscured the potential advantages of the framework. Interviewees reported that they found it especially challenging to formulate the key results. This is in line with other research that highlights the lack of practical advice on how to turn a qualitative objective into measurable key results [8]. The OKR tracker supported coordination across teams by providing transparency and alignment of goals between teams in the large-scale project.
As a result of being fully distributed, all scheduled and unscheduled coordination was performed virtually. We found that communication was frequent, accurate, and timely in both teams, and Slack was the key communication platform used. Further, reaching out on Slack was members' first choice to locate expertise and start problem-solving communication, both within teams and across the large-scale project. The threshold to ask for help was lowered, and more people could engage because they could see the questions being asked and issues discussed. In this way, Slack supports high-quality communication [5].
On the negative side, the members of team Bravo stated that the ad-hoc communication on Slack sometimes led to disorganized communication. This is in accordance with previous research demonstrating that coordination challenges related to communication are more evident in distributed teams compared to co-located teams [17]. Team members from both teams stated that it was difficult to find the right balance between chatting and initiating meetings for discussions, where chats and threads often ended up too long. This is consistent with the findings of a recent study [11], which showed that a lack of guidelines for using Slack resulted in coordination being confusing and frustrating for some team members.
Dikert et al. [20] found that choosing and customizing the agile approach was the number one contributor to success in large-scale agile transformations. In our large-scale set-up, the teams adapted their ways of working. Team Alpha and Bravo both continually evaluated how to improve and experiment. For example, Team Bravo replaced daily stand-up meetings with written Slackups and Jira with Trello. Team Alpha restructured their planning, and replaced Google slides with Miro. Also, the teams experimented with and modified the use of OKRs to fit their contexts, which made it easier to work with the framework over time.

Implications for Practice
We found that team members perceived setting the OKRs as difficult. This suggests that OKR training is vital, not only for team leaders, but for all team members to obtain shared knowledge. Further, OKR is an approach that takes time to adopt, therefore there is a need for teams to be supported over an extended period of time. In our case it took over a year before the teams started to see the benefits of the new practice.
Moreover, as many teams currently work distributed, all communication is performed digitally. The findings of this study show that transitioning from co-location to working remotely could quickly result in increased ad-hoc communication, disorganized chats, and lack of structure when using Slack. With proper routines, chat logs would be better structured and information easier to locate (such as the use of threads). Furthermore, the expectation of answering within a reasonable time frame would often urge team members to check Slack. By using channels and tags for cases of different priority and relevance, team members could reduce their context switching and potentially only be disrupted by inquiries that need urgent attention. Such routines could also encourage more unscheduled meetings instead of long threads of discussions. This yields increased focus and efficient decision-making, which could positively influence coordination in teams [11].
As observed in team Bravo, replacing the daily stand-up meeting with a written Slackup could help distributed teams in reducing disruptions further. It is relatively easy to implement and could assist teams in focus on complex work over longer periods of time. This could be especially valuable in large teams where it could be difficult to keep the meeting relevant for all.

Limitations
The data collected and the results presented in this case study were confined to two teams and their contexts. Considering "coordination mechanisms are dynamic social practices that are under continuous construction" [40], the results could be different if the data were collected at a different point in time.
Regarding this, it could have been interesting to observe the teams over a more extended period of time. A different case with a different context could also have provided different results. The observations and interviews were virtual, potentially excluding important body language and other valuable interactions in meetings. Considering that the teams were working distributed, this limitation would be hard to avoid.

Conclusions and Future Work
To be able to maintain productivity across teams in large-scale agile distributed projects, coordination is essential. By understanding how to affect coordination positively, organizations can introduce and adjust the practices which best fit their coordination needs. For example, they can introduce goal-setting frameworks and collaboration tools.
This study investigated how coordination was affected when distributed team members used the goal-setting framework OKRs and the collaboration tool Slack. While the implementation of OKRs was challenging, our results suggest that using the framework supports teams by allowing them to identify shared goals and frequently revisit these goals. The use of OKRs also strengthened shared knowledge, that, together with shared goals, is important for achieving high-quality communication.
Further, we found that the project members used Slack for sharing domain knowledge, pair programming, debugging, discussing problems, and for anything work related that required a person to locate knowledge outside of regular meetings. Therefore, Slack provides high-quality communication that enables coordination. However, team members from both teams stated that it was difficult to find the right balance between chatting and initiating meetings for discussions; some conversations became too long, and some interrupted focused work. Therefore, future work should investigate how to find the right balance for timely response in Slack versus working uninterrupted.
In this project, the teams could see the OKRs of other teams in an OKR tracker, but not everyone used this opportunity. Future work should therefore investigate how to increase the alignment of shared goals across teams in large-scale agile projects.