THE IMPACT OF USER EXPERIENCE WORK ON CLOUD SOFTWARE DEVELOPMENT

Kuusinen’s paper describes a single case study following a software development team, where the product owner and the user experience specialist are working closely together in a distributed setting. The case that the paper describes is interesting, however there is one major problem; lack of earlier research and theory to which the findings of this case would be reflected. Hence, this problem leaves this piece of research isolated and without any notable contribution.
In the following more detailed comments are given by chapters:
Introduction
This chapter lays quite nice justifications to why the topic is worth studying for in the Cloud context. However, the purpose and the aimed research contribution of this specific research could be stated more clearly. In this chapter references are used, but the referencing technique could be checked (e.g. there are sentences with same reference one after another and sometimes it is not so clear for what is the part of the sentence for which the reference is actually referring). Should the numbering of the references be ascending in relation to their order? I would also want to hear little bit more what made this case and the used software development model so special? What is the more traditional approach stated, but not explained? Also I would definitely want to see the term “user experience” defined somehow by the author (what is meant by it in this paper), as there are so many ways to interpret this concept. Reference to Agile (ref.3) should be more scientific as Agile is the central basics of this study. Literature includes lots of more applicable references. Referring to Agile manifesto leads one to consider the writer of the paper as an amateur, not a specialist.
Also: If there is no separate chapter describing the earlier body of literature relating to the topic, this should be done in the introduction.
Methods
In this chapter the data collection methods are described in a somewhat confusing manner. The chapter does not explain what is the overall methodology used in the study. Related to the data collection: Why interviews were only carried out in the beginning of the follow-up period? Does the follow-up period stand here for the actual case study or some period after collecting the data?
Project
This chapter is good in giving the context for the case. I would move this in the beginning of the method chapter.
Development model
The development model chapter continues giving the background information for the case under study. Could it be also included somehow with the project and methods chapters? As a separate chapter it might mislead the reader, thinking that these are already somehow findings presented here.
Agile roles (product owner, scrum master etc.) are described very thoroughly. However, they are basic information of agile methods and could be just referred to a proper source. Agile methods have been in use already about twenty years and are very well-known.
The chapter does not clarify, what was the author’s impact on the development model/process or was there any (this relates partly also to the lack of overall methodology description). Was the development model something that the project team invented/has developed by themselves or did the author have some impact on how (process/model) the development was done?
Second paragraph: “They represent the end user…” Who represents? The product owner? Why don’t the users in the customer’s site represent the end users?
Are the process phases (1. Go,…, 5. Live) somehow present at Figure 1? It is not clear whether these phases are something that the project team has defined or something that the author has defined based the collected data.
Survey findings
Chapter name could be maybe changed a bit as you are describing results utilizing both the quantitative (survey) and qualitative (interviews) data. There are very interesting findings presented (e.g. communication challenges, cultural differences, divergence in expectations and preferences related to UX work), however as there are not proper discussion (separate chapter) and reflection of the finding to previous research, the analysis seems quite shallow.
The study lasted 11 weeks, hence it would have been interesting to see how the collaboration between the UX specialist and other project members varied during this time.
Project success factors
Can you elaborate a bit how “user recommendation” was actually measured?
Somehow the success factor presented seem very subjective (from interviewees point of view). E.g. sentence “Both the product owner and UX specialist were convinced that the close co-operation from early on is a key success factor in the project…” -> How does the collected (objective?) data support this statement? What is the research’s contribution if findings merely report interviewees’ opinions?
The author talks about lean applications. I would like to learn more about them. Is the term launched just in this paper or are there any references?
Research limitations
Thorough description about the limitations. Change the sentence “We believe that approach we used gave detailed enough information…”, it does not sound very convincing.
Summary and conclusions
Move last paragraph to the beginning of the chapter. Can the described next steps be modified so that they sound more as future research challenges spurred from this study and not a “project plan”?

Editor decision

The Reviewer has given good advice on how to improve the paper.
We are looking forward for a revised version of the paper.

I want to thank the reviewer for very good comments. They helped me to improve the paper greatly.

The most significant changes in the paper are that I partially reanalyzed and rewrote the results and included a more proper conclusions section. The reported research period is now 16 weeks instead of the previous 11 weeks.

I explain the changes made based on the reviewer comments below. Reviewer comment is started with "R:" and my comment with "K:"


R: there is one major problem; lack of earlier research and theory to which the findings of this case would be reflected.

K: I added an earlier research section and reflected my findings in a discussion section.

*Introduction*

R: the purpose and the aimed research contribution of this specific research could be stated more clearly.

K: It is now stared clearlier.

R: the referencing technique could be checked (e.g. there are sentences with same reference one after another and sometimes it is not so clear for what is the part of the sentence for which the reference is actually referring). Should the numbering of the references be ascending in relation to their order?

K: Other papers in the journal seem to use alphabetical ordering of references. I am doing the same.

R: what made this case and the used software development model so special?

K: This case can be considered as a typical cloud software development project in many ways. In addition, it is a good example of successful approach that has led to major improvement compared to the earlier way of working.

R: What is the more traditional approach stated, but not explained?

K: The earlier approach is now shortly introduced.

R: Also I would definitely want to see the term “user experience” defined somehow by the author (what is meant by it in this paper)

K: The term is now defined in the beginning of the chapter "Background and Related Research"

R: Reference to Agile (ref.3) should be more scientific

K: I changed the reference.

R: Also: If there is no separate chapter describing the earlier body of literature relating to the topic, this should be done in the introduction.

K: I added a separate chapter describing the earlier body of literature.

*Methods*

R: The chapter does not explain what is the overall methodology used in the study.

K: Overall methodology is now explained in the beginning of method chapter.

R: Related to the data collection: Why interviews were only carried out in the beginning of the follow-up period?

K: The aim of the interviews was to gather information about the way of working in the project. We were not studying change over the research period. However, a validation session with the PO and UX specialist was arranged after the study period. This is now explained better in the paper.

R: Does the follow-up period stand here for the actual case study or some period after collecting the data?

K: It stands for the actual case study. The sentence is improved in the paper.

*Project*

R: This chapter is good in giving the context for the case. I would move this in the beginning of the method chapter.
R: The development model chapter continues giving the background information for the case under study. Could it be also included somehow with the project and methods chapters?

K: I moved these chapters as subchapters of the results chapter. The information was gathered from the PO, the interviews and process documentation during the research and it is not part of method (understood as the procedure to conduct the research) as such. As the development model chapter also included citations from the interviews (findings), I thought the best place for such data would be in the beginning of results chapter. I begin the results chapter with a short introductory text to describe the content of the chapter and the means to collect the data.

*Development model*

K: This is now a subchapter of results as well. As the background information included citations from the interviews, I thought it would be better to include it in results.

R: Agile roles (product owner, scrum master etc.) are described very thoroughly. However, they are basic information of agile methods and could be just referred to a proper source.

K: I removed parts of the description and referred to Scrum roles. I also revised the two first paragraphs to make them more compact.

R: what was the author’s impact on the development model/process or was there any (this relates partly also to the lack of overall methodology description). Was the development model something that the project team invented/has developed by themselves or did the author have some impact on how (process/model) the development was done?

K: the development model and process were invented by the project team themselves. This is now described better in the paper.

R: Second paragraph: “They represent the end user…” Who represents? The product owner? Why don’t the users in the customer’s site represent the end users?

K: The text has been removed.

R: Are the process phases (1. Go,…, 5. Live) somehow present at Figure 1? It is not clear whether these phases are something that the project team has defined or something that the author has defined based the collected data.

K: The process phases are now linked to Figure 1 in the text.

*Survey findings*

R: Chapter name could be maybe changed a bit

K: I changed the chapter name to 'Results'.

R: there are not proper discussion (separate chapter) and reflection of the finding to previous research,

K: A separate discussion chapter is now included. Findings are now reflected to some point.

R: The study lasted 11 weeks, hence it would have been interesting to see how the collaboration between the UX specialist and other project members varied during this time.

K: Variations over time were analyzed. As the project was iterative and the same practices basically repeated weekly, there was nothing interesting to report. I added a sentence about this in the results section.

*Project success factors*

R: Can you elaborate a bit how “user recommendation” was actually measured?

K: Users were asked after each iteration how likely they would recommend the application to their colleagues. This is now explained better in the paper.

R: Somehow the success factor presented seem very subjective (from interviewees point of view). E.g. sentence “Both the product owner and UX specialist were convinced that the close co-operation from early on is a key success factor in the project…” -> How does the collected (objective?) data support this statement? What is the research’s contribution if findings merely report interviewees’ opinions?

K: Research contribution in qualitative approach often is to dig and synthesize interviewees' opinions. The current version of the paper includes also a correlational analysis of survey findings. However, the survey findings also are based on participants' opinions since they were asked about their experiences (questions like "how implementable is the design" basically measure respondent opinion/ subjective experience).

R: The author talks about lean applications. I would like to learn more about them. Is the term launched just in this paper or are there any references?

K: By lean applications I referred to applications limited in scope and size; applications where only truly needed (in terms of user and business needs) features are development. I changed the wording of the sentence.

*Research limitations*

R: Change the sentence “We believe that approach we used gave detailed enough information…”, it does not sound very convincing.

K: I changed the sentence to sound more convincing. I also made some changes to the end of the paragraph adding information about the reliability of the research.

*Summary and conclusions*

R: Move last paragraph to the beginning of the chapter.

K: Moved.

R: Can the described next steps be modified so that they sound more as future research challenges spurred from this study and not a “project plan”?

K: I removed the "project plan" part from the summary.

The paper has improved notably from the initial submition. All comments have been taken into sufficient consideration.

I uploaded a new version. Correlations are recalculated and the paper is proofread.

Editor Decision

Congratulations!
Based on the reviewer comments your manuscript has reached high level of maturity. It is now accepted for publication in the forthcoming issue of the Communications of Cloud Software journal.

I uploaded a new version as requested (with larger images etc)