ETSI BOARD Project “CR Tools for implementation automation and quality improvement”

Conference Call #1 Report

Actions resulting from the meeting

Where discussion is captured below, points attributed to speakers are from Erik’s notes; these are not a verbatim transcription of statements by participants on the call.

Agenda

• Status Update
• Discuss known requirements
• Discuss timelines and ‘community expectations’ for acceptance of the new tools
• Next Steps
• Update from ETSI Secretariat on status
• Closing discussion

Status Update

The project scope was reviewed for progress and also suggested updates shown in bold below):

Supporting Board members: Christian Toche, Christian Hoymann, Colin Wilson, Dao Tian, Francesc Boixadera, Juan Montojo, Heinz PolstererTBD

1. Identify the current status & report to the ETSI BOARD, GA and potentially to the PCG on the status of this work. (I am the 3GPP IT Task force convener)
2. Provide additional input and support to ETSI secretariat to actively support and advance this effort.
3. Report and assess on the prospects of a solution ready for use by the time 6G specifications will be developed in 3GPP.
4. Consider the work done in oneM2M on CR Tools.

Time Frame: 63 months

There were no comments on this, so the changes appear to be acceptable.

The progress so far, in the past 2 months of the origal 3, was to: * organize and hold off-line discussions with those previously working on the topic, including Matthew Baker and Ultan Mulligan. This [DONE] * gather and provide requirements and background documents for information to the board project group. [DONE]

Discussion:

None

Discuss known requirements

The main requirement in the background of the project is to ensure that 3GPP can move beyond the constraints of a process that is now 25 years old. Only this way can the organization continue to grow to greater scale of work and contributors and provide timely specification updates following plenaries of excellent quality. For this reason, we seek to increase the speed at which CRs an be implemented, and to detect clashes between CRs on demand, especially before implementation. Clashes themselves are not always errors, indeed if there are alternative CRs submitted, the clash is intentional. Still these must be eliminated (only one path forward taken) in order for CRs to be implemented. Finally, detection of errors in CRs will lead to improved quality and also to elimnination of conditions that would prevent a CR from being implemented at all, e.g. the wrong base version of the specification is used to create the CR.

The existence of tools is somehow independent of how they will be used. Can we focus on the tool and its requirements and beyond some discussion of supported scenarios leave the discussion of how the tool is used to others (in 3GPP) to consider and decide?

Existing requirements for CR implementation automation and quality improvements as a result of the 3GPP survey in 2022:

The requirements following the closure of the NWM project from 3GPP community stakeholders was:

Objectives of this project

The primary goal of this project is to introduce automation to improve the efficiency of CRs implementation, reducing delays, errors and need for manual intervention.

Key objectives include:

1. Conducting automated verifications of CRs, encompassing:

• File naming conventions
• Meta data aspects: consistency checks between CR cover sheet versus data extracted from database (e.g. title specification) and CR body (e.g. list of clauses affected as declared by the author vs the list of clauses actually updated in the CR body)
• Body aspects: respect of drafting rules, no changes to stock template styles, use of correct reference version, verification that no changes were made without revision marks

2. Automatically identifying clashes with other open CRs

• (i.e. which have been verified but not implemented into the specifications), detecting conflicting changes at the lowest level of granularity possible, e.g. incompatible modifications of the same element (e.g. piece of text or figure). Any ambiguity or issues should be be identified to allow iterative modification of CRs and the tool to create a correct output.

3. Automating the implementation of a new version of a specification from the previous version plus a set of verified CRs.

Constraints

• No per-user license fees
• No software installation required

The survey provided the following types of requirements

1. CR creation and editing aspects (wysiwyg, support of all elements in 21.801 including code, etc.)
2. Collaboration aspects (change tracking, difference checking, etc.)
3. CR implementation and checking aspects.

While all were in scope of the NWM project, only 3 is in scope of the current effort to develop a CR tool for 3GPP. We have tools for 1 and 2 and the problems stated (scaling up, implementation speed, CR quality) are not specifically with CR creation and editing or Collaboration aspects.

This project will focus on (3) above, and reviewing and evaluating tools to satisfy them, once these are available.

Discussion:

Erik

We add scope for looking at the oneM2M methods. We will evaluate this, in the sense of coming up with our view of how the tools could benefit 3GPP and satisfy the requirements we list. We won’t say ‘the requirements presume Microsoft Word and the oneM2M tool does not work with such files so it is out of scope.’ However, we must take the requirements into account.

Heinz

It may be possible to find improvements beyond automated implementation and quality improvements.

Christian H

It is not ruled out that we find something better than using Microsoft Word.

Erik

It is not ruled out, but it is also not part of the scope of this project.

Heinz

An alternative could be better for people in 3GPP. It could improve CR checking automation. It is possible to have WYSIWYG capabilities arise from generated Microsoft Word files, from Markdown documents. There are also AI tools that can be used to e.g. summarize documents. Word documents can be transformed into Markdown also. Possibly also 3GPP specifications.

Matthew

There is a common thread - WYSIWYG is important. Markdown is straightforward and worth looking at.

It was observed that some kinds of tool usage will rely on functionality that could be required. So considering different usage scenarios may help or even be necessary to provided adequate feedback to the ETSI secretariat.

Discuss timelines and ‘community expectations’ for acceptance of the new tools

The following timeline was presented (Erik’s opinion, not the official agreed dates for Rel-21!)

Note that the initial phase of specification development is in the form of pCRs, so the first CRs will appear a year later, at the earliest.

The tool aims at CRs not pCRs, so the initial phase will require extensive work by the rapporteur and EditHelp before producing the first version of the specification. Specification quality depends on the quality of the initial specification, as well as that of CRs to it. So we have to get the initial version right.

Discussion

It was asked ‘why can’t we support pCRs, too?’

Erik

2 reasons - first, those responding to the survey did not say they needed to. Second, a pCR has no header and is informal, so any support for the tool would move the pCR closer to the formality of the CR and this is not generally seen as desirable. That said, I fully encourage the tool support for pCRs (I was in the minority who responded this way on the survey, unfortunately.)

There was interest in trying to support this, even if not in the initial version of the tool.

Next Steps

Discussion

Heinz

For a demo and assessment by December (ETSI BOARD 150) we need to have the demo soon, in the next 4 weeks.

Erik

Let’s schedule this.

NEW ACTION: A240917-1 - ERIK: schedule and arrange the oneM2M demo and discussion. ALL: indicate your interest in participation to ERIK.

Erik

It is clear that there is interest in Markdown - I use it extensively.

NEW ACTION: A240917-2 - ERIK: gather material concerning Markdown and start to consider this as part of the board project.

Update from ETSI Secretariat on status

Ultan

Automation and quality control can be added without or with only minimal impact on 3GPP work practices. For further improvements, we can look at that, but for now we focus on CRs to specifications, to achieve the timeline. The focus is on:

• CR verification
• Clash detection
• Implementation

The goal is to be ‘ready for 6G’ and the constraint is to ‘use DOCX files as input.’

Ultan

Do we need a ‘strong encouragement to use the tool’ in order for the program to be a success? Or to mandate it?

Erik

This gets close to the question how to use the tools and how far to go in deployment: on (and before) upload, for work during the meeting, at WG agreement of a CR or for TSG approval? Shouldn’t we aim to support all these scenarios, and then leave it up to 3GPP leaders when and how these are mandated?

Ultan

What about existing specifications (5G) that will become 6G? do we aim for a gradual inclusion of these for tool support? Will this be easy?

Somehow usability considerations have implementation implications.

Erik

Should we initiate a dialog on expectations in 3GPP?
Should we contact our companies internally and get feedback?
Should we ask 3GPP leaders for their views?

Matthew

If we gather information, would it really be representative? One WG chair who is quite vocal would be overrepresented if we consult with 3GPP leaders. The survey was quite representative.

The best thing to do for getting feedback is get something working into people’s hands as soon as possible.

Bettina and Heinz agreed strongly with getting something working to evaluate as soon as possible.

Ultan

We need to discuss with a ‘reference group’ as we develop a functional specification. The Board project could serve as this.

Closing Discussion

Topic: coordination with other OPs

The ETSI project could help draft input from the ETSI delegation to make clear what is going on, and reports on progress including (positive) evaluation.

Ultan

for evaluation, one could even invite members of other OPs to join the board project…

Topic: On Markdown and improvements beyond ‘CR automation and quality control on MS Word Documents’

Ultan

Our focus on tools to be ready next year doesn’t stop us from looking at what comes next.

Heinz

We have been looking at AI tools to help with specification and CR review. It is hard to be a comprehensive expert. A tool can produce summaries. Working with PDF and DOCX is difficult compared to text.