Integrating Requirements into the Codebase: Pros, Cons, and a Practical Guide with Cypress

Preface

Managing the increasing number of end-to-end (E2E) tests in software projects can become overwhelming, especially as scenarios grow more complex and overlap in various steps. This complexity often results in a reliance on the individual tester’s experience rather than a cohesive strategy. Additionally, requirements, which form the foundation of testing, are frequently incomplete, misplaced, or inadequately described. Consequently, before effectively managing the test suite, teams often need to revisit and refine the requirements, leading to a cycle of inefficiencies and unclear outcomes.

To address these challenges, we propose an enhancement to the workflow: storing each atomic requirement alongside the test codebase during test implementation. This adjustment not only clarifies the relationship between requirements and tests but also streamlines several aspects of test management, reducing dependency on external documentation and improving accuracy in test coverage calculations.

Table of Contents

Purpose

The purpose of storing requirements within the repository is to simplify the management of test cases and test suites. This approach enhances the traceability of requirements, ensures accurate test coverage, and fosters a consistent and collaborative environment for the team. By centralizing requirements in the same repository as the test code, we can:

Pros and Cons

Pros of Integrating Requirements with Codebase

Cons of Integrating Requirements with Cypress

Structure

The structure of requirements should reflect the structure of the tests. For instance:

requirements/  
├── api/  
├── ui/  
│ ├── req-common.json  
│ ├── req-action.json  
│ ├── req-audit-type.json  
│ └── req-audit-round.json

Index Convention

Indexes should simplify the search of particular requirement. Each part of index should be defined in convention. In future it will allow to implement automatic verification of indeces. Here is an example of convention, and index template:

Creation

The indexing system should streamline the search for specific requirements. Each part of the index must adhere to a defined convention, facilitating future implementation of automatic index verification. Below is an example of the convention and an index template:

Example

Instructions for Creation

  1. Identify the requirement and determine if it is common or specific to a subcomponent.
  2. If common, add it to the req-common.json file with a unique ID and description.
  3. If specific, add it to the appropriate JSON file under the relevant section and sub-component.
  4. Use the index convention to assign a unique identifier to the requirement.

Updating

Instructions for Updating

  1. Locate the requirement in the appropriate JSON file.
  2. Update the description or details as needed.
  3. If the update affects common requirements, ensure all references are consistent with the changes.

Deleting

Instructions for Deleting

Delete only outdated or obsolete requirements. Requirements not yet covered by tests should NOT be deleted.

  1. Locate the requirement in the appropriate JSON file.
  2. Remove the requirement entry.
  3. If the requirement is common and referenced by other requirements, update those references accordingly.

Example JSON Structure

Here’s an example of how the JSON files might look:

req-common.json

{
  "COMMON-BUTTON-1-1": "All buttons must have a consistent style.",
  "COMMON-ERROR-1-1": "Error messages should be displayed in red."
}

req-action.json

{
  "UI-ACT-1-1": "Admin-specific action must be logged.",
  "UI-ACT-1-2": "Action buttons must adhere to COMMON-BUTTON-1-1.",
  "UI-ACT-1-3": "Action error messages must adhere to COMMON-ERROR-1-1."
}

Integrating Requirements with Cypress

To integrate requirements into the Cypress testing framework, follow these steps:

Load Requirements: Load requirements from JSON files into Cypress tests. Tag Tests: Tag your Cypress tests with the corresponding requirement IDs. Validate Requirements: Ensure each test validates the corresponding requirement. Analyze Test Results: Create a script to analyze test results and extract requirement IDs.

  1. Load Requirements Create a utility function to load requirements from JSON files.
// cypress/support/requirements.js  
const fs = require('fs');
const path = require('path');

function loadRequirements(filePath) {
    const fullPath = path.resolve(__dirname, filePath);
    const rawData = fs.readFileSync(fullPath);
    return JSON.parse(rawData);
}

module.exports = {
    loadRequirements
};
  1. Use the Helper Function Create a helper function to format test descriptions using requirement IDs.
//cypress/support/descriptionFormatter.js  
function formatDescription(requirementId, requirements) {
    return `should validate ${requirementId}: ${requirements[requirementId]}`;
}

module.exports = {
    formatDescription
};
  1. Tag Tests Tag your Cypress tests with the corresponding requirement IDs.
// cypress/integration/action_spec.js  
const {loadRequirements} = require('../support/requirements');
const {formatDescription} = require('../support/descriptionFormatter');
const actionRequirements = loadRequirements('../requirements/ui/req-action.json');

describe('Action Tests', () => {
    it(formatDescription('UI-ACT-1-1', actionRequirements), () => {
        // Test implementation  
        cy.get('button').should('have.class', 'consistent-style');
    });

    it(formatDescription('UI-ACT-1-2', actionRequirements), () => {
        // Test implementation  
        cy.get('.error-message').should('have.css', 'color', 'red');
    });
});

  1. Extract Requirement IDs from Test Descriptions

  2. Analyze Test Results Create a script to analyze the test results and extract the requirement IDs from the test descriptions.

// analyzeRequirements.js
const fs = require('fs');
const path = require('path');

function extractRequirementIds(testResults) {
    const requirementIdPattern = /UI-[A-Z-]+\d{1,3}\-\d{1,3}/g;
    const requirementIds = new Set();

    testResults.forEach(test => {
        const matches = test.description.match(requirementIdPattern);
        if (matches) {
            matches.forEach(id => requirementIds.add(id));
        }
    });

    return Array.from(requirementIds);
}

function analyzeRequirements(requirementsFilePath, testResults) {
    const requirements = JSON.parse(fs.readFileSync(requirementsFilePath, 'utf-8'));
    const requirementIds = Object.keys(requirements);
    const usedRequirementIds = extractRequirementIds(testResults);

    const usageCount = {};
    requirementIds.forEach(id => {
        usageCount[id] = usedRequirementIds.filter(usedId => usedId === id).length;
    });

    const coverage = requirementIds.filter(id => usageCount[id] > 0);
    const uncovered = requirementIds.filter(id => usageCount[id] === 0);
    const redundant = requirementIds.filter(id => usageCount[id] > 1);

    return {
        coverage,
        uncovered,
        redundant
    };
}

// Example test results (replace with actual test results)
const testResults = [
    {description: 'should validate UI-ACT-1-1: Admin-specific action must be logged.'},
    {description: 'should validate UI-ACT-1-2: Action buttons must adhere to COMMON-BUTTON-1-1.'},
    {description: 'should validate UI-ACT-1-1: Admin-specific action must be logged.'}
];

const requirementsFilePath = path.resolve(__dirname, '../requirements/ui/req-action.json');
const analysis = analyzeRequirements(requirementsFilePath, testResults);

console.log('Coverage:', analysis.coverage);
console.log('Uncovered:', analysis.uncovered);
console.log('Redundant:', analysis.redundant);