Frontend Multi-process Component Compilation Principle

Compilation

Compilation supports multiple modes, including single component compilation and multiple component compilation. Single component compilation is further divided into compiling one component at a time and compiling multiple components at a time.

The usage method is as follows:

// Single time
npm run build Test 
// Multiple at single time
npm run build Test,Http,xxx
// Package multiple components
npm run build

The implementation principle is as follows:

Concurrent Build + Queue Scheduling Mechanism Design

In order to package multiple components more efficiently, a "task queue + concurrent process pool" mechanism is introduced to replace the static task grouping method. This mechanism has better CPU utilization and build stability.

Design Goals

Goal	Description
Maximize CPU Utilization	Dynamically concur according to CPU cores, build always runs at saturation, no resource waste
Dynamic Task Scheduling	Each component acts as an independent task, distributed to idle child processes by the scheduler, avoiding uneven tasks
Smaller Task Granularity	Each component is built independently, scheduling is more flexible, and troubleshooting is more precise

Implementation Core Logic

1. Initialize Task Queue

   • Push all widgetName into the queue
   • Support debug mode or parameter filtering component names

2. Create Concurrent Worker Pool

   • Read os.availableParallelism() as the number of concurrent threads
   • Each worker thread takes a component name from the queue and executes the packaging command

3. Process Event Listening

   • After the child process packaging is completed, trigger the next queue task
   • If the task fails, report an error and exit

4. Build Completion Callback

   • When all tasks are completed, perform version tagging and compress products (such as wwwroot.zip)

Technical Points

• Use spawn to start child processes and inject BUILD_WIDGET_NAME environment variable
• Support platform compatibility: use npm.cmd under Windows
• Task distribution logic is implemented in pure JS, avoiding dependence on queue middleware
• Use ora to display progress in real time, and chalk to beautify terminal output

Example Flowchart

TaskQueue[WidgetA, WidgetB, WidgetC, ...]
        │
        ▼
 ┌──────────────┐
 │  Process #1  │◄────┐
 └──────────────┘     │
 ┌──────────────┐     │
 │  Process #2  │◄────┘ ← Dynamic Distribution
 └──────────────┘
      ...

---

Build Script Function Description (buildWidgets.js)

This script is used to batch build components under the src/widgets directory, supports single component and multiple component parallel packaging, and generates a compressed package wwwroot.zip for deployment.

---

Basic Configuration

• baseDir: Cache directory .cache/wwwroot
• baseBuildFile: Packaged component grouping configuration file
• hostZipPath: Final generated compressed package path

---

Packaging Process

1. Initialize the environment, clean up old .cache and wwwroot.zip
2. Parse command line arguments, support comma-separated multiple component names
3. Identify component names in the widgets directory
4. Divide components equally according to CPU concurrency, and package in parallel by group
5. Each packaging process executes npm run build-lib
6. After all components are packaged, clone the assets repository to the cache directory
7. Write the packaged products to the wwwroot structure and generate a zip package

---

Additional Functions

• Support debug mode to replace resource paths
• Use ora, chalk to beautify console output
• Automatically generate version tags and clean up temporary files The source code implementation is as follows

import tag from './tag.js'
import { removeSync, ensureFileSync } from 'fs-extra/esm'
import { spawn } from 'node:child_process'
import { globSync } from 'glob'
import path from 'path'
import fs from 'fs-extra/esm'
import { writeFileSync } from 'fs'
import slash from 'slash'
import crossSpawn from 'cross-spawn'
import { zip } from './ZipAFolder.js'
import os from 'os'
import chalk from 'chalk'
import cloneDeep from 'lodash/cloneDeep.js'
import ora from 'ora'

const assetsUrl =
  'https://gitlab.syc-cms.com:8443/lmes-plugin/web/assets.git'
const baseDir = './node_modules/.cache/wwwroot'
const baseBuildFile = './node_modules/.cache/widgets.json'
const hostPath = slash(path.resolve(process.cwd(), baseDir))
const hostZipPath = slash(path.resolve(process.cwd(), './wwwroot.zip'))
let isDebugMode = false
// Total compilation process count
let buildSumCount = 0
let startTime = Date.now()
let widgetNames = []
// Remaining uncompiled data
let remainWidgetNames = []
let isMultipleBuild = false
let spinner = ora('Compiling, please wait...\n')

removeHostPackage()
buildWidgets()

/**
 * Compile components
 */
function buildWidgets() {
  const isWin = process.platform === 'win32'
  const argv = process.argv || []
  // In debug mode, map local 206 environment image address to code, can set VITE_STATIC_URL in .env environment
  if (argv.includes('debug')) {
    isDebugMode = true
    argv.splice(argv.indexOf('debug'), 1)
  }
  // Support passing multiple component names for packaging, npm run build QualityManagement,ProcessManagement
  const widgetName = argv[argv.length - 1]

  const widgetsPath = globSync(`./src/widgets/*/index.ts`)
  widgetNames = widgetsPath.map((file) => {
    const parts = isWin
      ? path.resolve(file).split('\\')
      : path.resolve(file).split('/')
    return parts[parts.length - 2]
  })
  const originWidgets = [...widgetNames]
  let remainWidgetNames = [...widgetNames]
  if (widgetName.includes(',')) {
    isMultipleBuild = true
    widgetName.split(',').forEach((name) => {
      if (!widgetNames.includes(name)) {
        console.error(chalk.red(`Component ${name} does not exist`))
        process.exit(1)
      }
    })
    widgetNames = widgetName.split(',')
    remainWidgetNames = cloneDeep(widgetNames)
  } else {
    if (!widgetNames.includes(widgetName)) {
      isMultipleBuild = true
    } else {
      isMultipleBuild = false
    }
  }

  try {
    fs.removeSync('./dist')
  } catch (error) {
    console.error('dist does not exist, continue packaging task')
  }
  buildSumCount = widgetNames.length

  function buildWidget(currentBuildWidgets) {
    const isWidgetArray = Array.isArray(currentBuildWidgets)
    // Package a component
    if (!isWidgetArray || (widgetName && originWidgets.includes(widgetName))) {
      runBuild(null, !isWidgetArray ? currentBuildWidgets : widgetName)
    } else {
      // Package multiple components, package in parallel according to CPU default parallelism
      const buildWidgets = divideArray(currentBuildWidgets)
      const slashPath = slash(path.resolve(process.cwd(), baseBuildFile))
      ensureFileSync(slashPath)
      writeFileSync(slashPath, JSON.stringify(buildWidgets, null, 2))
      for (let index = 0; index < Object.keys(buildWidgets).length; index++) {
        const widgets = buildWidgets[index]
        if (widgets.length) {
          runBuild(index)
        }
      }
    }
  }
  async function build(buildWidgetName) {
    if (!buildWidgetName) {
      const cpus = os.availableParallelism()
      const currentBuildWidgets = widgetNames.splice(0, cpus)
      buildWidget(currentBuildWidgets)
    } else {
      buildWidget(buildWidgetName)
    }
  }
  function start() {
    spinner.start()
    build()
  }
  start()

  /**
   * Run compilation
   * @param {*} nodeIndex Set packaging component start point
   */
  function runBuild(nodeIndex, widgetName) {
    // return new Promise((resolve,reject) => {

    const cmdParams = ['run', 'build-lib']
    cmdParams.push(isDebugMode ? 'development' : 'production')
    const run = spawn(
      process.platform === 'win32' ? 'npm.cmd' : 'npm',
      cmdParams,
      {
        // stdio: 'inherit',
        shell: true,
        env: {
          // Compiled component index mapping
          ...process.env,
          NODE_INDEX: nodeIndex,
          BUILD_WIDGET_NAME: widgetName,
        },
      }
    )

    run.on('close', async (code) => {
      // Add packaging hash
      remainWidgetNames.shift()
      if (
        (!widgetNames.length && !remainWidgetNames.length) ||
        !isMultipleBuild
      ) {
        spinner.stop()
        await tag()
        if (isMultipleBuild) {
          console.log(
            chalk.green(`\nAll components have been compiled, adding version tag, please wait...\n`)
          )
          console.log(chalk.green(`\nStart packaging wwwroot.zip package\n`))
          getHostPackage()
        } else {
          console.log(
            chalk.green(`\nTotal compilation time: ${(Date.now() - startTime) / 1000} seconds\n`)
          )
        }
      } else {
        const name = widgetNames.shift()
        if (name && widgetNames.length) {
          spinner.text = `Compiling component: ${chalk.green(
            name
          )}, remaining component count: ${chalk.green(widgetNames.length)}\n`
        }

        if (name && isMultipleBuild) {
          build(name)
        }
      }
    })
    // })
  }
}
/**
 * Get equally divided component data
 * @param {*} widgets
 * @param {*} cpus
 * @returns
 */
function divideArray(widgets) {
  // When packaging, operating the computer may lag
  const cpus = os.availableParallelism()
  let result = {}
  let dataPerKey = Math.floor(widgets.length / cpus)
  let remainingData = widgets.length
  for (let i = 0; i < cpus; i++) {
    let currentDataCount = Math.min(dataPerKey, remainingData)
    result[i] = widgets.splice(0, currentDataCount)
    remainingData -= currentDataCount
  }
  if (widgets.length) {
    widgets.forEach((widgetName, index) => {
      result[index].push(widgetName)
    })
  }
  return result
}
/**
 * Get host package zip package
 * @param {*}
 */
function getHostPackage() {
  const resourcesPath = slash(
    path.resolve(process.cwd(), `${baseDir}/resources`)
  )

  const widgetsPath = slash(path.resolve(process.cwd(), `${baseDir}/widgets`))
  const currentDistPath = slash(path.resolve(process.cwd(), './dist'))
  const isResources = fs.pathExistsSync(resourcesPath)
  const isWidgets = fs.pathExistsSync(widgetsPath)
  if (!isResources) {
    fs.mkdirpSync(resourcesPath)
  }
  if (!isWidgets) {
    fs.mkdirpSync(widgetsPath)
  }

  const git = crossSpawn.sync('git', ['clone', assetsUrl, '-b', 'develop'], {
    stdio: 'inherit',
    cwd: resourcesPath,
    shell: true,
  })

  if (git.status === 0) {
    fs.removeSync(slash(path.resolve(resourcesPath, './assets/.git')), {
      recursive: true,
    })
    const dirs = globSync(slash(path.resolve(currentDistPath, './**/*.js')))
    dirs.forEach((dir) => {
      const widgetName = slash(dir).replace(currentDistPath, '.')
      const widgetPath = slash(path.resolve(widgetsPath, widgetName))

      fs.copySync(slash(dir), widgetPath)
    })
    zipDir(hostPath, hostZipPath)
      .then(() => {
        console.log(chalk.green(`${hostZipPath} Compression successful`))
        fs.removeSync(hostPath, {
          recursive: true,
        })
        console.log(
          chalk.green(`\nTotal compilation time: ${(Date.now() - startTime) / 1000} seconds\n`)
        )
      })
      .catch((error) => {
        console.log(error)
      })
  }
}
/**
 * Compress zip package
 * @param {*} dir
 * @param {*} zipPath
 * @returns
 */
function zipDir(dir, zipPath) {
  return new Promise(async (resolve, reject) => {
    try {
      await zip(slash(dir), slash(zipPath))
      resolve()
    } catch (error) {
      reject(error)
    }
  })
}
/**
 * Delete host package
 * @param {*}
 */
function removeHostPackage() {
  if (fs.pathExistsSync(hostPath)) {
    fs.removeSync(hostPath, {
      recursive: true,
    })
  }
  if (fs.pathExistsSync(hostZipPath)) {
    fs.removeSync(hostZipPath, {
      recursive: true,
    })
  }
}

Packaging effect is as follows: