Continuous Integration

Extension integration tests can be run on CI services. The vscode-test library helps you set up extension tests on CI providers and contains a sample extension setup on Azure Pipelines. You can check out the build pipeline or jump directly to the azure-pipelines.yml file.

Automated publishing

You can also configure the CI to publish a new version of the extension automatically.

The publish command is similar to publishing from a local environment using vsce, but you must somehow provide the Personal Access Token (PAT) in a secure way. By storing the PAT as a VSCE_PAT secret variable, vsce will be able to use it. Secret variables are never exposed, so they are safe to use in a CI pipeline.

Azure Pipelines

Azure Pipelines is great for running VS Code extension tests as it supports running the tests on Windows, macOS, and Linux. For Open Source projects, you get unlimited minutes and 10 free parallel jobs. This section explains how to set up an Azure Pipelines for running your extension tests.

First, create a free account on Azure DevOps and create an Azure DevOps project for your extension.

Then, add the following azure-pipelines.yml file to the root of your extension's repository. Other than the xvfb setup script for Linux that is necessary to run VS Code in headless Linux CI machines, the definition is straight-forward:

trigger:
  branches:
    include:
    - master
  tags:
    include:
    - v*

strategy:
  matrix:
    linux:
      imageName: 'ubuntu-16.04'
    mac:
      imageName: 'macos-10.13'
    windows:
      imageName: 'vs2017-win2016'

pool:
  vmImage: $(imageName)

steps:

- task: NodeTool@0
  inputs:
    versionSpec: '8.x'
  displayName: 'Install Node.js'

- bash: |
    /usr/bin/Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
    echo ">>> Started xvfb"
  displayName: Start xvfb
  condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))

- bash: |
    echo ">>> Compile vscode-test"
    yarn && yarn compile
    echo ">>> Compiled vscode-test"
    cd sample
    echo ">>> Run sample integration test"
    yarn && yarn compile && yarn test
  displayName: Run Tests
  env:
    DISPLAY: ':99.0'

Finally, create a new pipeline in your DevOps project and point it to the azure-pipelines.yml file. Trigger a build and voilà:

You can enable the build to run continuously when pushing to a branch and even on pull requests. See Build pipeline triggers to learn more.

Azure Pipelines automated publishing

1. Set up VSCE_PAT as a secret variable using the Azure DevOps secrets instructions.
2. Install vsce as a devDependencies (npm install vsce --save-dev or yarn add vsce --dev).
3. Declare a deploy script in package.json without the PAT.

"scripts": {
  "deploy": "vsce publish --yarn"
}

4. Configure the CI so the build will also run when tags are created:

trigger:
  tags:
    include:
    - master
    - refs/tags/v*

5. Add a publish step in azure-pipelines.yml that calls yarn deploy with the secret variable.

- bash: |
    echo ">>> Publish"
    yarn deploy
  displayName: Publish
  condition: and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/tags/'), eq(variables['Agent.OS'], 'Linux'))

The condition property tells the CI to run the publish step only in certain cases.

In our example, the condition has three checks:

• succeeded() - Publish only if the tests pass.
• startsWith(variables['Build.SourceBranch'], 'refs/tags/') - Publish only if a tagged (release) build.
• eq(variables['Agent.OS'], 'Linux') - Include if your build runs on multiple agents (Windows, Linux, etc.). If not, remove that part of the condition.

GitHub Actions

You can also configure GitHub Actions to run your extension CI using the gabrielbb xvfb action. This automatically checks if Linux is the current OS and runs the tests in an Xvfb enabled environment accordingly:

on:
  push:
    branches:
      - master

jobs:
  build:
    strategy:
      matrix:
        os: [macos-latest, ubuntu-latest, windows-latest]
    runs-on: ${{ matrix.os }}
    steps:
    - name: Checkout
      uses: actions/checkout@v2
    - name: Install Node.js
      uses: actions/setup-node@v1
      with:
        node-version: 8.x
    - run: npm install
    - name: Run tests
      uses: GabrielBB/xvfb-action@v1.0
      with:
        run: npm test

GitHub Actions automated publishing

1. Set up VSCE_PAT as an encrypted secret using the GitHub Actions secrets instructions.
2. Install vsce as a devDependencies (npm install vsce --save-dev or yarn add vsce --dev).
3. Declare a deploy script in package.json without the PAT.

"scripts": {
  "deploy": "vsce publish --yarn"
}

4. Configure the CI so the build will also run when tags are created:

on:
  push:
    branches:
    - master
  release:
    types:
    - created

5. Add a publish job to the pipeline that calls npm run deploy with the secret variable.

- name: Publish
  if: success() && startsWith( github.ref, 'refs/tags/releases/') && matrix.os == 'ubuntu-latest'
  run: npm run deploy
  env:
    VSCE_PAT: $

The if property tells the CI to run the publish step only in certain cases.

In our example, the condition has three checks:

• success() - Publish only if the tests pass.
• startsWith( github.ref, 'refs/tags/releases/') - Publish only if a tagged (release) build.
• matrix.os == 'ubuntu-latest' - Include if your build runs on multiple agents (Windows, Linux, etc.). If not, remove that part of the condition.

Travis CI

vscode-test also includes a Travis CI build definition. The way to define environment variables in Travis CI is different from other CI frameworks, so the xvfb script is also different:

language: node_js
os:
- osx
- linux
node_js: 8

install:
- |
  if [ $TRAVIS_OS_NAME == "linux" ]; then
    export DISPLAY=':99.0'
    /usr/bin/Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
  fi
script:
- |
  echo ">>> Compile vscode-test"
  yarn && yarn compile
  echo ">>> Compiled vscode-test"
  cd sample
  echo ">>> Run sample integration test"
  yarn && yarn compile && yarn test
cache: yarn

Travis CI automated publishing

1. Set up VSCE_PAT as an encrypted secret using the Travis CI encryption key usage instructions.
2. Install vsce as a devDependencies (npm install vsce --save-dev or yarn add vsce --dev).
3. Declare a deploy script in package.json without the PAT.

"scripts": {
  "deploy": "vsce publish --yarn"
}

4. Add an after_script stage to the job that calls npm run deploy with the secret variable.

after_script:
- |
  echo ">>> Publish"
  yarn deploy

stages:
- name: after_script
  if: env(TRAVIS_TAG) =~ ^v

The stages property tells the CI to include stages when certain conditions are met.

In our example, the condition has one check:

• env(TRAVIS_TAG) =~ ^v - Publish only if a tagged (release) build that starts with the letter v.

Common questions

Do I need to use Yarn for continuous integration?

All of the above examples refer to a hypothetical project built with Yarn, but can be adapted to use npm, Grunt, Gulp, or any other JavaScript build tool.