node_modules caching is harmfull when using npm ci

[WolfSoko]

Beside caching node_modules with your caching workflow step the Agents for my repo needed always too long (4mins) to run the install-node-modules action.

The problem comes from the usage of npm ci to install node_modules based on the package-lock.json:

In an CI environment using npm ci sound correct, but the upfront caching of node_modules folder makes no sense at all.
It even takes more time because due to npm documentation npm ci deletes the node_modules folder for a clean installation process. @see npm ci documentation what takes more time when the folder is filled after a cache hit.

What can be cached to speed up npm ci is the global node cache $HOME/.npm from which npm ci can pull the packages faster.

I created my agents based on your runner templates. So I guess a lot of people are facing the same problem.

As a workaround I just replaced my usage of your install-node-modules workflow-step with a simple script that just runs npm install.
It brought down the execution time to 4sec when hitting the cache!
Fast execution was more important for me then consistency of the dependencies.

[JamesHenry]

The actions/cache documentation also say that it is recommended to not cache node_modules and instead cache the central cache (path dependent on OS): https://github.com/actions/cache/blob/main/examples.md#node---npm

@barbados-clemens please can you take a look?

[WolfSoko]

@JamesHenry That is exactly what the github setup-node action does when caching is configured to npm.

It leverages npm config get cache to determine the central cache folder and only caches that.

@barbados-clemens, @rarmatei
As a suggestion:
A possible and maybe fast and dependency consistent method might be,

1. cache node_modules and central cache.
2. Then try with npm install
3. if npm install changes package-lock.json revert changes and fallback to npm ci

[WolfSoko]

@rarmatei did you make any progress on this topic?

[rarmatei]

@rarmatei did you make any progress on this topic?

@WolfSoko apologies about the late reply, I missed this thread. This has been fixed and the docs were updated a couple of months ago with recommendations on caching the ~/.npm folder: https://nx.dev/ci/reference/launch-templates#full-example

We need to use the official NPM recommendations on caching, but if you want you can try caching the node_modules folder in your pipelines, like you explained above:

- name: Cache everything
- name: Cache everything
        uses: 'nrwl/nx-cloud-workflows/v5/workflow-steps/cache/main.yaml'
        inputs:
          key: 'package-lock.json'
          paths: |
            ~/.npm
            node_modules
- name: NPM Install
   script: npm install
- script: ./check-lockfile-and-run-npm-ci.sh