NodeJS modules shrink

I'm a newbie in NodeJS and I'm not really familiar with all this thing of "npm". I mean, I didn't expect that just for one "npm install" I will get 10Mb of 30 different folders. 

After a bunch of minutes when I processed this fact, I just realize that there are a lot of junk in those folders. They are not "compiled", let's call it this way... There are "readme" files, source files, test scripts, example applications, docs and so on!!! Many of modules has duplications of another modules inside their folders. Like “inherits”, “core-util-is”, “tedious” etc. Modules inside modules, Karl! And inside them – another modules, that already exist in other modules in the project. Whaaaaat???!!!

There is an example of one of them - I'm in "node_modules" and in "bl" folder, but there is another "node_modules" folder inside it as well as "test", "readme" and all this junk here. And all of this just for a 10 Kb JS-file!!! 

Ok, it tooks me another good bunch of minutes to accept this situation. After this “acceptance” I reasonably thought that there is some command, like “build project” that’s going to “compile” all those modules into some production form – only “dist” or "lib" folders, without duplications of references, preferably js-minimized...

Aaaaannnnnd! Nop, there isn’t.

So I found some script (“npm-dist”) that can “shrink” module somehow. Why “somehow”? Because it can’t handle circular dependencies. But anyway! It was something! Something that can help me to get rid of all this junk in modules and minimize a package size.

Note: All this story is in the frame of AWS lambda-scripts where is a limitations of size, per script and per region as well.

It wasn't my problem as you can see but it can be in the future when the system will grow and in each labmda will be those mess of junks.

So I found some script (“npm-dist”) that can “shrink” module somehow. Why “somehow”? Because it can’t handle circular dependencies. That are a lot in this project. But anyway! It was something! Something that can help me to get rid of all those junk in modules and minimize a package size.

The simple way to use this package is:

const Trim = require('npm-dist');

Trim("mssql");

But it's just for one package. So I wrote some few ways to do this in loop. 

First of all I used package_lock.json to go on it and do shrinks:

const Trim = require('npm-dist');

var packs = require('./package-lock.json');

Object.keys(packs.dependencies).forEach(d => {

    console.log(d);

    Trim(d);

});

 And I ran into this issue of circular dependencies on one of my modues. So I tried more direct approach - list of directories:

const Trim = require('npm-dist');

var fs = require('fs');

const packs = fs.readdirSync("./node_modules").filter(

    d => !d.startsWith(".") && d != 'npm-dist'

);

packs.forEach(d => {

    console.log(d);

    Trim(d);

});

I filtered them from system dirs started from point, from "npm-dist" it saelf and than I added problematic modules there - to avoid script to be terminated. 

There are results of some small project I worked: 

• there were 30 folders of different modules in "node_nodules"
• only one of them was problematic with dependencies issue
• they holded 4Mb of storage (except of problematic one that I took it through the same procedure independently and then merged it into the result main folder)
• there wad a new "distr" directory created as a result of script running
• it was 1Mb size

The problematic package, as I told already, I ran through the same procedure but in isoleted mode. Than I copied the result "dist" folder into a main "dist" folder. Some manual adjustments was requred during this procedure but it was worth of this effort.

I know that is not a some "CI/CD regulated" or some "Production oriented" procedure but it helps me to create a few minimized lambda layers that I'm using right now and I'm happy :-)

Happy coding, friends!