6 votes

What is the present state of NPM Packaging System?

As I recall, about 2-3 years ago I had experimented with what is called the npm install xyz through some tutorials, etc. The objective was to improve my tooling or toolchain at least with things like css/js minification, etc. along with Bootstrap customization. The idea was to then move on to other newer learnings like react, etc.

But I was soon disgruntled by the whole process! Neither bootstrap turned out to be an easy horse to tame, and NPM was like this astronomical universe of packages that keep on downloading into your folders! I soon lost taste of the whole thing and kept using my tried and tested stack of PHP scripts and using stock Bootstrap through CDNs.

But today, I want to give it another shot. Has NPM improved than earlier days? What is the best way to go about building a toolchain using it? Are there any useful guides that make the whole process easier to digest?

15 comments

  1. [2]
    Octofox
    Link
    I don’t think there was really anything wrong with it 2 years ago but from regular usage I haven’t seen it change much since then other than more packages moving to namespaces (very good). You...

    I don’t think there was really anything wrong with it 2 years ago but from regular usage I haven’t seen it change much since then other than more packages moving to namespaces (very good).

    You probably just need to push through the initial learning. It works perfectly fine once you work it out which shouldn’t take long. I’d start with the react guide and it will show you how to set it all up

    10 votes
    1. Greg
      (edited )
      Link Parent
      Agreed. Going back a few years beyond that there were some consistency issues, but since package-lock.json and npx were added I find that things pretty much just work. For anyone unfamiliar:...

      Agreed. Going back a few years beyond that there were some consistency issues, but since package-lock.json and npx were added I find that things pretty much just work.

      For anyone unfamiliar: package-lock.json is an autogenerated file that you check into source control. It contains the full dependency tree, including all derived dependencies, precise versions, and hashes so that you can be confident that what gets installed with a specific commit is always identical over time and across different systems. Way back in the mists of time (pre-2017) it would calculate that tree from scratch every time at runtime, which... honestly actually worked fine most of the time, but very occasionally broke things in difficult to diagnose ways, at times that were impossible to know or predict in advance.

      npx is used to execute a CLI tool from within the project's node_modules directory rather than installing it globally or hardcoding brittle paths to specific executables in your config. Prime example of where it's useful would be developing multiple React Native apps on the same workstation - rather than running react-native run-ios and having to remember which version of the CLI is installed system-wide, which is installed on the CI server, and potentially juggle compatibility issues between older and newer codebases, you just use npx react-native run-ios and it'll call the binary [edit: or script] that's stored with that particular project and library version.

      8 votes
  2. [12]
    vord
    Link
    I'm a newbie, but I've noticed that NPM isn't really uniquely bad, but just shows the symptoms of a modern language with tightly integrated package manager. Namely, that you'll get giant...

    I'm a newbie, but I've noticed that NPM isn't really uniquely bad, but just shows the symptoms of a modern language with tightly integrated package manager.

    Namely, that you'll get giant dependency chains of imports, as the easy import makes library makers more willing to import other dependencies rather than just working with the standard library. I first noticed with a moderate sized (but somewhat amateur) Rust project, where'd you have hundreds of dependencies, with many duplicate-but-different-version packages in the chain.

    I don't recall this being quite so common in Perl/Python compared to Rust/Node. And it's really not a problem per-se, but there definitely are security implications. And the disk usage is annoying.

    3 votes
    1. [5]
      Adys
      Link Parent
      It's not common in Python, because for the longest time it was a pain in the ass to release python modules. NodeJS made it easy, so more people made more, smaller libraries. And because people did...

      It's not common in Python, because for the longest time it was a pain in the ass to release python modules.

      NodeJS made it easy, so more people made more, smaller libraries. And because people did that, the language evolved around this concept of using small external libraries, and it kind of picked up from there.

      I believe the pendulum is slowly swinging back to a more reasonable state, but of course, there's a lot of legacy.

      4 votes
      1. [3]
        arp242
        (edited )
        Link Parent
        Last month I updated a Webpack/Vue project that wasn't worked on for three years and it went from ~1,200 dependencies to ~100 dependencies (updating it all was a ridiculous amount of effort...

        I believe the pendulum is slowly swinging back to a more reasonable state, but of course, there's a lot of legacy.

        Last month I updated a Webpack/Vue project that wasn't worked on for three years and it went from ~1,200 dependencies to ~100 dependencies (updating it all was a ridiculous amount of effort though). I also eliminated quite a few dependencies in the process such as webpack-dev-server and a few other small things (IIRC that didn't actually matter all that much in terms of dependency tree, but it helped a bit).

        So from this one datapoint, it seems slightly more reasonable. Still seems a lot for something that essentially does "vue-compile *.vue | minify >file.js", but ah well.

        1 vote
        1. [2]
          Adys
          Link Parent
          webpack-dev-server brings in a ridiculous amount of cruft. I think a lot of dev tooling has been streamlined quite nicely which is probably responsible for a lot of this

          webpack-dev-server brings in a ridiculous amount of cruft. I think a lot of dev tooling has been streamlined quite nicely which is probably responsible for a lot of this

          1. arp242
            Link Parent
            Okay, I did a quick test: % npm install up to date, audited 119 packages in 705ms % npm install webpack-dev-server added 195 packages, and audited 314 packages in 8s It does add quite a bit, but...

            Okay, I did a quick test:

            % npm install
            up to date, audited 119 packages in 705ms
            
            % npm install webpack-dev-server
            added 195 packages, and audited 314 packages in 8s
            

            It does add quite a bit, but still significantly less than the 1,123 before (checked out old revision to verify).

            I did have to build a somewhat esoteric solution to make all of this work smoothly on dev though (the Go backend will start webpack with watch, which will output build status as JSON which gets read and reported by Go, which also serves files from the dist directory. This sounds more complex than it is: it's actually very little code and quite simple, just a very non-standard way of doing it).

      2. vord
        Link Parent
        I think part of the problem is that there's a need for a division between micro and macro imports A micro-package acting more like a Co-pilot autocomplete, where the import gets embedded in your...

        I think part of the problem is that there's a need for a division between micro and macro imports

        A micro-package acting more like a Co-pilot autocomplete, where the import gets embedded in your code, ideally as a single file (or perhaps even just inline).

        A macro package being something beefy, like database abstraction.

        And that ideally, macro packaging doesn't support multiple versioning for cross-dependency. If there is a conflict, it forces the 'fix the conflict' rather than 'hide the conflict,' in the way a type-safe language prevents you from mis-typing variables.

    2. [2]
      Diff
      Link Parent
      Python and Go have very rich standard libraries, and Go's standard library includes a number of standard interfaces that people try to adhere to even in their third party libraries. I think they...

      Python and Go have very rich standard libraries, and Go's standard library includes a number of standard interfaces that people try to adhere to even in their third party libraries. I think they (and maybe Perl?) don't suffer from cascading dependency trees despite the same ease of imports because you just need fewer dependencies.

      4 votes
      1. skybrian
        Link Parent
        Go also has a culture of having larger modules that do more, and also not importing a module just for one convenience function. "A little copying is better than a little dependency" is a Go...

        Go also has a culture of having larger modules that do more, and also not importing a module just for one convenience function. "A little copying is better than a little dependency" is a Go proverb.

        Contrast with JavaScript where people would try to use small libraries to avoid code bloat in browser apps. That's no longer needed now because build tools are better at removing unused code, but it resulted in a different culture where small libraries that just do one thing were considered better.

        2 votes
    3. [2]
      Octofox
      Link Parent
      In older languages the dependancies were just invisible. You’d use libraries as compiled blobs where you have no visibility in to how many things those libraries imported themselves. And many...

      In older languages the dependancies were just invisible. You’d use libraries as compiled blobs where you have no visibility in to how many things those libraries imported themselves. And many projects just copy pasted libraries in to a vendor folder so again you’d see nothing.

      4 votes
      1. arp242
        Link Parent
        % ls -l /lib/**/*.so | wc -l 1354 There are all shared libraries on my system, and this count is inflated (e.g. /lib/zsh contains 37 dynamic .so plugins, gconv has a .so file for every encoding,...
        % ls -l /lib/**/*.so | wc -l
        1354
        

        There are all shared libraries on my system, and this count is inflated (e.g. /lib/zsh contains 37 dynamic .so plugins, gconv has a .so file for every encoding, etc.)

        Many JS projects have more than 1,354 dependencies.

        Firefox – a large program by any standard – links against 93 libraries:

        % ldd /lib/firefox/*(*) | grep -o '.*=>' | sort -u | wc -l
        93
        

        It does bundle a few dependencies, such as hunspell, and you need a few things to build it, so let's add a liberal 50% to that and say it had 150 dependencies, which is significantly less than many JavaScript-based projects of significantly smaller size.

        2 votes
    4. [2]
      noble_pleb
      Link Parent
      Exactly, PHP and Python are different in that their core libraries are so powerful that you can do a lot of things without ever using their package managers (PyPi/Composer). But with Node, you're...

      Exactly, PHP and Python are different in that their core libraries are so powerful that you can do a lot of things without ever using their package managers (PyPi/Composer). But with Node, you're absolutely crippled without npm. Even for basic things like database access which are taken for granted in other languages, you need an npm package here which is quite an anti-pattern.

      3 votes
      1. stu2b50
        Link Parent
        Part of is just that JS has a bit of a Jekyll and Hyde thing going on - nodejs is just one possible runtime. The whole other half is in browsers, where you're never going to have a large standard...

        Part of is just that JS has a bit of a Jekyll and Hyde thing going on - nodejs is just one possible runtime. The whole other half is in browsers, where you're never going to have a large standard library available by default in the runtime - it's split into three different runtimes after all. Node was always lightweight since it's basically just V8 ripped out of Chromium.

        I think Deno, which is in general trying to be highly opinionated to rectify some of node's issues, has more batteries included.

        4 votes