I hope one day we can get a widely adopted C and C++ package manager. The friction involved in acquiring and using dependencies with odd build systems, etc. is one of the things I dislike about the language. I’m aware on Linux things are a bit easier, but if it were as easy as “npm install skia”, etc. everywhere, I think many people would use the language more.
Rust has package management, but not the ecosystem yet. On the other hand, C/C++ has the ecosystem, but no standard way to easily draw from it.
Widely adopted source code manager requires a widely adopted build system. CMake is certainly a contender but the ecosystem is too fragmented even then & you have to do a lot to try to link disparate build systems together. Also C++ is a transitive dependency hell nightmare & any attempt to solve that (like Rust has) would break every ABI out there. Given how bumpy such breakages have been in the past, I don't think there's any compiler maintainer eager for it (even MSVC has decided to largely ossify their STL runtime ABI).
Conan is certainly a laudable attempt at something like this. Without access to their metrics though, it's hard to tell if they're continuing to gain meaningful traction or if their growth curve has plateaued. It's certainly not in use in any project at medium to bigger size companies I've worked at. By comparison, Cocoapods was pretty successful in the iOS ecosystem precisely because Xcode was the de facto build/project system.
I'm a longtime CMake user, but I think even within the CMake world, the solution is quite a bit more complicated than just "everything needs to be CMake", with a lot of hassles that arise when multiple generations of the tooling is involved, when you're trying to pass down transitive dependencies, when package X has a bunch of custom find modules with magic to try to locate system versions of dependencies but silently fall back to vendored ones.
The higher up the stack you get, the worse and worse these problems get, with high-level packages like Tensorflow being completely intractable:
Yup. 100% agree. I totally overlooked the shitshow you'll have managing the different versions of CMake a build might require. Somehow Bazel manages to escape that mess. I think that might be a better foundation, but getting everyone to port to that... it's a tall ask & there's many vocal people who are against improving the build system they work with (hell, I've met many engineers who grumble and strongly prefer Makefiles).
I'm obviously pretty biased having come from 10 years of doing infrastructure in the ROS world, but having spent a lot of time integrating this or that random library into a singular product-versioned build, I do quite like the approach of colcon:
Basically it has plugins to discover/build various package types (autotools, cmake, bazel, setuptools, cargo), and the "interface" between packages is just the output of whatever the standard install target is for a given package. This makes it totally transparent whether your dependency is built-from-source in your workspace, coming from /usr/local via a sudo-make-install workflow, or coming from /usr via a system package.
Under this model, you never pull a dependency as a "subproject" with invocations like include or add_subdirectory; it's always using a standard find_package invocation, where basically the only requirement on participating packages is that they cooperate with long-existing standards like CMAKE_PREFIX_PATH and CMAKE_INSTALL_PREFIX. Vendoring a library is then not making a copy of it in your project tree, but rather as sibling project within the shared workspace that colcon builds.
> Without access to their metrics though, it's hard to tell if they're continuing to gain meaningful traction or if their growth curve has plateaued.
Some public data that could be used as proxy for traction:
- Some companies using Conan in production can be seen in the committee for Conan 2.0 called the tribe: https://conan.io/tribe.html. That includes companies like Nasa, Bose, TomTom, Apple, Bosch, Continental, Ansys...
- With +1600 subscribers the #conan channel in the CppLang slack is consistently ranked in the most active channels every month: https://cpplang.slack.com/stats#channels
C++ is considered the industry leading language in many fields. I'm not sure how many more you would want (given that those fields that don't use C++ ARE probably better served with some other language).
I agree the build is painfull, but large orgs have for this reason specifically implemented build systems using nugets, conan/cmake or whatnot.
In personal projects I just download the prebuilt binaries of component libraries and drag and drop them to visual studio, minimizing hassle.
If you discard finesse and scalability as requirements you can actually jury rig a C++ project in a jiffy. You just need to let go of the idea that it must be "industry standard setup".
C++ used to be the industry leading language in many more fields, but it lost ground to other languages. Not a bad thing--"know thyself" and all that. But Rust seems like a credible threat to C++'s remaining niches (bury your head in the sand if you want), and C++ will need to evolve if it is to not lose further market-/mindshare. And it is evolving, as this article points out, but a huge glaring pain point in C++ development remains the build and package management tooling. The aforementioned build systems that large organizations operate aren't nearly as nice as, say, Cargo and I think a lot of greenfield projects who have to choose between cobbling together their own build tool to work with C++ and using Rust + Cargo off the shelf will choose the latter (other factors notwithstanding).
I will get worried when NVidia releases CUDA-Rust, and changes their GPGPUs from C++ memory model to Rust, Microsoft decides to rewrite WinUI in Rust, Apple moves Metal from C++ into Rust, or Unreal/Unity get rewritten in Rust.
> You write as if Rust vs. C++ was some sort of competition.
Competition exists all around us, all the time, whether we like it or not.
And the competition between C++ and Rust is very clear. I, for example, would likely be spending more time / effort on learning the latest C++ standards if Rust didn't exist. And likely hate my life a little bit, unless I could exclusively stick to C++ "the good parts" if such a subset exists and I didn't need to interface as much with existing C++ code.
My job is writing C++ and I love it. I've been working with C++ since the mid 90's and have grown very fond of the language and I'm pretty productive in it.
If C++ use declines, then there are fewer opportunities for me. So you can count me as a member of team C++.
I also write C++ for a living but feel no threat if I had to suddenly start writing C, C#, Java, Python, Rust, F#, Scala, or what have you. Sure, it would need learning a thing or two but basically they all are driving the same ARM or x86 based compute stack with exactly the same constraints due to computer architecture .
My focus has been to brand myself as "domain expert" in few algorithmic domains rather than "C++" expert so this may affect my point of view, though.
> feel no threat if I had to suddenly start writing C, C#, Java, Python, Rust, F#, Scala
It isn't necessarily a threat. I'm pretty comfortable in a bunch of different languages but just enjoy C++ more than the others. To use a car analogy, I have no problem driving automatic but I really like driving stick.
If you don't keep up with all this C++20 material, there will likewise be fewer opportunities for you. They will throw it at you in an interview, to check that you aren't some stubborn C++98 gunslinger.
I do my best! I really enjoy working in C++ and it's a pretty exciting time for me when a new standard rolls around and my compiler gets updated to support it.
Years. Some things are obviously useful right away and other things take me a lot longer to grok. For example, rvalue references have been around for a long time now and I still have to slow down when I see && in code.
> If use of C++ declines then I don't understand how that would make the language a lesser tool.
The hypothesis is that C++ will decline because it becomes the lesser tool (where "lesser tool" means it excels only in increasingly small niches) if it doesn't adapt. That said, the C++ community seems to want to adapt and remain relevant, as indicated by its significant progress over the last decade.
As for why someone might care about the usage of a programming language: because "ease of finding developers" and "quality and breadth of ecosystem" are major factors in deciding on new projects. I.e., "the best tool for the job" is often the one with the broader ecosystem and more developers, all else equal. So these factors feed back on each other.
> Why would it matter and to whom if C++ use would decline?
If the use declines to zero, then all the effort someone put into developing C++ compilers and related tooling had been for naught, as are C++ development skills.
Given the prevalence of C++ it is very hard for me to imagine a situation where the use would decline to zero.
The notion of how to write a specific language such as C++ is immaterial compared to the capability to design and implement complex software systems and those skills are quite portable between languages. The best employers tend to recognize this.
>I hope one day we can get a widely adopted C and C++ package manager. [...] , but if it were as easy as “npm install skia”, etc. everywhere,
It's not just the package manager (the command line tool) ... it's the canonical website source that the tool pulls from.
C++ probably won't have a package manager with the same breadth of newer language ecosystems like npm/Nodejs and crates.io/Rust because for 20+ years C++ was developed by fragmented independent communities before a canonical repo website funded by a corporation or non-profit was created. There is no C++ institution or entity with industry-wide influence that's analogous to Joyent (Nodejs & npm) or Mozilla (Crates.io & cargo)
- C++ for 20+ years of isolated and fragmented development groups creates legacy codebases --> then decades later try to create package manager (vcpkg? Conan? cppget?) that tries to attracts those disparate groups --> thus "herding cats" is an uphill challenge
- npm and crates.io exist at the beginning of language adoption allowing the ecosystem to grow around those package tools and view them as canonical
Go has a perfectly good package manager that works with sources hosted on GitHub and other sites -- there isn't any centralized place for people to publish sources, unlike the other package managers you mentioned.
Go's package manager also came years after the language became widely used, and it is now very widely adopted according to the most recent survey[0].
I think C++ could have a good, unified package management story. It would just require the major stakeholders to all care enough to make it happen, which seems to be the missing piece here.
Go has a small dedicated team that develops and designs the language. They take some input from the broader community but are still the one who decides how things evolve. They decided at some point that go modules was the way to go and everybody followed, because they are the authority who decides how Go evolves.
C++ does not have an equivalent, it's completely decentralized which results in more messy situation. As a result you have an open market where different people try to build different tools and approaches for their own problems, then try to get others to use them (similar to what Go had before go modules, we had lot of package managers to chose from at the time).
Instead of a top down decision it's a negotiation between the various actors. But the last thing we need is for the C++ standards committee to standardize a package manager. That would take forever to do, would result in a messy tool that tries to compromise with all the actors in some ways, make it very hard and slow to evolve over time and would likely result in a lot of pain, etc.
> C++ does not have an equivalent, it's completely decentralized which results in more messy situation.
That is the role of the ISO C++ committee, is it not? They are the major stakeholders. They would just have to care enough. They cared enough to release C++20, didn't they? It's not like they never get anything done, which seems to be the implication a lot of people make in this discussion.
> Instead of a top down decision it's a negotiation between the various actors.
My understanding is that the various committee members represent the disparate interests of the broader C++ community. I agree it would be very much like a negotiation.
That doesn't mean that it can't be done. This whole thread is discussing things that have been done by the C++ committee: C++20.
> But the last thing we need is for the C++ standards committee to standardize a package manager. That would take forever to do, would result in a messy tool that tries to compromise with all the actors in some ways, make it very hard and slow to evolve over time and would likely result in a lot of pain, etc.
You just summarized my feelings about C++ in general. I would much rather people use Rust or Go or any number of other languages instead of C++, depending on project needs. Such opinions are rarely taken well in threads like this, though, so...
I've been trying to be optimistic and point out that C++ could get package management. If the C++ committee process works well, then the package manager should also end up turning out well.
I'll leave the reader to decide how well they think the long term direction and guidance of the C++ standard has been going and apply that to their feelings of a hypothetical future package manager.
It's design by agreement vs design by decision. In the former you need people to agree. In a committee setting it means that not only do you have to make MSFT, Google, & Apple happy, it's also the various other people that happen to be part of that standard body (the group is large). You definitely pull from a larger group of experts, but it's mired in indecision hell & compromise. Often times a decision that solves 90% of problems is better than a decision that is perfect, but the way ISO is set up, decisions kind of have to be perfect.
That being said, the C++ standards body (at least under Herb?) has done a decent job modernizing their process to fight some of the gravitational issues they were having. They've formalized deprecation rules & tried to get over disagreements. The design by committee issues haven't gone away though - the mess with coroutines, modules, & concepts is a great example of that. The ISO process of language papers precludes even simple additions to the STL where you not only have to navigate standardese, but also manage the review process (that's why you have to find a champion on the standards body to help guide your review through the rigamarole).
My experience contributing to the Rust standard library by comparison was much easier - put up a drive-by diff adding a new (admittedly minor) API, some minor review comments, done & shipped. The whole process took 1-2 weeks, no standardese, no arguing with a large committee on the exact wording, etc.
> My experience contributing to the Rust standard library by comparison was much easier - put up a drive-by diff adding a new (admittedly minor) API, some minor review comments, done & shipped. The whole process took 1-2 weeks, no standardese, no arguing with a large committee on the exact wording, etc.
this is off course all great until two people working in different parts of the language do things slightly differently. Either works alone, but the whole of the language is inconsistent and hard to learn.
C++ has enough inconsistent parts already and so tries to be careful to make new things consistent with itself as best as possible. Even that has failed despite all the review of people looking for places to make things consistent. It is a hard problem to design a large language.
So, a key difference here is that in Rust, the standard library and the language designers are two different teams, with two different standards. The parent is talking about the standard library; there's a reasonably low barrier to entry to add something, but it is added unstably, and the bar to getting it to stabilize is higher. The language does not accept additions by "drive-by PR", the barrier to getting something to land, even in an unstable way, is much, much higher.
The whole language team has to sign off on these stabilizations and language additions, which is what keeps up that consistency you're talking about.
Yeah sorry. Should have been clear that I was talking about the standard library. I've got the chops to contribute standard library code - would never even think about trying to tackle implementing language changes. I don't have the time nor energy to deal with C++ standardese since the spec is an ancillary artifact describing the thing rather than the thing itself (the thing itself being the implementation & documentation).
Granted, this isn't necessarily everyone's experience in std as the change I implemented is well-worn/adopted by any condition variable implementation. Something more controversial/exploratory may have been pushed off into a crate first. I'm still impressed that it only took ~2 weeks to get [1] reviewed & into unstable. I wasn't even involved in the stabilization work/cosmetic renaming that it took to close out [2] which was driven by a community ask & the std maintainers doing a pass to make things consistent. Rust's velocity seems to be that they can deliver changes a full 1 year faster than C++ can (& likely faster if the community really asks for it). In my book it's largely owing to having 1 compiler & 1 standard library & the latter having a much more streamlined RFC process.
No sorry needed! I think you were clear that you were, just worth re-iterating that in Rust, these are two separate groups with similar but slightly different processes, and in C++, the standard contains both language + standard library. (Obviously C++ has working groups... point is that the two languages are similar, but different.)
Unlike C++, Rust has pretty strong conventions around formatting and naming, and these conventions are followed in almost all major libraries. Furthermore, most such small APIs tend to bake in nightly for a while before they are stabilized, and so get two rounds of review: once during the initial commit, and once during the push for stabilization.
Right. C++ has a chicken-and-the-egg problem in that neither its build not packaging ecosystem have even de facto standards. GitHub URLs don't solve either problem.
I think everyone agrees that a common build system is a necessary step if any of this is going to work.
Thinking about how much or how little would be required beyond a common build system in order to get a working package management system is still a valid thing to do.
I expect it is necessary to keep build configuration portable with respect to packaging and environment configuration. In other words, I expect downloading from URLs in Makefiles and CMakeLists to be a local maximum.
Keeping a parallel set of instructions or metadata that includes specific URLs and such might work, though. As long as you can skip all that when a system package, filesystem path, or git submodule is more appropriate.
>Go's package manager also came years after the language became widely used, and it is now very widely adopted according to the most recent survey[0].
Are you talking about "pkg.go.dev" and the "go get" command? Isn't there some path dependence in the history of events that's not comparable to C++? Consider:
- Go language: created by Google Inc
- "go get" syntax for package download designed and created by Google Inc
- "pkg.go.dev" funded by Google Inc and highlighted on "golang.org" website that's also run by Google Inc.
There is no business entity or institution in the C++ world that's analogous to Google's influence for Go + golang.org + "go get" + pkg.go.dev.
>It would just require the major stakeholders to all _care_ enough to make it happen,
But it's easier to care if there was an influential C++ behemoth that captured everyone's mindshare to move the entire ecosystem forward. C++ has no such "industry leader" that dictates (or heavily influences) technical direction from the top down.
> Are you talking about "pkg.go.dev" and the "go get" command?
No. I'm not talking about either of those. Your whole comment is, unfortunately, irrelevant.
pkg.go.dev is not a package repo. It's just a place for documentation to be rendered. It renders documentation from third party hosted code, such as on GitHub or elsewhere.
"go get" predates Go Modules, which is the current package management system. The whole original design of "go get" was to simply download code from somewhere on the internet, and place it in the right spot of the $GOPATH. This has nothing to do with a proper versioned package manager like Go Modules.
AFAIK, "go get" was also never really designed for Google's internal use cases. They use a monorepo that was perfectly content with $GOPATH, and all their code was developed in the monorepo to begin with. There was nothing for them to "go get", except for the rare outside dependency that they were embedding into their monorepo, I would imagine. I've never worked for Google, these are just things I hear about.
Go Modules was also not designed for Google. It was designed for the community, based on findings from community developed package managers for Go. Google has no real use for it — again, they use a monorepo.
Nowadays, "go get" can be used with Go modules, but in practice, it feels like it almost never is. Maybe someone would use that command to upgrade an existing dependency, instead of editing the `go.mod` file to change the version there?
So, your comment just shows that you haven't researched this enough. Yes, Go Modules was still guided by Googlers, who were even more in control of the language direction back then than they are now. Yes, change always causes some drama. But, I'm not really here to explain the history of Go package management...
I'm just saying that C++ could have a nice, distributed package management system, it would just require the major stakeholders to all care and work together on it. The ISO C++ language committee is a finite number of people. They are the major stakeholders, as far as the language direction is concerned.
If they didn't have the power to enact major language changes, we wouldn't be here talking about C++20.
The stakeholders for Go were able to develop a package manager that is distributed (an idea compatible with how all C++ code is scattered across the web these days), and that achieved broad adoption, and this was some years after the language went into wide use.
It’s an extremely relevant analogue for C++ to study, if the committee members wanted a package manager badly enough.
> But it's easier to care if there was an influential C++ behemoth that captured everyone's mindshare to move the entire ecosystem forward. C++ has no such "industry leader" that dictates (or heavily influences) technical direction from the top down.
You edited this in while I was replying, but I agree entirely. Getting the committee to agree to a package management solution would be much more difficult than having a single behemoth guide the decision. Does that mean it is impossible and therefore no one could do it? Everyone here talks like it is impossible, but it doesn't really seem to be.
>Yes, Go Modules was still guided by Googlers, who were even more in control of the language direction back then than they are now. Yes, change always causes some drama. But, I'm not really here to explain the history of Go package management...
>I'm just saying that C++ could have a nice, distributed package management system, it would just require the major stakeholders to all care and work together on it. The ISO C++ language committee is a finite number of people. They are the major stakeholders, as far as the language direction is concerned.
The ISO C++ committee can't learn from the history of Go modules community acceptance because they don't have the same power as Google. You seem to misunderstand what the C++ committee _is_. Yes, they have representatives from Microsoft/Google/Apple/Intel but the org is designed to review proposals from submitted papers. They are more like an ongoing academic conference rather than a devops team that runs websites.
We seem to be discussing 2 different abstractions of making a "package manager". With your emphasis on Modules, you seem to be only focusing on the tool. To repeat my gp comment, I'm also focusing on the canonical package repository (or index, or discovery engine).
>The Go team is providing the following services run by Google: a module mirror for accelerating Go module downloads, an index for discovering new modules, and a global go.sum database for authenticating module content. >As of Go 1.13, the go command by default downloads and authenticates modules using the Go module mirror and Go checksum database.
You misunderstood my cite of "pkg.go.dev" run by Google Inc but this is the part of your survey that I was referring to:
>The package discovery site pkg.go.dev is new to the list this year and was a top resource for 32% of respondents. Respondents who use pkg.go.dev are more likely to agree they are able to quickly find Go packages / libraries they need: 91% for pkg.go.dev users vs. 82% for everyone else.
The ISO C++ committee is not set up to implement a new website to make the above Go-specific paragraphs be a similar reality for C++ with a search & replace "s/Go/C++/g". Think about _who_ funds and provides paid people to actually run the "proxy.golang.org". It's Google Inc. The C++ committee doesn't have an equivalent situation.
Yes, the C++ committee can receive a proposal for new language syntax such as "std::unique" and after some back & forth commentary and debate, they say "approved" and then it's up to each C++ compiler vendor to then go and independently implement it on their own timeline. In contrast, if someone proposes "C++ should have a package manager", exactly _who_ will implement and maintain the canonical repo mirror? This is not independent lines of work that GCC, Clang, Microsoft, and Intel can do on their own. Even if we hypothetically extend the website "isocpp.org" to actually start hosting the canonical C++ repos instead of just blog posts about syntax proposals, _who_ is paying for it? Again, there is no single entity like Joyent/Mozilla/GoogleInc that raises their hand and says, "We'll set it up". I suppose we could imagine that the major players like MS+Google+Apple all contribute to a shared fund to pay for the repo mirror -- and the salaries for devops to remove malicious uploads -- but notice no other major language package manager Javascript/Rust/Go had to do it that way. So we have that friction of coordinating multiple corporations. Even if that website was set up, many existing C++ library writers (that existed for decades before a C++ package manager) wouldn't bother uploading their code to it. So that's another friction. E.g. Conan is supposedly the current winner of C++ package manager mindshare and ffmpeg is not on it.
I think the disagreement is rooted in how we compare ISO C++ committee vs Google Inc. To me, releasing a C++20 language specification* does not say anything about implementing a canonical repo* so that a command line tool magically works the way people expect.
EDIT to reply: >You can have a package manager without having a discovery tool or a central repo.
This means your conversation is focusing on the tool which isn't the abstraction I'm emphasizing.
>Package discovery tools are not very relevant to the discussion.
It's relevant if the particular person wondering "why C++ doesn't have a package manager?!?" uses a mental of model of how npm and cargo work. They don't have to know if it's github vs gitlab vs somewhere else. The tool just works without thinking about the location. That's what a canonical repo as a default convention for the client tool provides.
> This means your conversation is focusing on the tool which isn't the abstraction I'm emphasizing.
Your edit implies that I'm talking about a useless tool that can't do anything in the absence of Google, which simply isn't true.
The Go Modules tooling does not depend on any central resource to work. Google could shut down tomorrow, and nothing would change for existing projects. The go CLI tools would still be able to find, download, and verify the dependencies. I would still be able to add new dependencies, and the tooling would be able to fetch those.
What are you talking about, if not a functional package management system? Google's websites are nice, but they're not required for everything to Just Work.
Anyone in the C++ community could stand those websites up at any time after the package management tooling came into existence. They're not required for the functionality of the package manager.
> It's relevant if the particular person wondering "why C++ doesn't have a package manager?!?" uses a mental of model of how npm and cargo work. They don't have to know if it's github vs gitlab vs somewhere else. The tool just works without thinking about the location. That's what a canonical repo as a default convention for the client tool provides.
Go's CLI tooling literally doesn't provide any way to search for packages at all. You may think it's a requirement, but it's really not! Go requires you to know where the dependency is located, because Go sure doesn't unless you tell it!
It feels like I'm really awful at explaining things.
>Go's CLI tooling literally doesn't provide any way to search for packages at all. You may think it's a requirement, but it's really not!
You're still misunderstanding the level of abstraction I'm emphasizing for what a "package manager" means to many people.
Let's dissect the following command from the Javascript ecosystem:
npm install react
Notice that the end user does not need to know whether React is hosted on GitHub or GitLab or Facebook's own servers. He doesn't even have to do a google search. The npm command just "magically" gets the React library.
Exactly _how_ does npm do that? From _where_ does npm fetch? The _how_ & _where_ is what the majority of my comment is about. All your explanations of Go not working that way does not address that mental model at all.
So you have 2 concepts in a "package manager":
(1) npm, the client command line tool
(2) the canonical default repo that npm tool points to -- and it's a virtuous cycle of easy use and trust because almost everybody publishes to it. It has grabbed mindshare.
You keep saying Go doesn't need (2) but I'm saying that doesn't change the fact that many mentally include (2) of what a comprehensive package manager _is_.
I don't understand what's confusing about this... it's literally specified in the name of the package.
Russ Cox is hosting his packages at rsc.io, which is a personal domain name he owns. If you visit it with a normal browser, he just kicks you over to pkg.go.dev, because he doesn't want to put in the effort to make a website for your human consumption. He's just hosting some packages there.
I really, really feel like you need to spend some time with Go Modules. You don't really seem to be getting the decentralized nature of it. But it works, and it works well!
In this case, Russ Cox has a meta tag there that tells the Go tool to download it from GitHub: <meta name="go-import" content="rsc.io/quote git https://github.com/rsc/quote">
But there's nothing stopping him from actually exposing a git repo at https://rsc.io/quote, instead of just exposing a redirect.
By telling people to use that package URL, he has the flexibility to change how and where he hosts the package in the future.
Sure, someone would complain. Just because some people would complain about the absence of one feature that you say is impractical to implement doesn't mean you should avoid implementing the rest of the thing. That's the classic "throwing the baby out with the bathwater" thing. The benefits of a standardized package manager seem worth a few people complaining. I'm sure someone somewhere would probably even complain that they would rather be writing JavaScript or another, non-C++ language, no matter how good the C++ package manager is.
Go has proven that a good package manager can work without that feature. You say that feature is something the C++ Committee could never tackle. My whole statement has been "fine, learn from Go!" Instead, you keep harping on this nice-to-have feature and saying it can't be done.
Package management is solvable in a way that suits C++. It seems inevitable that standardized package management will eventually happen for C++.
I understand your perspective now, but I just don't think I agree with it.
> In contrast, if someone proposes "C++ should have a package manager", exactly _who_ will implement and maintain the canonical repo mirror? This is not independent lines of work that GCC, Clang, Microsoft, and Intel can do on their own. Even if we hypothetically extend the website "isocpp.org" to actually start hosting the canonical C++ repos instead of just blog posts about syntax proposals, _who_ is paying for it? Again, there is no single entity like Joyent/Mozilla/GoogleInc that raises their hand and says, "We'll set it up".
Literally no one is required to do any of that. That is the answer. Plain and simple.
> I think the disagreement is rooted in how we compare ISO C++ committee vs Google Inc. To me, releasing a C++20 language specification* does not say anything about implementing a canonical repo* so that a command line tool magically works the way people expect.
But I'm saying that's not how Go works at all. The dependencies are hosted on GitHub, GitLab, or wherever else.
There is no central package repo. There is no "canonical repo".
Package discovery tools are not very relevant to the discussion. You can have a package manager without having a discovery tool or a central repo. Searching GitHub to find a C++ package, then adding that repo as a dependency of your current project seems like it would be entirely reasonable, if C++ had a standard package manager that worked. Some community members might build a website to help you find popular packages... but that discovery tool doesn't interact directly with the packages at all.
proxy.golang.org is a proxy. No one publishes packages to it, and you don't have to use that proxy. You can use no proxy at all, which was the default once upon a time, or your company can host a proxy, or you can potentially find some random third party proxy online. The proxy isn't where packages are hosted -- it's just a means of accelerating downloads, if GitHub were slow, for example.
C++ code is hosted in a myriad of locations. The Go approach is to specify the GitHub repository that you're depending on, and that repo will be cloned by the package manager in your terminal. The `go.sum` file contains hashsums to verify that the dependency you downloaded is untampered with since the last time you fetched it, and those hashes can also be used by any proxy that happens to be used.
Go's package management system is truly distributed. It isn't centralized at all. Yet it still supports SemVer, downloading the correct, exact version of a dependency, checking the integrity of dependencies, recursively collecting dependencies of your dependencies, etc. All the features you would expect out of a package manager.
Unlike Cargo in Rust, someone can delete one of these repos from GitHub and cause a real mess. `go mod vendor` is an option for anyone who prefers to vendor their dependencies.
Google has certainly provided some nice web tooling around the Go Modules system, but none of it is integral to the type of package manager that I'm proposing would suit the C++ dependency model. Go's package manager is very distinct from what you were discussing with Cargo, NPM, and others. It's much more attuned to the problems that C++ faces, and it walked a similar path to what C++ will inevitably have to do.
Which honestly mostly works well, except when it doesn't. There are at least a few module/version combos where people have shifted a version tag on their repo over time, leading to the (unfortunate) reality that you end up with different modules depending on if you fetch it through a proxy, or directly. Not entirely surprising, this can (and will) cause build failures.
Source: I build lots of Go modules for fun (well, specifically, I have some automation to do it for me) and notice these things, when I get more failures than expected. http://github.com/vatine/gochecker for anyone wanting to play along from home (the 1.16 release report is on hiatus, as there was a LOT of things that didn;t work smoothly this time).
Then give me a package manager that can handle cross-compilation well.
Currently, it seems almost nobody is taking that into account when packaging their wares for consumption by CMake, or distribution by Conan. Or if they do give it some thought, it always ends up making dubious assumptions, like "Clang == Linux", or "MSVC == Windows".
I have one at work that does okay. If your project is based on cmake I can create the package very quickly, though most projects still don't create a cmake configuration file. If your project isn't cmake - well at best a day, and often I will spend weeks fighting your half baked build system that doesn't understand the common things I want to do. (in practice autotools projects have the options to cross compile but it doesn't actually work...)
I actually extremely dislike language specific package managers. I'm on Linux, the packages should be in my package manager. I don't want to maintain multiple package managers. nmp is actually the worst here.
> I actually extremely dislike language specific package managers. I'm on Linux, the packages should be in my package manager. I don't want to maintain multiple package managers. nmp is actually the worst here.
As a user of software that doesn't care how it's built, sure. But system package managers are not a solution for general development with C++, or any other language.
If I want to use C or C++ to create software, how do I use libraries that aren't available in a system package manager? What if I need a version of a library that's not available in my system package manager? There are answers here but they aren't good answers (build from source, using whichever of N build tools the project happens to use, or hope there are prebuilt libs hosted somewhere)
Relying on system package managers to contain dependent libraries makes cross-platform development a complete PITA (more that it already is). Now you need the specific versions of all your libraries in package managers on all platforms, which is a complete non-solution for real development.
Sorry, but I'm really not sure what you mean by "solved".
Nix is yet another (language agnostic) package manager with certain tradeoffs. But, if there is not an available Nix package for a specific version of a library I need to use - I'm out of luck.
Nix is not a build tool designed to work with arbitrary or latest development versions of libraries, for example. And, it will never solve that problem even _if_ it is technically capable of doing so - because there is no force in the world that would get all projects in all languages to use it.
> Nix is not a build tool designed to work with arbitrary or latest development versions of libraries, for example.
No, that's exactly what it's designed for and exactly how we use it. And it's not a build tool (it just calls your existing build tools under the hood), it's a system for keeping different versions of dependencies installed at the same time without errors.
> because there is no force in the world that would get all projects in all languages to use it.
You can write your own wrappers for the projects that are missing. It's a simple and idiomatic process.
> Nix is not a build tool designed to work with arbitrary or latest development versions of libraries, for example
Well it sort of is and it sort of isn't. The great thing about Nix is how the provided packages remain malleable. You can usually quite easily make a small override to a provided package to make it build from a specific revision of the source you desire, or add a custom patch, and Nix will just build it all for you then & there. Then you can go and rebuild bits of the rest of the distribution that depend on that using your custom version. If you so want.
Also, i'm terrified of this idea of "library-manager download code from internet and run on this machine", without all the tests and QA of individual dependencies like we have in Linux packages.
Also, i've seen so many times people adding dependencies to projects because they did not know the standard library already had what they needed. I get it, it is easier to "pip install foo" than to look for "foo" in the docs. I don't think any sane person can learn everything that is available in the standard library, but searching the docs is always insightful.
The problem is that system-specific package managers are an obstacle to making portable programs.
Even within Linux and BSD there are many flavours of package managers with slightly different naming schemes for their packages.
This fragmentation makes it impossible to have dependencies that just work. You need to either make users install things manually or every author has to probe multiple package names/locations using multiple tools.
Language-specific managers support all of the OSes and just work, especially for users of macOS and Windows (telling people their OS sucks may state a true fact, but doesn't solve portability problems)
The Linux model of package management doesn't work for newer languages. In particular it is heavily reliant on dynamic linking, which tends not to work when you have (a) an unstable ABI (b) generics (c) a culture of static linking.
It works fine, you just ship the static libraries. With static linking your binaries won't have dependencies anyway.
That's not to say the static linking craze is a good thing. We'd be far better off finding a way to dynamically link templates, so you get the security benefits of automatically updated dependencies that dynamic linking gives you.
Code library managers don't belong inside OS package managers (because you want hermetic builds), unless maybe you have some Nix-live multi-manager that can provide many environments.
> Rust has package management, but not the ecosystem yet. On the other hand, C/C++ has the ecosystem, but no standard way to easily draw from it.
I take your point, and I share your desire for a canonical, Cargo-like package manager and build tool for C++ (it's one of the reasons I pivoted out of C++ development); however, I don't think C/C++ "has the ecosystem" these days. It certainly has an ecosystem--C/C++ dominates its own niches, but there's a big world outside those niches and there aren't good packages for much of it. Meanwhile, Rust is growing like a weed both inside and outside of the C/C++ niches, and the package manager largely enables that rapid growth. Also, Rust has a good interop story for C/C++, allowing it to leverage the existing C/C++ ecosystem. Anyway, I hope this doesn't read as contrarianism--I just thought it was an interesting distinction.
Most of the places where C++ doesn't have good libraries I wouldn't want to use Rust anyway, that is the domain of managed languages, GUIs, distributed computing, Web development.
And for the stuff I use C++ for, COM/UWP, Android NDK, GPGPU shaders, Unreal/Unity, Rust tooling is yet WIP or requires to leave the confort of the existing C++ frameworks and IDE integrations.
What’s wrong with using Rust for “GUIs, distributed computing, web development”?
In the case of a GUI, I’d expect a modern Rust GUI toolkit binding to look like any other GUI toolkit binding: an FFI-like abstraction that parses its own declarative view format, and exposes handles from those parsed views for native controller methods to bind to. Y’know, QML, NIBs, XAML, those things. This kind of GUI toolkit doesn’t exactly have high requirements of the language it’s bound to. (And I don’t believe many people want the other, procedurally-driven kind of GUI toolkit in the year 2021.)
Re: distributed computing — I can see the argument for Rust being the antithesis of easy network “rolling upgrade” (e.g. via being able to recognize known subsets of unknown messages, ala Erlang); but pretty much all languages that support distribution are very nearly as bad in that respect. (Only the languages that have distribution that nobody else actually uses — e.g. Ruby, Python, etc. — are on Erlang’s side of the spectrum in this regard.) But in terms of pre-planned typed-message version migrations, Rust can do this more idiomatically and smoothly than many other languages, e.g. Go, Haskell, etc.
Re: web development — there’s actually a lot of activity in building web frontend SPAs using Rust compiled to WASM. Started with games, but has expanded from there. Not sure about web backends, but the argument is similar to distribution: you need to do it differently in a static compiled language, but of static compiled languages, Rust is really a pretty good option.
The productivity hit produced by having to deal with the borrow checker and the design constraints it imposes into application architecture.
I won't take a GUI framework without a graphical designer, or a component ecosystem from companies selling GUI widgets in 21st century.
Distributed computing, again when thinking about distributed calls a la Akka, Orleans, SQL distributed transactions, I rather have the productivity of a GC.
Web development with Rust is nowhere close to the stack provided by JEE, Spring, ASP.NET, Adobe Experience Manager, Sitecore, LifeRay, Umbraco, enterprise RDMS connectors, ...
Rust best place is for kernels, drivers and absolute no GC deployment scenarios.
> I won't take a GUI framework without a graphical designer, or a component ecosystem from companies selling GUI widgets in 21st century.
Well, yeah, what I’m saying with “these types of modern frameworks don’t impose very many constraints on the language” is that there’s no reason that Qt, UWP, Interface Builder, etc. can’t support Rust (or most other languages, really), because in the end the tooling is just generating/editing data in a declarative markup language, that the language’s toolkit binding parses. You don’t have to modify the tooling in order to get it working with a new language; you just need a new toolkit binding. Just like you don’t need to modify an HTML editor to get it to support a web browser written in a new language. Qt et al, like HTML, is renderer-implementation-language agnostic.
> Distributed computing, again when thinking about distributed calls a la Akka, Orleans, SQL distributed transactions, I rather have the productivity of a GC.
I think I agree re: the productivity multiplier of special-purpose distributed-computing frameworks. I don’t think I agree that it’s a GC that enables these frameworks to be productive. IMHO, it’s the framework itself that is productive, and the language being GCed is incidental.
But, either way—whether it’s easy or hard—you could still have one of these frameworks in Rust. Akka wasn’t exactly easy to impose on top of the JVM, but they did it anyway, and introduced a lot of non-JVM-y stuff in the process. (I’d expect that a distributed-computing framework for Rust would impose Objective-C-like auto-release-pools for GC.)
> Web development with Rust is nowhere close[...]
Web development with Rust isn’t near there yet, but unlike distributed computing, I don’t see anything about web development that fundamentally is made harder by borrow-checking / made easier by garbage-collection; rather the opposite. I fully expect Rust to eventually have a vibrant web-server-backend component ecosystem equivalent to Java’s.
> Rust best place is for kernels, drivers and absolute no GC deployment scenarios.
Those are good use-cases, but IMHO, the best place for Rust is embedding “hot kernels” of native code within managed runtimes. I.e. any library that’s native for speed, but embedded in an ecosystem package in a language like Ruby/Python/Erlang/etc., where it gets loaded through FFI and wrapped in an HLL-native interface. Such native libraries can and should be written in Rust instead: you want the speed of a native [compiled, WPO-able] language; but you also want/need safety, to protect your HLL runtime from your library’s code that you’re forcing to run inside it; and you also want an extremely “thin” (i.e. C-compatible) FFI, such that you’re not paying too much in FFI overhead for calls from your managed code into the native code. Rust gives you all three. (I see this being an increasingly popular choice lately. Most new native Elixir NIF libraries that I know of are written using https://github.com/rusterlium/rustler.)
I would use Rust for distributed computing and GUIs, and I wouldn't be surprised if it begins to break into the graphics/gamedev world in the next 5 years. Agreed that Rust is still immature in those areas today, but it seems to be on a pretty aggressive trajectory and it's only a matter of time before Rust begins chipping away in those domains.
I did some real-time embedded development (including distributed embedded) in a past life in C and C++, and I really expect Rust to break through in that domain in a big way even though it's incredibly conservative (C++ is still the new kid on the block). It will take some time and it's never going to "kill" C or C++ in that domain (especially considering all the hardware that exists that LLVM doesn't yet target), but I think Rust will carve out a swathe of the embedded space for itself.
Sure if you like to do stuff by hand, I rather use visual design tooling (think Qt Designer, Microsoft Blend) and have bigger fish to fry in distributed network calls than who is owning what, instead of using Akka, Orleans or Erlang.
> I rather use visual design tooling (think Qt Designer,
Oof, I did professional Qt development and Qt designer was basically a joke. Not sure if it improved, but I've never experienced a visual design tool that saved me time. Not that they can't exist, just that the implementation is usually too buggy to justify itself. I don't enjoy debugging XML that gets compiled to C++ (I think it's compiled, anyway--maybe it's parsed at runtime... I forget). In whatever case, if you build a visual design tool for C++, you can build one for Rust as well.
> bigger fish to fry in distributed network calls than who is owning what
Agreed that I don't think distributed is the sweet spot for Rust, but there are certain niches (high performance, low level, etc) where Rust would be valuable. Previously I worked in automotive which is basically a bunch of distributed embedded computers talking to each other over a CAN network, and Rust would have saved a lot of time and money. On the other end of the spectrum, you have high frequency trading where performance is so important that C++'s myriad problems are worthwhile, so certainly Rust could add value here as well.
Imagine something like Swift UI for Rust, including the live preview.
Now imagine how to implement such designer in a way that supports component libraries, without having the burden of using Rc<RefCell<>> everywhere, while allowing the user to rearrange the component widgets in any random ordering.
It's a tool to manage C++ dependencies (using CMake), created and maintained by Microsoft. A lot of open source projects are supported (you can see part of the list here: https://github.com/microsoft/vcpkg/tree/master/ports).
I did, about a year ago. The usability was questionable.
Their main workflow appears to be, all developers use that thing, everyone building packages from source code and using their own binaries. For large dependencies that’s a large waste of time if more than 1 person is working on the software. It’s possible to export built libraries as nuget packages, but these are tricky to consume.
Another thing, these ports (where they applying patches to third party open-source code to squeeze them into vcpkg) are fragile. I remember cases when packages didn’t build, either at all, or subject to conditions (half of what I tried was broken when I only wanted release configurations).
Without a standard ABI, having c++ binary packages is a huge pain, requiring multiple artifacts for every permutation of compiler, os, and platform. It's less painful today than in the past, simply due to fewer compilers, OSes, and platforms, but it is still a problem.
A common ABI doesn't save anything as we still need to build for ARM, and x86 (MIPS, RISCV are also out there and may be important to you). Those processors all have different generations, it might be worth having a build for each variant of your CPUs. Once you take care of that different ABIs are just a trivial extension. RPM and .deb have been able to handle this for years.
All developers on that team used 1 compiler (VC++ 2017 at that time), one OS (Windows 10), one target platform (AMD64). Compiler/linker settings are shared across developers.
I wanted vcpkg to export just the required artifacts (headers, DLLs, static libraries, and debug symbols), so only 1 person on the team (me) is wasting time building these third-party libraries. The team is remote, upload/download size needs to be reasonable.
Maybe a difference is that C++ can be very, very slow to build. And C++20 will likely results in even longer build time now that you have concepts and can have exceptions and allocations in a constexpr context.
Rust's package management is actually a downside to my adoption. I have a lot of C++, a home-grown package manager, and a large cmake based build system. Rust wants to replace all this, but that means shelling out to Rust's build system, which is a bit of a messy situation and means I need to learn a new build system with the language. (not hard, but another thing to learn). Our home grown package manager means we have our own local copy of everything - I have a hard requriement to be able to rebuild software for the next 15 years (there is a closet someplace with a windows XP service pack 1 PC so we can build some old software - God only knows if it will boot anymore). In the embedded space we need to support our old code for years, and you can't always upgrade to the latest.
"cargo vendor" enables you to embed your entire dependency tree into your repo instantly, and compilation will work from that. Over half of your comment seems to have been predicated on the assumption that this either wasn't possible or wasn't easy... so I think your perceptions of Rust's package management system are more of an impediment to you than the actual package management system.
While I will admit ignorance about the details of Rust, what you described doesn't solve my problem. We do not believe in mono-repo here, and have broken our system down into lots of small repos with custom tools to manage that. Checking in a copy of the dependency tree into each repo is not the right answer. I'm sure I can make this all work, but everything I've heard about Cargo is it will fight the way we have setup our system. We are not changing, while there are things I'd do different (use Conan - but that didn't exist until just after we rolled our system out, and is just enough different that it will be hard to switch), the system works for our needs.
Cargo will allow you to solve this problem half a dozen ways. I’m certain at least one of those ways would fit the patterns you’re describing.
But you keep coming back to this idea of how Rust has to fit your workflow perfectly and you’re unwilling to make any changes to have things work better with Rust...
If you’re unwilling to change anything at all, then it’s laughable to imply that you would use an entirely different programming language for anything, even if it fits your workflow exactly.
So, I just don’t see the purpose of this discussion. You’re basically saying that you’re not going to use Rust, no matter what Rust does or does not do. That’s neat?
It sounds like you're implying git submodules are actually a good thing... I think you're the first person who has implied such a thing to me before. Everyone I actually know agrees that submodules are basically never the right solution or a pleasant solution.
But, to your question, no. Where would the submodules even point? The dependency source code artifacts are stored "immutably" (except for takedown notices or extreme abuse cases) on https://crates.io. They aren't git repos, and there's nowhere for git to point.
Yeah: using submodules makes maintaining vendor patches (which, FWIW, I pretty much don't do and will move mountains to avoid... but like, I totally appreciate why people do them) really natural and easy. Like, you don't just want a copy of the code: you want to be able to participate in the development of the upstream code with the same level of tooling that they have, and submodules does that.
The approach here would be to declare the dependency on the git repo directly. Vendoring is still going to copy the stuff you're building into your project, but you'd keep those patches in the repository of the dependency, not on your vendored copy.
The key thing here is being able to do it through multiple levels of dependency, for which I see someone else provided me an answer that is actionable! \o/
People definitely have strong opinions on submodules, but it is nowhere near so one-sides: a ton of people hate them, and a ton of people swear by them. FWIW, all of the Rust libraries I use are available as git repositories. With many other package managers, I can tell them "don't use the upstream copy from the package repo: use the copy I have in this folder" in a trivial manner. I thereby don't really want "automation" around either downloading the code for me to mirror or for the submodules I want: I want to set it up and then configure it so it is all "honored"... and I could totally see the feature you talking about somehow only working one way (with automatic copies) instead of being flexible.
Yes, it can pick dependencies from checked out submodules, or git URLs directly. It has ways to patch individual dependencies anywhere in the dependency tree, and multiple ways to mirror or replace the whole crates.io index. It's pretty flexible in this regard.
I’m not here to argue one way or another on dependency vendoring. The person I replied to was making an inordinately big deal about how they keep code around forever and it compiles decades later, as if Rust dependencies were some ephemeral thing that would break your code by next Monday!
If they want to reproduce their workflow using Rust, cargo allows vendoring and many other solutions.
I think the barrier to entry in the problem domain for C++ is much higher than something like nodejs. Installing dependencies is the least of one’s worries there.
Also, how many dependencies are we talking about? Node apps have a million dependencies for, I think, stupid simple stuff that should just be reinvented in a given codebase. In a C++ app too many dependencies invites incompatible stylistic choices which I think will turn to a Frankenstein codebase.
In Go this isn’t a problem because of “go fmt” plus a simple language at its core.
No matter the implemented package manager solution, it has deal with different packages types: from a single class library (a single hpp file) up to monster libraries like ffmpeg.
In the case of ffmpeg, what the package manager should do? Download the sources and all its dependencies and build from scratch? This is very difficult and time consuming.
Because right now the alternative is going to the ffmpeg website, download and include the dll (and lib) or .so and a couple of .h files to your project. And that's pretty simple to me.
It's not that the package manager fixes the problem, it's that having 1 or maybe 2 or 3 canonical or popular package managers gets the implementer to fix the problem.
The implementer, who has extensive knowledge of their own build system runs that aspect and creates a package that conforms to a universally expected output.
It's an incredible difference going from C++, where you end up in the details of all kinds of repos and build systems, to something like C# with Nuget packages where it's a simple command or single click to start using someone else's code.
Consider that C/C++, being highly portable, has support for many platforms and architectures, including the possibility of cross-compiling.
I guess if a package manages works on all those architectures and platforms, then the implementer would have to support all of them, and it's not always the main objective.
> guess if a package manages works on all those architectures and platforms, then the implementer would have to support all of them,
Other "highly portable" languages handle this by simply having the developer include a manifest of the platforms their library works for. The package manager only shows compatible packages for the targeted platform.
"FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created."
After that:
"It contains libavcodec, libavutil, libavformat, libavfilter, libavdevice, libswscale and libswresample which can be used by applications. As well as ffmpeg, ffplay and ffprobe which can be used by end users for transcoding and playing"
Conan is probably the flagship C++ package manager and supports multiple build systems including cmake. Nuget/vcpkg is also usable but does not come with build system integration.
Vcpkg has great CMake integration. Further, the model of Conan of distributing a bunch of binaries honestly seems like the wrong approach for C++ where you have to juggle all different sorts of compilers, triples, and ABIs. We use a completely custom toolchain which pretty much rules out Conan.
The one annoying thing about vcpkg, though, is that all packages are described in the vcpkg source tree. There are no “repositories”. Customizing or adding custom packages requires using the somewhat annoying to use overlay system.
I’d prefer some sort of hybrid between the two, with packages distributed as source code but pulled from a repository. I believe this is how Rust’s Cargo works.
We'd also need the one operating system on the one architecture. Perhaps the central planning committee can make that a goal for their next five year plan?
Python and Ruby only work on the Python and Ruby interpreters, respectively, and those require an OS-specific way to install them. They work on multiple OSes and architectures as well as JavaScript or HTML.
Go and Rust work only on a very very limited set of OSes and architectures. That's fine if you're targeting one of those, but it turns out the vast majority of computers in the world are not vanilla rice-pudding desktop systems or vanilla rice-pudding desktop systems adapted for the server room. The argument that some other tool solves a limited set of problems with your tool in a limited and limiting way is a poor one if you're trying to promote a universal solution.
Java runs on many platforms (and billions of devices as Sun used to love to point out), yet packaging and dependency management are pretty much solved problems.
Where there's a will, there's a way. In the C/C++ community there's no will. It's time they admit that to themselves and everyone else.
But it's not necessarily centralized, Java package management tools can use many third party repositories, if needed, and can also use proxy/cache/mirror systems where for example a company can point all their package manager just to their official company repo and everything goes through it.
BTW, Java's not interpreted, it's compiled. Just not to native code.
