Once again I must object. I don't want to rehash all of #16181 but people may not read all my comments so I will discuss here briefly. The --ca-native option imports CAs in addition to other CA import options. Enabling it by default is going to change the behavior to always import those CAs even when the user has selected a different CA option. This changes curl behavior in a way that is undesirable (in my opinion) and, objectively, unexpected.

Apple did something similar in their SSL library (fork of LibreSSL I think it was) to import native CAs even if the user has specified their own and I thought it was a security issue, but Apple didn't think so. I think @bagder commented on this as well somewhere.

Adding a build-time fallback, like we already do with CURL_CA_FALLBACK, could work.

diff --git a/lib/vtls/openssl.c b/lib/vtls/openssl.c
index a2ab831..aa4288d 100644
--- a/lib/vtls/openssl.c
+++ b/lib/vtls/openssl.c
@@ -3384,7 +3384,11 @@ static CURLcode ossl_populate_x509_store(struct Curl_cfilter *cf,
        https://stackoverflow.com/questions/9507184/
        https://github.com/d3x0r/SACK/blob/master/src/netlib/ssl_layer.c#L1037
        https://datatracker.ietf.org/doc/html/rfc5280 */
+#ifdef CURL_NATIVE_CA_FALLBACK
+    if(!ssl_cafile && !ssl_capath && !ca_info_blob) {
+#else
     if(ssl_config->native_ca_store) {
+#endif
       const char *storeNames[] = {
         "ROOT",   /* Trusted Root Certification Authorities */
         "CA"      /* Intermediate Certification Authorities */

The user can already put --ca-native in their curlrc. Also CA native on Windows does not work properly unless the OS already has the root certificates in the store, and that is not always the case with Windows. It gets the root certs on demand, see #12303.

Unless I'm missing something the issue you raise is only present if curl is implicitly
picking up a undesired CA bundle file from somewhere on the disk (or from embed).
This is controlled by the user/packager by providing such file (or not), or explicitly
building with an embedded bundle, or explicitly passing a CA bundle via the
command-line (or curlrc, or envs).

The point of this PR is to make these implicit (and in many cases insecure), or explicit
ways of passing a CA bundle unnecessary, for most uses-cases.

I think explicitly passing extra CA roots is still a valid use-case. Disabling them
completely would break these options, which would be incompatible and IMO
unexpected.

(In case of Apple, they patched the TLS library, with an undisclosed source
code and unconditionally adding they CA bundles with no option to disable
or override. It's a different case AFAICS.)

If one doesn't want curl to pick up on-disk CA bundles by default, this build
option can be set in combination with CURL_DISABLE_CA_SEARCH=ON.
Ensuring that only the native CA is used by default. If one is explicitly provided
via the command-line, curlrc or env, it's used in addition.

edit:

The user can already put --ca-native in their curlrc. Also CA native on Windows does not work properly unless the OS
already has the root certificates in the store, and that is not always the case with Windows. It gets the root certs on
demand, see #12303.

They can, but in practice few people seem to be doing that.

As for #12303, either we can't really do anything about it and it affects every
TLS stack out there besides Schannel (I suppose?). Or maybe there is
a solution, which is probably easier to figure out with more exposure.

I don't understand your logic around this. The user can already use the curlrc to set default options. Adding this is like adding --ca-native to everyone's curlrc and their OS store's certificates would be used in addition to whatever CA options they specified (or the autodetected curl-ca-bundle.crt, CURL_CA_BUNDLE env etc). Users are not going to expect that. IMO the only advantage over a fallback is it compensates for bad user config.

I don't understand your logic around this. The user can already use the curlrc to set default options. Adding this is like adding --ca-native to everyone's curlrc and their OS store's certificates would be used in addition to whatever CA options they specified (or the autodetected curl-ca-bundle.crt, CURL_CA_BUNDLE env etc). Users are not going to expect that. IMO the only advantage over a fallback is it compensates for bad user config.

The goal of this option is to make it unnecessary to configure CA roots
manually (or only when really needed, e.g. for a custom cert). Because
the manual (or file-based) configuration is insecure by default, annoying
to configure, nearly impossible to configure securely, and as a consequence
in many cases it's not configured at all or not configured securely.

With this option enabled, it's possible to drop these manual methods.

Making CA roots actually useful and do what they were designed to do:
securing connections. And without being an annoyance.

Would the pre-existing manual configs cause issues if they remain enabled?
They may keep reducing security, otherwise they just remain redundant certs.
The file-based ones can be disabled at build-time, which is probably
prudent to do when enabling this option.

But, it's just an option, no one is forced to enable it at build time, and it's
disabled by default. It's also disablable via .curlrc. I don't see an issue with
providing an option for this.

@jay
The most use case of this option is on windows together with http3. Because now the relevant option is to do with backend that is not default with native certificate, like openssl.
Most of the people don't know about this option, and when they try to do ssl with openssl on windows, they think it broken.
I have this experience, also you can take a look on the my vcpkg PR with other people do it the same.
And now you want to do http3 on windows with backend that the native certificate is not enable by default, meaning you enable this option every time you do ssl with that backend, it same, as to compile it with default option.

Please understand that there is other people want to use curl different the way you want it.
You are welcome to enter my PR to read more.

Sure I understand this but if you switch to an SSL backend that is not the native backend (eg switch from Schannel to OpenSSL) then it is not going to be the same as using the native backend. OpenSSL takes a certificate bundle or certificate locations.

I agree. If we talking about specific on windows, openssl with the option enable native certificate, it not the same as using schannel. But I am talking about specific on windows, doing http3 on windows, then you need to use backend different from schannel, for example openssl, then you must use enable native certificate option always for doing ssl. For this use case we need / want enable native certificate always in the compilation time.

But, it's just an option, no one is forced to enable it at build time, and it's
disabled by default. It's also disablable via .curlrc. I don't see an issue with
providing an option for this.

The issue is it overrides the user's expectation by using those OS certs in addition to whatever certs they specified (or were detected as documented). I don't understand how a fallback is not sufficient for all of this. At this point I think we are just repeating ourselves and we disagree.

I can only speak for myself, but my expectation has always been that HTTPS
URLs just work. If I don't have to provide the browser's CA bundle myself
manually (or let a random one hang around somewhere in the PATH), it'd
be a relief, rather than overriding my expectations.

What actual issues can this cause in practice? Would these affect most users?
Would the effect be so devastating to prevent building a secure-by-default curl
configuration, as an option?

I can only speak for myself, but my expectation has always been that HTTPS URLs just work.

Sure I agree. curl is used many ways though and users may specify their own CA certificates or a limited set or location, they may not be passing a bundle with all known root CAs like cacert.pem to access every HTTPS URL. The user has some intention to use their CA certs, or some CA certs were autodetected as we document (we can assume some intention there). In practice the build option in this PR would make those builds of curl behave in a way that goes against the documented behavior of the CA options. I know a better out-of-box experience is desired but this is not the way to go about it.

As distro packagers, we just want to be able to build full-featured, out-of-the-box curl.exe with cmake --build, rather than need a bunch of non-standard steps to avoid building a garbage that doesn't work at all for the end users.

As curl.exe end users, we just want http3 enabled curl.exe to work just like Windows builtin curl.exe.

If someone feels there is something wrong with this, just disable the option when building, rather than block merging this PR. curl should not sacrifice the experience of the vast majority of users because of someone's edge use case.

Even if this PR was blocked, we would have patched curl anyway to get actual usable curl.exe.

Since this is a build option and not even set by default, I am in favor of it. curl users on all platforms are used to curl https://host using the system default CA store, and I think it is safe to assume that curl users on Windows would like that too.

Since this is a build option and not even set by default, I am in favor of it. curl users on all platforms are used to curl https://host using the system default CA store, and I think it is safe to assume that curl users on Windows would like that too.

That could be done with a fallback as I mentioned earlier in the thread. There's no reason to the change the user's --cacert choice. Why do it that way? I am very much against that and I'm surprised you aren't as well. Let's say we do this and we end up with builds of curl distributed on vcpkg that have this behavior enabled, and the user specifies --cacert private.crt with the expectation that they are connecting to their corporate or government server and now the OS store certs are used as well. It is backdoor behavior if a big govt wanted to cut the route. Do you not remember https://daniel.haxx.se/blog/2024/03/08/the-apple-curl-security-incident-12604/

There's no reason to the change the user's --cacert choice.

No there isn't. I wasn't aware this did. --cacert should override the default. I was under the impression that this changes the default, which is that the title says.

Looking shallowly as this proposal and

At the end, both X509_STORE_set_default_paths() and gnutls_certificate_set_x509_system_trust() serve exactly the same purpose and I don't understand why they are treated differently...

@michael-o don't mixup native-ca and "ca-fallback" though. The first one uses "the native CA store" (which primarily is a thing on Windows and macOS) while the second option more mysteriously uses "the TLS library's fallback" and the second is only an OpenSSL (hack) - that most typically would be used when you rely on some specific magic behavior of your OpenSSL.

We already have native-ca support in the OpenSSL backend, as this PR is about an option to set that by default.

@michael-o don't mixup native-ca and "ca-fallback" though. The first one uses "the native CA store" (which primarily is a thing on Windows and macOS) while the second option more mysteriously uses "the TLS library's fallback" and the second is only an OpenSSL (hack) - that most typically would be used when you rely on some specific magic behavior of your OpenSSL.

We already have native-ca support in the OpenSSL backend, as this PR is about an option to set that by default.

Ah, thanks for the clarification.

Let's say we do this and we end up with builds of curl distributed on vcpkg that have this behavior enabled, and the user specifies --cacert private.crt with the expectation that they are connecting to their corporate or government server and now the OS store certs are used as well. It is backdoor behavior if a big govt wanted to cut the route.

If it this is true, you should add an option that call something --cacert_private private.crt, that it will not store the certificate to os certificate manager. It more simpler.

In my opinion, as a security reason, --cacert private.crt should never store the certificate in os manager, even when the ca_native is on, only if the user specific ask it, for example --cacert_store private.crt

Because you already have an api that doing undesired behavior, I think you should start to deprecated the --cacert option, and add 2 option that explicit tell the user what he doing: --cacert_private and --cacert_store

As a philosophy of curl, doing only the things the user explicit ask for. This is not the way with --cacert that why I call it undesired behavior.

This patch does nothing else than allow enabling --ca-native option by default at build-time.
It doesn't change the semantics of existing options.

If someone is concerned by big-government (or big-corp) CA's installed in the OS, the native
CA store can be disabled with--no-ca-native (and its proxy counterpart, see OP), also via
.curlrc.

It seems fair to assume that those concerned by a planted/forced/malicious CA root on their
OS are a smaller group than those who merely want a working/secure HTTPS experience out
of the box. It's also plausible that the former group is aware of the options to avoid a malicious
CA root installed on their OS, since it affects browsers, and every other tool, not just curl.

If someone has access to mess around with your system certificate store, you're screwed. They obviously have administrator rights to the entire system and monitor all your network traffic. A curl.exe that doesn't use the system certificate store doesn't solve anything.

No that's not a given. It was an example of how an attacker could take advantage of using certificates that the user did not expect. That's not to say someone hacked your certificate store. Like the user trusts CA X only to connect to server Y.

@vszakats Can this be generalized that is works for both libcurl and curl? Configured once during build and then used by all consumers w/o configuring anything on lib or tool level?

@vszakats Can this be generalized that is works for both libcurl and curl? Configured once during build and then used by all consumers w/o configuring anything on lib or tool level?

Yes, it could be extended to libcurl. The issues this PR intends to help
fixing apply to libcurl too. Perhaps even more so, because with libcurl,
there is no automatic mechanism to setup the CA roots (such as picking
it up from disk or embedding them), therefore every libcurl user have
to do the CA root distribution and updates themselves, on Windows.
Which is brittle and hard to do, esp. securely, and it wouldn't suprise me
if many would just choose to explicitly disable security instead.

IMO it'd be nice to make this work by default in libcurl too, and may also
possibly be a cleaner solution. But haven't looked into the details.
I wanted to make the smallest possible step, and the curl tool seemed
like the best place for this.

@vszakats Can this be generalized that is works for both libcurl and curl? Configured once during build and then used by all consumers w/o configuring anything on lib or tool level?

Yes, it could be extended to libcurl. The issues this PR intends to help fixing apply to libcurl too. Perhaps even more so, because with libcurl, there is no automatic mechanism to setup the CA roots (such as picking it up from disk or embedding them), therefore every libcurl user have to do the CA root distribution and updates themselves, on Windows. Which is brittle and hard to do, esp. securely, and it wouldn't suprise me if many would just choose to explicitly disable security instead.

IMO it'd be nice to make this work by default in libcurl too, and may also possibly be a cleaner solution. But haven't looked into the details. I wanted to make the smallest possible step, and the curl tool seemed like the best place for this.

I dare you say that 90+% of the users are unable to comprehend what's needs to be done and what a CA is and so on.

The idea I had is that --with-native-ca at compile time could completely replace --with-ca-fallback.

Started #18567 to clarify terminology and, hopefully, get us to agree on the usability criteria. We need this to guide our implementations.