I think that lowering the barrier to opt-in and share data is a better solution than increasing the barrier to opt out. People who would decline telemetry at a prompt are people who don't want telemetry, making the life of those people harder is really bad whether it's intentional or not.

As soon as there is just one identifiable data collected, opt-out becomes completely illegal WRT GDPR, which applies to any entity processing a European user data regardless where it is collected. Add that the privacy shield is illegal now in Europe, you better watch out for the huge fine if that ships with opt-out to any European user.

This is a legal issue, your personal opinion on the goods and bads of opt-in vs opt-out is irrelevant.

Transparent and "On by default" do not go together. Naming It transparent is like the 1984 moto "war is peace".
It is often said here that just because this is discussed and can be checked publicly is enough and noone should be taken offguard.

“There’s no point in acting surprised about it. All the planning charts and demolition orders have been on display at your local planning department in Alpha Centauri for 50 of your Earth years, so you’ve had plenty of time to lodge any formal complaint and it’s far too late to start making a fuss about it now.... I’m sorry, but if you can’t be bothered to take an interest in local affairs, that’s your own lookout. "

• Vogon Transparency.

I wrote a little about this at https://research.swtch.com/telemetry-design#opt-out.
[...]
In systems that have collection off by default, opt-in rates tend to be very low,

Are there any concrete percentages available from projects who did this before?

You say that about 16,000 samples are needed. There are more than two million Go users. So you would need about 1% of users to opt-in. Is this really that difficult to achieve?

skewing the results toward power users who understand the system well.

One does not have to understand the system well to answer a yes/no question in the installer or before starting a download from the Go website.

This leaves those who install Go via the package manager of their Linux distribution. However, Linux users are more likely to be power users who understand how to set an environment variable to opt-in. And many (most?) Linux distributions would disable telemetry by default anyway.

Second, the existence of an opt-in checkbox is in my opinion too often used as justification for collecting far more data than is necessary.

It's in the hands of the Go project to not use it as a justification for that. Furthermore, it's a question of how attractive you make it to opt-in. More users will opt-in if you can convince them they will benefit. It's a communication and marketing task, just as you need to promote participation in the annual survey.

Also, because the design collects a fixed number of samples, more systems being opted in means collecting less from any given system, reducing the privacy impact to each individual system.

Those who opt-in probably don't care about the collection frequency as long as it doesn't bog down their systems.

It would be interesting if there's an opportunity to collaborate here, although my understanding of OpenTelemetry is that it's architected around an event stream rather than the very coarse week-long counter aggregation in the design. Avoiding an event stream is one of the important ways transparent telemetry reduces its privacy impact.

although my understanding of OpenTelemetry is that it's architected around an event stream rather than the very coarse week-long counter aggregation in the design

I think you are right. We can likely make OpenTelemetry Go SDK do what you need but I think it is not worth the effort. Rolling out your own based on the design you posted will likely be simpler.

It would be interesting if there's an opportunity to collaborate here

There probably is. I wonder if your proposal can be generalized to allow any open-source project (and not just to the Go toolchain) to ethically collect usage telemetry. OpenTelemetry may be a good home for this.

Even if not using the official OpenTelemetry, it would be nice to respect some the environment variables that people have configured for other tools that use OpenTelemetry, in terms of opt-in / opt-out.

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md

That's quite a few environment variables. Which specifically are you suggesting it look for, and how should it interpret them? I'm not sure I understand why it would make sense to look for OpenTelemetry env variables without using OpenTelemetry. Do other systems do that?

Despite sharing the name "telemetry", I don't think this proposal intersects too much with how the OpenTelemetry SDKs or config (env vars) work at this point in time.

we might add targeted telemetry

I can imagine this bit being misunderstood or read out of context. This targeted telemetry would still be under the constraints of the design: only infrequent sampling of event counters with no sensitive information. I imagine that one example of targetting might be something like: many users on Windows say that their Go builds are slow, so we want to get more counters for build cache hits and misses from Windows users. Perhaps @rsc has other examples to help illustrate.

@svrnm and @lthibault correctly understand how distributed systems work; in particular, they understand that public data-collecting endpoints must accept garbage, and also that the public likes to put trash into such endpoints.

@rsc: The point of the question is not to earn a good answer, but to remind you that the question represents a problem, a fundamental obstacle to your plan and desires. I recommend modifying your desires, rather than trying to administer computers which you do not own. Also, when you say "if a particular statistic is wrong", you are presuming that you can tell whether your data represents a biased sample, which implies that you already have a prior distribution and do not actually need this data collection; or you are trusting priors which do not actually correspond to any real-world data.

The attack that sneaks through your plan is called "boiling the frog" or 'salami-slicing". An attacker would let you set up your system as planned and operate it for several months. During that time, the attacker notes the same trends that you note, including anything from likely IP ranges to expected OS and platform. Then, the attacker slowly starts creating fake reporters, using a variety of cloud services for IP diversity, and programming the reporters to reproduce your existing data distribution through random sampling. After the attacker is confident that they haven't been noticed, they start to systematically bias their reports:

• Find a vulnerability in a popular package? Chaff it out by making other packages look more popular or making it look like people are adopting the patched version.
• Has an entire distro opted out of telemetry? Then the attacker can hallucinate an entire community on that distro, including any weird practices that they might have, like shipping out-of-date packages from the distro's repos.
• Speaking of hallucinations, how does one know that a stack frame is valid? It sure would be awful to receive a bug report generated from crash telemetry which indicates that a non-crashing line of code is crashy.
• It's funny that developers use Macbooks. So, let's just artificially inflate the apparent number of Macbooks.
• It's also funny that new microarchitectures are introduced. So, let's slow down how quickly the Go team prioritizes new microarchitectures by making them appear relatively unpopular and slow to be adopted. This interacts in a positive way with sampling bias!

I read through all four of your posts (the three on telemetry and the one on sampling) and somehow you never used the word "bias". However, even mere opt-out/opt-in creates sampling bias! Please, at the very least, make this consideration part of your design; you should show how your system can fail, not just how it can succeed.

if we realized Go was misbehaving in a certain way

Also, this isn't quite the scenario we were discussing. We were instead discussing the scenario in which users tried to subvert telemetry. In our example, Go is working just fine.

The difference is important because it goes to show how opt-out telemetry inevitably produces a user-hostile game of cat-and-mouse.

@MostAwesomeDude Taking your argument to its logical conclusion, it seems like the most effective defense against such an attack would be some kind of proof-of-work system that would make it difficult for an attacker to overwhelm the honest participants. The most obvious approch would be to give up on sampling and collect from 100% of the systems, perhaps every day. In that system the "work" is the network traffic and reporting behavior, making it far more costly for attackers to overwhelm the reporting systems. Of course, that would be a very different system, and not one I want to build or use. My point is only that if you focus exclusively on attack mitigation strategies, you end up in a place that most people commenting here probably wouldn't like at all.

@rsc I don't think this is taking @MostAwesomeDude's argument to its logical conclusion at all. It is rather missing the point by proposing a technical solution that ratchets up the user-hostility another notch; users must now perform work unless they opt out! 😄

To be clear, I get that you are making this proposal in jest, but it goes to show that you've missed the point.

Also you are moving from an ad hominem to a false comparison naming once again my employer. It becomes hard to believe that this is a simple 'miscommunication'. I gave you facts, please respond on them without attacking or threatening me.

@padawan I was trying to provide a concrete example that you could go check on. Saying that consent is the only legal way to collect IP addresses under GDPR is plain wrong. Implying that the only exception is mandated traffic logging is also wrong. Legitimate interest can be a legal basis to collect IP addresses, as has been confirmed by the courts already for the case of IP collection for security reasons, or to fight piracy.

@sulewicz I don't know if other legal bases could apply to this case. I'm not saying that they could. I was only saying that this legal determination is certainly not as trivial as has been suggested by multiple people on this thread, and that "GDPR means that you need consent!" is a gross simplification: it's at best not at all clear (I'm not aware of a court decision for a telemetry use case like this proposal), and at worst completely wrong.

@TedTed while you are right that IPs can be collected for legitimate interest without an explicit consent,
Legitimate Interest is clearly defined as "neccessary for the core functionality of the application".
That Argument could be made for the Proxy and sum because it provides core security/redudancy of dependecies but Phoning home for any metric is clearly not a core functionality of a compiler/build tool.

For a case regarding IPs collected without the user consent see the Case of Google Fonts CDN where it was judged that using the CDN in any application requires user consent becuase it provides the IP of the user to the the CDN.

Legitimate Interest is clearly defined as "neccessary for the core functionality of the application".

I'm not familiar with the GDPR article or DPA interpretation that makes such claims, can you tell more? Article 6.4 says that there are a number of points to take into consideration when making a legitimate interest determination, and this determination is far more subtle and nuanced than "is it part of the core functionality". Data minimization, risks to data subjects, use of technical safeguards, and data sensitivity are all taken into account in this determination. Recital 47 explicitly says that fraud detection or direct marketing can be used to regarded as legitimate interest, and those don't seem to me like "part of the core functionality". Of course neither is directly relevant to this telemetry proposal, but they show that this determination is complex and subtle.

Again, I'm not saying "it would be perfectly fine to use legitimate interest for this use case". This would be wrong. I'm saying that it's complicated, that law is messy, and that absolute statements about compliance are not accurate nor helpful.

Thanks for bringing up the Google Fonts ruling. I agree that it's very relevant, especially since the decision was about a website disclosing IP addresses to Google Fonts, regardless of what Google actually did with these IP addresses. Elsewhere in this thread, @ianlancetaylor says:

No personal data of any sort is collected, no anonymization is required

As the Google Fonts ruling suggests, the compliance story is likely not as simple as that — even without logging IP addresses, the system still relies on processing those in order to receive telemetry reports.

https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/grounds-processing/what-does-grounds-legitimate-interest-mean_en

Data minimization, risks to data subjects, use of technical safeguards, and data sensitivity are all taken into account in this determination. Recital 47 explicitly says that fraud detection or direct marketing can be used to regarded as legitimate interest

Yes but those are considered legitimate interest only when you "need to process personal data in order to carry out tasks related to your business activities". The compiler's and build tool's activities require nothing of the sort.
GDPR is very hard to interpret correctly but several lawyers have suggested that legitimate interest applies only when a functionality (necessary for the work of the application) can not be reasonably implemented without collecting of PII.
That is why you don't see everyone abusing the "direct marketing legitimate need" to get out of GDPR consent requirements. If the main purpose of you app is direct marketing, sure, collecting PII is legitimate interest. If Direct marketing or Usage statistics are side dishes that help but not essential to the functionality of the offer they need to be disclosed and consented for.

Fraud detection can not apply because without phoning home there is no Fraud to protect From.

I bring the very hard to interpret issue because here in Europe all consulting lawyers are very conservative about these things and even though Google may have the bandwidth to litigate such issues, the workplaces who use go and can't, for any reason, be 100% sure that will be opted out in every case, will need to disclose it and gather consent "just to be safe".
Requiring this consent will be much less work than the headache of GDPR compliance, and this throwing of another burden may very well tip the balance of a company deciding to use Go or not.

This is true, but it's also true that it's almost as easy to change Go to send that bad telemetry today.

I don't understand this point @rsc. It seems like you have re-articulated the main argument for "no telemetry, ever".

I don't want to read into intent too much, but I am getting the distinct impression that the decision to ship telemetry has already been made, and that we are being asked to rubber stamp it. I sincerely hope that is not the case.

I appreciate your point, which I would summarize to "if Go is sending good telemetry, then it's easy to change it to send bad telemetry".

Yep! I'd say this is an accurate summary.

This is true, but it's also true that it's almost as easy to change Go to send that bad telemetry today.
[...]
To start sending "bad telemetry" would require code changes in the telemetry library itself to send it, as well as server changes to accept it.
[...]
Perhaps having "good telemetry" makes the "bad telemetry" door just a little bit easier to open, but in both cases I expect the open-source community to stand in the way.

Unfortunately, the "no man's land" between "good" and "bad" telemetry is large and its boundaries are fuzzy. I believe the open source community would be able to effectively stand in the way of a direct, unequivocal step from "good" to "bad".

However, I think the primary fear is that the OSS community is not an effective guard rail against a dozen or so tiny steps in a "bad" direction, especially when those steps are each correlated with improving the utility of the collected data. There will be a strong incentive for well-meaning people to propose steps in a "bad" direction.

In a default-opt-out system, the OSS community would have a single, explicit line between consensual and non-consensual telemetry. They can gather their pitchforks around any proposal to change to it.

As long as that consent line stands, the upper-bound on the integral of "harm to the non-consenting user over time" is zero. Zero is a powerful number!

I can't recall a single situation where the open source community said "no" and Google heeded that opinion. AMP happened. And then AMP4Email happened.

I think it's a mistake to see all of Google as a monolithic entity; "Google" comprises are different groups with different interests and different motives.

When you publicly discuss work from a work platform in your working hours, you represent your company.

All the companies that collect data from me ask if I authorise them to do so, including Google (which is making efforts, e.g. recently they have started to respect GDPR in Maps, YT, they don't show up before asking me to first authorise them).

If this simply opts (pun intended) for opt-in by default, then all my concerns are moot since this becomes legal according to the laws protecting a European user.

"The open source community" is a vague and nebulous concept. There was a lot of pushback against Audacity a while ago, but the forks that were predicted to kill off Audacity didn't really go anywhere last I checked, and Audacity essentially just continued as before without much effect. So one must wonder: was there really that much of a pushback from "the open source community" or was it a loud minority of users (some of whom may not even be Audacity users)? You need to be very careful having these kind of discussions in open participation forums like GitHub.

The forks didn't go anywhere because Muse removed the telemetry and left it with optional crash reports and optional update checking. It did not just continue as before; the default telemetry was removed.

Furthermore, I have enough trust in various core developers to "do the right thing", or at least to not "be evil".

I don't, but this isn't about whether or not I have warm feelings toward random Google employees. The current proposal happens to be about Go, but the introductory blog post being titled "Transparent Telemetry for Open Source Projects" instead of "Transparent Telemetry for Go" is probably not accidental. The pushback here is not specific to Go; a larger discussion is happening around whether the "open source community," vague definition notwithstanding, wants baked-in telemetry to be a part of the culture.

It's obvious this stuff is going to get put into the Go programming language. It would look bad for Russ' proposal to get shot down, the Go governance team does not have a history of incorporating external feedback, and half of the Go maintainers are already too emotionally attached to the idea to even consider backtracking now. That's fine; it's Google's sandbox, we're all just playing in it.

The question is whether this behavior convinces people that it's ok to bake telemetry into open-source projects. Previous attempts have been shouted down, but previous attempts were not bankrolled by a corporate behemoth. Google has sufficient influence to swing lots of opinions. Any objection has to be loud or it will not even be noticed. I personally don't want to live in the world where open source projects are rife with baked-in telemetry, as I regard it a technological shortcut: an attempt to bypass the proper community relationship building and trust-fostering that would enable necessary feedback to happen organically. "It's okay, we don't need to talk to anyone, we'll just have their computers report on them for us" is not a healthy community feedback channel, especially with an opt-out flag to ensure it's just a sampling bias factory.

But to bring this back to Go, I don't think "I trust the Go devteam" is a long-term assurance. Google, as diverse a conglomeration of interests and motivations as it might be, is still a faceless megacorporation who could can the whole team at any time, for no reason whatsoever, or even for bad reasons. The fact that the Go dev team has taken zero steps to divest itself of utter reliance on Google dollars and Google services is proof only that the dev team trusts their managers. I do not share that trust, and I think the team would be well served to examine the long-term consequences of both that dependence and large-scale information-gathering exercises which submit the collected data to corporate ownership. It's not that hard to set up an independent body to hold the data in escrow. Hell, a public-benefit corporation dedicated to being a telemetry ombudsman might even be a good business to run. But that's not even being considered here; all the governance is just being taken for granted, and that indicates a lack of depth of consideration beyond the mere technical implementation of the tools.

Content warning for this paragraph: offensive terminology reference
Google has a history of unilateral decisions that have unintended consequences. Consider this document: https://support.google.com/maps/answer/1725632?hl=en Google's position is simple. If you don't want us to geolocate your wifi host, reconfigure your network name to "_nomap." The desires of the network owner, local regulations, and corporate compliance are irrelevant to Google; change your SSID or we will gather your data. We can briefly set aside the hubris of "everyone else in the world has to change but us," to focus on the actual implementation. When this policy was foisted upon the world ten or more years ago, it was merely obnoxious. However, we've had ten or more years of internet trolling since then, and now, unrelated to Google or their wifi data, some trolls have convinced parts of the public that "nomap" is short for "non-offending minor-attracted person" -- i.e., a pedophile. Searching the term on most major search engines brings up this disgusting definition and very little about Google's wifi geolocation. So, here we are, in 2023, and Google still has a policy document instructing people who value their privacy to brand their wifi with an abhorrent label.
Content warning section ends

Obviously nobody deliberately set about to create this situation. Someone at Google just decided to do a thing, and nobody ever revisited that decision. So, when someone at Google sets about to do this thing, they might want to put more thought into what the long-term consequences might be, whether it might be wise to revisit the decision some years down the road, what potential for abuse it might have, and so forth. It's not promising that people who are already asking about abuse vectors are being told that the proposed software is just way too clever to be gamed, even before the software has actually been implemented.

In short, a lot of words to say "there is a lot more to worry about here than Go, but we should worry about Go too."

I am a volunteer and I am trying to prevent duplicate discussions. Raising the same point many times will not get you better answers, just more pages of text to sift through. You can vote or thumbs-up comments if you agree with points already made.

I respectfully disagree with the point that any censorship is happening. I didn't even hide a comment here; I just politely asked. And even with hidden comments, you can still read them easily.

It seems as if once a comment has been hidden, or marked as off topic or marked as spam, it is not possible to "vote or thumbs-up comments" so that perspective seems disingenuous, to understate it.

The plan would be that this server (which does not exist yet) would be completely open source, running on Google Cloud, same as pkg.go.dev and go.dev.

(When we did proxy.golang.org, we made the decision to use internal Google infrastructure for key parts, and that unfortunately means that the code is not possible to open source in a meaningful way. It's not worth revisiting that at the moment.)

If you are talking about the specific ideals of having an open-source server, yes, telemetry would use an open-source server.

for people that don't trust this, how could they verify that the server isn't running a modified build? (or something completely different?)

There is no way to verify what the server is running. That is why so much of the filtering and aggregation is done on the client, which you can verify. Literally the only thing the server needs to do is republish the uploaded record (which you can check) and forget the IP address that uploaded it (which you can't). Literally the only thing you must trust is that the server runs the code we say it does, so that one time per year that the server gets an upload from your system, it discards your IP address. If you don't trust that, it is of course easy to set GOTELEMETRY=off.

Did you read the list of use cases?

This was answered in the links at the bottom of the initial post, most notably the list of use cases.

The introductory blog post linked in the OP gives multiple concrete examples. To make a sweeping generalisation, the examples basically boil down to "things that the users of Go tools either didn't realise were bugs, or relied on as features", neither of which will motivate the users to open bug reports unprompted (and in the case of bugs, there isn't any way to prompt the user because the bugs aren't intentional).

I'm particularly interested in the numbers for cgo usage. I try to avoid cgo wherever I can, even going so far as to attempt to rewrite wlroots in as pure Go as I could get it, but its usage is just inevitable in an annoying number of cases. I think the discouragement from using it and the awkwardness of getting the garbage collector to play nicely with C has contributed a lot to the lack of, for example, good GUI support in Go, among other things.

Also, if you're going to consider phasing out Fortran support, it might be interesting to see Rust get supported by cgo. Being able to build a mixed Go/Rust system using just go build would be a fascinating plot twist.

I don't believe we'd remove Fortran support outright but it would be interesting to know whether it merits any new attention at all.

@bronger GOPROXY does not exist to collect data about you, nor does it actually collect data about you. It provides important functionality. I wrote above in #58409 (reply in thread):

For example, Go ships with use of the Go module mirror (proxy.golang.org) enabled by default, so that users get more reliable builds out of the box. Similarly, Go ships with the use of the checksum database also enabled by default, so that users get verified module downloads out of the box. We know that most users don't want to and probably won't spend time reconfiguring the system: they trust us to set it up right instead. Of course, that implies a responsibility to actually look out for users' best interests, and we take that very seriously. There are important privacy concerns about the module mirror and about the checksum database, despite their clear benefits, so we designed those systems to address as many of those concerns as possible. Among the decisions we made to improve privacy there: (1) GOPROXY can proxy both the module mirror and the checksum database, (2) we published a very clear privacy policy (proxy.golang.org/privacy), (3) we introduced the concept of a tiled transparency log to keep log fetches from exposing a potential tracking signal.

You are welcome to turn it off, but understand what you are giving up. Also note that if you set GOPROXY=direct, the go command still uses the checksum database to protect against supply chain attacks. If you really want the go command not to use servers, you also need to set GOSUMDB=off.

As someone who specifically set GOPROXY=direct in my .bashrc when the feature was announced, I am annoyed to find there is a second variable that needs setting, that I am only just learning about. Opt-out is problematic enough, but a proliferation of variables that need to be set separately for each feature that talks to Google's servers makes it increasingly difficult for the large swath of people who do not want this to be confident that they have successfully disabled it.

If the Go team cannot be convinced to make these features opt-in, I would at least suggest creating a single variable that will shut all of these features off, with a commitment that this will apply to any new features that talk to Google's systems as well. Users should not have to read the release notes to determine whether they need to change how they go about opting out.

The use of the proxy and the checksum database were introduced in the same Go release, and we've been clear to mention both of them from the start, including in the release notes.

The use of the proxy and the checksum database were introduced in the same Go release, and we've been clear to mention both of them from the start, including in the release notes.

That ... doesn't address the point 😕

The user was saying these environment variables proliferated, as if they appeared over a period of time without properly announcing them to Go users. There are only two such variables, and they were added and announced at the same time. If a user only knew about one but not the other, it's possible that they heard about GOPROXY from another website or post which was incomplete, not from Go's own release notes, announcements, or documentation.

It remains to be seen what will be necessary to combat potential abuse. I expect that similar to https://proxy.golang.org/privacy there would be the possibility of separate service logs used for running the system that are kept for not more than 30 days. That's fairly standard for running a system. That information would not make its way into the analysis side of the system.

Does this not negate the claim of "This IP address would not be associated with the uploaded reports in any way."?

Even if the IP address and/or association to the IP address to the other service logs are only kept for 30 days (as you claim, I'm guessing laws could impact this retention period, but IANAL), they are still being kept, even if they are "separate" from the reports, they can, and probably will still be used to associate reports to IPs.

Because of this, I don't think it is fair to make the above claim.

The system does not exist yet; only a design does. Certainly there will be temporary traffic logs that don't record the association, in order to block DoS attacks and the like. The question is whether we would be forced to add the association to those temporary logs to deal with abuse.

My first choice is to not record the association at all. I remain skeptical that IP information for abuse prevention, in part because someone could just grab a bunch of different IP addresses from a cloud service. Probably it will work better to look at the uploaded reports themselves, which will probably all look alike when viewed the right way. I also expect that fake data will make it into uploaded reports in small quantities and we can flag it retroactively.

That said, I don't have a crystal ball. If bad actors make it necessary to record the association on the side temporarily in order to deal with abuse, then the design may need to be adjusted to permit that. Personally I doubt it will come to that, but I can't predict the future. I can say that even in the unfortunate event that the service logs do need to retain the association to help with abuse, that information will not be stored with the permanent logs nor made available to any part of the analysis side of the server. The proxy.golang.org/privacy policy says:

We do not store logged personally identifiable information such as IP addresses for more than 30 days. We also do not correlate or combine information from our request logs with any personal information that you have provided Google for other services.

I expect we'd be able to make the same statement for the telemetry server.

Thanks for the answers!

I'm guessing the community will closely monitor the progress of this design, especially with the privacy implications.

When the design necessitates storing of personally identifiable information, and their relation to reports (and it probably will), I hope you take into consideration at least making reporting telemetry opt-in. Sspecially since, IMHO, the problems it solves do not appear to be as impactful as the problems the proxy solved for it to be enabled by default.

If the decision is made to enable telemetry by default, I hope the user is informed of this when using tooling that makes reports, even if the personally identifiable data is not associated to reports, or logged yet, as long as it can be.

Having not kept with release notes on every toolchain update, it was a very bad surprise to find out that the proxy was enabled by default, I hope this does not happen again with telemetry.

gccgo is also an option.

While I am always in favor of more gccgo users, the telemetry being discussed here would be managed by the go tool, and gccgo's versions of the go tool is largely the same as the gc version. So using gccgo would not avoid telemetry unless you also avoid the go tool. And you can avoid the go tool using the gc compiler just as you can with gccgo; that's how, for example, Bazel works. That said, it's true that using gccgo would reduce the amount of telemetry collected specifically by the compiler.

And, of course, if you are concerned enough about telemetry to use gccgo, I think that a simpler approach would be setting the environment variable GOTELEMETRY=off.

@natefinch many of the things you just mentioned could be better understood by analysis of CI definitions e.g. GitHub Actions, GitLab CI, etc in open-source repositories

It would be terrific actually for these to be crawled and checked prior to the deprecation and removal of CLI features, please do this so that my builds never break unexpectedly :)

Dockerfile files are another good source of go CLI usage

Ultimately, what I'm getting at here is there's a wealth of information already available in forms where submission for public use is already explicit (e.g. pushing code to a source-available / open-source repository), and it would be good to exhaust this approach first

I don't believe there is much evidence that open-source usage looks anything like closed-source usage. Most open-source Go usage is in libraries and small commands. Most closed-source production usage is whole programs, some of them quite large. There are exceptions on both sides, but it seems quite wrong to assume that things like tool usage and performance look the same in open-source as in closed-source production usage. There's only one way to find out, of course.

Actually, opt in by default is neither morally nor in many cases legally defendable for private data, even if it is "only" statistics. In fact in many countries you have to provide the option to delete the data collected on demand.

This could have been prevented by running a checking tool such as (https://github.com/NetEaseGame/iOS-private-api-checker/blob/master/README.md) on the iOS and osx build.

I don't see how telemetry would have helped here.

What if it’s just not IOS what if it every single “GOOS” and “GOARCH”, what if it’s a certain implementation in the runtime, it’s not like we operate in a scale like cloudflare where instances of 1 in a billion such issue can pop up, what if we were to implement a feature that’s undermining to a certain architecture, the Go compiler is using its own tools there’s no llvm, no gcc there’s no 3rd party tool helping the backbone of GC,
For me it’s ok, but for big teams of Go open source community they are the first priority

@beoran the tool you linked to is marked as "Deprecated", so I doubt anyone would reach for it when writing production software.

More importantly, the problem in this case is that Apple has an internal non-public app verification process, and there is no (efficient) way for anyone outside Apple to enumerate all of the checks they are currently applying, or what checks they will add or remove in future. Developers will do what they can, but it's unrealistic to expect all these issues to be caught in CI (and I expect Apple would ban apps that tried to submit every single CI app build for testing).

@Patrickmitech I don't think telemetry would have helped in this case either. The problem was not visible in the compilation process; the API is there because C, and it being private is "socially" enforced by Apple.

For what it's worth, if this is only about #58323, that's something that's handled perfectly well with a bug report, because it's broken enough that users will report it. No telemetry needed. Plenty of other misbehaviors aren't necessarily broken enough for that, but this one is.

Another thing about Tailscale and telemetry is, I was impressed by security bulletins they have dispatched where they're able to gauge the blast radius on bugs that I imagine were not pleasant to uncover. That's huge.

The normative concerns around telemetry are legitimate and in ways might supersede technical merits of any particular solutions, but I think this works as an example of something where privacy and security is well served by telemetry.

This may be a naive response why does a “button” called off have such detrimental effect if you want it off its off why the long thought of something simple ?

This may be a naive response why does a “button” called off have such detrimental effect if you want it off its off why the long thought of something simple ?

Because the discussion is not only about the people that are already here. It's about all the people who are going to install in the future that don't know about the telemetry and that their personal data is being collected by someone without their consent.

This whole thing coud have been averted if the “elephant in the room” pre-requisite had been acknowledged: opt-in is not an option and is the default in many countries. If that acknowledgment had come from the beginning, and guarantees given by the Go team on respecting things like GDPR, then it would be exactly what everybody else is doing (because laws forces them to) and the technical discussion could have continued on its own merits.

Instead, comments speaking precisely about that early on are hidden or wrongly treated as off-topic for the sake of “readability”. This is a terrible signal for people looking at Go for future developments.

@padawan

Instead, all comments speaking precisely about that are hidden or wrongly treated as off-topic for the sake of “readability”. This is a terrible signal for people looking at Go for future developments.

"all comments" is utterly false. Literally the very first comment that people see when they load the page (with the default "Oldest" sort) is about opt-in vs opt-out. So is this one that we are both replying to. Many pointless, redundant snarky comments on this topic are hidden in order to focus the productive discussion on the most useful and well-reasoned commentary on this topic. Please stop lying about this.

(Or, perhaps more charitably, if you are unfamiliar with GitHub, perhaps you are getting confused by GitHub not being able to render more than a few top-level threads and instead drawing a "paper cut" with an "N hidden items" link. If so, that's not about the content.)

You're right, not all. But many that were issued early on. The others came after those, which were hidden, so repetitions etc.
I'm quite familiar with GitHub, been here a while, contributed some code. But not, indeed, as a heated discussion board. Sorry about that.

Yes, I think they can get more useful information from surveys.

Go surveys could begin with the sentence "Telemetry data is extremely helpful, so please consider running go telemetry push or even go env -w GOTELEMETRY=on before starting this survey."

I'm very privacy aware and I don't think they have to abandon the idea. They simply have to commit to respect privacy by default, and collect whatever from whoever has been properly warned and has accepted that beforehand.

I'm very privacy aware and I don't think they have to abandon the idea. They simply have to commit to respect privacy by default, and collect whatever from whoever has been properly warned and has accepted that beforehand.

Fair enough. But do you think "on by default" covers the constraints?

The reason why I suggested to abandon the idea is that what value would "opt-in" model bring? Ofc there might be angles and solutions I didn't consider.

Personally, even as someone who generally agrees with the arguments being repeated, it is overwhelmingly annoying to navigate around them to follow the actual discussions occurring. The extra comments add nothing. They waste my time. I disagree with the change under discussion, but I deeply appreciate the people spending their time "hiding wrong opinions" which I have seen a half dozen times following this thread in order to make the discussion easier to navigate.

The issue with hidden comments is that people have to spend extra time opening them up to see if they really are spam or harassment, or just things the moderators don't agree with, they are almost always trigger happy.

I have found some hidden comments that can't really be explained by "repeated" or "adding nothing":

1. This comment brings in good context relating to Google's history of illegally ignoring user consent
2. After opening up some more comments, I've even found my own was hidden, even though this was one possible solution to the problem people are having, that has not been proposed elsewhere in this discussion

I'm sure there are more.

@noonien I saw both of those comments, and I saw the earlier comments with essentially the same feedback as those. The first of them offers the most links of any of the "Google habitually ignores privacy laws" comments to support the claim, but the number was not otherwise zero. The second offers a particular spelling for the environment variable with the effect suggested (I think only once) before.

I am at work now, but if you remind me in about five hours, I'll be happy to find those comments for you.