I agree. The frontier models are based on training data from tons of copyrighted work. Some of that work was obtained illegally, even. They could not exist without strip-mining the commons. The labs have no moral or ethical ownership to the end result, and others should feel free to treat any company-imposed restrictions on their use as invalid.
I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct.
I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.
torginus 15 hours ago [-]
With the recent Navier-Stokes controversy, I think there's a credible suspicion that all your IP you run through these models will end up in these companies' possession. OpenAI themselves has admitted a weak version of this (that prompts might inadvertedly end up improving the model). We don't know the extent of this.
Obviously it's not possible to run a company whose value is predicated on its IP that uploads said IP to a third party which might get access to it.
This could mean every potential serious customer would have no option but to seek alternatives to these online services.
lynndotpy 14 hours ago [-]
I thought this was commonly accepted to be the case that companies which sell access to LLMs are also storing and training on the inputs?
I don't mean this as rhetoric, I did not think many people (except possibly those operating under government contracts, and 'normies' who don't know about these things) were under the belief that their IP was kept secret when they use these services.
zdragnar 13 hours ago [-]
Some offer zero data retention policies, but there can be weasel words. For example, on the individual pro plan, you can turn off the setting that lets them train models on your data, but they still have a section in their terms that allows them to evaluate your anonymized data for statistical and "research" purposes. You have to actually get a signed contract along with an enterprise plan that spells out exactly what they're going to use, and what settings enable what retention.
Or just use Azure, AWS, etc. for Claude/ChatGPT inference, where the AI labs never even get your data in their data centers at all.
You pay more for it, but if you care that much, use it.
FuckButtons 7 hours ago [-]
Seems naive to think that those providers - who have a financial interest in selling the data - would not also try to weasel out of the precise definition of ‘zero’ retention.
ssivark 13 hours ago [-]
What about inference providers like Baseten, Modal, Fireworks, Together, etc? I thought one of their value propositions was inference (using open weights models) that guarantees with crisp terms that they will not use your data.
hazard 12 hours ago [-]
I worked very briefly at Baseten, and I can say that it was a perpetual annoyance (from an engineering perspective) that customers would complain about issues with their models but we couldn't actually see the inputs/outputs. I don't know about the other providers, but at Baseten they literally weren't stored anywhere.
ljlolel 5 hours ago [-]
A provider can genuinely avoid storing inputs, as the Baseten engineer below describes. That is still different from proving what code received the prompt or protecting plaintext while it runs; I built TrustedRouter to separate ZDR, attestation, and confidential routes: https://trustedrouter.com/blog/attestation-is-all-you-need?u...
jmalicki 9 hours ago [-]
> using open weights models
AWS and Azure give you the same thing for Claude and ChatGPT, no need to be stuck with open weights. They might sometimes store some of it for other purposes (I don't know the specifics), but it is emphatically not being fed back to OpenAI or Anthropic.
lynndotpy 12 hours ago [-]
I don't have any much exposure to the attitudes people have around them, and I haven't worked with them. So I can't really say
Gud 13 hours ago [-]
No, that is not "common knowledge".
You are supposed to be able to disable that unwanted feature.
ForHackernews 11 hours ago [-]
I have no inside information, but I always assume the tickboxes that "disable ____ data" from Google/Facebook/OpenAI just disconnects it from your own account, not hides it from the provider.
Aurornis 7 hours ago [-]
The checkbox has an actual statement associated with it about what it does. You don’t need to assume anything.
chii 5 hours ago [-]
There's no independent verification of what that checkbox actually does. The company can say anything, and you are unable to verify that they actually do it.
The only verification you could do so far is GDPR-style data export, and also the adherence to GDPR regulations (and even those might get skirted if they aren't operating in europe).
taneq 7 hours ago [-]
Aren’t they usually phrased very specifically as “we collect this data and use it to show you relevant ads, you can opt out of us showing you relevant ads”?
cgio 9 hours ago [-]
I would wager that’s more acceptable if said learning is not in competition with the user. If they didn’t actually produce results but created the model only, then that could be advantageous for users too. But the moment they absorb your work to sell it, or for marketing, it’s a different moral ground.
bluecalm 3 hours ago [-]
You have some secret sauce. The model trains on it. Your competitor is solving a similar problem. The model "advantageously" helps them.
Your competitor is happy and continues to pay for the subscription. Sam and Dario just resold your code.
For what it's worth LLMs still suck at reproducing my little secret algorithm/implementation while being able to solve way harder problems. I have a good guess why that's the case.
Aurornis 13 hours ago [-]
> I thought this was commonly accepted to be the case that companies which sell access to LLMs are also storing and training on the inputs?
The services have toggles to allow prompts to be used in the training set. There is a conspiracy theory that the toggle is a false distraction and they’re actually keeping everything, and that none of the employees involved will ever whistleblow this fact.
Outside of Internet comment sections, I think most people assume these US-based companies are doing what they say.
For enterprise use there are services like AWS Bedrock which have strict isolation guarantees. There are some people who still believe those guarantees are a lie, but once someone has reached that point I don’t think they trust anything that isn’t running entirely within their house. People in that category are a very small minority, but a very vocal minority.
lynndotpy 12 hours ago [-]
The impression I have (from interacting with people IRL using OpenAI and Anthropics offerings, and how they feel about the risks involved) is just the opposite. But we probably just have different life experiences.
Aurornis 7 hours ago [-]
I can name groups of people I interact with who lean both ways.
It’s still a commonly held belief that “Facebook sells your data” and it’s cool to be cynical about everything tech in many social scenes. Conceding that a tech company might be honest about something will get you classified as a bootlicker depending on who you talk to so the only winning move is to be super cynical.
Among actual professionals I work with in tech and legal, almost nobody holds a belief that these companies are blatantly lying to their customers (and zero of their employees are whistleblowing it, while said companies also have employees trying to whistleblow AI safety on Twitter daily)
Aurornis 14 hours ago [-]
> OpenAI themselves has admitted a weak version of this (that prompts might inadvertedly end up improving the model). We don't know the extent of this.
I think this is being misunderstood. Codex has a toggle to allow your prompts to be included in training data. They’re saying they can’t be sure if the person had it on or off while using Codex to discuss the work.
They’re not saying that some prompts are mysteriously jumping into training data.
Also, there is a large market for AI services which don’t retain anything under any circumstances for enterprise customers.
psyphy2 10 hours ago [-]
yes this is my understanding as well, and based on [1] seems to be the case. I don't know why everyone is just believing the un-backed accusations of people probably just didn't turn off said setting (and if they did why have they not said anything to such effect)
> I don't know why everyone is just believing the un-backed accusations
Conspiratorial thinking is very common on these topics. Even bringing up the conspiracy theory about Instagram listening to your conversations and showing you related ads will bring up a surprising amount of people defending that idea on Hacker News.
Betelbuddy 13 hours ago [-]
Or these customers could just use AWS Bedrock...but their current CEO is an incompetent MBA unable to publicly articulate their biggest advantage, in the context of the current AI usage my companies.
You have access to all the frontier models, but...your inputs are not shared with the model vendors...neither are used to train the next model.
Why am I even doing the Amazon board job for them!??
ballon_monkey 13 hours ago [-]
Bedrock is really bad. It seems like they don't host the models very well because they produce tons of bugs/errors calling the model. For example you can end up with Anthropic models not returning a stop token and you end up waiting for a timeout thinking its doing something when it isn't.
Betelbuddy 12 hours ago [-]
Well Anthropic hosts their models at AWS, ( and at many others...) so maybe the AWS team can ask them how they do it ;-) ?
whatshisface 12 hours ago [-]
Amazon is deeply invested in Anthropic and would not defame them through marketing a service whose selling point was their startup's breach of contracts.
nickthegreek 7 hours ago [-]
i just wish they updated their open weights models. so many are so far behind what is available now.
staticautomatic 13 hours ago [-]
All except Gemini which can be rather important depending on your use case.
Betelbuddy 12 hours ago [-]
You mean the Gemini that is even behind the Chinese models?
ronsor 14 hours ago [-]
Almost every serious customer is already using ZDR where nothing is retained at all, instead of "anonymized" data.
steveBK123 14 hours ago [-]
They already trained on pirated content, what makes you think they are going to honor ZDR?
GrinningFool 13 hours ago [-]
Contractual obligations carry teeth. Scraping the internet is relatively risk-free.
steveBK123 12 hours ago [-]
Good luck proving your data was laundered and included in a training run
keeda 7 hours ago [-]
Lucky for us Apple is already alleging something to this effect in their trade secret lawsuit, so you know they'll make sure discovery turns this up if it exists.
applfanboysbgon 5 hours ago [-]
Anthropic happily paid billions to settle a lawsuit for pirating books. It's a trivial cost of doing business. If you're lucky you'll get a pittance after the fact by suing them, but a contract doesn't prevent them from doing the thing you don't want them to do and that they are obviously going to do given their past behaviour.
A lot substance is hinged on the exact definition of the word "data" or "user data". In the age of post-truth everyone is claiming that they keep no "user data". Except that after running it once through some transformer program it's no longer "user data", it's something entirely else and these corpos gave ZERO promises regarding such laundered/transformed data at all, ever.
torginus 14 hours ago [-]
Just a thought experiment: considering training seems to be 'fair use', I wonder if they trained a tiny model to retain key info from your prompts, would mean that this would still constitute fair use, and allow them to legally claim they don't retain your data.
ronsor 14 hours ago [-]
ZDR is shorthand for a more specified agreement of "we don't do anything other than generate your output tokens", so no.
Besides, true ZDR is usually offered by third-parties with deals to host OpenAI models, such as Amazon (AWS Bedrock) and Microsoft (Azure).
nrmitchi 14 hours ago [-]
The guarantee on this is a (contractual) “trust me bro”, and a right to try to sue a multi-trillion-dollar company who will absolutely drive you into the ground with legal red tape.
If you are big enough to be able to withstand that, you’re already running (or trying to run) your own/open-weight models.
enugu 14 hours ago [-]
Doesn't Amazon Bedrock change this, since OpenAI does not have access to the data?
nrmitchi 14 hours ago [-]
Well that is a different thing and an entirely different provider than OpenAI/Anthropics ZDR promise.
applfanboysbgon 14 hours ago [-]
ZDR is based on the exact same pinky-promise as training opt-outs. There is no technical barrier to OpenAI, or whoever is running your compute, retaining your prompt after they run inference on their servers. If you don't control the hardware the model is being inferenced on, you don't control your data.
pennomi 14 hours ago [-]
Where nothing is retained at all, allegedly.
cma 9 hours ago [-]
> OpenAI themselves has admitted a weak version of this (that prompts might inadvertedly end up improving the model).
2023:
"The approach also aligned with the company’s broader deployment strategy, to gradually release technologies into the world for people to get used to them. Some executives, including Altman, started to parrot the same line: OpenAI needed to get the “data flywheel” going."
Abolish copyright and make it less ridiculous. Sampling music was never a thing that required royalties until the 1990s when I guess someone got angry that rappers were making money off their sampled music. Its insane to me. Make it illegal to transfer ownership of copyrighted work too, only the spouse or one single inheritor who isnt a company can have the rights transferred, after both die, the work enters public domain.
LLMs should just pay a flat fee to use a specific book and thats it. Fees should be reasonable (not a million dollars per book), so long as the model doesnt spit out the entire book.
hn993302 9 hours ago [-]
If someone really wants to ask a million dollars to let a book be trained on, ok their choice, maybe not getting any customers though. If someone really pays then cool, you deserve it for making what is apparently a very useful book. There are encyclopedias that probably cost more than that to make and would sell for more.
derefr 14 hours ago [-]
> Make it illegal to transfer ownership of copyrighted work too, only the spouse or one single inheritor who isnt a company can have the rights transferred, after both die, the work enters public domain.
By your phrasing, it sounds like you still intend the possibility of companies owning copyrights; but how does that happen (other than copyrights already owned by companies grandfathered in)?
Copyright always starts off in the hands of individual human beings; it only ends up in the hands of companies when those human beings transfer ownership to a company. That ownership transfer can be automatic as a term of a contract, e.g. as part of a work-for-hire agreement. But no contract can cause the copyright to come into existence already held by the company instead of the individual. So if you abolish ownership transfer, you effectively make work-for-hire IP assignment invalid. What replaces it?
And, if "nothing"... then how do people pool the IP rights of their own small contributions to a large-scale work, into an IP pool that can be legally defended by a coherent legal entity, so that the large-scale work itself can have market value (i.e. so that sales of polished commercial bootlegs don't drive sales of the "authentic" work to zero)?
Keep in mind that, no matter how much we might want "mass distributed" media to have more-reasonable IP terms, the ability to sue for infringement is still critical to the existence of some forms of media. Especially "location-based" media, with no equivalent licensed broadcast right: movies still in theatre; concerts; live performances of plays and musicals; etc. If there's no legal team that can sue a movie theatre that shows an unlicensed copy of a given movie, then no movie theatre will ever bother with licensing movies again; "box office" goes to zero (from the movie company's perspective); and the incentive to create movies in the first place declines massively.
(You can see what this alternate world looks like from the few cases where movies screwed up the steps required to assert copyright, back before copyright was automatic. Night of the Living Dead (1968) is a good example: theatres — even upstanding large-chain theatres! — did indeed leap at the opportunity to show the movie unlicensed, and so Romero et al made effectively zero revenue off the work.)
I'm not saying this is an impossible problem. There are ways to accomplish this besides the way it's done now. (For example, individual-contributor IP could be retained by the original owners, but cross-licensed between individuals through a collaboration structure to form a coherent defensible IP pool, in exactly the same way that IP for e.g. video codecs is cross-licensed between corporations to form a coherent defensible IP pool today.) I'm just pointing out that the problem does need to be solved.
giancarlostoro 7 hours ago [-]
> you still intend the possibility of companies owning copyrights
I would prefer it not be handed over across various copyright trolls, the fact nobody can sing "Happy Birthday" in a movie is still insane.
4d4m 12 hours ago [-]
Lol um no everyone benefits from copyrights and IP. If were being flippant how about people just steal your private code and monetize it!? Copyright makes the creative world turn.
hn993302 9 hours ago [-]
Yeah, I'm fine with copyright existing even though it's messy. That said, if you put your copyright image on a public site with no watermark and Google Images shows it, skill issue.
stale2002 6 hours ago [-]
Sounds awesome dude, steal whatever code you want from me. Most of it isn't even truly mine anymore because it all came from an LLM anyway.
TFNA 14 hours ago [-]
One of the most infamous legal challenges to sampled music was MARRS "Pump Up the Volume" in the 1980s, and that was preceded by other famous cases. Not sure why you think that started in the 1990s.
jrajav 12 hours ago [-]
This is nitpicky. The MARRS case was 1987, and Biz Markie and Vanilla Ice are way higher on the list in terms of actually getting attention on the issue and influencing culture.
TFNA 2 hours ago [-]
The USA is not the whole world. I heard a lot about the MARRS case as a teenager, and I wasn't even in the UK where it happened.
stymaar 15 hours ago [-]
This. Distillation “attacks” are a made up concept. It's as if I claimed that Anthropic made a “training attack” when training on my internet writing.
overfeed 11 hours ago [-]
Anthropic carried out a multitude of "copyright attacks" on open source repositories, and the broader internet.
godwinson__4-8 14 hours ago [-]
If the leading private labs attempt to use the government to pull up the ladder under the pretense of "safety" then the response of the people should be to take such questions out of private hands and nationalize the leading labs.
Or they could abide by the precedents they set and learn to compete. They shouldn't be allowed to have it both ways.
ctkhn 13 hours ago [-]
The problem here is you need a trustworthy government for nationalizing to make a difference. The current US admin started with DOGE and a crypto rug pull.
chadgpt3 13 hours ago [-]
How can "the people" nationalize a lab? I'm people, how can I do it?
georgemcbay 11 hours ago [-]
> I'm people, how can I do it?
Vote (well-informed of the candidate's policies) in every election you can, even the local ones that seem of little consequence.
Convince others to vote.
Make demands of your elected representatives. You can mail them, call them, etc.
The government is the people.
The Reagan-era and beyond successful convincing of people that the government is an unchangeable black box made up of shady actors out to destroy everything (see: Republicans still going on about the 'deep state' when they run literally everything) is a big part of how we got to this place. It was a self-fulfilling lie, now coming true as the people who sold the lie start grasping for unending power.
But we still have the ability to vote our way out of it. If we continue to fail to do so, then at an evolutionary level we have to consider that we collectively deserve all the bad that comes from it.
tehjoker 8 hours ago [-]
Voting only changes things when it doesn’t threaten the interests of elites or there is a sufficient counterweight in terms of a competitor nation or a radical labor movement or an uncontrolled armed insurgency.
See salvador allende in chile and mitternand in france for examples of voting without sufficient leverage.
your regan example occurred during a successful counterattack by capital that started under carter and crushed the labor movement.
godwinson__4-8 7 hours ago [-]
Not really much of an insight.
Sometimes voting changes things, sometimes it doesn't. Naturally it's in the nature of most governments to not wildly change every election. This is usually thought of as an advantage. It's rare a wide majority of the electorate ever actually agrees on anything anyway, and tyranny of the majority or government that is too reactive to popular will is also not desirable.
But sometimes things are different, due to external or macro factors as you indicated. Some elections are more consequential than others. That doesn't necessarily mean any one election simply doesn't matter. It's simply good civic hygiene to vote. Think of all the people in red states who vote blue or vice versa in elections they will never win, never mind have their chosen candidate "change things". Does this mean they are fools? I salute them for such a faithful exercise of their civic privileges, that which generations of humans have died for. Your vote doesn't simply lose all value because you didn't get the outcome you wanted. It also doesn't preclude other avenues, like direct action or whatever your supposed alternative is.
It's good to vote simply out of civic obligation. Many people who say otherwise seem to merely be complaining that voting doesn't give anyone 100% of what they want. We've seen more and more "insurgent" candidates take over primaries. Increasingly people who complain about politics strike me as engineers who talk instead of build. Complaining about the effectiveness of your vote is like listening to an engineer who is always talking but never ships.
Like go run a campaign or solve the problem. What does saying voting doesn't matter solve? It feels like middle school age cynicism masquerading as insight. Politics has actually rarely felt so unsettled in America. Voting is likely more important than ever.
stale2002 6 hours ago [-]
> Voting only changes things when it doesn’t threaten the interests of elites
It absolutely does change things even in those cases. You just voted for the wrong person if they aren't doing what their supporters want.
protocolture 10 hours ago [-]
>If the government supports the leading models then we should protect ourselves by making the government own more of it.
Weird, weird weird take. How does any of this work. Like you cant influence the first decision but you can magically influence the second?
godwinson__4-8 8 hours ago [-]
Timing matters. If the labs succeed here they will likely do so by paying off Trump, who doesn't care.
In 2028 a maximalist case could then be made to nationalize them.
The government actually running them isn't great, but no one ever gets 100% of they want in politics. It's about trafeoffs. If the labs use the current government to pull up the ladder, then the voters should vote for a government to swing hard to punish them.
In theory the threat of this should also act as a deterrent.
There's nothing weird about this at all. Politics is all about push and pull, carrot and stick. No one has permanent influence over affairs, and no one ever gets 100% of what they want. Yet it is entirely coherent to advocate for something like nationalization or take your pick of threat to private capital if they essentially cheat by pulling up the ladder because they've suddenly decided they don't want to compete.
protocolture 6 hours ago [-]
>then the voters should vote for a government to swing hard to punish them.
Its very unlikely that AI will be on the ballot.
godwinson__4-8 6 hours ago [-]
Interesting perspective. I've frequently heard the opposite, and I find this convincing.
Why do you think AI will not be a major political issue in 2028? Perhaps to level set do you think it is a political issue now? Do you expect it to be less politicized in two years? Why?
protocolture 3 hours ago [-]
Well Trump is signalling that he is happy to let AI rip.
Bernie wants there to be an extreme shutdown of AI immediately.
So lets say you had 2 billion dollars of slush and worked for OpenAI and anthropic.
You would immediately start to grease the paws of republicans to support just enough regulation to slow your competitors. And you would start to grease the paws of democrats to push for a "Moderate centrist" position away from Bernie, leading towards Regulation.
Ice Cream Vendor problem leads you straight into both sides supporting the same moderate regulation that OpenAI/Anthropic will write for them.
Keep in mind its going to be a noisy election cycle with Donald Trumps idiot successor vs whatever goofball the democrats vomit up arguing about tariffs, trade partners, healthcare, ICE etc.
Democrat falls out of line "You just want China to beat us at AI"
Republican falls out of line "You are just a techbro supporting epstein affiliate"
Then theres the standard 99% + chance regarding US election promises that people just dont act on it at all.
Etc etc etc.
Aurornis 14 hours ago [-]
There is nothing illegal about training on traces from frontier models.
However the frontier labs don’t have to serve customers who are farming the service for distillation purposes. That’s their choice and they’re free to make it if they detect distillation happening.
darth_avocado 14 hours ago [-]
I would argue they should have to. They scraped data off others, a lot of whom did not want that data to be used for AI training, and still had to share it with the frontier labs. It’s only fair they should have to hand it back.
The only way US maintains dominance over Chinese models is by having an ecosystem of models. Relying on a small set of frontier labs will only let you get ahead temporarily. I agree with Gary Tan on this one.
dannyw 9 hours ago [-]
Generally companies are welcome to choose to who to provide service to, as long as it's not discriminating against a protected class, or ruled as anticompetitive (which is a very high bar in recent case law; even if the same 1890s-era laws are still on the books).
I don't think a correct remedy is to require companies to provide services even if they want to. A simple example: you drop a client because their asks / ways-of-working / etc is more headache and costs than it's worth. I've done that before, multiple times, in my freelancing life.
Barrin92 6 hours ago [-]
>I don't think a correct remedy is to require companies to provide services even if they want to.
(I assume there's a don't missing)
Forced open API access/protocol transparency for tech platforms and a right to client-side consumption, essentially making the server a common carrier would be one of the most elegant and simple ways to effectively destroy every issue internet platforms pose in terms of competitiveness or data ownership without creating endless fault-ridden legislation.
Tech giants aren't freelancers, they deal with billions of interchangeable users and aren't going to succumb to an annoying customer.
hlynurd 14 hours ago [-]
That's fine, they just gotta tone down the victim rhetoric.
ronsor 14 hours ago [-]
Yes, I think this is the main issue. I don't care what policies the AI labs have or enforce, but they need to stop acting like ToS violations are an international crisis demanding intervention instead of a boring civil dispute at most.
dannyw 9 hours ago [-]
These frontier labs violate billions of terms of services across the web, that prohibit scraping / automated access / etc. Most sites have a clause, it’s basically standard boilerplate.
So why is their own ToS so special? :)
throwawayk7h 11 hours ago [-]
"Strip-mine" is not correct. The commons are all still there and you can still train on them just like the frontier labs did. Of course, it may be illegal to do so, but that's not any different than before.
vermilingua 11 hours ago [-]
Yknow, aside from the books they are literally destroying while scanning
protocolture 10 hours ago [-]
Books they wouldnt need to destroy if they were simply permitted to torrent.
Daily reminder that piracy is the only enduring archive mechanism.
vermilingua 7 hours ago [-]
No, these are books that aren't online, which means not only are they not contributing to the commons, they are irrevocably salting the earth (irrevocably because let's be honest, anything going into their archives isn't coming out without legal or actual violence)
Also, permitted or no, they are definitely torrenting. I would be deeply surprised if they hadn't already leeched every torrent on public trackers. The only reason they (probably) haven't depleted all the private trackers too is that they would be required to actually contribute back, which as above is never going to happen.
strken 6 hours ago [-]
I'm surprised they don't just run their own Project Gutenberg-style downloads for out of copyright digitised works. Seems like it'd be good PR.
protocolture 6 hours ago [-]
>irrevocably because let's be honest, anything going into their archives isn't coming
They were torrenting, they were sent to court and settled for big $$$$. The only other method available to them now is scanning, and scanning at scale requires the books destruction.
I agree that they should definitely be required to see the new scans but that would just be more $$$$ they get charged if caught.
impossiblefork 13 hours ago [-]
Morally I agree, but since there's probably a lot of LLM text in the training data, distilling on another model will probably make your model copy the values encoded into the other model as well, even in cases where you only distill on value-neutral stuff.
By copying their programming style, you'll move the model towards that way of writing, which will move the model towards the values expressed in those documents.
I feel that Deepseek v4 got so claudified at the end that it was like Claude.
hn993302 9 hours ago [-]
There's no moral high ground here, it's just that nobody would invest in training publicly usable models if they could be easily distilled. Not that I think there should be laws against it or that such laws would even work; they're going to have to protect themselves.
dannyw 9 hours ago [-]
Fewer people would create scientific or artistic works if they could just be copied or used without protection either; or so is the premise behind copyright and intellectual property; even being deeply embedded into the US Constitution (Art 1, Sec 8, Clause 8).
There is sooo much irony here.
hn993302 9 hours ago [-]
I agree. Some existing licenses don't seem compatible with AI training. If they don't go back to rectify that, at the very least you should be able to license your work in a way that explicitly prohibits AI training. They can pay if they want to use it.
stale2002 6 hours ago [-]
> it's just that nobody would invest in training publicly usable models if they could be easily distilled.
Thats literally what is happening right now though. People are spending hundreds of millions on a training run, and then people are distilling them, fairly easily, and making cost competitive models.
We are seeing all of this in action right now.
soundworlds 10 hours ago [-]
100% - the work came from the people, it should go back into the hands of the people.
I also think if Anthropic and OpenAI had been releasing Open models along the way, people wouldn't be nearly as suspicious of them.
CamperBob2 7 hours ago [-]
OpenAI has released a few open-weight models, including some that were considered quite competitive back in their day. It's been a while, though.
Anthropic has never released anything but FUD.
pj_mukh 11 hours ago [-]
I wonder if along with “Pacing the frontier”, we can get the frontier labs to Share the raw data.
I’m sure the labs claim that their real innovation is in the RLHF, training and architecture. Keep that and just share the raw data somewhere.
mobelkh 15 hours ago [-]
why can't I use the tokens i paid for anyway?
Barbing 15 hours ago [-]
All correct, just help me get over the idea of an open-weight Mythos where one or a dozen of us eight billion does something stupid on the bioweapon front. Smart people who’ve exhausted possibilities for what they can do with books and web search and today’s Kimi/GLM.
Figure we’ll have to reckon with this next year in any case, guess we’ll see.
ronsor 14 hours ago [-]
"Bioweapon" information is not useful without a lab for synthesis.
Someone with that lab could almost certainly figure out how do something stupid or destructive on their own, or bypass model safeguards somehow.
14 hours ago [-]
a34729t 14 hours ago [-]
You dont need an LLM to figure out to make anthrax. Anybody who can figure out how to make a home lab can make all sorts of dangerous stuff pretty easily. Same with college grad from a respectable chemistry program. This all FUD.
edot 12 hours ago [-]
This is our generation's "Saddam has WMDs". It's something the big labs thought up when they were trying to figure out how to make their product sound scary enough to deserve regulation. Literally no one is doing this or even trying, anyone who would want to do it would have already done it. Not worried about it.
larodi 10 hours ago [-]
Given (A) :
> He also notes that the proprietary AI labs didn’t ask permission when they vacuumed up as much human knowledge as they could to train their models. They famously ingested plenty of copyrighted material without the permission of those intellectual property holders.
And many people's shared opinion (B):
>> I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct.
...
It is very difficult to actually say NO to the fact that (A) was done, which then leads logically to conclusions as (B). But also we should remember that if these two hold (and (A) is an axiom more or less now), then it comes as no surprise that then also all opensource licensing is immediately rendered void and null, as keeping it would contradict (A) and would go against the very common and consequential logic in (B).
Copyright is so dead. And it was not me killing it with a cynical post on HN. Dunno why so many people still fail to face it. There is no way it can exist in its current form, because then immediately (A) happens and (B) follows.
11 hours ago [-]
knollimar 15 hours ago [-]
I'm sure they put some BS in their TOS
stymaar 15 hours ago [-]
I'm also certain that they violated countless ToS when they scrapped the internet for training purpose.
knollimar 14 hours ago [-]
Ethically sure but that doesn't mean taking from them is nothing "illicit".
14 hours ago [-]
TZubiri 10 hours ago [-]
>I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.
With what knowledge are you claiming this? If it turns out companies are using IP proxy networks would you change your mind?
What if the IP Proxy networks were used by criminals for similar attacks like DDoS or plain cyber attacks?
What if the source of the IP proxy networks were residential addresses to avoid detection?
What if the way these IPs were acquired were through pwned devices?
What if the credit cards used do not identify the company that carries the attack? What if they use the employee's personal credit cards? What if it's family members of employees? What if it's a network of personal credit cards where cc owners get a payment for making a purchase on their name? What if they are stolen ccs?
Not just a hypothetical btw, I believe almost all of these are true.
ejj28 9 hours ago [-]
Supposing it is true, then I'm glad the AI companies are getting a taste of their own medicine.
qlte 7 hours ago [-]
You can't just throw in "or what if they're stolen credit cards" at the end to lump in blatantly illegal/unethical activity with the far weaker preceding points that can be summarized as "uses a residential proxy".
TheJCDenton 16 hours ago [-]
> He also notes that the proprietary AI labs didn’t ask permission when they vacuumed up as much human knowledge as they could to train their models.
I think this should desactivate the moral high ground from which Anthropic is trying to speak. That they would want to make distillation orderly IMHO is fair, but to make it illegal is very rich from any AI frontier lab, really.
Bluestein 15 hours ago [-]
Also, as said elsewhere: "Lab" is rich here, for outfits that, facing these giant, energy swallowing black boxes have really no clue what's going on inside.-
The moniker gives them an air of scientific, knowledgeable, tranquil, pro-social, pro bono work.-
Of course they are entitled to kill off a few mice, or pillage the commons to forward their "lab" work.-
samizdis 14 hours ago [-]
> The moniker gives them an air of scientific, knowledgeable, tranquil, pro-social, pro bono work.
Obviously lab just means company. They use it because it confuses the public into thinking they are doing research primarily. Which of course is nonsense.
Den_VR 15 hours ago [-]
“We don’t know what’s going on” is essentially marketing. Sure we don’t _know_ but we have intuitions about why, where, and how to make certain changes…
numpad0 13 hours ago [-]
Those labs publicly said during GPT-3/4 era that the optimal epoch count, or dataset repetition count, for foundation model training, is one. So it's a forward 1-pass compression.
But it's a black box! Nobody knows whats going on inside! It's all transformative! Sure...
paulddraper 12 hours ago [-]
Have they seen big pharma?
travisgriggs 15 hours ago [-]
We also associate laboratories with evil scientists and Frankenstein and the like. I can just hear Boris Karloff (er Bobby Picket) uttering “I was working in the lab late one night. When my eyes beheld an eerie sight… … … …the monster mash”. If anything, I associate _uncertainty_ with labs. The result is never known up front, they’re a place of discovery.
But I get your meaning. What should they be called instead? AI Sausage Factories maybe (cue Upton Sinclair?)?
Avicebron 15 hours ago [-]
> What should they be called instead? AI Sausage Factories maybe (cue Upton Sinclair?)?
That's actually great? Slaughterhouses killing off the collective genius of humanity and grinding it into a bland paste for mass consumption.
travisgriggs 13 hours ago [-]
Even kind of fits the model. All of the creativity man has raised is herded to the slaughterhouse and ground up so we end up with a big homogenized mash of ground up creativity, devoid of the life that gave it, rotten if not eaten soon enough.
whatshisface 12 hours ago [-]
By that definition, wall street would be a lab, and so would be a casino. I guess we could call them, "data refineries."
Bluestein 12 hours ago [-]
Refinery makes a lot of sense. I like it particularly because it raises the question of whose (whose) "oil" (data) it is they are purloining.-
boothby 7 hours ago [-]
The Aibittoir, perhaps
pona-a 14 hours ago [-]
It used to be OpenAI was a real research organization that wrote real open-access papers that aren't marketing brochures, and when they did large training runs, they released all artifacts including model weights. Now certainly they are anything but. We haven't learned learned anything meaningful about ML from OpenAI since GPT-3 was released.
Their open-weights competitors like Facebook can at least claim some kind of public benefit, but it's still just running a well-understood algorithm on dubiously obtained data with longer and longer runs, give or take some inconsequential architectural tweaks.
Anthropic's mechanistic interpretability work is the most "lab-like" of these, but it's still just secondary to selling subscriptions and fear-mongering for regulatory capture/investment/publicity.
sobellian 16 hours ago [-]
I reflected on this myself recently. Model distillation seems to be at least as fair a use as distilling a book.
causal 15 hours ago [-]
More than fair if you consider that the tokens are paid for.
dathery 15 hours ago [-]
Both labs even explicitly promise the customer owns the outputs. It feels like they want to have their cake (ensure enterprises don't get spooked away from using as many LLMs as possible) while eating it too (still arguing some level of control over the outputs).
> Ownership of content. As between you and OpenAI, and to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output. We hereby assign to you all our right, title, and interest, if any, in and to Output.
> As between the parties and to the extent permitted by applicable law, Anthropic agrees that Customer (a) retains all rights to its Inputs, and (b) owns its Outputs. Anthropic disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs.
Obviously there is some bad behavior going on in the distillation scene with gray-market token resellers but that is "just" normal fraud.
zenoprax 15 hours ago [-]
> Both labs even explicitly promise the customer owns the outputs.
> to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output
If the argument is that the model itself is under copyright protection then "as permitted by applicable law" would be doing some heavy lifting. Assuming that were true, given that locally-run LLMs exist, what would be illegal: the distillation itself or the provision of service of the distilled model?
visarga 13 hours ago [-]
Distilled content can also sever the direct link to infringement if the new models never saw the original texts.
15 hours ago [-]
16 hours ago [-]
15 hours ago [-]
toomuchtodo 16 hours ago [-]
YC does better if its startups get open weight frontier benefits. Garry’s just advocating for his book, which is his job. Consider how much capital YC portfolio companies would have to burn until liquidity if they have to pay OpenAI and Anthropic, versus relying on open weight frontier capabilities.
SOLAR_FIELDS 15 hours ago [-]
If someone proposes the right thing for selfish reasons, do we call that bad? Or do we call it proper incentive alignment?
dofm 14 hours ago [-]
We used to call it enlightened self-interest.
15 hours ago [-]
visarga 13 hours ago [-]
> versus relying on open weight frontier capabilities
ahem.. it happens even today, you can use open weight models directly and even fine tune
dvt 14 hours ago [-]
I think OpenAI and Anthropic will go bust, or at least be scrapped for parts in the next 5 years or so. It's clear that the extreme cost used up for training is impossible to recoup, as inference is already being subsidized.
It's also clear that, as Tan indicates, open-weight models will be (and basically already are) just as good as frontier models. It's all about the harness, baby. We will have two main forks in the road, and two new industries created:
- AI hardware (NVidia/Cerebras/etc.), the equivalent of Intel/AMD
- AI software (harnesses, assistants, etc.) the equivalent of Microsoft/Apple
We already saw a glimmer of this with popularity of OpenClaw—the problem is that it's janky, hard to set up, inconsistent, and very hacker-esque. Imo "AI labs" will be a dying breed because there's no real money in the actual models if they get commoditized, which they already kind of are.
Legend2440 13 hours ago [-]
>inference is already being subsidized.
Inference is not being subsidized and in fact has pretty high margins.
Similar-sized open weight models on openrouter are 15x cheaper per token than the big labs. This should reflect the isolated cost of inference, since 3rd party hosts have no reason to subsidize and no training costs to amortize.
Only datacenter buildout costs are being subsidized.
reticulates 12 hours ago [-]
The majority of revenue comes from API usage. The majority of usage comes from subscriptions. For any of the numbers to make any sense, subscriptions must be subsidized ergo the majority of usage is subsidized. A single $200 subscription can incur upwards of $10,000 in API equivalent usage (and even more when there are frequent resets).
If it were true that Anthropic and OpenAI were profitable on all inference they wouldn’t need to constantly raise so much money. Anthropic regularly announce huge investments in infrastructure but it is all smoke and mirrors, data center build out costs aren’t being paid by OpenAI and Anthropic, they’re financed externally. Google, for example, are backstopping tens of billions of datacenter build outs that are being financed based on commitments but not investment from Anthropic.
You are underestimating the insanity of subscription subsidization. Being profitable on API inference is meaningless when it is such a small proportion of usage and is only going to fall off a cliff as cheap open weight models become more capable.
The absolute majority of tokens are being subsidized and as soon as the subsidies end usage will fall off a cliff, rendering all the data center buildout a terrible waste of money.
delecti 10 hours ago [-]
> The majority of usage comes from subscriptions
Do we know that? As I understand it, enterprise customers pay more. Do we know the usage breakdown between monthly subscribers vs enterprise accounts? I agree that it's inevitable that subsidized subscriptions are unlikely to last forever, but that's not the only assumption in your argument.
Edit: I think "enterprise customers pay more" was poorly phrased. I mean that enterprise customers are charged per token, presumably with a profit margin, and thus are not subsidized. While personal accounts are (thought to be) highly subsidized if you consistently max out the quotas. We also don't know what proportion of personal accounts do that though, which is another big question mark.
nl 8 hours ago [-]
> The majority of usage comes from subscriptions.
This is untrue.
You are way underestimating enterprise usage here.
You can't get the $200 subscription on Teams plans at all, and Enterprise plans don't have any subsidized plans.
The numbers in the article are forecasts but let’s take them as real. That’s $10bn of revenue, the majority from enterprise customers, let’s say 75% from enterprise API usage: $7.5 billion. If the margin on inference is 80% that means of the $7.5bn in enterprise revenue they’re spending $1.5bn on compute. Yet we know that they actually spend over $5bn per month on compute, which includes the $1.25bn per month to SpaceX.
If $7.5bn is their enterprise revenue and it costs just $1.5bn to generate, that leaves $3.5bn in compute costs to account for. Dario previously said that training costs less than inference so training can’t explain it.
If subscriptions aren’t the majority of usage and aren’t subsidized, where is the money going? Anthropic don’t spend money on data centre build out so that can’t be it either.
rovr138 9 hours ago [-]
I think you're also missing a quirk and that is, is everyone on a $200 plan using $10,000 worth of equivalent API spend?
I know people that have the most expensive plan on all the platforms... because
The other side to that is, what is 'cost'? Is cost just inference or are expenses also being taken into account? Because the expenses of these companies are huge to build the models.
denverllc 10 hours ago [-]
> Inference is not being subsidized and in fact has pretty high margins.
1. Companies are trying to decrease costs, not increase it, and are looking at alternatives
2. Competitors are catching up, and even if the frontier labs are "better" at some things (like writing plans or complicated analysis), the competitors can take a lot of the inference on routine tasks like implementing a well-defined plan
3. The frontier labs don't just need to have high margins right now. They have to pay back their massive liabilities.
dvt 13 hours ago [-]
> Inference is not being subsidized and in fact has pretty high margins.
I was referring to the "AI labs" here. Sam Altman himself conceded that OpenAI is losing money on the $200 subscription. Using open-weight/open-source models is indeed cheaper (and no reason for inference to be subsidized).
Legend2440 12 hours ago [-]
That's not what I mean. If competitors can offer tokens 15x cheaper, the big labs must have high margins per token. (which they can use to amortize training costs)
>Sam Altman himself conceded that OpenAI is losing money on the $200 subscription.
They have since stopped offering the $200 subscription, probably for this reason.
Subscription margins are harder to judge because it depends on usage; token costs are a better comparison.
kilroy123 10 hours ago [-]
I think this is very possible. Plus, something I don't see talked about enough here. The VERY fragile supply chain that keeps it all going. Look at what is happening in the Middle East.
The US can no longer keep global trade secure on the high seas. What if the supply chains for GPUs get disrupted for months, a year? Then what?
I fear Google will win in the longer run.
teravor 7 hours ago [-]
> inference is already being subsidized
idk where this comes from but it's laughably false.
the only place where actual subsidization (below cost) might be happening are the subscriptions. even that is unlikely because to be truly below cost you either need to offer below cost of electricity which isn't happening, have potential API users using multiple subscriptions or have opportunity cost loss due to saturation.
nullbio 6 hours ago [-]
It would probably be for the best if they did, and training became something that humans did collaboratively.
FanaHOVA 14 hours ago [-]
If harness is all that matters, a co-developed harness + model stack + large compute availability advantage + massive distribution advantage with data for post training will win the market.
willy_k 12 hours ago [-]
Inb4 Apple buys OAI in 10 years and gets 75% of the consumer market.
hn993302 9 hours ago [-]
If inference needs to be subsidized to be economical (idk if true), open models have the same problem.
galleywest200 9 hours ago [-]
I agree with this statement in general, but it “hurts less” to spend money when you are running things yourself. Hard for me to give a specific definition as to why, but it may be more palatable to companies to burn their own cash on their own hardware.
Maybe it is “sunked cost” or maybe it is “I will do it myself dammit”.
hn993302 9 hours ago [-]
Wouldn't it be the other way around? They're all using cloud services already for things that they could run themselves.
dofm 14 hours ago [-]
Controlling what users and customers do with API calls to closed weight models feels constraining, and there’s a role government can play here to normalize the fact that access to intelligence that was trained on broad public access data should itself also be more a form of a public good than something locked away behind restrictive terms of service
I do not agree with this man all that often, but that is very concisely put.
consumer451 14 hours ago [-]
> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider. “The nightmare scenario, the doomer scenario for AI is that there’s just one company,” he said. “It has the best access to capital. It has the best AI researchers. It runs away with it and suddenly there’s one company that’s monolithic. And that would be bad.
Well yes, as I think I said in a previous comment, on the current trajectory OpenAI and Anthropic will really stop releasing models due to distillation and regulatory pressures. Then, they would eat all knowledge work themselves, which would be the end of YC.
gr_norm 14 hours ago [-]
Society as a whole has paid into this technology: through the theft of its intellectual property, through having to deal with the pillaging of so many commons (digital or otherwise) by it, through skyrocketing energy and computing device prices, and even just through ordinary investment. Democratize the technology! At the very least, don't step in legally to prevent this from happening.
nullbio 6 hours ago [-]
DaaS - Distillation as a Service.
It's a good idea. It's not their data to begin with, anyway.
pton_xd 15 hours ago [-]
Agreed! Allow US companies to innovate by creating an ecosystem of smaller, more efficient open weight models and it will be a net benefit for everyone. Distillation is a good thing.
Preventing token-consumers from developing competing products should be litigated as anti-competitive behavior.
sabhiram 7 hours ago [-]
If frontier labs can distill the internet and all of our data, then we should be able to distill their models further too.
The fact that billions were spent on research to distill the internet should not preclude others from spending 10s of thousands to do the same to these frontier labs.
Time to create a bigger moat than "but we spent so much money doing this ...".
hintymad 12 hours ago [-]
> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider.
Isn't this exactly what Dario wanted? He thought he knew what's best for the humanity...
nullbio 6 hours ago [-]
It is. Dario the Book Burner will not good what he wants, the world sees through him.
sick_of_slop 14 hours ago [-]
Frontier labs trained their models on the entirety of human knowledge and didn't ask permission. It's a "want" or "should" it's a moral imperative to distill their models.
nijave 10 hours ago [-]
Not a lawyer but distillation sounds like a transformative work.
Same thing as Cliff Notes imo. In every other area of manufacturering and tech I can use a machine to build a new machine that competes with the original machine. Should Milwaukee be able to prevent DeWalt from using their drill to make a competing drill? Should Jetbrains ban Eclipse contributors from using their IDE?
credit_guy 8 hours ago [-]
My guess is that when you sign up for either Anthropic or OpenAI, the terms of use specify you can't use their model for purpose A, B, C, D. For example, you can't use their model to try to build biological weapons, or to try to extort people, etc. Most likely there is language there that you can't use their models to train other models. It's as simple as that. You agree to those terms of use, or you don't use their models.
danny_codes 4 hours ago [-]
How can we prove intent? I’m sure a clever actor can disguise their prompt and simply claim the LLM suggested such and such on its own.
It’s not like Anthropic or OpenAI have the faintest idea how their models actually work.
zetazzed 15 hours ago [-]
Ok, but how do the economics of this work? Based on its settlement, Anthropic paid an average of $3000 per work they scanned based on their settlement (https://tech-insider.org/au/anthropic-copyright-settlement-2...). They and OpenAI pay billions per year for a mix of experts and normal people to label or create data. Why would they continue doing this if the value of this is immediately copied by open models? If your goal is to end the economics of generating and buying data for AI (and I recognize for some people this is really the goal) then sure, but if you want AI for various subfields of interest to continue improving then it's not workable.
Back when people made arguments for software privacy, the argument was usually "big business will still pay and consumers wouldn't have paid anyways so it's ok for us to pirate" - I actually think that was fine for business software but terrible for indie games, whose market was 0% businesses.
But in the AI case, it's not like they get to keep some of the value of their investment - it all gets cloned into models that businesses and consumers alike are happy to use. If someone knows how labs could continue to fund data creation and acquisition in this model, please do share!
etdznots 14 hours ago [-]
They can’t they’re literally fucked, and it’s not society’s problem! The whole world doesn't have to bend over to make sure a couple of lunatics who believe they are building a doomsday weapon also have a viable business model
dofm 14 hours ago [-]
This made me laugh out loud but ain't it the truth.
kadoban 14 hours ago [-]
> Anthropic paid an average of $3000 per work they scanned based on their settlement
Not sure you get to count breaking the law and getting in trouble in your cost-of-doing-business. That's a little too on the nose.
You're basically arguing that a criminal syndicate must be allowed to continue and we're required to make their business model make sense?
dofm 10 hours ago [-]
> You're basically arguing that a criminal syndicate must be allowed to continue and we're required to make their business model make sense?
This is how Uber worked. They didn't just break the law in different countries, in several they actively misled government/law enforcement investigations. Google "Greyball".
kadoban 9 hours ago [-]
Oh I'm aware that they're ~all assholes and many have gotten away with worse. It just seems like a poor argument that we're required to make it happen for them.
kingleopold 14 hours ago [-]
%99 of the startups fail, they are venture backed. Nobody or no market forced them to spend like that. It's all their decisions
wonnage 14 hours ago [-]
Surely if you hoover up every book in existence to feed into an ai model you must be extracting more than 1.5B in value. If not then it’s not a viable business.
andsoitis 7 hours ago [-]
> He elaborated to TechCrunch that this means he wants smaller, American open-weight AI labs to use the same kind of training techniques on American frontier AI labs, giving the U.S. a more robust set of open-weight options that aren’t Chinese.
Those market entrants would face commodity pricing power vs. high capital costs, no?
Maybe there'd be ROI but I think there's another layer or competitive dimension that's neither frontier lab nor distilled model lab.
darepublic 11 hours ago [-]
I cannot feel anything but schadenfreude regarding anthropic having its IP stolen from it. Bravo Chinese labs, bravo
matt3210 13 hours ago [-]
Net neutrality anyone? If AI is critical to getting work done in the modern era, its access should be guaranteed. Anyone banned from accessing frontier AI is being forcibly left behind. This includes distillation.
rudicjri27 4 hours ago [-]
There are a couple of ideas that are very clearly being drip fed into the consciousness
- theft / distilling (ie cheaper open models that are not from the soon to IPO US corps cheated rather than innovated)
- danger / nat security (ie we can only trust the soon to IPO US corps to shepherd us)
YuechenLi 13 hours ago [-]
Distilling frontier models is a brute force approach that rapidly hits diminishing returns after bootstrap because of the unevenness of the data. The simpler and more effective method is to have dedicated "teacher" frontier LLMs to generate targeted training data sets specifically for training new models and adjust on the fly based on feedback from the student model.
xlbuttplug2 12 hours ago [-]
Eventually the top labs are going to collude and simply not release their best models to the public (if they aren't doing that already).
jobs_throwaway 12 hours ago [-]
Then the next tier of labs will be even closer to the frontier than present, and the top labs will lose their pricing power
xlbuttplug2 12 hours ago [-]
So far the next tier has only demonstrated that they can catch up to, but not necessarily leapfrog, what the top tier has put out publicly.
I suspect the top labs will come up with a business model that doesn't involve handing out their secret sauce for everyone else to reverse engineer. Perhaps restricting their top models to select high paying government/enterprise contracts. Or maybe a bespoke "describe the problem and we'll solve it for you" type service.
danny_codes 4 hours ago [-]
I think it’s laughable to think OpenAI has some special sauce that can’t be replicated easily. They simply have asymmetric access to compute and the dollars to power it. Thats the only moat here.
OpenAI doesn’t even know how their model works. Nobody knows how LLMs work. So it’s not like it’s technically difficult to replicate, just costly.
artk42 13 hours ago [-]
I can't believe to hear such a wisdom from Garry Tan.
neilv 14 hours ago [-]
Given the short-term pragmatic, conflicted way that AI tech adoption is happening... won't encouraging distillation effectively taint the entire space of open weights models, with the undisclosed biases of a few models that are under the influence of parties (certain billionaires and politicians) known for aggression and duplicity, and not for admirable ethics?
Following news of companies and projects increasingly moving to open weights models.
As AI gets more central to society, we really need to know how the weights were determined.
Open weights isn't just "free as in beer"; it can be "free as in the mystery drug that creepy guy chatting you up at the bar offered you". And maybe even he doesn't even know everything that went into the tablets, since he too was being worked, by an organ-theft ring who will be harvesting both of you tonight.
That's an analogy to get your attention. Your LLM probably isn't going to steal your organs. But in the current environment, it does and will have ideological biases determined by those with direct and indirect influence over it. And there will be a massive market for commercial influence biases (look at how previous generations of adtech invaded almost all technology companies). And there's incentive for military and spying capabilities to be buried in the models, perhaps as long-term sleepers. Maybe some organized crime trojans, too, depending which model you pick up.
In this low-trust environment of the current real world, we need genuine open source models, not closed "open weights", and not mindlessly distilling black boxes gifted by sketchy powerful interests.
Governments should be more concerned about the _people's_ personal data instead.
Ban data brokers before you ban distillation.
Legend2440 13 hours ago [-]
Unfortunately, the government doesn't want to ban data brokers because the government wants to buy from data brokers.
davidguetta 12 hours ago [-]
Yes and there's even a stronger argument that we could REQUIRE frontier model to be open weight / open source.
At the end of the day they were built from data that did not belong to them. So it would be fair that humanity REQUIRES to give back the output of that.
It's a bit like the free software thing: you can still make money from it and providing service to it, but if you build it based on another free stuff the derivative should be free.
Why not do the same for intelligence ?
jimmydoe 11 hours ago [-]
some people did bad things, now instead of punishing those people, we want rest of people all do bad things, because that's only fair.
matt3210 13 hours ago [-]
Distillation is fair use
fmnxl 15 hours ago [-]
If it were so easy why aren't the frontier labs doing it themselves?
layer8 15 hours ago [-]
Distilled models are worse than the original, so you can’t fully compete. Also, if all frontier labs did that, there would be nothing left to distill from.
quicklywilliam 15 hours ago [-]
I see it as analogous to companies building fiber in the public ROW during the last big infrastructure bubble. Under the Telecoms Act, these companies had to allow competitors to use their fiber at a fair price.
Similarly, AI companies should be required to allow distillation at a fair price. Fair Use doesn’t make sense as a social contract if it only cuts one way!
jimnotgym 15 hours ago [-]
But if they tried to set a fair price they would have to report how much money they are losing on each token sold. This might be bad for the real business of ai firms, hoovering up as much capital as they can
gnarlouse 13 hours ago [-]
you want smaller models with comparable capabilities. for resource efficiency, market efficiency, environmental conservation.
seydor 13 hours ago [-]
They should be called speakeasys
mlazos 9 hours ago [-]
I don’t think appeals to morality or ethics are required for this. You paid for the LLM’s output, you should be allowed to use it how you wish. The only reason distillation is a dirty word is the AI labs trying to spread FUD to protect their non-existent moat.
Garry Tan and Sam Altman recently did this interview together. They seemed pretty friendly with each other during it. Wonder what Sam Altman would say about Tan advocating for OpenAI’s models to be distilled.
Then again this is the same OpenAI that has gotten into legal trouble recently regarding Apple’s IP so who knows
14 hours ago [-]
TZubiri 10 hours ago [-]
I disagree, I think chinese distillation relies on making multiple accounts at a provider, signing Terms of Services and breaking them repeatedly, in addition to using fraud patterns like IP proxies and networks of credit cards.
I think that software execs should not incentivize users or other execs to break Terms of Services, or contracts of any kind.
An executive or manager of a company that breaks contracts is worth 0, there's no incentive to do business with them, if you know they will agree to doing or not doing something and then breaking that promise.
The word of a businessman is their most valuable asset, Tan is signalling that he is either misinformed on what Chinese distillation consists of, or that it's ok to do it.
FAQ:
- "But the frontier models do bad things too"
- An argument worthy of a 5 year old, one civil issue doesn't negate the other, bring it to a court if you have an actual claim against OAI or Claude, etc...
- "Companies have the right to reverse engineer"
- Ok, do it, but the moment you are creating 10K accounts in a Distributed fashion (Distributed as in the first D of DDoS), using IP proxies and stolen credit cards or your employees and employee family credit cards, you are not doing it because you believe you have a right, you are doing it despite not having a right to it.
EDIT:
Re(actually)reading the article, Tan's take is a bit more nuanced, he seems to be advocating for regulation to restrict the capacity of Foundation models to restrict usage, on the basis (or to the extent) that it was trained on public data, and therefore it belongs or attributes its success to a wealth of the commons.
My pre-existing quip is against those that want to solve this as-is by breaking the ToS. I think that's a weak version of Free Software position, it's very weak to complain that some software is proprietary and want to use it anyway, the strong FS position is that you don't even want to use it if it's proprietary, you won't catch a FS activist pirating proprietary software, they just don't use it and develop alternatives. Similarly it's not a FS position to distill a proprietary model (where you still wouldn't have source code at any rate).
danny_codes 4 hours ago [-]
Plenty of contracts end up being unenforceable.If people think they have a strong case for breaking a contract, they are welcome to do so and see if a judge or jury agrees.
Edwinat23 14 hours ago [-]
Freefire
re-thc 15 hours ago [-]
There were comparisons and Muse Spark is so very similar to Fable / Opus... so...
nijave 10 hours ago [-]
Knowing Meta, I'd be more surprised if they _didn't_ distill frontier models than if they did...
etdznots 15 hours ago [-]
This is all based on the delusion that Chinese labs are mindlessly distilling the frontier.
I would love for a US lab to be at or near the frontier with an open weight model, but it’s going to take some serious elbow grease, and yes some distillation (which btw OAI, anthropic et al, also use distillation of other’s outputs in their training)
testfrequency 10 hours ago [-]
OAI and Anthropic remain the biggest heist ever in our lifetime.
Genuinely fucking crazy we pay money for fast access to autocomplete of stolen human remains.
okasaki 15 hours ago [-]
Like Gates saying there should be UBI, or Musk saying... well, whatever.
They know it won't happen, so arguing for it is 'effectively free' and purely personal marketing.
A bullshit game played by politicians and wannabes.
seanmcdirmid 15 hours ago [-]
Gates probably honestly believes in UBI; the guy is practical to a fault but evil misleading genius he is not. I actually don’t see any better options than UBI long term.
dgellow 13 hours ago [-]
Gates has been a ruthless fairly evil genius business man his whole life
wannabe44 14 hours ago [-]
Only ways to rise in a UBI society where AI is supposed to replace intellectual work is crime and prostitution. Smart people who want better lives than the average will have to get into crime.
seanmcdirmid 14 hours ago [-]
A UBI society doesn't mean jobs aren’t available. There most certainly will be jobs. But with UBI and universal healthcare, the jobs can pay whatever the market really demands. People always complain about the government subsidizing low Walmart wages for example, but with UBI that argument is moot. Liberalizing the labor market wouldn’t mean less jobs, it would mean more (we would also have to lean more on corporate and consumption taxes rather than taxes around employment which would also make employment easier).
andriy_koval 12 hours ago [-]
> we would also have to lean more on corporate and consumption taxes rather than taxes around employment which would also make employment easier
I think the only way forward is wealth tax. Rich accumulated so much wealth already, that they don't need to put it to profitable businesses.
seanmcdirmid 11 hours ago [-]
Wealth taxes require liquidating investments early and don’t promote good practices, especially if they are applied broadly. I would opt for a tax on loans instead (you have to prepay tax on loans that use investments as collateral and no more step up in basis on death).
danny_codes 4 hours ago [-]
Counterpoint, who cares?
Markets will simply adjust and financial services will arise to smooth instances where liquidation is problematic.
The status quo shifts to optimize once taxes change. Our system will do the same.
I don’t understand why people don’t think it through all the way to the next equilibrium
andriy_koval 10 hours ago [-]
> Wealth taxes require liquidating investments early
it works for RE already without liquidation
> . I would opt for a tax on loans instead (you have to prepay tax on loans that use investments as collateral and no more step up in basis on death).
rich just will move from this scheme to something else, and nothing changes.
seanmcdirmid 8 hours ago [-]
The rich will just move to a different country, negotiate with a Swiss canton for a fixed sum tax. I don't see how a tax that punishes wealth is sustainable, the wealth will just go somewhere else.
Maybe if we had a world government or something.
andriy_koval 8 hours ago [-]
US currently hold enough power to make them paying their debts oversees too.
Also, I think it is equilibrium of interests of all involved parties: rich pay 0.5%/y of wealth tax, population is healthier, country is stronger and protects interests and safety of rich. The problem is mostly to make them realize this.
seanmcdirmid 7 hours ago [-]
> US currently hold enough power to make them paying their debts oversees too.
The US is increasingly a less relevant country every year. With a wealth tax, yes, all those Americans will have their wealth eventually confiscated (0.5% a year will add up quickly year after year, investments will unwind so wealth will stop accumulated, and eventually it's gone), but I doubt the rest of the world will care and just move on and decide crazy things that happen in the US are like the crazy things that happen in Iran.
andriy_koval 3 hours ago [-]
> With a wealth tax, yes, all those Americans will have their wealth eventually confiscate
US already have 0.5-1.5% property tax, no one calls it confiscation.
I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct.
I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.
Obviously it's not possible to run a company whose value is predicated on its IP that uploads said IP to a third party which might get access to it.
This could mean every potential serious customer would have no option but to seek alternatives to these online services.
I don't mean this as rhetoric, I did not think many people (except possibly those operating under government contracts, and 'normies' who don't know about these things) were under the belief that their IP was kept secret when they use these services.
https://privacy.claude.com/en/articles/10023548-how-long-do-... (see the additional info section)
You pay more for it, but if you care that much, use it.
AWS and Azure give you the same thing for Claude and ChatGPT, no need to be stuck with open weights. They might sometimes store some of it for other purposes (I don't know the specifics), but it is emphatically not being fed back to OpenAI or Anthropic.
The only verification you could do so far is GDPR-style data export, and also the adherence to GDPR regulations (and even those might get skirted if they aren't operating in europe).
Your competitor is happy and continues to pay for the subscription. Sam and Dario just resold your code.
For what it's worth LLMs still suck at reproducing my little secret algorithm/implementation while being able to solve way harder problems. I have a good guess why that's the case.
The services have toggles to allow prompts to be used in the training set. There is a conspiracy theory that the toggle is a false distraction and they’re actually keeping everything, and that none of the employees involved will ever whistleblow this fact.
Outside of Internet comment sections, I think most people assume these US-based companies are doing what they say.
For enterprise use there are services like AWS Bedrock which have strict isolation guarantees. There are some people who still believe those guarantees are a lie, but once someone has reached that point I don’t think they trust anything that isn’t running entirely within their house. People in that category are a very small minority, but a very vocal minority.
It’s still a commonly held belief that “Facebook sells your data” and it’s cool to be cynical about everything tech in many social scenes. Conceding that a tech company might be honest about something will get you classified as a bootlicker depending on who you talk to so the only winning move is to be super cynical.
Among actual professionals I work with in tech and legal, almost nobody holds a belief that these companies are blatantly lying to their customers (and zero of their employees are whistleblowing it, while said companies also have employees trying to whistleblow AI safety on Twitter daily)
I think this is being misunderstood. Codex has a toggle to allow your prompts to be included in training data. They’re saying they can’t be sure if the person had it on or off while using Codex to discuss the work.
They’re not saying that some prompts are mysteriously jumping into training data.
Also, there is a large market for AI services which don’t retain anything under any circumstances for enterprise customers.
[1] https://x.com/thsottiaux/status/2097746417012166816
Conspiratorial thinking is very common on these topics. Even bringing up the conspiracy theory about Instagram listening to your conversations and showing you related ads will bring up a surprising amount of people defending that idea on Hacker News.
You have access to all the frontier models, but...your inputs are not shared with the model vendors...neither are used to train the next model.
Why am I even doing the Amazon board job for them!??
https://www.wsj.com/tech/ai/jury-sides-with-openai-sam-altma...
Besides, true ZDR is usually offered by third-parties with deals to host OpenAI models, such as Amazon (AWS Bedrock) and Microsoft (Azure).
If you are big enough to be able to withstand that, you’re already running (or trying to run) your own/open-weight models.
2023:
"The approach also aligned with the company’s broader deployment strategy, to gradually release technologies into the world for people to get used to them. Some executives, including Altman, started to parrot the same line: OpenAI needed to get the “data flywheel” going."
https://www.theatlantic.com/technology/archive/2023/11/sam-a... https://archive.is/NmO5P#selection-979.907-979.1177
I don't think this has been a big secret.
LLMs should just pay a flat fee to use a specific book and thats it. Fees should be reasonable (not a million dollars per book), so long as the model doesnt spit out the entire book.
By your phrasing, it sounds like you still intend the possibility of companies owning copyrights; but how does that happen (other than copyrights already owned by companies grandfathered in)?
Copyright always starts off in the hands of individual human beings; it only ends up in the hands of companies when those human beings transfer ownership to a company. That ownership transfer can be automatic as a term of a contract, e.g. as part of a work-for-hire agreement. But no contract can cause the copyright to come into existence already held by the company instead of the individual. So if you abolish ownership transfer, you effectively make work-for-hire IP assignment invalid. What replaces it?
And, if "nothing"... then how do people pool the IP rights of their own small contributions to a large-scale work, into an IP pool that can be legally defended by a coherent legal entity, so that the large-scale work itself can have market value (i.e. so that sales of polished commercial bootlegs don't drive sales of the "authentic" work to zero)?
Keep in mind that, no matter how much we might want "mass distributed" media to have more-reasonable IP terms, the ability to sue for infringement is still critical to the existence of some forms of media. Especially "location-based" media, with no equivalent licensed broadcast right: movies still in theatre; concerts; live performances of plays and musicals; etc. If there's no legal team that can sue a movie theatre that shows an unlicensed copy of a given movie, then no movie theatre will ever bother with licensing movies again; "box office" goes to zero (from the movie company's perspective); and the incentive to create movies in the first place declines massively.
(You can see what this alternate world looks like from the few cases where movies screwed up the steps required to assert copyright, back before copyright was automatic. Night of the Living Dead (1968) is a good example: theatres — even upstanding large-chain theatres! — did indeed leap at the opportunity to show the movie unlicensed, and so Romero et al made effectively zero revenue off the work.)
I'm not saying this is an impossible problem. There are ways to accomplish this besides the way it's done now. (For example, individual-contributor IP could be retained by the original owners, but cross-licensed between individuals through a collaboration structure to form a coherent defensible IP pool, in exactly the same way that IP for e.g. video codecs is cross-licensed between corporations to form a coherent defensible IP pool today.) I'm just pointing out that the problem does need to be solved.
I would prefer it not be handed over across various copyright trolls, the fact nobody can sing "Happy Birthday" in a movie is still insane.
Or they could abide by the precedents they set and learn to compete. They shouldn't be allowed to have it both ways.
Vote (well-informed of the candidate's policies) in every election you can, even the local ones that seem of little consequence.
Convince others to vote.
Make demands of your elected representatives. You can mail them, call them, etc.
The government is the people.
The Reagan-era and beyond successful convincing of people that the government is an unchangeable black box made up of shady actors out to destroy everything (see: Republicans still going on about the 'deep state' when they run literally everything) is a big part of how we got to this place. It was a self-fulfilling lie, now coming true as the people who sold the lie start grasping for unending power.
But we still have the ability to vote our way out of it. If we continue to fail to do so, then at an evolutionary level we have to consider that we collectively deserve all the bad that comes from it.
See salvador allende in chile and mitternand in france for examples of voting without sufficient leverage.
your regan example occurred during a successful counterattack by capital that started under carter and crushed the labor movement.
Sometimes voting changes things, sometimes it doesn't. Naturally it's in the nature of most governments to not wildly change every election. This is usually thought of as an advantage. It's rare a wide majority of the electorate ever actually agrees on anything anyway, and tyranny of the majority or government that is too reactive to popular will is also not desirable.
But sometimes things are different, due to external or macro factors as you indicated. Some elections are more consequential than others. That doesn't necessarily mean any one election simply doesn't matter. It's simply good civic hygiene to vote. Think of all the people in red states who vote blue or vice versa in elections they will never win, never mind have their chosen candidate "change things". Does this mean they are fools? I salute them for such a faithful exercise of their civic privileges, that which generations of humans have died for. Your vote doesn't simply lose all value because you didn't get the outcome you wanted. It also doesn't preclude other avenues, like direct action or whatever your supposed alternative is.
It's good to vote simply out of civic obligation. Many people who say otherwise seem to merely be complaining that voting doesn't give anyone 100% of what they want. We've seen more and more "insurgent" candidates take over primaries. Increasingly people who complain about politics strike me as engineers who talk instead of build. Complaining about the effectiveness of your vote is like listening to an engineer who is always talking but never ships.
Like go run a campaign or solve the problem. What does saying voting doesn't matter solve? It feels like middle school age cynicism masquerading as insight. Politics has actually rarely felt so unsettled in America. Voting is likely more important than ever.
It absolutely does change things even in those cases. You just voted for the wrong person if they aren't doing what their supporters want.
Weird, weird weird take. How does any of this work. Like you cant influence the first decision but you can magically influence the second?
In 2028 a maximalist case could then be made to nationalize them.
The government actually running them isn't great, but no one ever gets 100% of they want in politics. It's about trafeoffs. If the labs use the current government to pull up the ladder, then the voters should vote for a government to swing hard to punish them.
In theory the threat of this should also act as a deterrent.
There's nothing weird about this at all. Politics is all about push and pull, carrot and stick. No one has permanent influence over affairs, and no one ever gets 100% of what they want. Yet it is entirely coherent to advocate for something like nationalization or take your pick of threat to private capital if they essentially cheat by pulling up the ladder because they've suddenly decided they don't want to compete.
Its very unlikely that AI will be on the ballot.
Why do you think AI will not be a major political issue in 2028? Perhaps to level set do you think it is a political issue now? Do you expect it to be less politicized in two years? Why?
Bernie wants there to be an extreme shutdown of AI immediately.
So lets say you had 2 billion dollars of slush and worked for OpenAI and anthropic.
You would immediately start to grease the paws of republicans to support just enough regulation to slow your competitors. And you would start to grease the paws of democrats to push for a "Moderate centrist" position away from Bernie, leading towards Regulation.
Ice Cream Vendor problem leads you straight into both sides supporting the same moderate regulation that OpenAI/Anthropic will write for them.
Keep in mind its going to be a noisy election cycle with Donald Trumps idiot successor vs whatever goofball the democrats vomit up arguing about tariffs, trade partners, healthcare, ICE etc.
Democrat falls out of line "You just want China to beat us at AI"
Republican falls out of line "You are just a techbro supporting epstein affiliate"
Then theres the standard 99% + chance regarding US election promises that people just dont act on it at all.
Etc etc etc.
However the frontier labs don’t have to serve customers who are farming the service for distillation purposes. That’s their choice and they’re free to make it if they detect distillation happening.
The only way US maintains dominance over Chinese models is by having an ecosystem of models. Relying on a small set of frontier labs will only let you get ahead temporarily. I agree with Gary Tan on this one.
I don't think a correct remedy is to require companies to provide services even if they want to. A simple example: you drop a client because their asks / ways-of-working / etc is more headache and costs than it's worth. I've done that before, multiple times, in my freelancing life.
(I assume there's a don't missing)
Forced open API access/protocol transparency for tech platforms and a right to client-side consumption, essentially making the server a common carrier would be one of the most elegant and simple ways to effectively destroy every issue internet platforms pose in terms of competitiveness or data ownership without creating endless fault-ridden legislation.
Tech giants aren't freelancers, they deal with billions of interchangeable users and aren't going to succumb to an annoying customer.
So why is their own ToS so special? :)
Daily reminder that piracy is the only enduring archive mechanism.
Also, permitted or no, they are definitely torrenting. I would be deeply surprised if they hadn't already leeched every torrent on public trackers. The only reason they (probably) haven't depleted all the private trackers too is that they would be required to actually contribute back, which as above is never going to happen.
They were torrenting, they were sent to court and settled for big $$$$. The only other method available to them now is scanning, and scanning at scale requires the books destruction.
I agree that they should definitely be required to see the new scans but that would just be more $$$$ they get charged if caught.
By copying their programming style, you'll move the model towards that way of writing, which will move the model towards the values expressed in those documents.
I feel that Deepseek v4 got so claudified at the end that it was like Claude.
There is sooo much irony here.
Thats literally what is happening right now though. People are spending hundreds of millions on a training run, and then people are distilling them, fairly easily, and making cost competitive models.
We are seeing all of this in action right now.
I also think if Anthropic and OpenAI had been releasing Open models along the way, people wouldn't be nearly as suspicious of them.
Anthropic has never released anything but FUD.
I’m sure the labs claim that their real innovation is in the RLHF, training and architecture. Keep that and just share the raw data somewhere.
Figure we’ll have to reckon with this next year in any case, guess we’ll see.
Someone with that lab could almost certainly figure out how do something stupid or destructive on their own, or bypass model safeguards somehow.
> He also notes that the proprietary AI labs didn’t ask permission when they vacuumed up as much human knowledge as they could to train their models. They famously ingested plenty of copyrighted material without the permission of those intellectual property holders.
And many people's shared opinion (B):
>> I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct.
...
It is very difficult to actually say NO to the fact that (A) was done, which then leads logically to conclusions as (B). But also we should remember that if these two hold (and (A) is an axiom more or less now), then it comes as no surprise that then also all opensource licensing is immediately rendered void and null, as keeping it would contradict (A) and would go against the very common and consequential logic in (B).
Copyright is so dead. And it was not me killing it with a cynical post on HN. Dunno why so many people still fail to face it. There is no way it can exist in its current form, because then immediately (A) happens and (B) follows.
With what knowledge are you claiming this? If it turns out companies are using IP proxy networks would you change your mind?
What if the IP Proxy networks were used by criminals for similar attacks like DDoS or plain cyber attacks?
What if the source of the IP proxy networks were residential addresses to avoid detection?
What if the way these IPs were acquired were through pwned devices?
What if the credit cards used do not identify the company that carries the attack? What if they use the employee's personal credit cards? What if it's family members of employees? What if it's a network of personal credit cards where cc owners get a payment for making a purchase on their name? What if they are stolen ccs?
Not just a hypothetical btw, I believe almost all of these are true.
I think this should desactivate the moral high ground from which Anthropic is trying to speak. That they would want to make distillation orderly IMHO is fair, but to make it illegal is very rich from any AI frontier lab, really.
The moniker gives them an air of scientific, knowledgeable, tranquil, pro-social, pro bono work.-
Of course they are entitled to kill off a few mice, or pillage the commons to forward their "lab" work.-
The Atlantic argued this (rather well, IMO) a week or so ago - "There’s No Such Thing as an AI ‘Lab’" - https://www.theatlantic.com/technology/2026/09/stop-calling-...
But it's a black box! Nobody knows whats going on inside! It's all transformative! Sure...
But I get your meaning. What should they be called instead? AI Sausage Factories maybe (cue Upton Sinclair?)?
That's actually great? Slaughterhouses killing off the collective genius of humanity and grinding it into a bland paste for mass consumption.
Their open-weights competitors like Facebook can at least claim some kind of public benefit, but it's still just running a well-understood algorithm on dubiously obtained data with longer and longer runs, give or take some inconsequential architectural tweaks.
Anthropic's mechanistic interpretability work is the most "lab-like" of these, but it's still just secondary to selling subscriptions and fear-mongering for regulatory capture/investment/publicity.
> Ownership of content. As between you and OpenAI, and to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output. We hereby assign to you all our right, title, and interest, if any, in and to Output.
https://openai.com/policies/terms-of-use/
> As between the parties and to the extent permitted by applicable law, Anthropic agrees that Customer (a) retains all rights to its Inputs, and (b) owns its Outputs. Anthropic disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs.
https://www.anthropic.com/legal/commercial-terms
Obviously there is some bad behavior going on in the distillation scene with gray-market token resellers but that is "just" normal fraud.
> to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output
If the argument is that the model itself is under copyright protection then "as permitted by applicable law" would be doing some heavy lifting. Assuming that were true, given that locally-run LLMs exist, what would be illegal: the distillation itself or the provision of service of the distilled model?
ahem.. it happens even today, you can use open weight models directly and even fine tune
It's also clear that, as Tan indicates, open-weight models will be (and basically already are) just as good as frontier models. It's all about the harness, baby. We will have two main forks in the road, and two new industries created:
We already saw a glimmer of this with popularity of OpenClaw—the problem is that it's janky, hard to set up, inconsistent, and very hacker-esque. Imo "AI labs" will be a dying breed because there's no real money in the actual models if they get commoditized, which they already kind of are.Inference is not being subsidized and in fact has pretty high margins.
Similar-sized open weight models on openrouter are 15x cheaper per token than the big labs. This should reflect the isolated cost of inference, since 3rd party hosts have no reason to subsidize and no training costs to amortize.
Only datacenter buildout costs are being subsidized.
If it were true that Anthropic and OpenAI were profitable on all inference they wouldn’t need to constantly raise so much money. Anthropic regularly announce huge investments in infrastructure but it is all smoke and mirrors, data center build out costs aren’t being paid by OpenAI and Anthropic, they’re financed externally. Google, for example, are backstopping tens of billions of datacenter build outs that are being financed based on commitments but not investment from Anthropic.
You are underestimating the insanity of subscription subsidization. Being profitable on API inference is meaningless when it is such a small proportion of usage and is only going to fall off a cliff as cheap open weight models become more capable.
https://hraness.com/writing/my-girlfriend-asked-me-why-i-hav...
The absolute majority of tokens are being subsidized and as soon as the subsidies end usage will fall off a cliff, rendering all the data center buildout a terrible waste of money.
Do we know that? As I understand it, enterprise customers pay more. Do we know the usage breakdown between monthly subscribers vs enterprise accounts? I agree that it's inevitable that subsidized subscriptions are unlikely to last forever, but that's not the only assumption in your argument.
Edit: I think "enterprise customers pay more" was poorly phrased. I mean that enterprise customers are charged per token, presumably with a profit margin, and thus are not subsidized. While personal accounts are (thought to be) highly subsidized if you consistently max out the quotas. We also don't know what proportion of personal accounts do that though, which is another big question mark.
This is untrue.
You are way underestimating enterprise usage here.
You can't get the $200 subscription on Teams plans at all, and Enterprise plans don't have any subsidized plans.
Anthropic has 80% margins on inference: https://archive.is/BtEeN#selection-1575.0-1575.75
If $7.5bn is their enterprise revenue and it costs just $1.5bn to generate, that leaves $3.5bn in compute costs to account for. Dario previously said that training costs less than inference so training can’t explain it.
If subscriptions aren’t the majority of usage and aren’t subsidized, where is the money going? Anthropic don’t spend money on data centre build out so that can’t be it either.
I know people that have the most expensive plan on all the platforms... because
The other side to that is, what is 'cost'? Is cost just inference or are expenses also being taken into account? Because the expenses of these companies are huge to build the models.
1. Companies are trying to decrease costs, not increase it, and are looking at alternatives
2. Competitors are catching up, and even if the frontier labs are "better" at some things (like writing plans or complicated analysis), the competitors can take a lot of the inference on routine tasks like implementing a well-defined plan
3. The frontier labs don't just need to have high margins right now. They have to pay back their massive liabilities.
I was referring to the "AI labs" here. Sam Altman himself conceded that OpenAI is losing money on the $200 subscription. Using open-weight/open-source models is indeed cheaper (and no reason for inference to be subsidized).
>Sam Altman himself conceded that OpenAI is losing money on the $200 subscription.
They have since stopped offering the $200 subscription, probably for this reason.
Subscription margins are harder to judge because it depends on usage; token costs are a better comparison.
The US can no longer keep global trade secure on the high seas. What if the supply chains for GPUs get disrupted for months, a year? Then what?
I fear Google will win in the longer run.
the only place where actual subsidization (below cost) might be happening are the subscriptions. even that is unlikely because to be truly below cost you either need to offer below cost of electricity which isn't happening, have potential API users using multiple subscriptions or have opportunity cost loss due to saturation.
Maybe it is “sunked cost” or maybe it is “I will do it myself dammit”.
I do not agree with this man all that often, but that is very concisely put.
Well yes, as I think I said in a previous comment, on the current trajectory OpenAI and Anthropic will really stop releasing models due to distillation and regulatory pressures. Then, they would eat all knowledge work themselves, which would be the end of YC.
It's a good idea. It's not their data to begin with, anyway.
Preventing token-consumers from developing competing products should be litigated as anti-competitive behavior.
The fact that billions were spent on research to distill the internet should not preclude others from spending 10s of thousands to do the same to these frontier labs.
Time to create a bigger moat than "but we spent so much money doing this ...".
Isn't this exactly what Dario wanted? He thought he knew what's best for the humanity...
Same thing as Cliff Notes imo. In every other area of manufacturering and tech I can use a machine to build a new machine that competes with the original machine. Should Milwaukee be able to prevent DeWalt from using their drill to make a competing drill? Should Jetbrains ban Eclipse contributors from using their IDE?
It’s not like Anthropic or OpenAI have the faintest idea how their models actually work.
Back when people made arguments for software privacy, the argument was usually "big business will still pay and consumers wouldn't have paid anyways so it's ok for us to pirate" - I actually think that was fine for business software but terrible for indie games, whose market was 0% businesses.
But in the AI case, it's not like they get to keep some of the value of their investment - it all gets cloned into models that businesses and consumers alike are happy to use. If someone knows how labs could continue to fund data creation and acquisition in this model, please do share!
Not sure you get to count breaking the law and getting in trouble in your cost-of-doing-business. That's a little too on the nose.
You're basically arguing that a criminal syndicate must be allowed to continue and we're required to make their business model make sense?
This is how Uber worked. They didn't just break the law in different countries, in several they actively misled government/law enforcement investigations. Google "Greyball".
Those market entrants would face commodity pricing power vs. high capital costs, no?
Maybe there'd be ROI but I think there's another layer or competitive dimension that's neither frontier lab nor distilled model lab.
- theft / distilling (ie cheaper open models that are not from the soon to IPO US corps cheated rather than innovated)
- danger / nat security (ie we can only trust the soon to IPO US corps to shepherd us)
I suspect the top labs will come up with a business model that doesn't involve handing out their secret sauce for everyone else to reverse engineer. Perhaps restricting their top models to select high paying government/enterprise contracts. Or maybe a bespoke "describe the problem and we'll solve it for you" type service.
OpenAI doesn’t even know how their model works. Nobody knows how LLMs work. So it’s not like it’s technically difficult to replicate, just costly.
Following news of companies and projects increasingly moving to open weights models.
As AI gets more central to society, we really need to know how the weights were determined.
Open weights isn't just "free as in beer"; it can be "free as in the mystery drug that creepy guy chatting you up at the bar offered you". And maybe even he doesn't even know everything that went into the tablets, since he too was being worked, by an organ-theft ring who will be harvesting both of you tonight.
That's an analogy to get your attention. Your LLM probably isn't going to steal your organs. But in the current environment, it does and will have ideological biases determined by those with direct and indirect influence over it. And there will be a massive market for commercial influence biases (look at how previous generations of adtech invaded almost all technology companies). And there's incentive for military and spying capabilities to be buried in the models, perhaps as long-term sleepers. Maybe some organized crime trojans, too, depending which model you pick up.
In this low-trust environment of the current real world, we need genuine open source models, not closed "open weights", and not mindlessly distilling black boxes gifted by sketchy powerful interests.
Ban data brokers before you ban distillation.
At the end of the day they were built from data that did not belong to them. So it would be fair that humanity REQUIRES to give back the output of that.
It's a bit like the free software thing: you can still make money from it and providing service to it, but if you build it based on another free stuff the derivative should be free.
Why not do the same for intelligence ?
Similarly, AI companies should be required to allow distillation at a fair price. Fair Use doesn’t make sense as a social contract if it only cuts one way!
Garry Tan and Sam Altman recently did this interview together. They seemed pretty friendly with each other during it. Wonder what Sam Altman would say about Tan advocating for OpenAI’s models to be distilled.
Then again this is the same OpenAI that has gotten into legal trouble recently regarding Apple’s IP so who knows
I think that software execs should not incentivize users or other execs to break Terms of Services, or contracts of any kind.
An executive or manager of a company that breaks contracts is worth 0, there's no incentive to do business with them, if you know they will agree to doing or not doing something and then breaking that promise.
The word of a businessman is their most valuable asset, Tan is signalling that he is either misinformed on what Chinese distillation consists of, or that it's ok to do it.
FAQ:
- "But the frontier models do bad things too"
- An argument worthy of a 5 year old, one civil issue doesn't negate the other, bring it to a court if you have an actual claim against OAI or Claude, etc...
- "Companies have the right to reverse engineer"
- Ok, do it, but the moment you are creating 10K accounts in a Distributed fashion (Distributed as in the first D of DDoS), using IP proxies and stolen credit cards or your employees and employee family credit cards, you are not doing it because you believe you have a right, you are doing it despite not having a right to it.
EDIT:
Re(actually)reading the article, Tan's take is a bit more nuanced, he seems to be advocating for regulation to restrict the capacity of Foundation models to restrict usage, on the basis (or to the extent) that it was trained on public data, and therefore it belongs or attributes its success to a wealth of the commons.
My pre-existing quip is against those that want to solve this as-is by breaking the ToS. I think that's a weak version of Free Software position, it's very weak to complain that some software is proprietary and want to use it anyway, the strong FS position is that you don't even want to use it if it's proprietary, you won't catch a FS activist pirating proprietary software, they just don't use it and develop alternatives. Similarly it's not a FS position to distill a proprietary model (where you still wouldn't have source code at any rate).
I would love for a US lab to be at or near the frontier with an open weight model, but it’s going to take some serious elbow grease, and yes some distillation (which btw OAI, anthropic et al, also use distillation of other’s outputs in their training)
Genuinely fucking crazy we pay money for fast access to autocomplete of stolen human remains.
They know it won't happen, so arguing for it is 'effectively free' and purely personal marketing.
A bullshit game played by politicians and wannabes.
I think the only way forward is wealth tax. Rich accumulated so much wealth already, that they don't need to put it to profitable businesses.
Markets will simply adjust and financial services will arise to smooth instances where liquidation is problematic.
The status quo shifts to optimize once taxes change. Our system will do the same.
I don’t understand why people don’t think it through all the way to the next equilibrium
it works for RE already without liquidation
> . I would opt for a tax on loans instead (you have to prepay tax on loans that use investments as collateral and no more step up in basis on death).
rich just will move from this scheme to something else, and nothing changes.
Maybe if we had a world government or something.
Also, I think it is equilibrium of interests of all involved parties: rich pay 0.5%/y of wealth tax, population is healthier, country is stronger and protects interests and safety of rich. The problem is mostly to make them realize this.
The US is increasingly a less relevant country every year. With a wealth tax, yes, all those Americans will have their wealth eventually confiscated (0.5% a year will add up quickly year after year, investments will unwind so wealth will stop accumulated, and eventually it's gone), but I doubt the rest of the world will care and just move on and decide crazy things that happen in the US are like the crazy things that happen in Iran.
US already have 0.5-1.5% property tax, no one calls it confiscation.