OpenBSD's De Raadt Pans 'Incredibly Bad' Disclsoure of Intel CPU Bug (itwire.com) 366
troublemaker_23 quotes ITWire:
Disclosure of the Meltdown and Spectre vulnerabilities, which affect mainly Intel CPUs, was handled "in an incredibly bad way" by both Intel and Google, the leader of the OpenBSD project Theo de Raadt claims. "Only Tier-1 companies received advance information, and that is not responsible disclosure -- it is selective disclosure," De Raadt told iTWire in response to queries. "Everyone below Tier-1 has just gotten screwed."
In the interview de Raadt also faults intel for moving too fast in an attempt to beat their competition. "There are papers about the risky side-effects of speculative loads -- people knew... Intel engineers attended the same conferences as other company engineers, and read the same papers about performance enhancing strategies -- so it is hard to believe they ignored the risky aspects. I bet they were instructed to ignore the risk."
He points out this will make it more difficult to develop kernel software, since "Suddenly the trickiest parts of a kernel need to do backflips to cope with problems deep in the micro-architecture." And he also complains that Intel "has been exceedingly clever to mix Meltdown (speculative loads) with a separate issue (Spectre). This is pulling the wool over the public's eyes..."
"It is a scandal, and I want repaired processors for free."
In the interview de Raadt also faults intel for moving too fast in an attempt to beat their competition. "There are papers about the risky side-effects of speculative loads -- people knew... Intel engineers attended the same conferences as other company engineers, and read the same papers about performance enhancing strategies -- so it is hard to believe they ignored the risky aspects. I bet they were instructed to ignore the risk."
He points out this will make it more difficult to develop kernel software, since "Suddenly the trickiest parts of a kernel need to do backflips to cope with problems deep in the micro-architecture." And he also complains that Intel "has been exceedingly clever to mix Meltdown (speculative loads) with a separate issue (Spectre). This is pulling the wool over the public's eyes..."
"It is a scandal, and I want repaired processors for free."
"I want repaired processors for free" (Score:5, Insightful)
You know, he's not wrong. This is, in impact, way bigger than Intel's FDIV fiasco and that ended up in recalls.
Re:"I want repaired processors for free" (Score:3, Funny)
Free i5s and i7s! I want to believe!
Re:"I want repaired processors for free" (Score:5, Interesting)
Re:"I want repaired processors for free" (Score:5, Interesting)
Oh I agree 1000%. It's not a freebie, it's Intel living up to the implicit contract to provide a CPU with the performance it was benchmarked when I bought it and not allow user mode stuff to read kernel memory.
In the UK you could make an argument that a processor with that bug was 'not fit for purpose'. Of course it's in the US that a class action suit has the highest chance of success and outside the US Intel will probably follow the US lead.
It'll be interesting to watch. Then again all my Intel chips are soldered to laptop motherboards. And rather elderly laptops at that - it's not like I'm going to convince Intel to convince Asus and Apple to recall motherboards that are out of warranty and do BGA rework to replace the CPUs.
However if I had machines with socketed CPUs and I was in the US I'd join a class action suit. Mind you Intel will presumably claim KPTI and its equivalents on Windows and macOS fix the security problem and any change in performance doesn't violate any sort of contractual agreement. Which they may or may not get away with. I think they probably will.
Re:"I want repaired processors for free" (Score:5, Interesting)
Some people are seeing >50% performance loss. Take a look at this graph: https://www.epicgames.com/fort... [epicgames.com]
Clearly they are going to need to spend some serious cash on upgrading their servers. The thread is full of players who can't connect.
Interestingly Intel's CPU data pages contain benchmarks. It will be interesting to see if they update them.
Re: "I want repaired processors for free" (Score:4, Insightful)
How far does the recall go? Should there just be a recall for Meltdown, or does that also extend to Spectre?
There wasn't a software workaround for the FDIV bug, which is why there was a recall. The F00F bug did have a software workaround, which is why there wasn't a recall for that bug. Meltdown also has a software workaround, though one with a potentially significant performance hit. Meltdown seems more like the F00F bug in that respect. Arguably, Spectre is a better candidate for a recall than Meltdown. Although there is a software workaround for it (see retopline), it cannot be implemented just by patching the operating system.
The problem here is that some of these features were designed over 20 years ago, when security wasn't as much of a priority. The feature worked and didn't present obvious security issues, so nobody tried to fix what didn't seem to be broken. It wouldn't surprise me at all if many other potentially serious vulnerabilities were lurking in hardware.
Re:"I want repaired processors for free" (Score:4, Interesting)
Re:"I want repaired processors for free" (Score:5, Insightful)
Intel simply cannot afford to recall all affected processors. Do not expect it to happen because it won't. They will obscurity-by-corporate-speak [youtube.com] their way out of this in a way that could make Enron's obfuscated lies look tame. If there were no software mitigation they would have few straws to grasp at, but the OS workarounds give them a tiny escape door and you better believe that they'll hire a whole crew of bulldozers to force this massve elephant through it.
Re:"I want repaired processors for free" (Score:3)
Re:"I want repaired processors for free" (Score:2)
As I recall the Pentium recall was option, i.e. you had to contact them to get your CPU replaced rather than them actively contacting you.
That seems like a reasonable way to handle it. Most people probably won't care enough to get their CPU replaced, but those who want the fix can get it. Of course it gets tricky with soldered on CPUs, for those they would just have to offer compensation.
Re:"I want repaired processors for free" (Score:2)
What if they replace 1 billion $1 dollar processors? Because if you manufactured a processor from 10 years ago with todays technology, it would probably be in the $1 range... And they wouldn't even have to replace them for free, just give me an option, I would gladly pay $50-$100 to have a drop in replacement of my sandy bridge that used 20% of the power and probably didn't even need a heatsink that is manufactured with the latest technology, but they won't, because they know I'm forced to upgrade to a latest offering, and buy a new cpu, motherboard, power supply, memory, etc They hardware isn't not even what annoys me, it's the reconfiguration and reinstallation of all of the software, for little to no speed improvement...
Re:"I want repaired processors for free" (Score:5, Informative)
Bwahahahahaaaa
it doesn't work that way at all.
Old designs would be for different process technologies. As the tech changes the DRCs (Design rule checks) change as well.
You can't run a design on a process it wasn't made for, the resulting product simply won't work correctly (if at all).
If the CPU was designed for a gate width of 35nM then it was designed with biasing around that gate width's leakage. If you then try to spin that part on a 14nM fab the biasing of the gates is all wrong and it will (likely catch fire) not work at all because of such high leakage.
Additionally, price doesn't scale the way you imply. A wafer start costs about $1K. Doesn't matter what process you run on it (it does, but not really all that much). The cost per part is based on the number of functional parts per wafer at the end. Thus going from an 8 to a 12 inch wafer lowers cost even though the process change requires a $2.2bn fab to be built, you have gone from 201 sq inches to 452 square inches, over *double* the yield.
Same thing from process shrinks, you cut the area used by your transistor gates and you make the die smaller, then you can fit more on a wafer.
Thing is, Intel may not even have fabs capable of making the older parts any more, even if they wanted to. Process tech has evolved, IDK if they even have an 8" fabs left...
To just "redesign" the part for the new process is not realistic either.
TLDR: To make an old part will cost the same or more than it did when it was the latest and greatest.
Re:"I want repaired processors for free" (Score:3)
I'm inclined to agree, and I suspect Intel will negotiate (in court?) to do a recall of (say) all the chips on a device still in warranty or some such. That way there's at least a reasonable way to do it in a practical sense. Having a recall that you can't actually perform isn't really going to help anyone.
Not sure about the US, but in the UK there's provision in the law for stuff you couldn't possibly have known about. That is, let's say you make a teleporter and the world starts using it. If (in a few years time) it turns out that using it causes cancer, then you can't be held accountable if there was no way you could have known about it. Of course, once you did know about it and tried to cover it up, then you're absolutely liable. It's possible Intel could claim that they only found out about this problem in (say) 2014, thus they're only liable for the last three and a bit years of chip sales - but if some of the comments above are true, then they knew the risk when they designed the thing, and so almost certainly confirmed the problem sometime shortly afterwards.
I also wonder if I'd now be able to ask Apple for a replacement Macbook, as it's still in warranty and is 'broken'. This is somewhat untested because they could argue that it's still perfectly fine to use (perhaps in certain conditions), whereas I bought a machine which I believed would have reasonable security. Even if this whole idea was a bit shaky, it might cause Apple to go after Intel for all the laptops they've sold over the last few years. I'm sure they're currently talking to them about the chips they've ordered but haven't taken delivery of yet, and quite probably the last few batches they did receive though.
This may also set a precedent for all those dodgy webcams that turn out to be insecure... could be interesting.
Re:"I want repaired processors for free" (Score:2)
Because otherwise you're being asinine. There's no such thing as perfect products or products that won't break down or bug-free software or bug-free hardware.
Re:"I want repaired processors for free" (Score:3)
...security is paramount, usability is not.
This is exactly reversed: without something to use there is nothing to secure.
In other words, before making a car safe one must first have a car. So, while important (and different people place different importance on it) security is not the paramount.
Re:"I want repaired processors for free" (Score:5, Insightful)
Isn't he? Firstly clearly the distribution was too wide as it was given there was a moratorium on disclosure scheduled for tomorrow to allow all patching to be in place in advance.
Secondly he has in the past jumped the gun on responsible disclosure, parroting OpenBSD as the secure alternative patting himself on the back for being the first.
Thirdly there are multiple groups now that refuse to work with him for this very reason. The OpenSSL team also disclosed to others before OpenBSD for the same reason.
He shat in his bed, and now is complaining that he has to sleep in it.
Re:"I want repaired processors for free" (Score:3, Insightful)
Firstly clearly the distribution was too wide as it was given there was a moratorium on disclosure scheduled for tomorrow to allow all patching to be in place in advance.
You largely validated his posting with this one sentence. This is exactly what he's complaining about. How do you expect patching to be in place in advance for OpenBSD if the kernel developers weren't notified? Since OpenBSD isn't Tier-1, so they weren't notified. Apparently, only Microsoft was notified in advance, which is a clusterfuck so big, that should be reason enough to prove willful negligence by Intel.
I completely agree with Theo.
Re:"I want repaired processors for free" (Score:5, Insightful)
How do you expect patching to be in place in advance for OpenBSD if the kernel developers weren't notified?
You're missing the point. The OpenBSD team would be notified if they cooperated with the temporary embargoes that are in place to provide vendors time to patch before attacks are developed and deployed. They haven't, in the past, so they're no longer in the group that gets advance notice.
Re:"I want repaired processors for free" (Score:5, Insightful)
You largely validated his posting with this one sentence. This is exactly what he's complaining about.
And if you don't take that sentence out of context you'll have seen the point. What happened to OpenBSD and Theo is the fault of precisely one person: Theo.
Hell when we discussed this on Slashdot there were a lot of posters saying that Theo's actions at the time would hurt the OpenBSD community as people would not disclose the vulnerabilities to them. Looks like they were right too.
I agree the OpenBSD community is in a bad place. I also agree with Theo, but only in that his actions have spoken louder than his words.
Re:"I want repaired processors for free" (Score:2)
I won't argue he's wrong, but I think as fast as CPUs change you'd have to have across the board reductions in workload capacity by a significant number (ie, the 30% touted initially) to be able to claim harm and justify a recall.
So far what I've read is that performance in some storage applications has taken a small hit, but generally it's not a meaningful hit overall in CPU performance.
And even if you could prove a performance hit, how reliable are the numbers? Workloads vary a lot. And if you're not operating at the limit of your hardware, is going from 45% idle CPU to 25% idle CPU really a performance "loss"? It's arguable that you had surplus capacity to begin with and that unless you frequently max out your CPUs, you didn't really lose any capacity.
Re:"I want repaired processors for free" (Score:2)
Datacenters which already patched are apparently getting a performance hit in the high two-digits. Epic, for example, has people complaining [epicgames.com] because users cannot connect to game servers after the patch doubled servers CPU load.
Re:"I want repaired processors for free" (Score:3)
It's hard to understand if this is Epic finding a convenient whipping boy for their problems or an actual problem associated with the patch. There's always complaining about gaming server performance, outages, etc, so it's not like they couldn't have other problems and that this is just being used as an excuse.
Re:"I want repaired processors for free" (Score:3)
No, it is not. There were a number of F00F bug workarounds [rcollins.org], all implementable at OS level and with negligible (if any) performance hit.
Meltdown is nothing like it. The performance impact depends largely on the load type the CPU experiences, but it is estimated to range between 5% and 30% - which is terrible.
Re:"I want repaired processors for free" (Score:3)
There's no simple solution for Spectre, as is it much more widespread and affects pretty much every modern CPU. The only viable way is some sort of software-based mitigation.
De Raadt's rant was about Meltdown though, and he's absolutely right. Meltdown is a Intel-only fuckup; someone decided that protection domains should not apply to execution speculation in order to boost performance.
He and Linus are Spot On (Score:5, Insightful)
He's also dead right in that Intel has been mixing up the two issues, Meltdown and Spectre, deliberately, so they could tell everyone that it wasn't just Intel that was affected, and they also gave the impression that Spectre had been fixed when it was Meltdown that had been mitigated - with a patch that creates unacceptable performance problems, to a lesser or a greater extent.
Yes, all processor manufacturers are affected by Spectre, but it is Intel that is mostly affected because they implemented speculative loads badly without much attempt at segregation. They've also attempted to pass this off as 'historical architectural decisions we can do nothing about, but it is working as designed'.
Comment removed (Score:5, Informative)
Re:He and Linus are Spot On (Score:5, Informative)
As you state, it's important to rely on the original sources. Here is each CPU vendor's response to the security issues:
https://www.amd.com/en/corpora... [amd.com]
https://www.intel.com/content/... [intel.com]
https://developer.arm.com/supp... [arm.com]
Here are two corrections to make:
1) Meltdown:
One of your bold statements "AMD and ARM are not affected" is untrue. See here, from ARM directly:
https://developer.arm.com/supp... [arm.com]
ARM has confirmed that A75 is vulnerable to Meltdown. In addition, A15, A57, and A72 are vulnerable to a variant of Meltdown (Variant 3a) which ARM has added. ARM has stated that they believe this variant is NOT exploitable, however, there is already userspace code out there that can do some limited exploits:
https://github.com/lgeek/spec_... [github.com]
AMD is not affected by Meltdown, in any form. From AMD's press release:
https://www.amd.com/en/corpora... [amd.com]
2) Variant 1: While other vendors may require application changes to address this issue, AMD appears to be able to address this with an OS update, based on their post:
https://www.amd.com/en/corpora... [amd.com]
Summary:
Variant 1: Some manufacturers (ARM) appear to not be able to fix it and are recommending compiler changes, but AMD will fix this in OS updates. Unclear how Intel is addressing this vulnerability.
Variant 2: Correct, from what I can tell.
Variant 3 (Meltdown): Affects nearly all Intel (within the last 10 years) and ARM A75 chips. AMD not affected.
Variant 3a (Modified Meltdown): Affects a larger set of high performance ARM chips
Finally, Intel has done a terrible job (intentionally?) at conflating the two issues, which is unfair. These are 3 separate security issues, with their own priorities and impacts. If you read Intel's official press release for this issue, there's no differentiation between variants 1-3, like there is for AMD and ARM:
https://www.intel.com/content/... [intel.com]
Re:He and Linus are Spot On (Score:4, Interesting)
I skimmed both papers, and that seems to about sum it up. Though I would add that all three attacks cause speculative execution of a construction like; "x = array[ *pointer ];", to push memory from an array into or out of cache based on the data loaded from the victim pointer. So combining the announcement does make some sense, as the details of any of those variants might point people to rediscovering the others.
I was impressed with the work put into variant 2. Tricking the CPU branch predictor into running ROP-like gadgets within a higher privileged process, then using cache access timing to work out what happened. It almost sounds like bad sci-fi dialog, yet they actually did it. And yes, the attack complexity sounds comparable to similar ROP stack smashing exploits.
Variant 2 is being patched in compilers. Both gcc and clang [llvm.org] are working on patches (that might already be released?) that avoid any speculative execution of indirect branching. Using a trick documented by google to patch the stack with the destination address, and then return. So now we just have to recompile *everything* that has access to privileged / sensitive memory contents to hopefully prevent attackers doing anything useful with branch poisoning. Of course there will be a performance hit, as no indirect branches can be correctly predicted.
Personally I would say that the problem with variant 2 is sharing the branch predictor between domains. Branches taken in one process, influence how branches in other processes are predicted. I can understand that in a modern OS, multiple processes end up running the same library code, so this may have been a deliberate decision. But, if these tables were stored per-thread and context switched, this problem would probably have never been exploitable.
The Spectre paper did suggest that they had found some evidence of something like variant 2 on an AMD CPU. But I believe that the inner workings of AMD's branch predictor are not as easily deduced as Intel's. So the researchers took the easiest route and attacked 3 different Intel cores instead. That doesn't mean that nobody will ever work out how to pull off an attack though.
Correction needed (Score:5, Informative)
>is hard to believe they ignored the risky aspects. I bet they were instructed to ignore the risk
The specific issue that Pentium line CPUs: a) do privilege check asynchronously; b) do it only for the "winning" execution branch was very well known among CPU design community.
Intel architects even bragged about that as their "innovation" in industry journals and filled a number of patents for that (this is the reason amd privilege checker runs on all branches)
Re:Correction needed (Score:4, Interesting)
Re:Correction needed (Score:2)
System calls where always slow because they used to be called via a software interrupt call.
SYSCALL is an in x64 instruction that speeds this up, introduced by AMD.
Speculative execution predates SYSCALL by about 5 years.
System calls are now slower because the kernel memory now has to be mapped and unmapped when the system call enters/leaves rather than be mapped all the time. This has to be done because memory that was marked as privledged can now be accessed by user programs i.e. memory protection no longer works and the only fix for the problem is to unmap the memory.
Speculative execution does n't mean we have have this problem, AMD managed to do it fine. No one can say this is by design, if it is by design then it should be documented since 1995 that the MMU protection can be bypassed.
Re:Correction needed (Score:5, Informative)
System calls where always slow because they used to be called via a software interrupt call.
And software interrupts were slow because they were not considered branches by early branch predictors and so triggered a complete pipeline flush equivalent to a branch mispredict (followed immediately by another branch, which SYSCALL removed). Intel addressed this by treating software interrupts as normal branches for the branch predictor, with an extra hint that they changed privilege level. This gave a small improvement to the Pentium, but was a huge boost on the Pentium 4, where the pipelines were long and deep enough that they had up to 140 instructions in flight at a time and having to flush all of those for a system call was painful.
Speculative execution does n't mean we have have this problem, AMD managed to do it fine. No one can say this is by design, if it is by design then it should be documented since 1995 that the MMU protection can be bypassed.
Speculative execution across ring changes is the root cause of this. AMD doesn't do this because Intel patented it, told AMD, and didn't include it in their cross-licensing agreement. You can bet that AMD was just waiting for the patent to expire before doing it, because without it you have to wait until all branches up to the system call have been retired before you can perform the transition. The MMU protection isn't bypassed, because the instructions that would be bypassing the MMU protection are cancelled. There is a side channel that allows you use the changes in cache behaviour to determine what the values in memory would have been.
Re:Correction needed (Score:2)
Comment removed (Score:2)
Core issue is trust (Score:4, Interesting)
Intel makes a monumental decision to benefit the short-term interest of their corporation at the long-term expense of their customers, then tries to weasel out of a equitable fix for their customers? It's not only their product that can't be trusted, it's their judgement at all levels. Heads need to roll at Intel for this....
Re:Core issue is trust (Score:4, Interesting)
Re:Core issue is trust (Score:5, Informative)
It was also a short and long term benefit of their customers. Are you willing to pay Intel back for the extra performance they provided by their same decision that you are deriding today?
Eh? We've already paid the Intel for the performance. Intel CPUs are that bit more expensive than equivalent AMD CPUs, performance is why they commanded the price premium.
Customers trusted Intel that the performance was gained with no cost to security, a reasonable assumption. I'm computer literate, and I'm shocked that this can even be an issue. How the hell do speculative memory accesses leak through kernel memory protection?
So I'm not sure what you think is to be paid back.
Re:Core issue is trust (Score:3)
Intel makes a monumental decision to benefit the short-term interest of their corporation at the long-term expense of their customers, then tries to weasel out of a equitable fix for their customers? It's not only their product that can't be trusted, it's their judgement at all levels. Heads need to roll at Intel for this....
Intel's stock price is down about $2 US since this became known, so I don't think I'd bet much on heads rolling. The problem is that most of the world is not IT geeks and for those non-iT people, many of them are scared of AMD. They know nothing about CPUs, so to them Intel is the safe choice because only Intel advertises on TV. Oddly, this may actually increase their fear of AMD because, as pointed out, Intel has managed to convince everybody that all CPUs have both problems so non-IT people are likely to think that Intel will probably fix it quickly and who knows when those weird "other companies" will do so.
Dream on (Score:5, Interesting)
"It is a scandal, and I want repaired processors for free."
And I want a pet unicorn. Come to think of it, unicorns are about as real a thing as a "repaired processor" since they physically cannot be repaired. He wants a replacement processor which almost certainly is never going to happen. Basically he's asking for every processor produced in the last 20 years to be replaced for free. If you think that's realistic I've got a bridge to sell you.
There will be plenty of legal action over this and the results of that will be the full extend of any compensation. Furthermore to get compensation he will have to show actual harm incurred. Simple fact is that at least so far there has been little to no tangible harm from this problem to date so standing will be an issue for anyone who sues. This might change in the coming months/years but until it does the chip makers aren't going to pay a dime to replace anyone's chip - flawed or otherwise.
Re:Dream on (Score:2)
"It is a scandal, and I want repaired processors for free."
And I want a pet unicorn. Come to think of it, unicorns are about as real a thing as a "repaired processor" since they physically cannot be repaired. He wants a replacement processor which almost certainly is never going to happen. Basically he's asking for every processor produced in the last 20 years to be replaced for free. If you think that's realistic I've got a bridge to sell you.
Ironically, it is realistic for one obvious reason; consumers are fucking lazy.
When you look at the actual affected processors vs. the number of people who will actually get off their ass and make a claim to have a processor replaced for free, it becomes very clear how affordable this really is. People are lazy, and don't give a shit. 75% of consumers won't even know or understand what the fuck "Meltdown" is a month from now, no matter how many times it's broken down in laymans terms on the evening news and morning talk shows. Two months from now 95% of people will have forgotten about it.
Good point but... (Score:3)
When you look at the actual affected processors vs. the number of people who will actually get off their ass and make a claim to have a processor replaced for free, it becomes very clear how affordable this really is.
A good point but not as good as you make it seem. Quite a lot of affected chips are not user serviceable for reasons such as being soldered to the board. This means that it isn't a simple matter of sending a customer a new chip. In many cases the boards in question won't be available for any price so the most a chipmaker could do would be to pay for a replacement. A repair simply isn't going to be an option in many cases.
But you are right, 99.99999% of people will probably have forgotten about this in a week and the only people concerned about it are the 5 or so of us who still read slashdot. Most people won't have even heard about the problem and most of those who do wouldn't know how to deal with it anyway. Basically it will go away when the current generation of PCs is retired in a few years.
Re:Dream on (Score:3)
Replacing the CPU on on one host often puts every system in the rack at risk. Most household systems can stand a loss of a few % of performance with a patched kernel. Server rooms filled with racks and blades, such as a major data center hosts, can mean unscrambling rats' nests of cabling to extract a host, opening it up, edging blocking components out of the way, releasing the heat sinks, replacing the CPU, _replacing the thermal paste_, and re-attaching the heat sink, closing the system up, and testing it. Much like taking your car in for an oil change, this creates a real risk of making mistakes and requiring additional effort or replacement parts. It also creates a risk of failures in the weeks after the CPU replacement, especially if the installer mishandles the thermal paste.
The risk is compounded in environments with poorly configured cabling, such as those shone here: https://www.cepro.com/photo/th... [cepro.com]
Completely halting and then cold booting servers is not a zero risk operation. Hosts, or arrays, that have been stable for years will fail to reboot and may even be unrecoverable in an environment where systems are not rebooted regularly and discovered earlier. Mechanical parts, such as fans, and spinning hard drives, are most likely to fail during such a restart. Old clock batteries can expire and fail to set time properly on reboot, old power supplies can fall out of spec and fail to handle start-up voltage requirements, the list of potential problems is extensive.
Replacing CPU's in a production environment can be as great a risk as the security issues of these Intel bugs.
Re:Dream on (Score:2)
I wonder how many people actually requested a replacement processor from Intel back when they had a recalled the FDIV bug, and how many of those users actually replaced their processor once they got it. Are there any numbers out there on that? Replacing a CPU is a pain in the ass, so I wonder how many people went through the effort to do it.
Re:Dream on (Score:3)
I mean replacing a CPU is a non-trivial task in a desktop, nevermind impossible for laptops which are the majority of purchases. And even if you could find someone else to do it you have to give up your device while they do, and you can bet that won't be a quick procedure.
How many components of your car, or washing machine, boiler, oven have *you* considered replacing? Even if you know what components there are in there, and what flaws they might have? Would you even know what particular components are in your car's engine management unit, for example? Because that's how most people view computers - they bought a Dell, not an Intel.
It's not "fucking lazy", it's "fucking have other shit to do, and don't have the technical expertise or time".
Re:Dream on (Score:2)
Everyone running an intel CPU will suddenly discover that they have 5-30% less processor than they paid for. In many cases, that difference would have resulted in going with the AMD processor. That is a real economic harm that deserves compensation.
That doesn't mean justice will be done, the courts often bend over backwards and grab their ankles for large corporations, but morally and ethically, Intel owes a lot of people a pile of cash.
Product liability is a funny thing (Score:2)
Everyone running an intel CPU will suddenly discover that they have 5-30% less processor than they paid for.
Untrue. The hardware is unchanged and can run exactly as fast as it always did. Security updates and software slow machines all the time. You'll have to credibly explain why this case requires special consideration. Easy to claim here but not so easy in front of a judge. Frankly most people aren't even going to notice and the few that do notice are going to be corner cases, mostly in large corporations with adequate resources to deal with the problem. From a 10,000 foot view it's not clear why this is meaningfully different than any other serious malware incident. It's not as if it is the first time a flaw has been taken advantage of in hardware.
In many cases, that difference would have resulted in going with the AMD processor. That is a real economic harm that deserves compensation.
I'm afraid it's not nearly that easy. Product liability [wikipedia.org] is a funny thing. First off there is not likely any strict liability [wikipedia.org] here so there is no automatic requirement for compensation. Second, there is unlikely to be any physical danger to anyone. Third, while there is a cost to dealing with the issue it isn't clear that the cost is out of line with other serious security issues including malware. Finally, you still haven't shown the economic harm. I'm sort of playing devil's advocate here just to illustrate the challenges here. In principle I agree with you but the law very likely may not agree with us.
Re:Dream on (Score:2)
He wants a replacement processor which almost certainly is never going to happen. Basically he's asking for every processor produced in the last 20 years to be replaced for free. If you think that's realistic I've got a bridge to sell you.
How is this different (aside from magnitude/number of units sold) from Takata's airbag recall? I wasn't affected by an exploding airbag, but I still get new airbags in my Dodge. Interestingly, it's apparently still going on [usatoday.com]
I suspect a recall this large would bankrupt Intel, much like the airbag recall is bankrupting Takata. We've seen our automakers get bailed out because they were deemed "too big to fail", but maybe Intel failing could be a good thing (though I don't know of anyone that can simply step up to replace them).
Nothing like Takata (Score:3)
How is this different (aside from magnitude/number of units sold) from Takata's airbag recall?
Nobody is going to die from this mistake. Pretty big and important difference there. Product defects that result in actual provable fatalities tend to get a lot more scrutiny.
I suspect a recall this large would bankrupt Intel, much like the airbag recall is bankrupting Takata.
Won't happen and while this is a serious issue, it isn't THAT serious. I expect Intel will pay some cash to settle some class action lawsuits (and so will some other chipmakers) but that's probably about as far as it will go unless there are revelations we haven't heard yet.
Re:Dream on (Score:2)
It's a single data point, but an interesting one: Engaget's Facebook page posted yesterday about Intel's new processors with AMD graphics. Every single comment below the post asked why on Earth anyone would buy an Intel processor without any assurances the Meltdown/Spectre flaws were taken care of.
The fallout hasn't even begun to really hit on this.
Re:Not only replaced. (Score:2)
All my Intel processors are out of warranty, and that must be true for a humongous number of people. Since my processors are out of warranty, I don't expect a free replacement - I expect a low cost replacement. The cost to me of replacing the machines would be huge, so I am happy to pay (say $50) for a plug in replacement that ups the performance to what can be achieved in the same socket with today's technology. There is no point in forcing Intel to re-engineer old architectures - that benefits no one.
If Intel are not going to supply paid replacements at reasonable cost, then I expect them to be forced to supply for free all necessary info to anyone else who wishes to make and sell replacements to me on pain of jailing every single shareholder.
If they are not willing to co-operate with their customers in producing an equitable fix, the whole lot of them are guilty of unjust enrichment, obtaining pecuniary advantage by deception, fraud, conspiracy to defraud, and anything else a bunch of greedy corporate lawyers can invent.
If Intel goes bust as a result of this, we can expect the official receiver to sell off the patents to someone who will sell us cheaper and more reliable processors (probably not Cyrix).
Intel need to be wetting their knickers at the very least.
Backdoor-free processors for free? (Score:4, Interesting)
So do I. I want backdoor-free processors without payment. I will send Intel the faulty processors.
Intel CPU Backdoor Report (Jan. 1, 2018) [slashdot.org]
My opinion: Intel is a world-class company, with poor top-level management. Brian Krzanich is not the kind of person who is necessary. He is not a person with enthusiasm for technology combined with the social ability to lead a large company. One story about Krzanich: Intel CEO sold all the stock he could after Intel learned of security bug [arstechnica.com].
Paul Otellini, the previous CEO, was worse, in my opinion. Otellini "joined the finance department in 1974 [mercurynews.com]" I complained about Otellini 11 1/2 years ago in a Slashdot comment: More Intel employees should say in public what they have told me in private: Intel CEO Paul Otellini is not a competent leader. He lacks social ability. [slashdot.org] (June 09, 2006)"
Intel's health and strength is important to everyone on the planet, it seems to me. The technological part of the company can be excellent, but recent top management has not been able to handle the challenges.
The underlying issue, it seems to me, is that the process of choosing new CEOs tends to be defective. Perhaps all employees should have 50% of a vote, with the board of directors having 50%.
Fuck intel! (Score:3, Interesting)
I dont care how much better their next CPUs might be, im jumping ship for AMD on my next upgrade. I did the same after NVidia fucked me over.
"I bet they were instructed to ignore the risk" (Score:5, Interesting)
I was one of those who called "no way" at first, but just yesterday I found this quote [danluu.com] from an Intel engineer. It was originally posted in a reddit thread [reddit.com] but has since been deleted - but not before being confirmed by other former engineers at Intel.
Re:"I bet they were instructed to ignore the risk" (Score:3)
If I may, I'd have to call this an anecdote rather than a quote. The description is from years after the Intel meeting, and doesn't have direct quotes of speech or writing of the personnel involved in the policy change.
With that understood about the anecdote's provenance, it is _completely_ believable. It is precisely the sort of mandate that can save a company in the short term, preserving the jobs and careers and technological development the company is doing, at the risk of a deadly failure down the road. It's the sort of business risk assessment that occurs on an annual basis when testing standards and guidelines are set. It also occurs on a daily basis when security practices are created: do we accept the risk of a breach today, while this is unpatched, versus the risk of service failure or loss of business during system updates?
Re:"I bet they were instructed to ignore the risk" (Score:3)
Time for open hardware (Score:2)
Sure, but... (Score:2)
Re:Sure, but... (Score:2)
Why do people say this? Intel is sitting on $17 BILLION in cash. Intel apologists are incredible.
Re:Sure, but... (Score:2)
All we can do is wait for the next CPU and see if better testing was done.
Then upgrade.
The global buying cycle of CPU upgrading has a few options:
For people who play computer games and need a fast CPU:
Trust the same brand to get a new CPU thats works next time.
Try another existing brand that has a different, new CPU that is tested and reward them.
Want security and know to walk away form the brands that fail?
Find, create a better CPU and get a fab to make that for CPU use that has to be secure.
Re:Sure, but... (Score:2)
Re:Sure, but... (Score:2)
Software is part of the problem (Score:2)
I'm no fan of Intel but I can't help but see Meltdown as at least partly the fault of OS vendors. They decided to keep kernel and user memory in the same address space for performance despite known problems with branch prediction and despite KPTI being an option.
I doubt Intel ever claimed prediction could not be detected nor forced kernel devs not to use KPTI. Usually when a vulnerability is found in a software architecture you blame the software.
Re:Disagree (Score:4, Interesting)
Re:Disagree (Score:5, Insightful)
Re:Disagree (Score:4, Insightful)
Standing up for your principles can have a cost attached, and I suspect we've just seen what that was for Theo and the BSD developers.
Re:Disagree (Score:5, Insightful)
Re:Disagree (Score:3)
Re:Disagree (Score:5, Informative)
Sure it does. If you want to keep something quiet until you are ready to announce it, then you DO NOT tell any of the people who have a track record of spilling the beans.
When has FreeBSD ever disclosed a security vulnerability under embargo? FreeBSD has a security officer and a secteam group that are the only ones that have access to any embargoed security information and have separate infrastructure from the rest of the project for preparing fixes. Only people who have signed the relevant NDAs are allowed access to anything shared with this group and they are normally given information about embargoed security issues as a result.
Regardless of where you personally stand on the idea of embargos and standing up for principles, Theo refused to go along with an embargo previously and it was quite likely that he wouldn't do so this time either
You do realise that FreeBSD and OpenBSD are entirely different projects, run by different people, with different infrastructure and different codebases and that Theo De Raadt has no connection to the FreeBSD project?
Re:Disagree (Score:3)
It also wasn't helped that there was no real coordination in releases.
There was, the problem is exactly what they were trying to avoid. It was under non-disclosure back in November. The actual full disclosure date wasn't supposed to be until tomorrow (9th Jan) so coordinated releases could take place. Unfortunately someone jumped the gun and we're left with the clusterfuck that this has become (Microsoft patches released, Apple patches released, UEFI patches not ready yet, Linux patches scheduled for original date).
This one of the very few downsides of Opensource. It's difficult to release bug information under NDA as efforts to patch the bugs result in the bug being revealed. That and Theo shat the bed with his past irresponsible patching of a bug early that was covered under a coordinated release schedule.
Secrets are only kept when few people know about them.
Re:Disagree (Score:2, Troll)
Re:Disagree (Score:2)
Yeah, that pissed a lot of people off.
Re:Disagree (Score:5, Informative)
Why did OpenBSD silently release a patch before the embargo?
OpenBSD announced an errata on 30 August 2017 that silently prevented our key reinstallation attacks. More specifically, patches were released for both OpenBSD 6.0 and OpenBSD 6.1.
We notified OpenBSD of the vulnerability on 15 July 2017, before CERT/CC was involved in the coordination. Quite quickly, Theo de Raadt replied and critiqued the tentative disclosure deadline: “In the open source world, if a person writes a diff and has to sit on it for a month, that is very discouraging”. Note that I wrote and included a suggested diff for OpenBSD already, and that at the time the tentative disclosure deadline was around the end of August. As a compromise, I allowed them to silently patch the vulnerability. In hindsight this was a bad decision, since others might rediscover the vulnerability by inspecting their silent patch. To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.
Re:Disagree (Score:3)
With a vulnerability disclosure of that scale, I can forgive a researcher for making a poor decision. Maybe you spend months to years bug hunting and make one discovery.
But someone who deals with a near constant stream of high severity bugs, should take a more considered position.
Re:Disagree (Score:3)
Wider disclosure of a vulnerability means a greater chance that it's leaked to those who would exploit it for malicious purposes.
This has a very poor underlying assumption: White hats always know about exploits prior to Black hats. Considering the latter are far better funded, have better incentives, and don't disclose; I think the betting is that it is already used for malicious purposes.
Additionally, this is the information age, not the dead tree, secret squirrel, stamped "for your eyes only" days. In this age, does it really matter if 100 or 10000 people know a confidential piece of information? You only need ONE individual in that set to provide the information to malicious actors. Additionally, there are a multitude of incentives and multitude of avenues for sharing with little chance of getting caught.
Given the above, full disclosure is far better than limited disclosure.
Also, due to the nature of the BSD license, it's code is in a ton of software around the world. The assumption that it isn't widely used is wishful thinking of the unknown.
Re: Disagree (Score:2)
According to whom?
Re:Freedom demands Open Hardware also (Score:5, Insightful)
This is a question of quality, not idealism and perverse incentives.
We aren't talking about IME here. You seem to be blindly assuming that Open hardware is always free of faults.
Re:Freedom demands Open Hardware also (Score:4, Interesting)
This is a question of quality, not idealism and perverse incentives.
We aren't talking about IME here. You seem to be blindly assuming that Open hardware is always free of faults.
This is a question of quality. You seem to be blindly assuming that starts and ends with hardware faults. It does not, and it was the main point Theo was making here. Quality also has to do with how you handle faults when they happen.
And I'd sure as shit trust an open community a lot more than a proprietary closed one hell-bent on protecting profits at all costs. How many more bugs does Intel know about right now that they refuse to disclose because it might affect stock price? I rest my case.
Re:Freedom demands Open Hardware also (Score:4, Informative)
We have seen what the best names in some sectors of the computing community did for security for years.
PRISM (surveillance program) https://en.wikipedia.org/wiki/... [wikipedia.org]
Open software and open hardware is a good start at having a few people have a look at computer security.
The big brands keep failing.
Generations of failed hardware, junk encryption, CPU's, OS and networking. Backdoors, trapdoors.
Re:Obligatory:Intel CPU Backdoor Report (Jan 1 201 (Score:2)
Please stop spamming.
Re:Freedom demands Open Hardware also (Score:4, Informative)
Not sure about others but some are available for purchase.
"SiFive has declared that 2018 will be the year of RISC V Linux processors" [slashdot.org] - Linux Now Has its First Open Source RISC-V Processor, Slashdot.
To answer AC's question a few moths later: "What's the big advantage with RISC over ARM or x86?"
Meltdown, Spetre.
Re:Freedom demands Open Hardware also (Score:5, Informative)
ARM is a RISC architecture, and plenty of RISC architectures suffer from Spectre. Meltdown is an Intel-only bug -- AMD doesn't have it because they implemented an obvious security rule, and presumably Cyrix and other x86 implementations didn't either.
Mixing brand name and technology (Score:2)
Linux Now Has its First Open Source RISC-V Processor, Slashdot. To answer AC's question a few moths later: "What's the big advantage with RISC over ARM or x86?" Meltdown, Spetre.
ARM is a RISC architecture, and plenty of RISC architectures suffer from Spectre
I think the GP was using RISC as an abbreviation for the RISC-V brandname, not RISC as in the architecture design approach. Indicating an advantage of RISC-V CPUs over ARM or x86 CPUs.
Re:Freedom demands Open Hardware also (Score:2)
Open hardware is going to be hard (Score:5, Insightful)
Open Hardware doesn't fix problems in silicon that has already been manufactured. It might help with the next generation but it won't prevent bugs from appearing in the first place.
Bear in mind that the reason Open Source software works so well is that the marginal cost of (re)production is close to zero and that there are (comparatively) minimal capital costs. Really you just need a PC and a lot of time. Open Hardware is a worthy goal but it's going to be a LOT trickier to pull off in the real world for mostly economic reasons. Furthermore hardware isn't protected by copyright for the most part. It's protected by patents and those are expensive. Worse once someone has one on a piece of kit they can basically shut down any open hardware that uses that idea for the next 20 years.
Re:Open hardware is going to be hard (Score:4, Interesting)
Of course, when Linux was new the argument was that an OS was just too big for a bunch of Free Software fans to manage. Only a big corporate structure could support development of anything as complex as an OS.
Open hardware is harder, but probably not impossible. It isn't a magic cure all, but it would tend to be free of corporate decisions like "we need 10% more performance, cheat here and nobody will notice" simply due to the open nature.
The patent swamp is a problem for that, but given how dependent the world is on secure digital hardware now, it's time to review the patent system. It may even become politically possible since it's to the point now where non-free hardware is hindering corporate profits.
Not the same (Score:5, Insightful)
Of course, when Linux was new the argument was that an OS was just too big for a bunch of Free Software fans to manage.
You are making a false equivalency here. Making and distributing software is COMPLETELY different than making and distributing hardware. The economics could not be more dissimilar. The legal protections (patents vs copyright) are different. The amount of up front capital required is different. You can modify software after it has been release but you cannot do that with (most) hardware. Basically just because it worked out well for software is does not mean it will work out well for hardware. Hope for the best of course but it's likely to be a difficult nut to crack.
Only a big corporate structure could support development of anything as complex as an OS.
Ultimately that turned out to be true. Basically all the developers of linux and most other major OSS projects are employed at large tech firms (and a few large foundations) and are paid to maintain them. It isn't a bunch of hobbyists in their garages.
Open hardware is harder, but probably not impossible.
Not impossible but for non-trivial applications it appears pretty close to it. The obstacles are predominately economic ones and some legal ones and they aren't minor obstacles. I'm not about to hold my breath for patent reform anytime soon and the patent swamp is a real problem. And the economics of making and distributing hardware are immutable. I think Open Hardware is a very worthy goal but it's going to be quite the challenge.
Re:Open hardware is going to be hard (Score:2)
Open Hardware doesn't fix problems in silicon that has already been manufactured. It might help with the next generation but it won't prevent bugs from appearing in the first place.
This is 100% correct. However, unlike with Intel, you would be able to get a 1:1 replacement with only the hardware issue fixed.
Open Hardware is a worthy goal but it's going to be a LOT trickier to pull off in the real world for mostly economic reasons.
Also 100% correct. That said, people are working out the details on drastically reducing the cost of microfabrication. It won't replace cutting edge mass production but it will make it prototyping and inspection of subsystems a lot easier. The idea here is to identify hardware/timing exploits before they ever make it mass production.
Re:Open hardware is going to be hard (Score:2)
All we can do is fix the next generation. Bugs can be prevented with much better questions, people doing things with new CPU's and sharing their test results.
The big brands have failed. The OS have failed.
Patent infringement (Score:5, Informative)
No, it's not tricky to pull off.
If it wasn't tricky to pull off then it would have already been done on a wide scale. I'm not saying it's impossible but it is going to be a much tougher nut to crack than open software. Mostly for economic reasons rather than technical ones.
- Research and make use of expired patents extensively, file new ones defensively.
Who is going to do this? Who has the funding and more importantly the incentive to do this? IBM received 8000 patents in 2016 and numerous other tech companies received thousands more each [fortune.com]. Exactly how do you plan to match that sort of pace? How do you plan to produce anything really useful without infringing on a pile of those patents? Not to mention fending off the flesh eating lawyers that give those patents teeth...
It's more capital intensive than software, but it's also not that expensive either.
I'm a certified accountant and an industrial engineer. I do cost accounting for a living. It is a LOT more expensive than software no matter how clever you are. There is a reason gross margins in manufacturing hardware are far thinner than in software. You don't escape these costs by just doing design either. Someone eventually has to make the product and that will require substantial capital. Then you have the cost of distributing the product. Unlike software which can be sent across the net for nearly free, hardware has to be shipped, stored and turned into products, all of which cost non trivial amounts of cash. If you think it isn't substantially more expensive than making and distributing software you haven't done the math.
Cost of outsourcing (Score:5, Insightful)
From a big picture perspective, the making of the hardware has already been detached from the design of the system.
Doesn't matter. You still have to make it and that still will cost money. Doesn't matter if you make it in house or if you hire someone else to do it. If doesn't matter if you have the secret formula for Coke, you still have to put sugar water in bottles and ship it somewhere which is expensive. It's FAR harder to bootstrap the funding for an open source hardware design than open source software.
Would a manufacturer take the risk of making a huge investment that relies on Open Source designs? They already do. Most mobile phones are entirely worthless without Android, an Open Source software.
You're conflating issues. You can already send an open source chip design to a chip fab or a hardware design to a contract manufacturer. My day job is general manager of a contract manufacturer (wire harnesses) so I'm more than passingly familiar. But just because you have outsourced production doesn't mean that the costs for it go away. Your analogy to Android is a meaningless one here.
Just because someone else makes it doesn't make the patent swamp go away. Open source software works precisely because how copyright law is written. The GPL and every other license basically only works because of copyright law. That doesn't apply to hardware. To protect hardware designs you have to get patents on the design and that costs serious money. Not only that you have to avoid infringing other companies patents which is not a trivial exercise when companies like IBM, Google, Apple, etc are getting thousands of new ones every year.
Companies that rely heavily on open source software can do so because they have an alternative revenue source. Typically service or engineering - sometimes ads. What is the alternative revenue source for open hardware? Service? Maybe but the revenue streams aren't quite as clear for open hardware. And even if they become clear it still doesn't solve the capital costs and patent issues.
I'm not saying it's impossible but it definitely will be difficult for open hardware to achieve the sort of success we've seen with open software.
Re:"I bet they were instructed to ignore the risk" (Score:5, Insightful)
Funny, both me and my friend worked at companies where we were told to ignore risk. Why would intel be different?
Re:"I bet they were instructed to ignore the risk" (Score:2)
Funny, both me and my friend worked at companies where we were told to ignore risk. Why would intel be different?
There is risk and there is risk. In the work I do risk means that someone could die. It may be that in your and your friends work risk means some people might not be able to access their social media platform for a few hours and not be able to post pics of their cats. For Intel the risk is that their systems run unchanged for multiple years at a time and have a huge market share and could require a hardware fix in order to correct any issues that come up.
Re:"I bet they were instructed to ignore the risk" (Score:3)
Funny, both me and my friend worked at companies where we were told to ignore risk. Why would intel be different?
Because all companies are not identical? Do you think everything you and your (imaginary?) friend have in common is also common to every company?
Also, what do you mean by "risk"? Do you think it's reasonable (or even possible) to eliminate all risk of a security breach? When did anyone accomplish that?
Re:"I bet they were instructed to ignore the risk" (Score:2, Funny)
Oh, Linus said it so that means he's an expert at CPU design. That's like saying competent engineers don't work at failed CPU companies yet Linus worked at Transmeta, so go figure.
Re:"I bet they were instructed to ignore the risk" (Score:5, Interesting)
To quote Linus "A *competent* CPU engineer would fix this by making sure speculation doesn't happen across protection domains."
That's bullshit. When Intel introduced speculation across protection domains everyone including Linux was happy because it reduced system call costs. Without this, as soon as you get to a syscall / sysenter instruction, you stall the pipeline until all pending instructions have been committed. On a modern Intel CPU with close to 200 instructions in flight at a time, that's a measurable performance overhead.
We've known for a long time that side channels of this kind were possible, but not that they were performant. The new attacks are not interesting because they're side channels that allow data to be disclosed, they're interesting because they're side channels that allow disclosure far faster than previously believed. CPU designers believed that this kind of attack could only be exploited to get a bit every few seconds, at which rate it's not really worth trying as an attack and is pretty easy for software to spot (hmm, why is this thread at 100% and triggering insane numbers of cache misses? Looks malicious...). Now we know that you can use these attacks to get data at about 0.5MB/s, so you can scan the whole of memory in a few minutes.
Re:"I bet they were instructed to ignore the risk" (Score:2)
When Intel introduced speculation across protection domains everyone including Linux was happy because it reduced system call costs
Is that the case? Was this attack known, and deemed an acceptable risk because of the incredible low rate at which data could supposedly be extracted? I remember some papers around 2015 on an attack involving data leaking between processes through the cache, but that did not rely on speculative execution IIRC.
Re:"I bet they were instructed to ignore the risk" (Score:5, Interesting)
Was this attack known, and deemed an acceptable risk because of the incredible low rate at which data could supposedly be extracted?
Not this specific attack, but it was known that any source of nondeterminism in a processor was a source of side channels. These were largely ignored because these attacks get lots of papers at top security conferences but are really hard and slow to carry out in practice. Most of the existing attacks used the cache, but there are others involving things like the fact that computation on denormals is much slower than on normal floating point values (a fun one of these lets you scrape browser contents via WebGL and I don't believe has been mitigated yet in spite of being published a couple of years ago).
Speculative execution was known to be a potential source of these side channels for a while. Now that it's shown to be feasible, expect a lot more similar attacks.
Re:"I bet they were instructed to ignore the risk" (Score:4, Informative)
Bullshit you say, and yet it's only Intel and a few, comparatively insignificant ARM chips which are affected by meltdown, which btw, was what Linus was referring to.
Ye, because Intel patented the technique and didn't license it to anyone else.
I can only presume AMD is an imaginary entity in your little world, because they apparently managed to solve all these impossible problems without handing out the keys to the kingdom to everyone who asked for them.
Nope, AMD pays a higher penalty on system calls, though they mitigate this to some extent by having shorter and narrower pipelines.
Re:Lot of speculation in his dire statement (Score:2)
Re:Lot of speculation in his dire statement (Score:3)
Risk isn't just a matter of how likely the discovery is, but how serious it is and how widespread the negative impact is. Given that Meltdown affects a huge population, even a tiny chance of discovery represents a huge risk, Intel doesn't want to do a recall since even chipzilla would be sunk by the cost. There's a big risk for you. Next time you consider Intel, just remember, you bought intel before and right now they're pointing at you and saying "HA-HA".
Care to spin the wheel again?