(Un)Marked Mac OS
I give up. After upgrading to OS X 10.9.3, I've experienced the least reliable computer of my life. I've had 16 kernel panics in 4 days, usually when plugging in remote displays, but sometimes just spontaneously.
In a meeting yesterday, a coworker exclaimed how reliable and awesome 10.9.3 was, and that he didn't have any kernel panics at all. I fantasized that his macbook spontaneously panic'd as he said that, proving him wrong in theatrical fashion, preferably with a loud bang and smoke. (It would also show that it wasn't just me.)
I settled for some debugging to see why his macbook was different, but he shooed me away from his keyboard as he was about to do an IMPORTANT DEMO, and didn't want me to jinx it. He plugged in a data projector.
Instant kernel panic.
Shell scripts are normally written and run on Linux systems but they’re almost as at home on a Mac system as they are on a Linux system. You can write a Shell script on macOS, and you can run it easily, far more easily than you can on a Windows 10 system. Note: The headings on this list indicate the Macintosh System bundle names; the bullet points indicate the version of the System File included in that bundle. This is to make it clearer for people searching for specific bundle versions as opposed to System File versions. Finder File versions are not indicated. 1 Classic Mac OS 1.1 Macintosh System Software (0 - 0.3) 1.1.1 System File 1 1.1.2. NOTE: these extras are not part of the FLAC project. Most (except those marked $) are freely available but distributed under their authors' own terms. Check out the links page for an even larger list of software supporting FLAC. How to Ping on Mac OS. This wikiHow teaches you how to ping an address from your Mac. Pinging an address will tell you approximately how strong your connection to an address is. If you want to see how many gateways are between your.
He was now the third co-worker to experience this after upgrading to 10.9.3.
tl;dr: Jump to Update 8 for the workaround.
OS X saves diagnostic reports for each panic in /Library/Logs/DiagnosticReports. Note the dates:
These can be useful to browse for patterns. Maybe it's just one application which I can stop using?
No, it's completely random. This looks a lot like bad or misseated DRAM, so our helpdesk performed what they called a 'shell swap'. This means they replace everything except the SSD. Or put differently, they take out the SSD and put it in a 'known to be good' macbook pro, to see if the panics continue.
They continued.
Here's an example report:
So, CR2 (0x000000000000007a) looks bogus, leading to the panic. I can't infer much more than that, since I'm missing kernel symbols, and the stack trace is hex.
It would be nice if the kernel wasn't stripped, or, if it was easier to get the Kernel Debug Kit (please put them on opensource.apple.com). I'd help debug this further, but can't without symbols (at least, easily).
A few of us are now have the recurring 10.9.3 panics. After sending Apple many diagnostic reports with additional details, I'm switching back to 10.9.2. 10.9.3 is toxic with remote displays.
A coworker suggested the fix.
While I hope Apple fix this in 10.9.4, I'm now leery of OS X updates. I'd feel a lot better if I could debug this on my own further.
I should add that I've used Apple products and OS X for many years, and have been impressed by the reliability and quality of their work. 10.9.2 on the same laptop worked fine, and I'd have prior OS X releases with hundreds of days of uptime. I'd still recommend their products, as I hope this experience was an outlier.
Update 1
Symbols for this stack below (thanks Rasmus!)
Update 2
Based on the hackernews comments, some people have hit this and others haven't, and using external displays is a factor. Some have said that they have had issues with 10.9.2 as well, or all the 10.9 series. Perhaps it is a problem with a particular graphics card driver (myself and my coworkers have the retina Macbook Pros), but that's just a guess. The decoded stack above includes HFS calls, but that makes no sense, unless the graphics driver was stepping on random memory (which are among the worst panics to debug).
EDIT: After learning that this doesn't affect everyone, I changed the title of this post from 'Is Toxic' to 'Recurring Panics'. I also removed the words 'infected' and 'disease', which were off-putting.
Update 3
I got another new macbook pro, running 10.9.2 (although, a different kernel version), and switched using migration assistant. It worked great, initially. Then I had five panics in a row when connecting to different remote displays. In case migration assistant moved over some corrupted preferences, I got another new macbook pro, with 10.9.2, and just began using it fresh, and was still able to reproduce the panics (running only Firefox and Chrome, this time). I don't know if these panics are the same as what I had on 10.9.3, since I only have hex dumps to compare. 10.9.3 seemed to panic much more easily.
I'm a little fed up of the typical macbook debugging technique: switch things until the problem goes away. I'm also fed up with seeing stack traces that are inscruitable hex. I want to read the stack traces, and understand what the kernel is doing that led to the panic. So I studied how Rasmus had translated my earlier stack, and I learned that the default kernel (mach_kernel) does have some symbols, which aren't used in the diagnostic reports. Excellent. I can write a quick helper tool for translating stacks.
Update 4
I wrote kernel_diagreport2text.ksh, a tool that translates symbols from OS X kernel diagnostic reports using two different techniques. It is here on github.
Here's my new 10.9.2 macbook panics, summarized by kernel_diagreport2text.ksh:
I wish I had this tool earlier! It uses atos(1) for symbol translation, and decorates remaining addresses with kernel extension names (eg, 'in com.apple.driver.AppleUSBHub') if available in the diag report. It does not need the Kernel Debug Kit installed, although if it is, you should get more symbols translated.
That output is for a default 10.9.2 system, and while many symbols are missing, we can still learn a lot. All of these panics are in VM, and don't look the same as the 10.9.3 panic that was translated earlier.
To run this yourself, download (or save) the raw script. Then open up Terminal (which is under Applications->Utilities) for a command line, and you can run it on your saved kernel diagnostic reports. The steps are likely something like this (depends where your browser has downloaded the file):
This script is (obviously) not an official Apple diagnostic tool, and is provided as-is with no warranties or guarantees. It does not need to be run as root.
Update 5
29-May-2014. I've partially translated my 10.9.3 panics, using my kernel_diagreport2text.ksh tool described earlier. Here are some key examples:
The others showed similar stacks. These are also all in VM, either page faults or an explicit panic() call in the VM_PAGE_QUEUES_REMOVE macro.
To translate my 10.9.3 diag reports with the kernel_diagreport2text.ksh script, I needed a copy of the 10.9.3 mach_kernel (I'm back on 10.9.2), and to edit the script to point to it (update: that's now the -f option). Apple's auto update had already downloaded the 10.9.3 update, putting it in /Library/Updates/031-02348/OSXUpd10.9.3.pkg. That pkg file turned out to be Russian dolls: a xar, containing a bzip2 file, containing a cpio archive, which contained the 10.9.3 mach_kernel.
If you ever want to do something similar yourself, you really want to make sure the mach_kernel matches what you have in the diag report, otherwise the translations will be incorrect. Eg:
Those match! The source for xnu-2422.100.13 isn't out yet, but when it is, it should be under opensource.apple.com/source/xnu. I've been browsing the earlier version to get a handle on the VM code.
Update 6
After a quick browse of the VM code, it looks like a double free, based on the VM_PAGE_QUEUES_REMOVE panic. It's possible the other VM panics are manifestations of the same bug. These are nasty bugs to debug, as the engineer must track down the earlier free. This is harder than it sounds, as free's occur so frequently. I could run out of memory trying to log them for later lookup.
But after using DTrace on the vm_page_free_prepare_queues path, I'm not sure it's a double free using the vm_* interface, as mem->free was not set. Which suggests something even nastier – someone else is stepping on memory, perhaps zero'ing it out. Now if two paths are fighting over the same memory, and if I'm lucky, they do so in either order with the 2nd hitting the panic. Which could mean I've already captured both paths in the earlier panics. The IOBufferMemoryDescriptor::free() could be the non-VM path that's freeing memory, coming from IOAcceleratorFamily2.
This is just speculation - I don't know what the real cause is yet. But given the nature of the panics (external displays), the partially-translated stacks, the open source xnu kernel, and DTrace to test theories, I have a lot of clues. The hardest part is finding time to put into this.
Update 7
Here's an older panic excerpt (OS X 10.5.8):
Notice something? This has arguments for the stack functions, which are incredibly useful when doing panic analysis, especially when all we have to go on is the panic report file. So where did these args go in 10.9?
From the xnu-2422.90.20 source, osfmk/i386/AT386/model_dep.c:
PRINT_ARGS_FROM_STACK_FRAME needs to be set. It's set in the same file:
(shakes fist.) No comment, no explanation, just zero that guy. Why?
I think the reason can be explained by an earlier kernel version, xnu-2050.9.2:
Ah. Stack frame arguments are architecture specific, and the code was written for i386, but not x86.
It should be a fairly easy enhancement for Apple to add the x86 code, and greately enhance all kernel panic reports.
Update 8
30-May-2014. I have a workaround in hand for Mavericks 10.9.2: turn off Firefox hardware acceleration. With this off, my panics have now stopped. Perhaps this works on 10.9.3 as well, if it is the same panic. The setting is under Preferences->Advanced->General.
Other applications use hardware acceleration as well, so you may need to disable it elsewhere, if you believe you have this bug. For me, the easiest way to duplicate the panic was to run a youtube video full screen in Firefox, then plug in a remote display, and close the laptop lid. A few cycles of that usually led to the same type of panic I was hitting earlier. And with hardware acceleration disabled, it worked fine.
I still feel this was much worse in 10.9.3 (if I had more time, I'd confirm). The 10.9.3 update did say that it 'Improves 4K display support', which may have modified the hardware acceleration routines.
I suspect hardware acceleration was freeing or stepping on memory it shouldn't, which led to the VM panics. I'd like to write more about this, including where I was in the analysis, and ideally identify the root cause, but now that I have a workaround it's no longer a priority to work on. I may do a follow up blog post later when I have the time. I wanted to share the workaround ASAP.
(unmarked Mac Os Catalina
This information has been filed as Apple bug ID 17082120. (This is in addition to the 30 or so Kernel diag reports I've sent their way.)
Latest News: Check out what’s new for business this fall. Learn more about what's new for business
When the world changes, business changes too. Apple hardware, software, and services work together to give your employees the power and flexibility to do whatever needs doing — wherever that may be.
With great power comes great productivity.
Apple hardware, software, and services work together to deliver a seamless experience that just works. You can start a project on Mac and finish it on iPad, use your screens side by side to extend your workspace, and even draw with Apple Pencil on your iPad or use your iPhone to make live updates on your Mac. And it’s all compatible with apps from Microsoft and Google, so your team has everything it needs to get any job done.
Mac
Bring your biggest projects to life. Every Mac is designed for powerful performance — so you can build complex spreadsheets, create stunning presentations, or multitask across multiple projects.
iPad
Get power that outpaces most PC laptops in a design that goes everywhere. Scan merchandise, visualize models in 3D, and breeze through work when you multitask with Split View.
iPhone
Do incredible things on the go. Visualize 3D projects using augmented reality. Collaborate with your team on Keynote presentations. And stay connected with FaceTime, Messages, and Mail.
Apple Watch
Stay connected at a glance. Handle notifications as they pop up with a single tap, track Messages, and get the most out of apps for work and wellness.
Apple TV
Turn your best work into a cinematic experience. Put important presentations and>
Success Story - Capital One
Capital One
When people love what they do, what they do is amazing.
Anything’s possible with apps.
Apple devices come with powerful apps built in. The App Store offers even more tools for almost any job — from sales and engineering to fixing jets and building skyscrapers. And the Apple developer platform gives businesses the power to create custom solutions that the world has yet to see.
Built-in Apps
Notes, Siri Shortcuts, and Reminders make simple things even easier, like signing and scanning documents to share and adding a sketch with Apple Pencil on iPad.
App Store
Over 235,000 business apps help you get any job done, like Cisco Webex and Microsoft Excel for daily needs and industry-specific tools like Shapr3D and Scandit for specialized tasks.
Custom Apps
Build your own game-changing apps using cutting-edge technology for whatever your business needs.
Success Story - BSH
BSH
Custom apps make employees, and customers, happier.
Zero-touch
deployment is a
snap for IT.
Apple Business Manager makes Apple devices exceptionally easy to deploy and manage. IT can push apps and create Managed Apple IDs, and employees can customize their devices on their own.
Security first, second, and third.
Apple devices and platforms are designed to keep your personal data and corporate information secure. Key security features, like hardware-based device encryption, can’t be disabled by mistake. Touch ID and Face ID make it easy to secure every device. And because many of these features are enabled by default, employees and IT won’t need to perform extensive configurations.
Success Story - BDC
BDC
iPad made it possible to close small business loans onsite.
Apple Business
Manager
Deploy devices and apps and create Managed Apple IDs for every employee in one place.
AppleCare for
Enterprise
Support your IT team, repair or replace hardware, and train employees to get the most out of every Apple product.
Apple Financial
Services
Find flexible terms and end‑of‑lease options to get the most value from your investment.
Apple Professional Services
Get all your Apple devices up and running with hands‑on help from Apple engineers.
Apple Training
Prepare your IT team and in‑house developers to deploy Apple products and build custom apps.
Privacy
Every Apple product is built from the ground up to protect your privacy. We don’t create user profiles, sell personal information, or share data with third parties to use for marketing or advertising. And apps share only the information that you authorize.
Environment
Apple products are designed to reduce our impact on the planet while maximizing performance and strength. We strictly monitor our supply chain during manufacturing, are careful to design for energy efficiency, and work to make our products as recyclable as possible.
Accessibility
We build Apple products to empower everyone. Every device, every piece of software, and every service is created with accessibility features built in. Because when everyone can participate in the ways that work best for them, people and businesses are at their best.
How to Buy
We’re here to help you find the best, most cost‑effective solution for your business, whether you’re a one-person team or 10,000 strong.
Apple at Work
in action.
(un)marked Mac Os 11
See how businesses are using Apple products to create extraordinary experiences for their employees and their customers.