Debugging the FreeBSD Kernel Transparently

An anonymous reader writes "To debug the FreeBSD kernel core dump efficiently, it is best to set up a remote debugging session between a development machine and the target machine, and remotely debug the kernel using serial communication. This article shows you how you can debug as many kernel images as you want; it becomes transparent to users once debugging starts, and your system's performance is not affected at all."

Surely...

If this news day gets any slower it's just going to just stop.

I was just about to post this story....

Hehe, good to see others saw this as an influential story of the day.
However, I have a question that perhaps others can answer.
What does the diagram actually work out?
Listing 1. 25-Pin NULL modem cabling

2 3 Transmit Data
3 2 Receive Data
4 5 Request to Send
5 4 Clear to Send
6 20 Data Set Ready and Carrier Detect
7 7 Signal Ground
20 6 Data Terminal Ready

What does that mean? I thought serial -> serial is merely a connection. Do I have to solder something?

Regards

Re:I was just about to post this story....

What does the diagram actually work out?

This is a Null modem adapter [wikipedia.org]. It allows you to connect to terminals, such as computers, together without a modem. It is the serial equivalent of a ethernet crossover cable.

gdb remote is useful

This is a very cool feature in gdb that is very commonly used to debug various embedded systems etc. Sometimes gdb remote uses a serial port or tcp from the host to a stub running in the target (as is the case with Redboot).

Also, gdb remote is often used to talk to a server that mimics a gdb remote stub and then turns the commands into some other connection into the target (eg. a JTAG debugger).

All up, this makes debugging embedded systems a lot simpler than it would otherwise be.

Stop pissing about

Get yourselves a system, with a proper debugger [bell-labs.com]

debugging kernels

This article shows you how you can debug as many kernel images as you want

including zero.

Unfortunately

Jayesh V. Rane works as a Systems Software Engineer for IBM and has five years of experience in product development, mostly working on network programming and storage management software. He has worked extensively on FreeBSD and AIX operating systems. Jayesh is currently working with the IBM India Systems and Technology Lab in Pune, India. You can contact him at [e-mail]. Unfortunately Jayesh doesn't understand that compiling your kernel with -g will in fact affect the systems performance and is currently looking for a job where he can pretend to be a kernel engineer. Contact Jayesh at [e-mail]

Wyse terminals work well for this.

About 5 years ago I used an old Wyse terminal [wyse.com] I got at a junkyard for exactly this job using a homemade cable, it worked beautifully. Once complete I then used the terminal to control and update the server from the living room couch.

I also used the same terminal and SoftIce to *ahem* "debug" several windows applications.

Wyse terminals (at least the older ones) are excellent gear, very sturdy, nice keyboards, though the monochrome monitor was a bit burnt in on the one I used. An industrial strength terminal for a rock solid OS. A match made in heaven. FreeBSD is an excellent OS.

Transparent to users?

Something tells me that if the kernel has dumped core and dropped down into a debugger, the users are going to notice something. Just a hunch.

It was a dark day for Linux

It was a dark day for Linux when Linus said he didn't believe in kernel debuggers.

Last time I had a good look at what was on offer while developing a filesystem for multiple operating systems, I compared the tools for Linux to Windows, MacOSX, IRIX and Solaris.

Windows was clearly the best for remote kernel debugging with windbg and I don't see that changing anytime soon. A fully fledged debugger, automatic download of kernel symbols from Microsoft and your own repository, reliable capture of dumps etc. That and the API documentation made the black box nature of the O/S a much smaller issue than the open source community would like us to believe. I was always so much more confident if we had a crash due to our software at a customer site that with Windows we had the best chance of capturing and hopefully identifying the problem.

For live debugging IRIX was the best with icrash. Its port to Linux, lcrash, was not as good and reliable at the time, maybe its better now but the lack of symbol files it need on Linux always made it frustrating. And now IRIX is dead.

I've heard some really good things about dtrace, so Solaris and Mac look to have the best live debugging and tracing tool there is today...I see some progress on Linux ports [sun.com] which is great to see.

But when it comes to kernel debugging on Linux, the picture is still bleak.

quantum

maybe the mere act of viewing the event will change it... is debugging ultimately futile?

Re:Who cares?

Go away to another land and find your fortune cookie u kinky kernel.

Re:DDB

DDB is in FreeBSD, too, since always. In fact, DDB was ported from Mach to 386BSD, which predates the FreeBSD/NetBSD split / diverge, which again predates the NetBSD/OpenBSD split.

Eivind.