NetBSD To Support Kernel Development In Lua Scripting 311
An anonymous reader writes "NetBSD 7.0 will support the Lua scripting language within its kernel for developing drivers and new sub-systems. A Lua scripting interpreter is being added to the NetBSD kernel along with a kernel API so developers can use this scripting language rather than C for developing new BSD kernel components. Expressed reasons for supporting a scripting language in a kernel were rapid application development, better configuration, and "modifying software written in C is hard for users." In a presentation it was said that Lua in the kernel will let users explore their system in an easy way."
users? (Score:5, Insightful)
"modifying software written in C is hard for users."
i thought *users* had no business writing device drivers?
of course if they had said "modifying software written in C is hard for programmers." it would gather a fierce shitstorm.
Re:users? (Score:5, Interesting)
Re:users? (Score:5, Funny)
So let's invent a sandboxed layer that can interface with the kernel, but can't do damage to it? Also, make the API stable, unlike innards of the kernel. Let's call this novel idea "userspace" or such.
Re:users? (Score:5, Interesting)
Interesting - failure of user space in this way is exactly why we have zero-days.
I would like to see this happen - but several things make it improbable:
1. Von Neuman architecture. As long as data and instructions exist in the same space - poorly written apps will allow abuse of it.
2. Complexity of current software. The more complex the software, the more likely a bug will exist in it that allows #1. Given how programmers stitch together preexisting modules without understanding what is being done on the underlying system - I only expect that to continue expanding.
It should be instructive that Java was supposed to be that sandboxed layer...and it has so many zero-days it looks like swiss cheese.
Now - how would we avoid that and make an unhackable userspace?
Re: (Score:3)
Interestingly, they leverage Bare Metal OS [returninfinity.com] and the coding is done mostly in Assembly (although C is possible). I think the fattening of all kernels is making these kind of projects look more interesting.
Re:users? (Score:5, Insightful)
Re: (Score:3, Funny)
The moment you even expect that users should, could, would or even remotely in their most maddened fever dreams WOULD WANT to modify the kernel, you lost it.
Though it's NetBSD users. Not Apple iPad users.
Re: (Score:2, Insightful)
go fuck yourself. I'm and Linux and BSD user, I've taken grad courses in kernel architecture of Minix, System V, BSD, VMS. I've modified the kernel of many OS. Not everyone is a windows wuss.
Re: (Score:2)
oh? I also work on my car, home electronics & wiring, carpentry and even plumbing at times. so by your logic I'm an auto manufacturer, fabrication facility, housing contractor.....??
Re: (Score:3)
oh? I also work on my car, home electronics & wiring, carpentry and even plumbing at times. so by your logic I'm an auto manufacturer, fabrication facility, housing contractor.....??
Yes. In that case you are an amateur auto manufacturer, fabrication facility and housing contractor.
Re:users? (Score:4, Interesting)
"It seems crazy, but this may not be as braindead as it seems."
I think that is very much a matter of opinion. And my opinion is: it *IS* just exactly as braindead as it seems.
Just what we need: "modern" software bloat in the kernel. [sarcasm] Thanks, but no thanks.
I have right here a program that is more than 45 times the size of the entire hard drive from one of our office computers back in 1994... and that hard drive had a full install of that year's Microsoft Office, plus WordPerfect and Lotus 1-2-3, with lots of room to spare for files.
Sure... that was '94. But is there REALLY call for programs that large, and by the way slow? And that is by no means the largest application I have on my machine right now. It's on the larger side but I have some many times as big.
Re: (Score:3)
A monolithic kernel will inevitably get bloated over time as more features and drivers are added. This is a bad thing because any malfunction in any component has the potential to mess up the entire running kernel. Using a scripting language for less performance-intensive parts is an attempt to mitigate this risk. Whether it's a good idea remains to be seen.
Re:users? (Score:5, Funny)
I only have 640 KB you insensitive clod!
Re:users? (Score:4, Interesting)
Re:users? (Score:5, Insightful)
Keep in mind that NetBSD users are quite different from Windows/Macintosh/Linux users (at least on average).
"Hard" may also be a reference to the implementation, rather than the language. Interpreted languages tend to bypass the compile and link part of the development process. This means that interpreted languages are easier to develop with and compiled languages are hard to develop with (at least in some respects). I'd also comment on the arcane nature of C, but I haven't used lua so I don't know if it is any better.
Re: (Score:2)
Lua is specifically designed to be an easy embeddedable scripting language when one needs a light scripting language but not a full featured programming language. Luas are way way easier.
Re: (Score:3)
Lua is all over the Logitech Squeezebox by SlimDevices. It's Tcl for the new generation.
Re: (Score:2)
Lua is a full-featured programming language, though - more so than many other common embedded languages, and on par with Python. Semantically it's actually somewhat similar to JavaScript , except without all the shitty design decisions.
Re: (Score:3)
It is a really simple syntax and reasonably efficient interpreter, and metatables allow a surprising amount of OO design and easy C integration. One complaint is I wish they hadn't made it so minimal as to combine hashes and arrays into one type. I kind of wonder if that's going to be a pain when writing more traditional C-ish driver-type code in this project.
Re: (Score:3)
I don't think that it is really true. I think Lua could evolve to be as strong in applications programming as Python but they choose to fill different niches and so the languages evolved along different paths.
Lua uses automatic coercion in all instances which makes programs faster to write and harder to maintain.
Python has a very well defined OO structure.
Lua doesn't have the huge libraries of functionality Python does
Lua's lack of high performance arrays
Lua's lack of unicode is a big problem for applicati
Re:users? (Score:5, Interesting)
i thought *users* had no business writing device drivers?
My non programmer wife wrote a custom device driver in AppleScript, not even knowing what a device driver is. Those sorts of languages can be amazing empowering to end users, and yes there are reasons that end users might want a custom device driver if it is easy enough to accomplish. Though this makes more sense for OSX than NetBSD, I doubt there are many if any non programmers who use NetBSD.
I had to write a custom device driver for Windows once to get something more like a Unix ethernet driver i.e. bypass some normally quite excellent parts of the Windows networking stack. I was doing software not hardware so I guess I'd consider myself an end user. And that was a pain in the ass.
Re: (Score:3)
No. Read http://www.google.com/search?q=transcription+foot+pedal&tbm=isch [google.com]
Re: (Score:2)
the stalker game series: all game scripting and i think even the ai of npcs is in lua.
Performance (Score:5, Informative)
Won't that kill performance?
I mean that's the reason we still write kernels in C. Every cycle counts..
Re: (Score:2)
Lua is the fastest available scripting language for the task that NetBSD needs it for. It compiles to a byte code, is designed for efficiency and most importantly has excellent hooks for C integration. I have to commend the NetBSD team for their decision - hopefully it will be very successful and lead to a similar implementation in Linux.
Re: (Score:2)
Perl is dying, Larry killed it.
lua vs perl stats (Score:3)
http://benchmarksgame.alioth.debian.org/u32/benchmark.php?test=all&lang=perl&lang2=lua [debian.org]
In some tests, one is faster than the other, but by no more than 4x to 1/9th each way.
At least they didnt choose ruby.
Re: (Score:3, Informative)
Lua is designed to be a embedded companion language to C. You still use C where it makes sense (i.e. pointer manipulation/performance). You use Lua for expressing higher level logic more easily. Lua is very fast. Writing the same higher level logic in raw C might not be any faster because you have to build up all the infrastructure that Lua has already provided and optimized like associative arrays.
Re:Performance (Score:5, Informative)
Kernel code is like lots of other code: a lot of it is executed rarely and not performance critical. There's setup code, occasional permissions checking, etc.
In addition, this is not to replace plain C modules (although it may be useful for prototyping), it's for complex and dynamic configurability.
Right now, there are two ways of handling that. One is to invent complex configuration files and data structures, sometimes even little interpreters. The kernel is full of those. They are error prone, complicated, and require a lot of effort to maintain. The other is to put complex decision making into user-space, but that involves context switches and other overhead. It also means lots of special-purpose and redundant code, and lots of documentation and complexity. The kernel uses both of those strategies extensively, one reason why it's so big.
Putting Lua+LuaJIT in the kernel is likely faster than either of those approaches, while at the same time also being easier to maintain and document.
Re:Performance (Score:4, Insightful)
A lot of it is executed rarely because at least half the code is drivers and one usually doesn't have but one sound card, one (or two) network card, etc vs the hundreds of sound card drivers, hundreds of network card drivers, etc. Further, a major point of the kernel is that it, by design, isn't supposed to be executed a lot because most CPU time should be devoted to user space programs. So, to that end, you could say it's not "performance critical". But, that's more in line with the point that precisely because kernels are generally developed so well, they rarely have "performance critical" code. Once you start throwing scripting into it, I think that goes out the window.
To this, I could see some benefit. But, honestly, I'd feel it safe and saner if it were a language that was reducible to a FSA. The point that "Dangerous code can be disabled at the byte-code level (prevention of endless loop DoS etc)" in the slide really only works in that context, anyways. In any case, Lua is nothing of the sort.
Which is an argument for making a good, universal FSA language. It won't remove the being error prone or complicated because that's the nature of the beast and papering over it with a Turing complete language is no sort of answer. But having them all share the same language makes sense from a maintenance standpoint. For example, the BPF was repurposed for syscall filtering--a little admission that I'm not sure entirely where BPF stands language wise although I don't think it's turing complete--which could probably be refactored into a more general filtering language which could be refactored into an even more generic FSA language which could likely be used to refactor a lot of other code.
Generally speaking, context switches aren't a big deal in most OSs/kernels today. Yes, certainly having context switches means an extra overhead and stuffing some code or even all code in the kernel would certainly speed it up. And certainly if a lot more code moved into user space would be measurably slower. But kernel design today seems to try to have a balance between the two, especially with considerations of stability and security. The fact is, lots of code is special-purpose because it's for specific hardware. And code being redundant is more a function of maintaining multiple trees--which admittedly having a single kernel allows a lot of unity--but there's no reason that out-of-kernel-space means out-of-the-kernel-source if the code is still rather kernel specific.
After all, the whole point is precisely that a lot of potential kernel code is "executed rarely and not performance critical" yet remaining outside kernel memory gives the system greater protection and hence speeds up development and generally makes the code easier to maintain. Documentation is a wholly separate beast, in any case, and that's more a point of maintainers or distributors demanding good documentation before code is accepted.
Re: (Score:2)
I'm not sure, but there might be non-C system code in OS X (Objective-C) and Windows (C++, C#). In FreeBSD, there's Forth in the bootloader (does that count?).
I think you meant: Linux developers only use C for system programming.
Re: (Score:2)
Re: (Score:2)
True it is not going to be as efficient as C, but when we get to device drivers we are not necessarily talking 'every cycle counts' anymore and other considerations start to factor in.
Re: (Score:2)
Re: (Score:2)
Pointers are pretty essential to a kernel.
Re: Performance (Score:5, Funny)
So we use it for performance. Ok I get it now.
Re: Performance (Score:3, Interesting)
True. Assembly can be faster than C. In the same way it can be more powerful than C.
Bottom line is C provides a solution for the complexity of large scale development, without sacrificing the performance, while at the same times still let's the developer be in complete control of his code and data structures.
Re: (Score:3, Insightful)
Multiple execution units, long pipelines and modern caching makes it hard for assembly to be faster than C. The human brain can not visualize the pipeline like a compiler can.
Re: (Score:2)
Re: (Score:2)
On microcontrollers -- sure as heck I can beat any compiler in assembly, without using anything but paper and pencil, and an assembler (eventually). On a modern Intel core? Yeah, but it'll take me 10x longer than on a microcontroller, and I may need to use tools like cachegrind to check my work in progress.
Re: (Score:3)
Re: (Score:3, Insightful)
Re: (Score:2, Interesting)
Not really. It's more subtle than that. Straightforward machine code and more explicit code can help performance, or it can hurt. For example, function pointers are slower than inlinable c++ templates or a JIT that can inline across library boundaries or based on observed call frequencies at runtime (all of these at the cost of memory usage). It's not nearly as simple or binary as you're suggesting.
Re:Performance (Score:5, Informative)
Re: (Score:3)
Re:My P4 (Score:4, Interesting)
It depends upon what he needs the computer for. If it is a server, then there is no need for a GUI. Even with respect to application software, there are a multitude of options out there. Writing can be done in text editors, then processed with TeX or troff (for formatted output to a printer, when necessary). There are a multitude of email clients and several web browsers. I've never really tried it, but you can even do graphical design with languages like PostScript and POV. Media? Again, you have access to many audio players. It's much more gimmicky in nature, but you can even play videos on text consoles.
Re: (Score:3)
And, of course, very few development tools actually depend upon a GUI.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
oh no not a P4, that thing is like stone age!
seriously though I often use lua on a 90Mhz pentium1 8 meg laptop for various test's where I may need to log data once an hour for 3 months and I dont want to waste a real computer
its faster than qbasic, its hardly noticeable against a poorly written compiled program, its not even terrible to a decently written compiled program. lua is a pretty darn fast interpreted language, and since its all bound to C on the backend anyway its easy to rapid develop with lua a
Re: (Score:2)
long overdue (Score:5, Interesting)
This is a good thing, and it's long overdue. Linux should do the same.
It's actually pretty easy to get Lua running in the kernel and call kernel APIs; but to make it really useful, it needs additional kernel hooks and callbacks, and that requires cooperation from the core kernel developers.
Re: (Score:3, Interesting)
Exactly.
With certain algorithms I have LuaJIT code that runs faster than the best C code I can create (I'm speaking with 25+ years of C experience here).
Re: (Score:3, Interesting)
LuaJIT can be seriously fast. But usually when C is slow, it's some compiler flag, or perhaps a bunch of unpredictable branches or random access pattern in the inner loop.
Pretty often the LuaJIT trace machine code generated is similar to what you'd get from gcc. That's how good it is.
Re: (Score:2)
Especially that the trace is equivalent to gcc profile-guided optimizations. It can rock your world.
Re: (Score:2)
Sounds like you are an idiot.
Re:long overdue (Score:4, Insightful)
They use Lua for the non-performance effecting parts of games. They'd never consider using it in the graphics code, which is the bottleneck.
Re: (Score:2)
Re: (Score:3)
They use Lua for the non-performance effecting parts of games. They'd never consider using it in the graphics code, which is the bottleneck.
Does World of Warcraft lay out its entire UI with LUA? Then render the actual 3D graphics with C/C++? Isn't that exactly like what we're talking about here for drivers-- high-level control in LUA and the heavy lifting in C?
Re: (Score:2)
A lot of low-level driver code is not actually performance-critical. Sure, you don't want to be writing video or audio drivers in this thing, but for something that just needs to send a bunch of commands over USB, why not?
Re: (Score:3)
I don't know what you mean by "pretty abysmal, performance wise". Lua is one of the fastest scripting languages around, even without LuaJIT.
Re: (Score:2)
Lua is one of the fastest bytecode interpreted scripting languages around. Which is a pretty low bar when you compare to even mediocre JIT VMs, not even to mention precompiled code.
Re: (Score:2)
But I'm not comparing it to scripting languages -- I'm comparing it to other things that are in use in a particular application domain. In industrial process control, there's generally no distinction between a scripting vs. non-scripting language. Everything is written in the same environment, and the languages all translate to same byte code that is then either run on a bytecode VM or compiled to native code. If you want to get rid of that, you can't just get rid of a small part of it -- it's not worth it.
Re: (Score:2)
Legacy codesys-based industrial PLCs, where I evaluated LuaJit as a replacement, have arrays limited to 64 kbytes in size, even when targeting 32 bit Intel platforms. Whatever limitations LuaJIT has, save for 2.0 teething pains, are not even on my horizon :)
Re: (Score:3)
I would be happier if the major window managers supported Lua. I'm not sure I would want to mess with device drivers.
Re:long overdue (Score:4, Insightful)
It's not mainly for messing with device drivers (although it may be useful for that too).
It's mainly useful for all the stuff that people right now create /proc devices, ioctls, and user-level kernel-related daemons for. Those mechanisms are both slow and complicated. With Lua in the kernel, a lot of that stuff can be handled much more easily, without the overhead of context switches and with much less code.
It probably won't replace all the bloated configuration stuff that already exists, but at least when writing new device drivers and new functionality, developers could save a lot of time and effort.
Great idea. (Score:5, Interesting)
I honestly think that this is a great idea. It's a novel approach to extending a kernel, especially considering NetBSD doesn't have that big of a market share and expressly focusses on supporting as many platforms as possible. Prototyping new features or certain drivers in a portable scripting language may give them a leg up in the functionality aspect of the race.
Also, LUA is super fast, small, easy to learn, concise and the C API/embedding it is straight forward.
You may knock it all you want, "because kernel land is C land", but after all bias being said and flamed, this may turn out to be a really fun, interesting and possibly very useful idea.
Re: (Score:2)
Uhm... (Score:2)
Okay, sure. But...why Lua? :/
Re: (Score:2)
Indeed. What is wrong with java script?
Re: (Score:2)
There's nothing wrong with Javascript
Very funny
Re: (Score:2)
LUA is an extremely fast scripting language that was designed to be embedded in C.
It can probably be used to handle the non-realtime components of the kernel quite easily
Re: (Score:2)
Because it's written entirely in portable ANSI C, very small, and very fast?
This is cool already, but with RUMP, OMG (Score:2, Insightful)
This is awesome. What is old is new again. Lisp machines were some of the most beloved equipment for os and ai hackers because of the ability to get to the underbelly easily. This is no different. Making the operating system more approachable makes doing operating systems research easier. This is a great idea.
Besides, it's a natural extension of the RUMP kernel paradigm. If you can run a netbsd kernel, as part of a userspace process on linux, then it only makes sense to be able to talk to these 'kerne
Modifying C? (Score:2)
I dont know what packages are going to be bound with this, but in many cases your right back in C adding bindings for lua. Thats one of the good points about lua, is that it can be tied to anything, but the bare bones basic lua is not too much more complicated than a batch file
Tags (Score:5, Funny)
Surely this article deserves the whatcouldpossiblygowrong tag more than any other this year. And possibly last year too...
Wikipedia (MediaWiki) is also deploying Lua (Score:3)
MediaWiki developers are almost ready for Lua scripting to be enabled for all Wikipedia and related sites, and It has already been deployed to http://mediawiki.org./ [mediawiki.org.] Lua was chosen mostly because of how easy it is to sandbox and limit memory consumption.
http://www.mediawiki.org/wiki/Lua_scripting/Tutorial [mediawiki.org] -- Introduction
Sorta interested in this... (Score:2)
I like to think I'm okay at C. I certainly don't generally find it particularly intimidating. I've written kernel code, though not much. I maintain a program that emulates root privileges by intercepting syscalls -- and which works on Linux and OS X. I'm basically clear on how this works.
But when I do iOS/Android stuff, I have a pretty strong preference for Lua, because I don't always want to be thinking about pointers. And yes, LuaJIT is... plenty fast. The mere fact that it's a JIT implementation means th
Re: (Score:2)
You realize that in most of those higher level languages *everything* is a pointer. So you're not actually saving yourself anything except calls to delete. Most of the difficult parts of pointers you still need to worry about.
Re: (Score:2)
I have tried several times to figure out what on earth you are saying, and I've come up empty.
In C, it is possible for me to allocate an object, fail to free it, and end up having leaked memory. It is also possible for me to retain a pointer to something that was freed, and get surprising behaviors from that. I can overrun buffers. Heck, I occasionally do, although it's usually pretty rare for me.
None of these are possible in Lua.
I am not sure what the "difficult parts of pointers" you're thinking of are, b
Re: (Score:2)
Me too and I'm not interested in learning Lua or more APIs. I think C is where it should remain; however, this could prove extremely useful for development, debugging and experimentation - especially for any kind of driver. I think it should be an option; although, it is likely many users would turn it on because of the number of drivers or features that would not have been ported to C yet. Eventually, things can be ported or even automatically translated to C code. For rapid and more stable developme
I suggested this years ago for Linux (Score:3)
With the BSD's, there no such thing as a tainted kernel. But with Linux, non-GPL drivers are actively discouraged. If they were to add scripting to the Linux kernel, it would actually make it easier to write proprietary drivers, so they'd never go for that. Forget the fact that it would make it a hell of a lot easier to sandbox drivers so they don't clobber other kernel memory. Forget the fact that code in a higher-level language is smaller; countless drivers would suffer no appreciable performance hit by being written in an interpreted language (especially Lua), so you'd have one small interpreter and many shrunken drivers, which would make the kernel memory footprint smaller. Forget the fact that mutually exclusive access to shared variables could be made implicit in a majority of cases, eliminating a whole class of bugs in one shot. No. We've got our precious GPL to protect. (Oh, and forget the fact that nVidia basically gets a free pass on this.)
DSL's are already in the BSD kernels (Score:2)
Here is a prime example http://en.wikipedia.org/wiki/Berkeley_Packet_Filter [wikipedia.org]
Lua is a very clean, small, and relatively fast object oriented language. It is probably the best possible choice for a general purpose dynamic language to embed in a kernel.
Caen We Get (Score:2)
How about
peek(xx)
poke(xx)
The best aprt of Commodore BASIC
Re: (Score:3, Funny)
Linus?
Re: (Score:2)
Why LUA? Why not implement BASIC while you're at it?
You meant that sarcastically but you do realize in the CP/M and early PC days BASICs were implemented in hardware!
Re: (Score:2)
BASICs were implemented in hardware!
WTF? My 6502 system had BASIC in the ROM, not the CPU.
Re: (Score:2)
Apparently, a lot of people in the software business have trouble with pointers. That includes Linux, and its many, many security breaches and flaws in its history. Of course, I'm not calling people stupid. I'm simply stating what is widely known. Safe programming is hard. Pointers are error-prone. People who deny that are either superbly arrogant or ignorant of the many serious events in the software industry. If you can avoid unsafe code by using a safer language, than you ought to. You don't seem to have
Re: (Score:2)
Because Lua suffers from none of those shortcomings.
Re:Stupid NOT (Score:5, Interesting)
I don't know how much Unix/BSD/Linus code you have read or written, but I have been developing core code, Scheduler, MM and Drivers since 1980 and this is a great idea, long overdue.
The Abstractions are now well understood and most code dos't need to be quick, but safe, putting PRE in kernels will clean up acres of cockamamie code now written in C, most of which just runs at boot time and is then freed. ... there are exceptions, you cant't have the garbage collector running whenever it likes, eg in the middle of an interrupt routine, but managed memory, not C, C++ scoping is the way to stop slow memory leaks.
Put simply you are an idiot and dont know what you are talking about.
MFG, omb
Re: (Score:3)
But if it doesn't need to be quick, why is it in the kernel in the first place? I would have thought that if you are tempted to put a scripting language in the kernel, it's a pretty good indication that the division between kernel and non kernel is somehow wrong.
Re: (Score:3)
Re: (Score:3)
Won't somebody think of the precious CPU cycles!
Re:Stupid (Score:5, Interesting)
Re: (Score:2)
There are many other languages that have well-defined object lifetime semantics - the most obvious way to get that is to use some form of reference counting. Now, the standard Lua interpreter actually uses a mark & sweep GC, but they might change that.
sandboxed lua (Score:3)
Lua interpreter instances are small. In a kernel environment, you'd like run lots of tiny Lua interpreters, one for every "task" you want to perform. You can impose limits on memory and CPU usage on each instance. Under an out-of-memory condition, you can also ask each of them to reclaim a maximum amount of memory, via a hook and via the GC interface. And if that fails, you can start killing them off one by one in order of least importance, just like you might with other processes or functions. The fact tha
Re: (Score:3)
And yet it is so useless without the concept of destructors.
s/useless/cumbersome
Re: (Score:2, Insightful)
No, if you understand how Lua glues C code together, this is brilliant.
Re: (Score:3)
What? have you any idea how many interpreters and the like 'glue' C code together, just about all of them and many do it better such as JavaScript.
Hahaha. Lua was *designed* to do just that. BTW, take a look at how LuaJIT extends what Lua already provides. You can directly manipulate C structures from LuaJIT - define new structure types *at runtime*, malloc them, fill them in, pass them arround, whatever you need to do with them.
Not to mention that JavaScript or ECMAScript is utilized my millions.
Lua is a MUCH better language than ECMAScript. Just consider issue number one: variable scoping. Lua: Essentially derived from the Scheme LET for or the functional languages (ML family style LET). Javascript: horrible hack, va
Re: (Score:2)
you are pretty ignorant if you think COBOL or LISP compiled to assembly is slower than C code