Grinberg did this experiment on a ATmega1284p, 8-bit RISC microcontroller clocked at 24MHz and equipped with 16KB of SRAM and 128KB of flash storage. Since the RAM was too low, he added 30-pin 16MB SIMM to the machine and a 1 GB SD card to host Ubuntu image.
Minimum system requirements for running Linux are considered to be a 32bit system with proper Modern Memory Management Unit (MMU) and 1 MB RAM. The experiment was done to prove that Linux can successfully run on an 8 bit micro.
To get the world's slowest Linux Computer running, he had to write an ARMv5 emulator which supports 32bit processor and MMU. A similar machine can be made very easily and everything should come in about $20. He has also provided full source code and a porting guide.