Metasepi team meeting #13: NetBSD driver using Haskell

1. Metasepi team meeting #13:
#13:
NetBSD driver using Haskell
Metasepi Project / Kiwamu Okabe

2. Who am I?
☆ http://www.masterq.net/
☆ Self employed software engineer
☆ Founder of Metasepi Project
☆ A developer of Ajhc Haskell compiler
☆ A Debian Maintainer
☆ 10 years' experience in developing
OS using NetBSD.

3. Agenda
☆ [1] Demo
☆ [2] What is Ajhc?
☆ [3] What is Metasepi?
☆ [4] What is compiler to build OS
☆ [5] NetBSD driver using Haskell
☆ [6] Ajhc is the best?
☆ [7] From Arafura to Bohai

4. [1] Demo
☆ NetBSD audio driver play sound
☆ The driver's interrupt handler
rewrited using Haskell
☆ Watch the movie at following
http://bitly.com/netbsdajhc
☆ The source code at following
github.com/metasepi/netbsd-arafura-s1

5. Demo architecture

6. [2] What is Ajhc?
http://ajhc.metasepi.org/
☆ Ajhc := Arafura designed jhc
☆ jhc := John's Haskell Compiler
☆ http://repetae.net/computer/jhc/
☆ Jhc outputs binary that has lowmemory-footprint and runs fast.
☆ Good for embedded software.

7. Who is John?
☆ John Meacham
☆ http://repetae.net/

8. Why need Ajhc?
☆ GHC is de facto standard on Haskell.
☆ GHC := Glasgow Haskell Compiler
☆ http://www.haskell.org/ghc/
☆ Why need another Haskell compiler?
☆ To develop kernel named "Metasepi".

9. [3] What is Metasepi?
http://metasepi.org/
☆ Unix-like OS designed by strong type.
☆ Using ML or more strong type lang.
Haskell http://www.haskell.org/
OCaml http://caml.inria.fr/
MLton http://mlton.org/
. . . and suchlike.

10. Why need Metasepi?
☆ We have already Linux or Windows.
☆ But the developers are suffering.
☆ If use the kernel changed by you,
☆ you will get many runtime error.
☆ Difficult even to reproduce it.

11. Doesn't OSS have good quality?
☆ "The Cathedral and the Bazaar"
☆ "Given enough eyeballs, all bugs are
shallow."
http://cruel.org/freeware/cathedral.html
☆ But if you develop your own product
reusing OSS...

12. Low quality out of OSS umbrella

13. Type safety
☆ Less runtime errors.
☆ "数理科学的バグ撲滅方法論のすすめ"
http://itpro.nikkeibp.co.jp/article/COLUMN/20060915/248230/

14. Kernel wants type desperately
☆ Kernels are developed with C lang.
☆ Error on user space => SEGV
☆ Error on kernel space => Halt!
☆ Should design kernel with the
greatest care.
☆ C language is safe?

15. [4] What is compiler to build OS
☆ Need strong type.
☆ Need flexibility such as C language.
☆ Create it if there are not!
☆ From scratch? No thank you...
☆ Look for our compiler base.

16. Want POSIX free compiler
Programs to print "hoge" on terminal.
The lesser depends on POSIX, the
smaller values.

17. Jhc output has only 20 undef
$ nm hs.out | grep
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
"U "
_IO_putc@@GLIBC_2.2.5
__libc_start_main@@GLIBC_2.2.5
_setjmp@@GLIBC_2.2.5
abort@@GLIBC_2.2.5
ctime@@GLIBC_2.2.5
exit@@GLIBC_2.2.5
fflush@@GLIBC_2.2.5
fprintf@@GLIBC_2.2.5
fputc@@GLIBC_2.2.5
fputs@@GLIBC_2.2.5
free@@GLIBC_2.2.5
fwrite@@GLIBC_2.2.5
getenv@@GLIBC_2.2.5
malloc@@GLIBC_2.2.5
memset@@GLIBC_2.2.5
posix_memalign@@GLIBC_2.2.5
realloc@@GLIBC_2.2.5
setlocale@@GLIBC_2.2.5
sysconf@@GLIBC_2.2.5
times@@GLIBC_2.2.5

18. Jhc is translator to C language

19. Easy to cross build

20. Survive burning out
Let's develop with dogfooding style.
(The method is called "Snatch".)

21. [5] NetBSD driver using Haskell
☆ Snatch target: HD Audio driver
sys/dev/pci/hdaudio
|-- Makefile
|-- ceareg.h
|-- eldreg.h
|-- files.hdaudio
|-- hdafg.c
|-- hdafg_dd.c
|-- hdafg_dd.h
|-- hdaudio.c
|-- hdaudio_config.h
|-- hdaudio_ids.c
|-- hdaudio_ids.h
|-- hdaudio_mixer.h
|-- hdaudio_pci.c
|-- hdaudio_pci.h
|-- hdaudioio.h
|-- hdaudioreg.h
|-- hdaudiovar.h
`-- hdmireg.h

22. Rewrited the driver entirety?
☆ No, partly

23. Where is Haskell code? (cont.)
netbsd-arafura-s1
# <= based on NetBSD original source tree
|-- BUILDING
|-- GNUmakefile
# <= Wrapper for building kernel
|-- Makefile
|-- Makefile.inc
-- snip -|-- metasepi
|
|-- sound
|
|
|-- Epopsan-signal.mp3
|
|
`-- TheMojitoofFluorescence.mp3
|
`-- sys
|
|-- conf
|
|
`-- files.haskell
|
|-- hsdummy
# <= C code to support Ajhc runtime
|
`-- hssrc
# <= Haskell code located
|
|-- Arch
|
|
`-- I386
|
|
|-- I386
|
|
|
`-- BusDma.hs
|
|
`-- Include
|
|
|-- BusDefs.hs
|
|
|-- Cpu.hs
|
|
`-- Types.hs

24. Where is Haskell code? (cont.)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|-|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|-|
|
|
|-|
|
Dev
|-|-|-|
|
`--
Auconv.hs
AudioIf.hs
Ic
|-- Ac97reg.hs
`-- Ac97var.hs
Pci
|-- Auich
|
|-- Intr.hs
|
`-- Ptr.hs
|-- Auich.hs
|-- Auichreg.hs
`-- Hdaudio
|-- Hdaudio.hs
|-- Hdaudioreg.hs
`-- Hdaudiovar.hs
Kern
|-- KernMutex.hs
|-- SubrKmem.hs
`-- SubrPrf.hs
Lib
`-- Libkern
`-- Libkern.hs

25. Where is Haskell code?
|
|
|
|
|
|
|
|
|
|
|-|-|-|-|-|-|-|-|-|-`--
|-|-|
`--
regress
rescue
sbin
share
sys
tests
tools
travis-ci
usr.bin
usr.sbin
x11
Main.hs
Metasepi
`-- EitherIO.hs
Sys
|-- Audioio.hs
|-- Bus.hs
|-- Device.hs
|-- Errno.hs
|-- Proc.hs
`-- Types.hs

26. How to build
☆ Install some packages
$ sudo apt-get install ffmpeg qemu gcc
☆ Install Ajhc and git clone
$ git clone https://github.com/metasepi/netbsd-arafura-s1.git
$ cd netbsd-arafura-s1
☆ Build QEMU image
$ make bootcd

27. Run the kernel
Run kernel on qemu
$ make qemu
>> NetBSD/x86 BIOS Boot, Revision 5.9 (from NetBSD 6.1.1_PATCH)
>> Memory: 639/1047544 k
--snip-Created tmpfs /dev (1490944 byte, 2880 inodes)
erase ^?, werase ^W, kill ^U, intr ^C
This image contains utilities which may be needed
to get you out of a pinch.
#
Kick play.sh script to play sound
# ./play.sh

28. Build process

29. How to call C (figure)

30. How to call C (code)
-- File: metasepi/sys/hssrc/Sys/Bus.hs
busSpaceRead4 = c_bus_space_read_4
busSpaceWrite4 = c_bus_space_write_4
foreign import ccall "hs_extern.h bus_space_read_4"
c_bus_space_read_4 ::
BusSpaceTagT -> BusSpaceHandleT -> BusSizeT -> IO Word32
foreign import ccall "hs_extern.h bus_space_write_4"
c_bus_space_write_4 ::
BusSpaceTagT -> BusSpaceHandleT -> BusSizeT -> Word32 -> IO ()
-- File: metasepi/sys/hssrc/Arch/I386/Include/BusDefs.hs
type BusSizeT = CSize
newtype {-# CTYPE "struct bus_space_tag" #-} BusSpaceTag =
BusSpaceTag ()
type BusSpaceTagT = Ptr BusSpaceTag
type BusSpaceHandleT = VaddrT

31. Trace pointer chain
/* C code */
static int
auich_open(void *addr, int flags)
{
struct auich_softc *sc;
sc = (struct auich_softc *)addr;
mutex_spin_exit(&sc->sc_intr_lock);
sc->codec_if->vtbl->lock(sc->codec_if);
// -- snip --
-- Haskell code
auichOpen :: Ptr AuichSoftc -> Int -> IO Int
auichOpen sc flags = do
mutexp <- p_AuichSoftc_sc_intr_lock sc
codecif <- peek =<< p_AuichSoftc_codec_if sc
lock <- peek =<< p_Ac97CodecIfVtbl_lock =<< peek =<<
p_Ac97CodecIf_vtbl codecif
mutexSpinExit mutexp
call_Ac97CodecIfVtbl_lock lock codecif
-- snip --

32. But need assistant code!

33. Tool to generate the code
https://github.com/ajhc/struct2hs
☆ Generate Haskell code scanning C
$ struct2hs sys/dev/pci/auich_extern_SNATCHED.h | tail
offsetOf_Pdevinit_pdev_attach :: Int
p_Pdevinit_pdev_attach :: Ptr Pdevinit -> IO (Ptr (Ptr (FunPtr
(Int -> IO ()))))
p_Pdevinit_pdev_attach p = return $ plusPtr p $
offsetOf_Pdevinit_pdev_attach
foreign import ccall "dynamic" call_Pdevinit_pdev_attach ::
FunPtr (Int -> IO ()) -> Int -> IO ()
foreign import primitive "const.offsetof(struct pdevinit,
pdev_count)"
offsetOf_Pdevinit_pdev_count :: Int
p_Pdevinit_pdev_count :: Ptr Pdevinit -> IO (Ptr Int)
p_Pdevinit_pdev_count p = return $ plusPtr p $
offsetOf_Pdevinit_pdev_count

34. [6] Ajhc is the best?
☆ No.
☆ Depended on GC
☆ Too buggy compiler
https://github.com/ajhc/ajhc/issues
☆ Can't use MonadTrans
☆ Debug hard for strong optimization

35. Before compiling (Haskell)
hdaudioIntr :: Ptr HdaudioSoftc -> IO Int
hdaudioIntr sc = do
intsts <- hdaRead4 sc e_HDAUDIO_MMIO_INTSTS
if intsts .&. e_HDAUDIO_INTSTS_GIS /= 0 then hdaudioIntr' sc
intsts
else return 0
hdaudioIntr' :: Ptr HdaudioSoftc -> Word32 -> IO Int
hdaudioIntr' sc intsts = do
when (intsts .&. e_HDAUDIO_INTSTS_CIS /= 0) $ do
rirbsts <- hdaRead1 sc e_HDAUDIO_MMIO_RIRBSTS
when (rirbsts .&. e_HDAUDIO_RIRBSTS_RINTFL /= 0) $ do
mutexp <- p_HdaudioSoftc_sc_corb_mtx sc
mutexEnter mutexp
hdaudioRirbDequeue sc True
mutexExit mutexp
when (rirbsts .&. (e_HDAUDIO_RIRBSTS_RIRBOIS .|.
e_HDAUDIO_RIRBSTS_RINTFL) /= 0) $
hdaWrite1 sc e_HDAUDIO_MMIO_RIRBSTS rirbsts
hdaWrite4 sc e_HDAUDIO_MMIO_INTSTS e_HDAUDIO_INTSTS_CIS
when (intsts .&. e_HDAUDIO_INTSTS_SIS_MASK /= 0) $ do
mutexp <- p_HdaudioSoftc_sc_stream_mtx sc
-- snip --

36. After compiling (C language)

37. Debug in kernel
☆ Hardware state is sometime buggy
☆ Not only debug software
☆ but also hardware!
☆ Gap between Haskell and C code
☆ Can't debug pure Haskell code
☆ Need to debug generated C code
☆ Where is more better compiler?

38. [7] From Arafura to Bohai
☆ We are in Arafura iteration
☆ Arafura := Ajhc + Snatch + NetBSD
☆ Use "ATS" compiler on the next
☆ Let's start Bohai iteration!
☆ Bohai := ATS + Snatch + NetBSD

39. What is ATS language?
http://www.ats-lang.org/
☆ Has syntax like ML
☆ Compiled into C
☆ Dependent type
☆ Linear type
☆ Without GC
☆ Without any runtime

40. Japan ATS User Group
http://jats-ug.metasepi.org/
☆ Translating ATS documents
☆ ATS propaganda in Japan
☆ Join us!

41. Hongwei said... (cont.)
☆ Hongwei Xi as ATS author
☆ Associate Professor
☆ at Boston University
Date: Mon Dec 23 11:40 JST 2013
Hi Metasepi-chan,
I spent quite some time today browsing metasepi.org.
I am really interested in your Metasepi project, partly because I
myself wanted to implement NetBSD in ATS about 5 years ago.
Unfortunately, I never had time to get the project started as I
needed to spend so much time on ATS2. By the way, I had planned to
use the very same approach which you call "Snatch". I had also
considered Minix but I chose NetBSD because it was a real OS.

42. Hongwei said... (cont.)
I think I know first-handedly the dilemma you are currently
one hand, you want to be able to fully focus on writing the
kernel. On the other hand, you also need to add features to
constantly to address all kinds of issues that keep popping
which undoubtedly makes it very difficult for you to focus.
in. On
Ajhc
up,
I would highly recommend that you use ATS to implement NetBSD
kernel. Unlike jhc, there is no semantics gap between ATS and C.
In particular, they both use the same native unboxed data
representation. Once you become familiar with ATS, you can readily
visualize the C code that your ATS source is to be compiled into.
ATS is truly an ideal language for the kind of "Snatch" approach
you want to take to re-write NetBSD. If you take the lead, then I
will be happy to "chip in" :)
I also spent some time reading documentation on jhc. Personally, I
feel that there is simply too much uncertainty to use it in real
kernel implementation. Features like GC could make the kernel
highly unpredictable, scaring away potential users.

43. Hongwei said...
I think that we both believe C is the right language for systems
programming. The problem with C is that it is too difficult to
write correct C programs. ATS is designed to allow the programmer
to correctly write the kind of C programs he or she wanted to
write in the first place. While jhc generates C code, the kind of
C code it generates may not be suited for kernel. This is what I
call a semantics gap.
I write this message in the hope that we could join effort in
doing something that has not been done up to now: Writing a real
kernel in (largely) functional style that can truly deliever
safety-wise as well as performance-wise.
Cheers,
--Hongwei