Minimal Zephyr OS on RISC-V and other tests

Intro

This is a followup article on my endeavors to run RISC-V soft core on Lattice ECP5 series FPGA.

I moved all my changes to a new github repository. This contains all modifications I’ve made to the original litexOnColorlightLab004 code.

The CPU reports that it’s running at 198MHz (It is a bit too high to be believable, but its working). I also managed to enable a few of the hardware components connected to the FPGA:

    • The 4MB SDRAM is running at 66MHz (running it at higher frequency makes the memory test fail).
    • Configured the CPU with 32KB cache.
    • Enabled the 4MB SPI flash, so now it can be used to binaries.
    • Enabled one of the Ethernet PHYs (net-boot works about half the time).

I also managed to flash the FPGA configuration to the flash, so you don’t have to load it every time.

In my previous article I had some trouble with my hacked JTAG probe. It was not working very well – about 2/3 of the time it would not upload the FPGA configuration correctly etc. The issue turned up to be quite simple – I was using bad cables to connect the blue pill board to the JTAG connector.

First thing’s first, assuming you have all litex and yosys software installed from the previous article. Simply clone the following github repository: https://github.com/ghent360/riscvOnColorlight-5A-75B.git

Go to the riscvOnColorlight-5A-75B folder and run:

./base.py --build

Build the test firmware:

cd firmware
make
cd ..

And load it with the lxterm tool (run this in a separate terminal):

~/.local/bin/lxterm /dev/ttyUSB0 --kernel firmware/firmware.bin

Reload the FPGA configuration and reset the board with:

./base.py --load

I switched the default cable type to ‘dyrtyJtag’ so you don’t have to type it anymore.

You wold see a new boot scree with the following text in the lxterm window:

[LXTERM] Starting....
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jul 26 2020 02:12:52
 BIOS CRC passed (6b395de0)

 Migen git sha1: 7bc4eb1
 LiteX git sha1: 1fdffdfd

--=============== SoC ==================--
CPU:       VexRiscv @ 198MHz
BUS:       WISHBONE 32-bit @ 4GiB
CSR:       8-bit data
ROM:       32KiB
SRAM:      8KiB
L2:        32KiB
MAIN-RAM:  4096KiB

--========== Initialization ============--
Ethernet init...
Initializing DRAM @0x40000000...
SDRAM now under software control
SDRAM now under hardware control
Memtest at 0x40000000...
[########################################]
[########################################]
Memtest OK
Memspeed at 0x40000000...
Writes: 461 Mbps
Reads:  382 Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading firmware/firmware.bin to 0x40000000 (14852 bytes)...
[LXTERM] Upload complete (3.2KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
Executing booted program at 0x40000000

--============= Liftoff! ===============--

Hello RISC-V core on Lattice EPC5 FPGA
Testing software built Jul 26 2020 02:15:37

Available commands:
help                            - this command
reboot                          - reboot CPU
led                             - led test
dhry                            - Dhrystone benchmark
RUNTIME>

I added the ability to run the Dhrystone benchmark to the test firmware. You can try it by issuing the ‘dhry’ command. On my board it looks like this:

RUNTIME>dhry

Dhrystone Benchmark, Version 2.1 (Language: C)

Program compiled without 'register' attribute

Execution starts, 200000 runs through Dhrystone
Execution ends

Final values of the variables used in the benchmark:

Int_Glob:            5
        should be:   5
Bool_Glob:           1
        should be:   1
Ch_1_Glob:           A
        should be:   A
Ch_2_Glob:           B
        should be:   B
Arr_1_Glob[8]:       7
        should be:   7
Arr_2_Glob[8][7]:    200010
        should be:   Number_Of_Runs + 10
Ptr_Glob->
  Ptr_Comp:          1073756824
        should be:   (implementation-dependent)
  Discr:             0
        should be:   0
  Enum_Comp:         2
        should be:   2
  Int_Comp:          17
        should be:   17
  Str_Comp:          DHRYSTONE PROGRAM, SOME STRING
        should be:   DHRYSTONE PROGRAM, SOME STRING
Next_Ptr_Glob->
  Ptr_Comp:          1073756824
        should be:   (implementation-dependent), same as above
  Discr:             0
        should be:   0
  Enum_Comp:         1
        should be:   1
  Int_Comp:          18
        should be:   18
  Str_Comp:          DHRYSTONE PROGRAM, SOME STRING
        should be:   DHRYSTONE PROGRAM, SOME STRING
Int_1_Loc:           5
        should be:   5
Int_2_Loc:           13
        should be:   13
Int_3_Loc:           7
        should be:   7
Enum_Loc:            1
        should be:   1
Str_1_Loc:           DHRYSTONE PROGRAM, 1'ST STRING
        should be:   DHRYSTONE PROGRAM, 1'ST STRING
Str_2_Loc:           DHRYSTONE PROGRAM, 2'ND STRING
        should be:   DHRYSTONE PROGRAM, 2'ND STRING

Microseconds for one run through Dhrystone: 4.11 
Dhrystones per Second:                      243186.4 
DMIPS                                       138.40 

RUNTIME>

An alternative command to load the FPGA configuration a bit faster is

openFPGALoader -c dirtyJtag -r build/colorlight_5a_75b/gateware/colorlight_5a_75b.bit

If you are happy with the configuration you can save it to the flash memory and it will be loaded automatically when you power the board on:

openFPGALoader -c dirtyJtag -f build/colorlight_5a_75b/gateware/colorlight_5a_75b.bit

How to compile and run Zephyr OS

If you have never heard of Zephyr OS you can read more about it on the project page. In short it is real-time OS similar to mbed and FreeRTOS. Why not Linux? Well the board has limited memory (4MB) and running Linux would be a challenge – not impossible, but not practical.

Most of the instructions are taken from this article. I did some modifications that are needed for the Colorlight board.

Install the Zephyr prerequisites – instructions from this article. Follow the steps up to #5. Skip step #4.3, we already did the udev rules in my previous tutorial. We would build the sample code a bit differently. Why, you ask? Well, the Zephyr OS already supports a variation of the litex generated RISC-V SoC, that however is different from the hardware configuration we have, so we’ll have to modify some things to get it to work.

Modify the board DTS code

In my repository folder (riscvOnColorlight-5A-75B) there should be a sub-folder called zephyr. Copy these modified files to the following locations:

$COLORLIGHT/zephyr/riscv32-litex-vexriscv.dtsi to zephyrproject/dts/riscv/riscv32-litex-vexriscv.dtsi
$COLORLIGHT/zephyr/litex_vexriscv.dts to zephyrproject/boards/riscv/litex_vexriscv/litex_vexriscv.dts

Why these changes. We are editing the hardware configuration for the OS to match the hardware we have generated with the litex software. In the future I hope I can automate this step.

Note if you are wondering what are all the magic numbers in the dts file. They are addresses in memory where the control registers for the peripherals are. You can find a full list of these registers in a file called ‘csr.json’ in the $COLORLIGHT/build/colorlight_5a_75b folder.

Modify the board configuration

Go to the hello_world sample project  in zephyr and create a build folder:

cd zephyrproject/samples/hello_world
mkdir build
cd build
cmake -DBOARD=litex_vexriscv ..

Now before you continue we have to change the OS default configuration – this removes support for networking and some device drivers that we don’t have.

Copy the simplified .config file:

$COLORLIGHT/zephyr/.config to zephyrproject/samples/hello_world/build/zephyr/

Build the Zephyr OS code

Now we are ready to build the hello_world example:

cd zephyrproject/samples/hello_world/build
make

If the build completes with no errors there should be a binary file zephyr.bin in the build/zephyr folder.

Let’s try to run the compiled binary on the FPGA board. Start the lxterm command in a new terminal window:

cd zephyrproject/samples/hellow_world/build
~/.local/bin/lxterm /dev/ttyUSB0 --kernel zephyr/zephyr.bin --speed 38400

Now reset your FPGA board – either load the FPGA configuration again or if you have stored the configuration in flash, power down and then back up.

You would see the following text in the ‘lxterm’ boot screen:

...
--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading zephyr/zephyr.bin to 0x40000000 (41184 bytes)...
[LXTERM] Upload complete (3.2KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
Executing booted program at 0x40000000

--============= Liftoff! ===============--
*** Booting Zephyr OS build zephyr-v2.3.0-1314-g52f993de1172  ***
Hello World! litex_vexriscv


uart:~$ 

Congratulations you have booted the zephyr OS, executed a sample app and now you have access to the Zephyr simple shell module.

You can press <tab><tab> to get a list of the shell commands. It is not much but as far as examples go it is enough.

Enjoy

Running RISC-V core on small FPGA board

Intro

I took a detour from my usual 3D printing and motor driving hobby this weekend. I have always been fascinated by the flexibility of FPGAs. They are this magical hardware you can make do whatever you want. Though the tools for programming FPGAs have been lets say “complex”.

A while back (14 years or so) I did learn a bit of VHDL. I managed to program one Xilinx CPLD to decode IR signal from a remote control. But the experience was quite an endeavor. Now to present day there has been quite a bit of progress and I managed to compile and run a RISC-V core in one day.

The hardware

A few months ago I read an article on hadkaday.com about the Colorlight 5A-75B board. Chubby75 managed to reverse engineer the schematics and program the FPGA to blink a led. While not earth shattering demo, the board itself is quite interesting. The FPGA is from the Lattice EPC5U family with 25k LUTs, quite modest in size, but there is 4MB of SDRAM and two gigabit Ethernet PHYs + a whole bunch of IO pins. The board’s original purpose is to drive led matrix panels, which is also interesting thing to do, but that is not the point of this article.

I ordered two of these from aliexpress.com and after a month or so they arrived. At the time there was not much to do besides blink the led on the board. I did try to find more info how to get started with risc-v, but at the time it was quite involved with importing VexRiscv CPU implementation and a whole bunch of configuration I had no clue how to do. So the boards went on the shelf waiting for more inspiration.

The software

Note: I use Ubuntu 20.04 for my developer workstation and the instructions would assume you have similar Linux setup.

My patience was rewarded. Some clever and dedicated people did all the hard work. Enter project LiteX. This is fascinating work wich removes a whole lot of menial tasks with regards to IP configuration. You can read a more detailed comparison with Xilinx’s Vivaldo product on bunnie’s blog.

This is the short version of the tutorial. I’ll walk you trough the steps in more details assuming (like me) you are not a seasoned FPGA designer.

The JTAG cable and software

First thing you need to configure an FPGA chip is JTAG cable. Sadly there are so many options and not a single perfect one. The one I followed is called DirtyJTAG – a project to utilize a STM32 blue pill board as USB to JTAG adapter. Why that one – well I have a stash of blue pill boards in one of my drawers for such an occasion.

First you have to flash the DirtyJTAG firmware on your blue pill board. Luckily the instructions on the DirtyJTAG project page are well detailed. I opted for compiling the firmware and flashing with stlinkV2 adapter, but you can download a pre-compiled binary and use serial port to upload the firmware.

Next you have to install the UrJTAG software. Don’t install the ubuntu package – it is not up to date and does not have the DirtyJTAG cable support. You have to download the latest source code and compile it. This page has instructions how to do that. You will have to add rules to grant access to your USB blue pill board, so you don’t have to run the jtag command as root all the time.

The following may seem like black magic, but I’ll try to explain what it does:

as root create a file named /etc/udev/rules.d/99-dirtyjtag.rules, for example use this command:

sudo nano /etc/udev/rules.d/99-dirtyjtag.rules

add the following two lines to the file:

# dirty JTAG 
ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="1209", ATTRS{idProduct}=="c0ca", MODE="0664", GROUP="plugdev"

Save the file and exit the editor. Wait, what just happened? There is a subsystem in Linux called udev that manages plug an play events. We added a rule to that subsystem saying when a USB device with the selected vendor ID and product ID is plugged in the “owner group” of that device should be the¬† group called “plugdev” also the group would have read/write access to that device.

Now to complete this step we need to run tree more commands:

sudo udevadm control --reload-rules
sudo udevadm trigger
sudo usermod -a -G plugdev $USER

The first two would force the udev subsystem to reload the rules from the files. The third line would add the current user to the plugdev group so we can get access to the device.

Unfortunately for the last step to take effect you would have to log out and log back in the Linux OS and no opening a new terminal window does not count.

With all that out of the way, you should be able to plug your blue pill JTAG adapter and run the ‘jtag’ command. You should see something like:

$ jtag
UrJTAG 0.10 #
Copyright (C) 2002, 2003 ETC s.r.o.
Copyright (C) 2007, 2008, 2009 Kolja Waschk and the respective authors

UrJTAG is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
There is absolutely no warranty for UrJTAG.

warning: UrJTAG may damage your hardware!
Type "quit" to exit, "help" for help.

jtag>

Enter the ‘cable dirtyjtag’ command, then ‘detect’. Assuming all is configured it would take a while and return nothing, but you won’t see any errors at this point. If you see anything that says ‘libusb…..something open something’ the most likely cause is that the device access is not correct and you have to redo the udev rules.

The openFPGALoader software

Next we have to install and configure the openFPGALoader software (I know the almost camel case name is confusing). This code can use the JTAG cable to load new FPGA configuration to the board. The steps to compile and install are straight forward. Don’t disable anything, use the defaults.

Cable hookup

At this point you are ready to connect the JTAG cable to the 5A-75B board.

Use the following diagram:

STM32 (blue pill)
JTAG 5A-75B
PA0 TDI J29
PA1 TDO J30
PA2 TCK J27
PA3 TMS J28
PA4 TRST (not used)
PA5 SRST (not used)
3.3V (do not use)
GND J34

You do need the separate power to the 5A-75 board as shown on the big black connector in the lower right corner. I tried to use the 3.3V from the blue pill board, but it was not enough to power the 5A-75 reliably.

Power the blue pill adapter (connect the USB). Run the ‘jtag’ command from the previous section. Send the ‘cable dirtyjtag’ and ‘detect’ command, this time you should see:

$ jtag

UrJTAG 2019.12 #a10945c7
Copyright (C) 2002, 2003 ETC s.r.o.
Copyright (C) 2007, 2008, 2009 Kolja Waschk and the respective authors

UrJTAG is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
There is absolutely no warranty for UrJTAG.

warning: UrJTAG may damage your hardware!
Type "quit" to exit, "help" for help.

jtag> cable dirtyjtag
jtag> detect
IR length: 8
Chain length: 1
Device Id: 01000001000100010001000001000011 (0x41111043)
Manufacturer: Lattice Semiconductors (0x043)
Unknown part! (0001000100010001) (/usr/local/share/urjtag/lattice/PARTS)
jtag>

Blink LED test (optional)

The canonical blinking led test can be performed at this time. Clone the following GitHub repository https://github.com/kholia/Colorlight-5A-75B

Go the Colorlight-5A-75B folder and run:

openFPGALoader -c dirtyJtag blink.bit

Yes the name here is slightly different while UrJTAG uses ‘dirtyjtag’ as cable name on the openFPGALoader the J is capital.

This command would load the blink.bit configuration file on the FPGA and you would see the green led on the board blinking.

Note: this FPGA configuration is compiled for the version 7.0 of the 5A-75B board, if you have a different board version this will not work for you as the led is connected to a different pin. To compile the blink example for a different board revision you would need the yosys software.

The yosys software

Do not install the ubuntu packaged version of yosys, it is old and does not have all the feature we need to run LiteX.

First we would need to install a few dependent projects: Project Trellis and nextpnr. Start with Project Trelis. The instructions are very simple. The only mistake I made was to skip the –recursive flag when cloning git the repository. Just run the commands exactly as described in the project page and it would be fine.

Next is the nextpnr code. In the project page there are several configurations described, we only need the ECP5 configuration. Make sure you have all the prerequisites installed. Scroll down to the nextpnr-ecp5 section and run the 3 commands to build and install the software.

Finally we are ready to compile the yosys software. Read the project page instructions in the Setup section. Then clone the repository and run:

make config-clang
make
make test

If all tests pass at this point you can run

sudo make install

The LiteX software

Finally we are approaching the end of the software configuration journey. This is the last step. You can follow the LiteX software installation page. I’ll repeat the instructions with small modifications that helped me.

Make a folder where you want to install the LiteX software – it downloads quite a few sub packages. Let’s call this folder ‘litex’:

$ mkdir litex
$ cd litex
$ wget https://raw.githubusercontent.com/enjoy-digital/litex/master/litex_setup.py
$ chmod +x litex_setup.py
$ ./litex_setup.py init install --user

Now it would complain that ~/.local/bin is not in the path. Add it:

$ export PATH=$HOME/.local/bin:$PATH

Download the risc-v gcc compiler:

$ ./litex_setup.py gcc

At the end it would tell you to add the risc-v gcc compiler to the path by running this command:

$ export PATH=$PATH:$(echo $PWD/riscv64-*/bin/)

Verify that you can run the risc-v gcc compiler:

$ riscv64-unknown-elf-gcc --version

Now we are ready to compile the RISC-V SoC and upload it to the board.

Compiling the LiteX RISC-V SoC

Exit the LiteX folder and checkout the litexOnColorlightLab004 github repository.

Run the following command to build the FPGA configuration. We would build the configuration to verify all software pieces are correctly installed, do not load it to the FPGA yet.

Go to the litexOnColorlightLab004 folder and run:

./base.py --build

Hopefully all should go well and there would not be any errors. Now you have to make some decisions. The original code would user the pins on the J1 connector for serial communication, however to do so it needs to modify the hardware on the board. The modification involves removing the U28 buffer and soldering some bridge wires in its place. See the prerequisites section.

Another option is to use the J19 connector, but we’ll have to sacrifice the button and the led. I personally went with that direction since it is less invasive.

Code modifications to use J19 connector:

changes in base.py
line 38 - delete: "platform.add_extension(_serial)"

line 46 - change the following 2 lines:
            integrated_main_ram_size = 0x4000,
            uart_name                = "serialJ1")
to
            integrated_main_ram_size = 0x4000)

line 51 - change:
        self.submodules.crg = CRG(platform.request("clk25"), ~platform.request("user_btn_n"))
to
        self.submodules.crg = CRG(platform.request("clk25"))

line 55,56,57 - delete these 3 lines:
       user_leds = Cat(*[platform.request("user_led_n", i) for i in range(1)])
       self.submodules.leds = Led(user_leds)
       self.add_csr("leds")

changes in firmware/main.c

line 93: change the code in led_test(void) to

static void led_test(void)
{
	printf("led_test is disabled\n");
#if 0	
	int i;
	for(i=0; i<32; i++) {
		leds_out_write(i);
		busy_wait(1);
	}
#endif	
}

We removed the user led and the custom serial port definition. We also removed the reference of the user button.

Now we can continue with the original instructions. Build the FPGA configuration again:

./base.py --build

Build the RISC-V firmware code

cd firmware
make
cd ..

Connect serial cable to the J19 connector. Please use 3.3V USB serial adapter not real RS232. Don’t forget to connect TX on the adapter to RX on the J19 connector and RX on the adapter to TX on the J19.

You can run minicom to read the serial output. Connect at 115200 bps 8N1.

Load the FPGA firmware to the board:

./base.py --load --cable dirtyJtag

After it completes you should see the LiteX bootloader come up on the serial port:

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jul 18 2020 22:47:39
 BIOS CRC passed (48dcc56e)

 Migen git sha1: 731c192
 LiteX git sha1: 63c19ff4

--=============== SoC ==================--
CPU:       VexRiscv @ 25MHz
BUS:       WISHBONE 32-bit @ 4GiB
CSR:       8-bit data
ROM:       32KiB
SRAM:      8KiB
MAIN-RAM:  16KiB

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
No boot medium found

--============= Console ================--

litex> 

This is the output of the LiteX BIOS code. It tries to load the RISC-V code from the serial port but fails.

Note: sometimes the openFPGALoader would load the configuration on the FPGA, but it would fail to reset it. This was an issue with the cables I used to connect the blue pill board to the JTAG port. I changed the wires and the stability improved a lot.

To complete the boot process, exit the minicom program and run the following command (while still in the litexOnColorlightLab004 folder):

lxterm /dev/ttyUSB0 --kernel firmware/firmware.bin

Note: in my setup the USB to serial adapter is /dev/ttyUSB0, in you case it may be something different for example /dev/ttyACM0.

This is a serial terminal which intercepts the firmware load request and is going to download the file firmware/firmware.bin to the RISC-V CPU.

Now load the board configuration again running:

./base.py --load --cable dirtyJtag

You will see the following output from the lxterm program:

[LXTERM] Starting....

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jul 18 2020 22:47:39
 BIOS CRC passed (48dcc56e)

 Migen git sha1: 731c192
 LiteX git sha1: 63c19ff4

--=============== SoC ==================--
CPU:       VexRiscv @ 25MHz
BUS:       WISHBONE 32-bit @ 4GiB
CSR:       8-bit data
ROM:       32KiB
SRAM:      8KiB
MAIN-RAM:  16KiB

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading firmware/firmware.bin to 0x40000000 (7856 bytes)...
[LXTERM] Upload complete (9.0KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
Executing booted program at 0x40000000

--============= Liftoff! ===============--

Hello RISCv software defined CPU
Testing software built Jul 18 2020 23:12:43

Available commands:
help                            - this command
reboot                          - reboot CPU
led                             - led test
RUNTIME>

Enter the “led” command:

RUNTIME>led
led_test is disabled
RUNTIME>

Remember that we disabled the led test earlier? This shows that our modified code is executing on the CPU.

~Enjoy