DigiTekXplorer - abCore16 Toolchain

The abCore16 project is a complete, vertically integrated 16-bit computer system designed and written entirely in Python. It constitutes a full software and hardware ecosystem, featuring a custom-designed CPU architecture, a C-like high-level programming language, and a robust toolchain to support the entire development lifecycle.

The abCore16 toolchain is a set of Python-based software tools responsible for taking the programs you write in a high-level C-Like language (SSL) and transforming them into a binary format that our 16-bit abCore16 Microprocessor Simulator can understand and execute. The toolchain consists of four main components: the C-Like Compiler, the SAL Assembler, Simulator, and the Machine Code Disassembler. 

The Workflow:
C-like Code (.ssl) -> Compiler -> Assembly Code (.sal) -> Assembler-> Machine Code (.bin) -> Simulator (simulator output to tool terminal)

The C-like High-Level Language (SSL)

This is the primary language for application development. It is designed to be familiar to C programmers but tailored for the abCore16 architecture.

• Data Types: A single 16-bit integer type is used for all variables.
• Variables: Supports global and function-local variable declarations using the var keyword.
• Functions: Full support for function definition (func) and calls, including parameters and return values. The compiler automatically manages stack frames, calling      conventions, and register preservation.
• Control Flow: Implements if/else, for, and while statements.
• Arrays: Supports the declaration and use of global, one-dimensional arrays.
• Abstraction: Completely abstracts away direct register management and memory addresses from the programmer.

The PLY-Based Compiler

This is the most complex component of the toolchain, responsible for translating the C-like SSL into SAL.

• c_ply_compiler.py: Uses the PLY (Python Lex-Yacc) library.
  • Lexer: Tokenizes the input source code into a stream of tokens (e.g., IDENTIFIER, NUMBER, WHILE, LPAREN).
  • Parser: Applies context-free grammar rules to the token stream to validate the program's syntax and build an Abstract Syntax Tree (AST).
• ast_nodes.py: Defines the Python classes for every node in the AST (e.g., FunctionDefinitionNode, ForNode, ArrayAccessNode). The AST is a tree structure that represents the program's semantic meaning.
• code_generator.py: A class that implements the Visitor design pattern. It traverses the AST from top to bottom, visiting each node and emitting the corresponding SAL      assembly code. It manages symbol tables, scopes, label generation, and the      allocation of temporary registers.

Once SSL is compiled into SAL, the Simple Assembler converts this textual SAL code into the binary machine code for the 16-bit abCore16.

• Purpose: To translate SAL instructions into a compact, numerical, binary machine code format, resolving symbolic addresses for the 16-bit architecture.
• Input: The SAL code string from the SSL Compiler.
• Process: Two-Pass Assembly
  1. First Pass: Symbol Table Construction
     • Reads SAL lines, preprocessing them (stripping SAL comments starting with ;).
     • Identifies label definitions (e.g., MY_LOOP:).
     • Calculates 16-bit Byte Offsets: Determines the byte length of each SAL instruction (1, 2, 3, or 4 bytes based on its opcode and operands, accounting for 16-bit immediates/addresses).
     • Builds a Symbol Table: Maps each label name to its calculated 16-bit byte offset.
     • Reports errors like duplicate labels.
  2. Second Pass: Machine Code Generation
     • Re-iterates through SAL instructions.
     • For each instruction:
       • Converts mnemonic to its 1-byte numerical opcode.
       • Operand Parsing:
         • Register operands (R0-R7) are converted to their 3-bit numerical codes (0-7).
         • Immediate values prefixed with # (for LOAD, SHL, SHR) are parsed as decimal.
         • Direct address operands (for STORE, LOADM, INM, OUTM) are parsed as decimal or 0xHEX (as output by the compiler).
       • Operand Encoding:
         • 16-bit immediates (for LOAD) are split into Low Byte and High Byte, then emitted (Little Endian: Low Byte, then High Byte).
         • 16-bit addresses (for STORE, LOADM, INM, OUTM, and resolved labels for jumps/calls) are split into Low Byte and High Byte, then emitted (Little Endian).
         • 8-bit immediates (shift amounts) are emitted as a single byte.
       • Appends the resulting byte sequence to the machine code.
       • Reports errors for undefined labels or values out of appropriate ranges.
• Output:
  1. Binary Machine Code File (e.g., myProg.bin): Contains the raw machine code bytes.

  
  

  Assembly Listing File (e.g., myProg.asm): Shows SAL alongside calculated 16-bit byte offsets and the symbol table (with 16-bit offsets 

The Simple Disassembler provides a way to translate the binary machine code back into a human-readable SAL format, aiding in verification and understanding.

• Purpose: To verify assembler output and inspect machine code.
• Input: A .bin machine code file.
• Process:
  1. Load Machine Code: Reads bytes from the .bin file.
  2. Label Pre-Pass: Scans the machine code to identify target addresses of jump/call       instructions and creates placeholder labels (e.g., L_010Ah).
  3. Sequential Disassembly:
     • Fetches opcode byte and determines instruction format using an internal table.
     • Fetches operand bytes, reassembling 16-bit immediates/addresses from byte pairs (Little Endian).
     • Converts numerical register codes (0-7) back to R0-R7.
     • Formats Operands for SAL Output:
       • 16-bit Immediates (for LOAD): "#0xHEXVAL".
       • 8-bit Immediates (for SHL/SHR): "#DECIMAL".
       • 16-bit Addresses (for memory ops and resolved jump/call targets): "0xHEXVAL" (or label name if identified).
     • Constructs SAL lines, prepended with their 16-bit byte offset.

     
     

     Output: A multi-line string of disassembled SAL code, written to a _disassembled.sal file. 

The Simulator acts as the virtual hardware for our system. It loads and runs the binary files produced by the Assembler.

Simulated Components:

• Registers: A bank of eight 16-bit general-purpose registers (R0-R7).
• Special Registers: A 16-bit Program Counter (PC) and Stack Pointer (SP).
• Data Memory: A block of memory used for global variables and the stack.
• CPU Flags: The four status flags (ZF, SF, CF, OF) are updated after every relevant instruction.

The Execution Cycle:
The simulator runs in a main loop that endlessly performs the classic Fetch-Decode-

Execute cycle:

1. Fetch: Reads the byte at the address pointed to by the PC. This is the opcode.
2. Decode: Identifies the instruction and fetches any additional operand bytes required.
3. Execute: Performs the action of the instruction, such as adding two registers, loading a value from memory, or jumping to a new address. This may involve updating      registers, memory, and flags.

I/O Simulation:
Input and Output are handled via Memory-Mapped I/O (MMIO). Specific memory addresses (0x00FE for input, 0x00FF for output) are treated specially. When the program reads from the input address, the simulator prompts the user for input. When it writes to the output address, the simulator prints the value to the console.

This section demonstrates the execution of a file called test_array.ssl and the abCore16's toolchain output as it progresses from compilation to simulation.

Test Program

The file test_array.ssl is a short test program written in the C-Like language.  The file test_array.ssl is shown below:

// test_arrays.ssl
// A program to test the new global array functionality.

// Declare a global array of 5 words and a global variable for indexing.
var my_array[5];
var i;

func main() {
   // --- Test 1: Direct write and read with constant indices ---
   print 1000; // Marker for start of test 1

   my_array[0] = 50;
   my_array[1] = 60;
   my_array[4] = 90; // Test writing to the last element

   print my_array[0]; // Expected output: 50
   print my_array[1]; // Expected output: 60
   print my_array[4]; // Expected output: 90

   // --- Test 2: Using a variable as an index ---
   print 2000; // Marker for start of test 2

   i = 2;
   my_array[i] = 77; // Write to my_array[2]

   print my_array[2]; // Expected output: 77

   // --- Test 3: Overwriting a value and reading it back ---
   print 3000; // Marker for start of test 3

   my_array[0] = 55; // Overwrite the first element

   i = 0;
   print my_array[i]; // Expected output: 55
}

Program Execution

In the PyCharm Terminal I entered:

python main.py test_array.ssl

Terminal Output

After executing the command above, the abCore16 toolchain generated the following output in PyCharm's Terminal:

Found source file: test_arrays.ssl

============================================

STARTING C-LIKE SSL (.ssl) TOOLCHAIN FOR: C-like SSL File 'test_arrays.ssl' 

============================================

--- COMPILER (PLY-based via c_ply_compiler.py): Compiling C-like SSL ---

--- COMPILER (PLY-based): Generated SAL successfully ---

PLY COMPILER: Generated SAL saved to 'test_arrays_from_ply.sal'

(Appended HALT to PLY-generated SAL for assembler)

============================================

STARTING TOOLCHAIN FOR PRE-COMPILED: Compiled from C-like SSL File 'test_arrays.ssl' (via PLY)

============================================

--- ASSEMBLER: Assembling SAL Code ---

--- Assembler: Starting First Pass ---

--- Assembler: First Pass Complete ---

Assembler: Assembly listing written to 'test_arrays.asm'

--- Assembler: Starting Second Pass ---

Assembler: Machine code successfully written to 'test_arrays.bin' (219 bytes)

Assembler: Assembly listing written to 'test_arrays.asm'

--- DISASSEMBLER: Disassembling 'test_arrays.bin' ---

DISASSEMBLER: Output successfully written to 'test_arrays_disassembled.sal'

--- SIMULATOR: Loading and Running Binary File ---

SIM MMIO OUTPUT (0x00FFh): 1000

SIM MMIO OUTPUT (0x00FFh): 50

SIM MMIO OUTPUT (0x00FFh): 60

SIM MMIO OUTPUT (0x00FFh): 90

SIM MMIO OUTPUT (0x00FFh): 2000

SIM MMIO OUTPUT (0x00FFh): 77

SIM MMIO OUTPUT (0x00FFh): 3000

SIM MMIO OUTPUT (0x00FFh): 55

--- SIMULATOR: Simulation Log ---

--- Sim Start (DataMemWords:8192, StackBase:2000h, StackLimit:1F00h) ---

MMIO In: 00FEh, MMIO Out: 00FFh

Sim: Binary 'test_arrays.bin' loaded (219B).

PC=0000h SP=2000h Flags:[ZF=0 SF=0 CF=0 OF=0] | Op:0x70(CALL) | Regs:[R0:0000, R1:0000, R2:0000, R3:0000, R4:0000, R5:0000, R6:0000, R7:0000]

CALL: Pushed RetAddr(0x0003h) to Stack[SP=0x1FFFh]. JMP to 0x0004h.

PC=0004h SP=1FFFh Flags:[ZF=0 SF=0 CF=0 OF=0] | Op:0x60(PUSH) | Regs:[R0:0000, R1:0000, R2:0000, R3:0000, R4:0000, R5:0000, R6:0000, R7:0000]

PUSH: R5(0x0000) to Mem[SP=0x1FFEh].

.

.

.

PC=00D9h SP=1FFFh Flags:[ZF=1 SF=0 CF=0 OF=0] | Op:0x71(RET) | Regs:[R0:0000, R1:0000, R2:0000, R3:0000, R4:0000, R5:0000, R6:1000, R7:0037]

RET: Popped RetAddr(0x0003h) from Stack[SP(old)=0x1FFFh]. JMP.

PC=0003h SP=2000h Flags:[ZF=1 SF=0 CF=0 OF=0] | Op:0xFF(HALT) | Regs:[R0:0000, R1:0000, R2:0000, R3:0000, R4:0000, R5:0000, R6:1000, R7:0037]

HALT: CPU Halted by instruction.

--- Sim Finished (CPU HALTED cleanly) (67 cycles) ---

Final PC: 0x0004h

Final Regs: [R0:0x0000, R1:0x0000, R2:0x0000, R3:0x0000, R4:0x0000, R5:0x0000, R6:0x1000, R7:0x0037]

Final SP: 0x2000h

Final Flags: ZF=1 SF=0 CF=0 OF=0

--- SIMULATOR: Final Simulator Register States ---

--- Final Simulator State (print_final_state) ---

Reg R0 : 0 (0x0000) (0000000000000000b)

Reg R1 : 0 (0x0000) (0000000000000000b)

Reg R2 : 0 (0x0000) (0000000000000000b)

Reg R5 : 0 (0x0000) (0000000000000000b)

Reg R6 : 4096 (0x1000) (0001000000000000b)

Reg R7 : 55 (0x0037) (0000000000110111b)

SP: 0x2000h (8192) PC:0x0004h (4) Flags: Z1S0C0O0

Halted: True, Clean Halt: True

Data Mem Sample (size 8192):

(All sampled mem is zero or stack empty at base)

-------------------------------------------

============================================

TOOLCHAIN COMPLETE FOR: Compiled from C-like SSL File 'test_arrays.ssl' (via PLY)

============================================

Successfully processed 'test_arrays.ssl'.

Here is a detailed breakdown of all the files generated by the complete abCore16 toolchain, along with the function that creates each one and its specific purpose in the ecosystem.   The toolchain is designed to produce a coordinated set of files that support software simulation, hardware implementation on an FPGA, and detailed debugging at every stage.

1. Primary Build Artifact

This is the core output of the compilation and assembly process.

• File: [base_filename].bin
  • Generated By: SimpleAssembler (in simple_assembler.py)
  • Purpose: This is the executable machine code for the abCore16 microprocessor.  The .bin file is a binary representation of the .ssl or .ab programs written by the user. It is a raw binary file containing the sequence of bytes (opcodes and operands) that the processor executes directly. It serves as the primary input for both the software simulator and the FPGA memory file generator.

2. FPGA/Hardware Implementation Files

These files are generated from the .bin file and are specifically for use with Xilinx Vivado and an FPGA.

• File: [base_filename].coe
  • Generated By: generate_mem_files (in generate_mem_files.py)
  • Purpose: This is a Xilinx Coefficient File used to initialize the Block RAM (BRAM) in the FPGA during synthesis and implementation. When you program the FPGA, the contents of this file are loaded into the on-chip memory, effectively "burning" your program into the hardware. The file is padded with NOP (0x00) instructions to match the full size of the BRAM.
• File: [base_filename].hex
  • Generated By: generate_mem_files (in generate_mem_files.py)
  • Purpose: This is a standard hexadecimal memory file used for behavioral simulation in a Verilog/SystemVerilog environment like the Vivado Simulator. It is read by the $readmemh system task in your testbench. This allows you to quickly test new programs in simulation without having to re-generate the BRAM model, dramatically speeding up the hardware development cycle.

3. Simulation and Verification Files

These files are generated during the software simulation phase to help verify program correctness and debug its execution.

• File: [base_filename].txt
  • Generated By: main.py (using data collected from MicroprocessorSimulator)
  • Purpose: This is a comprehensive, human-readable log of a program's execution in the software simulator. It contains two key sections:
    1. MMIO OUTPUT: Placed at the top, this section provides a clean, final record of all data the program wrote to the Memory-Mapped I/O output address. It's perfect for quickly verifying the program's final results or for use in        automated testing.
    2. Simulation Log: A detailed, cycle-by-cycle trace of the processor's state, including the Program Counter (PC), register values, flag statuses, and the        instruction being executed at each step. This is invaluable for low-level debugging of program flow and logic.

• File: [base_filename]_disassembled.sal
  • Generated By: SimpleDisassembler (in simple_disassembler.py)
  • Purpose: This file provides a round-trip verification of the toolchain. It takes the final .bin machine code and translates it back into human-readable Simple Assembly Language (SAL). By comparing this file to the original or intermediate SAL file, you can confirm that the assembler and disassembler are working correctly and consistently.

4. Intermediate and Debugging Files

These files are generated as part of the toolchain process and are primarily used for debugging the tools themselves.

• File: [base_filename].asm
  • Generated By: SimpleAssembler (in simple_assembler.py)
  • Purpose: This is an Assembly Listing File. It shows the original source assembly       code side-by-side with the memory offsets (addresses) and the exact machine code bytes generated for each instruction. It is essential for debugging the assembler, understanding instruction encoding, and resolving issues with labels and offsets.
• File: [base_filename]_from_ply.sal
  • Generated By: compile_c_ssl_string_to_sal (in c_ply_compiler.py), saved by main.py
  • Purpose: This file is an intermediate artifact created only when compiling from a high-level, C-like .ssl file. It contains the Simple Assembly Language (SAL) code that the PLY-based compiler produced. Its purpose is to allow developers to inspect the compiler's output, making it possible to debug the C-like language compiler itself.