# Embedded System Test Automation Framework

This is a C++ application designed for testing embedded systems using UART communication. The system allows you to load and execute test cases that interact with the UART interface according to predefined protocols.

## Table of Contents

- [Overview](#overview)
- [Key Features](#key-features)
- [Supported Embedded Toolchains](#supported-embedded-toolchains)
- [Project Structure](#project-structure)
- [Requirements](#requirements)
  - [Development Environment](#development-environment)
  - [Build and Deployment](#build-and-deployment)
- [Configuration](#configuration)
  - [Test Stand Settings](#test-stand-settings)
  - [Test Plan](#test-plan)
- [Running the Test Stand](#running-the-test-stand)
- [Test Case Syntax](#test-case-syntax-json)
  - [Processing Text Data Received via UART](#processing-text-data-received-via-uart)
  - [Processing Binary Data Received via UART](#processing-binary-data-received-via-uart)
- [Reporting](#reporting)
- [Notes (summary)](#notes)

## Overview

Embedded Systems Test Automation Framework is a console-based C++ application for automated testing of embedded systems through UART communication.

The framework supports:

* execution of predefined test plans;
* transmission of binary commands to the target device;
* reception and validation of UART responses (if text format) or parsing of binary C structures directly from the target memory;
* generation of text logs and HTML reports;

The framework is configured entirely through JSON files and can be easily integrated into automated verification environments.

## Key Features

- **Binary Command Support**: Send raw `.bin` payloads directly to the microcontroller.
- **UART I/O Communication**: UART I/O for interacting with microcontrollers.
- **Sequence Validation in text format**: Validates incoming messages by matching specific patterns:
  - **Start Message**: Initial handshake/trigger.
  - **Sequence**: Ordered sequence of expected intermediate messages.
  - **End Message**: Completion criteria for the test case.
- **JSON-Driven Configuration**: Easily define test cases, UART parameters, and test plans in JSON format.
- **Flexible Execution**:
  - **Grouping**: Organize tests into suites (e.g., --group=smoke, --group=full).
  - **Repeats**: Built-in support for stress testing by repeating cases (using --repeat=N).
- **Advanced Logging**: Generates detailed text logs and visual HTML reports for every run.

## Supported Embedded Toolchains

The framework itself is compiler-independent.

However, automatic binary structure parsing is currently implemented for firmware generated by:

* IAR Embedded Workbench
* ELF/DWARF compatible toolchains

The parser extracts:

* variable addresses;
* structure sizes;
* nested structures;
* array dimensions;
* primitive data types.

The generated metadata are stored as JSON files inside:

```text
./iar_files/
```


## Project Structure

```text
├── cmd
│   ├── cmd_abort.bin
│   ├── set_interface_params.bin
│   ├── set_settings_NAND_micron_MT29F16G08AJADAWP.bin
├── compiler
│   ├── Makefile
│   ├── Makefile.Debug
│   ├── Makefile.Release
│   ├── data.bin
├── config.json
├── configs
│   ├── cmd
│   ├── test_cases
│   │   ├── test_UART_cmdSetDataInterface.json
│   │   ├── test_abort.json
│   │   ├── test_readSnapshot_HK.json
│   │   ├── test_read_NANDctrlOper.json
│   │   ├── test_set_micron_MT29F16G08AJADAWP.json
│   └── test_plan.json
├── iar_files
│   ├── ENTRY_LIST_map.json
│   ├── NANDctrlOper_t.json
│   └── Snapshot_HK_t.json
├── logs
│   ├── 12_05_2026_15_48_16.log
│   ├── report.html
│   ├── test_cases.log
│   └── test_stand.log
├── python
│   └── iar_parser
│       ├── elf_parser.py
│       ├── map_parser.py
│       ├── requirements.txt
├── reports
├── src
│   ├── core
│   │   ├── cli_parser.cpp
│   │   ├── cli_parser.h
...
│   │   ├── uart.cpp
│   │   └── uart.h
│   ├── embedded_test_stand.pro
│   ├── embedded_test_stand.pro.user
│   ├── gtest_dependency.pri
│   ├── main.cpp
│   └── tests
│       ├── test_runner.cpp
│       ├── uart_fixture.cpp
│       └── uart_fixture.h
└── src.rar
```

## Requirements

### Development Environment
The project was developed and tested on Windows 10/11 using the following software:

| Component | Version |
|----------|---------|
| Qt | 5.15.2 |
| Qt Creator | 9.x or newer |
| Compiler | MinGW 8.1.0 (64-bit) |
| Debugger     | `gdb.exe`            |
| Build system | qmake                |
| qmake | Qt 5.15.2 |
| mingw32-make | MinGW 8.1.0 |
| Python | 3.9+ |
| GoogleTest | Latest stable version |

#### Qt Modules

The following Qt modules are required:

* QtCore
* QtSerialPort

#### Python Dependencies

Python scripts located in:

```text
python/iar_parser/
```

To install the required dependencies, run the following command:
```bash
pip install -r requirements.txt
```

#### GoogleTest

The project uses GoogleTest sources directly:

```text
GOOGLETEST_DIR=C:\path\to\googletest
```

### Build and Deployment

The project is built using **qmake**.

Generate Makefiles:

```bash
qmake embedded_test_stand.pro
```

Build:

```bash
mingw32-make -j8
```

Clean:

```bash
mingw32-make clean -j8
```

The build directory used during development:

```text
./compiler
```

After successful compilation, Qt runtime libraries should be deployed using:

```bash
windeployqt embedded_test_stand.exe
```

In Qt Creator the following deployment step is used:

```text
Command:
windeployqt

Arguments:
%{buildDir}/release/embedded_test_stand.exe

Working directory:
%{buildDir}/compiler
```

This command automatically copies:

* QtCore.dll
* Qt5SerialPort.dll
* platform plugins
* required MinGW runtime libraries
* other Qt dependencies

to the executable directory.


## Configuration

### Test Stand Settings  
See `./config.json` file:
```json
{
  "UART": {
    "RS485_address": 1, 
    "COM_port": "COM14",
    "COM_baudRate": 921600,
    "COM_bits": 8,
    "COM_parity": 3,
    "COM_stopBits": 1,
    "COM_flowControl": false
  },
  "Path": {
    "MK_elf_file": "C:/Danila/work/sputnik_test/src/sputnik/Debug/c.out",
    "MK_map_file": "C:/Danila/work/sputnik_test/src/sputnik/Debug/sputnik.map",
    "test_plan_path": "C:/Danila/work/embedded_test_stand/configs/test_plan.json",
    "test_cases_path": "C:/Danila/work/embedded_test_stand/configs/test_cases/",
    "test_log_path": "C:/Danila/work/embedded_test_stand/logs/",
    "stand_report_html_path": "C:/Danila/work/embedded_test_stand/logs/report.html",
    "map_parser": "C:/Danila/work/embedded_test_stand/python/iar_parser/map_parser.py",
    "elf_parser": "C:/Danila/work/embedded_test_stand/python/iar_parser/elf_parser.py",
    "binary_cmd_path": "C:/Danila/work/embedded_test_stand/cmd/"
  }
}
```
- **`UART`** - Define UART hardware parameters
- **`Path`** - Define path settings
  - **`MK_elf_file`** - the result of compiling the microcontroller software in the form of en elf-file (needed if using `"protocol_ID": 2` in `test case json file`)
  - **`MK_map_file`** - the result of compiling the microcontroller software in the form of map-file
  - **`test_plan_path`** - the path with test plan json file
  - **`test_cases_path`** - the path with test cases json files
  - **`test_log_path`** - the path to which log files will be written
  - **`stand_report_html_path`** - the path and filename to which html-report will be written
  - **`map_parser`** - the path and filename where python-file is located. This python-script parse map-file (from compiler for microcontroller software)
  - **`elf_parser`** - the path and filename where python-file is located. This python-script parse elf-file (from compiler for microcontroller software)
  - **`binary_cmd_path`** - the path with binary data will be transmitted to microcontroller through UART


### Test Plan 
See `./configs/test_plan.json` file. This file define test groups and their associated test cases, for example:  
```json
{
  "groups": {
    "smoke": [
      "test_UART_cmdSetDataInterface.json",
      "test_readSnapshot_HK.json",
      "#test_erase_all.json"
    ],

    "full": [
      "### set NAND settings ###",
      "test_UART_cmdSetDataInterface.json",
      "test_start_March_FTE_by_blocks.json"
    ]
  }
}
``` 
- **`"smoke", "full"`** - user defined groups containing sets of test cases
- **`"#test_abort.json"`** - ignored test
- **`"test_readSnapshot_HK.json"`** - example of user test case 
- **`"### set NAND settings                       ###"`** - comments

## Running the Test Stand
Before starting the application make sure that:

* the target embedded device is powered on;
* the UART interface is connected correctly;
* the COM port specified in `config.json` exists and is not used by another application;
* all configuration files and binary command files are present;
* ELF and MAP files of the embedded firmware are available if binary structure parsing is used (`protocol_ID = 2`).


Execute the built binary with command-line arguments:  
```bash
./embedded_test_stand.exe --delay 10 --group full
```
or
```bash
./embedded_test_stand.exe --delay 10 --group smoke --case test_UART_cmdSetDataInterface
```
**Available Flags**:  
- `--group <GROUP_NAME>`: Specify a test group (e.g., `smoke`, `full`).  
- `--case <TEST_NAME>`: Run a specific test case (overrides group).  
- `--repeat <N>`: Repeat the test case `N` times.   
- `--delay`: <N> delay `N` msec between each test cases 

expects the following files to be present:

```text
config.json
configs/
├── test_plan.json
└── test_cases/
    ├── test_readSnapshot_HK.json
    └── ...
```

The application loads:

* global settings from `config.json`;
* a list of test cases from `test_plan.json`;
* individual test descriptions from `configs/test_cases/*.json`;
* UART command binaries specified in the `"binary"` field of each test case.

After startup, the application:

1. Opens the UART port.
2. Sends binary commands to the embedded target.
3. Receives UART responses.
4. Parses received binary structures or text.
5. Validates structure fields according to the `struct_check` rules.
6. Generates console and HTML test reports.

---

## Test Case Syntax
The general view of the test case depends on the type of processed data, received via UART:
- binary data (see below `"protocol_ID": 2`)
- text data (see below `"protocol_ID": 1`)

### Processing text data, received via UART
Each test case file (for example, `test_start_March_FTE_by_blocks.json`) defines the expected UART sequence:  
```json
{
  "meta": {
    "name": "start_March-FTE_by_blocks",
    "description": "start March-FTE by specified blocks",
    "parameters": {
      "_comment": "protocol_ID = 1 for text format from UART, see protocol_ID_en",
      "protocol_ID": 1,
      "version": 1,
      "cmd_from_binary_file": true,
      "cmd_code": 21,
      "mode_March_FTE": 2,
      "begin_block": 20,
      "end_block": 25
    }
  },

  "binary": "start_March-FTE_by_blocks_20_25.bin",

  "expectations": {
    "start": {
      "type": "json",
      "fields": {
        "msgType": 2,
        "cmd": 21
      }
    },

    "sequence": [
      {
        "type": "regex",
        "value": "start march-FTE in block \\d+(, LUN 0, Target 0)?"
      }
    ],

    "end": {
      "type": "contains",
      "value": "NAND Flash test stand ready to receive new commands"
    }
  },

  "timeout_msec": 35000
}
```  
- **`"timeout_msec"`** - UART response timeout. Messages may arrive in parts over a specified time interval. After the timeout expires, data reception stops, and the message parts are combined into a final string as they arrive, which is then analyzed in its entirely
- **`"meta"`**: `"parameters"` - user defined test case parameters
  - `"protocol_ID"` - defines the type of data received from the UART that needs to be processed
  - `"version"` - reserved for future realization
  - `"cmd_from_binary_file"` - always `true`, because command data file for transmit via UART is taken form the binary file now, see `"binary"` field below
  - `"cmd_code"` - command code (user information) \
  **Additionally, the user can add other fields here in order to distinguish test cases**
- **`"binary"`** - The binary data file name, whose contents will be transmitted via UART. Corresponding file path is taken from `"binary_cmd_path"` field, see `./config.json`
- **`"expectations"`** - Validates incoming messages by matching specific patterns \
  - **`"start"`** Find the initial specified substring at the beginning of the message from UART.
  - **`"sequence"`** Find ordered specified sequence of expected intermediate messages from UART.
  - **`"end"`** Find specified terminate substring of massage from UART. 
    - *`"type"`** - specifies the type of message to compare. Possible meanings: `"contains"`, `"regex"`, `"json"` \
    `"contains"` - find the substring, specified in the "value" field in the entire message received via UART \
    `"json"` - find the substring in json-format, specified in the "value" field in the entire message received via UART \
    `"regex"` - find the substring by using regular expression, specified in "value" field in the entire message received via UART 


An example message from UART for passing this test case:
```
{"msgType": 2,"cmd": 21}

 NAND Flash manufacture: MICRON
 block Endurance (when using internal ECC): 100000 cycles
**************************************************************

 start test March-FTE in NAND Flash 
 algorithm test:
 1. ERASE BLOCK - erasing one block
 2. READ(~D)n - read all pages of one block 3 times to check for data equality to the 0xFF pattern, incrementing address
 3. PROGRAM(D) - programming pages of one block with pattern data, incrementing address
 4. READ(D)n - read all pages of one block 3 times to check for data equality to written pattern data, incrementing address
 5. READ(D) - repeat read all pages of one block, incrementing address
 6. ERASE BLOCK - erasing one block
 7. READ(~D)n - read all pages of one block 3 times to check for data equality to the 0xFF pattern, incrementing address
 8. PROGRAM(D) - programming pages of one block with pattern data, DECREMENTING address
 9. READ(D)n - read all pages of one block 3 times to check for data equality to written pattern data, DECREMENTING address
 10. READ(D) - repeat read all pages of one block, DECREMENTING address

 March-FTE test will check the following blocks: from: 20 to: 25, LUN: 0, target: 0
 **************************************************************
start march-FTE in block 20, LUN 0, Target 0
start march-FTE in block 24, LUN 0, Target 0
TEST FINISH!
NAND Flash test stand ready to receive new commands
```

### Processing binary data, received via UART
Each test case file (for example, `test_readSnapshot_HK.json`) defines the expected structure, received via UART:  
```json
{
  "meta": {
    "name": "read_Snapshot_HK",
    "description": "read Snapshot_HK structure",
    "parameters": {
      "_comment": "protocol_ID = 2 for binary format data from UART, reading C structure from memory, see protocol_ID_en",
      "protocol_ID": 2,
      "version": 1,
      "cmd_code": 27,
      "struct_name": "Snapshot_HK",
      "struct_type": "Snapshot_HK_t",
      "cmd_from_binary_file": true,
      "addr_user_set": true,
      "address": "0x08125600",
      "size_user_set": true,
      "size_bytes": 222
    }
  },

  "binary": "data.bin",
  
  "struct_check" : [ {
        "field": "size",
		"type": "uint16",
		"op": "==",
		"expect": 222
    }, {
        "field": "counter",
		"type": "uint32",
		"op": ">",
		"expect": 0
    }, {
        "field": "uart.errorLastOperation",
		"type": "uint8",
		"op": "==",
		"expect": 0
    }, {
        "field": "rmap.error",
		"array": true,
		"type": "uint16",
		"op": "==",
		"exclude": [0, 22],
		"expect": 0
    } ],

  "timeout_msec": 100
}
```  
- **`"timeout_msec"`** - UART response timeout. Messages may arrive in parts over a specified time interval. After the timeout expires, data reception stops, and the message parts are combined into a final string as they arrive, which is then analyzed in its entirely
- **`"meta"`**: `"parameters"` - user defined test case parameters
  - `"protocol_ID"` - defines the type of data received from the UART that needs to be processed
  - `"version"` - reserved for future realization
  - `"cmd_code"` - command code (user information) \
  - `"struct_name"` - a variable having the type of the structure being checked as in the source embedded code (for example: "Snapshot_HK")
  - `"struct_type"`: - the name of the type of the structure being checked as in the source embedded code (for example: "Snapshot_HK_t")
  - `"cmd_from_binary_file"` - always `true`, because command data file for transmit via UART is taken form the binary file now, see `"binary"` field below
  - `"addr_user_set"` - if `true`, means checked variable address is taken from "address" field (see below). If `false` means checked variable address is taken from parsed compiler map-file (by using python-script) from `./iar_files/ENTRY_LIST_map.json`
  - `"address"` -  checked variable address in hex format, for example: "0x08125600"
  - `"size_user_set"` - if `true`, means checked structure size is taken from "size_bytes" filed (see below). If `false` means checked structure size is taken from parsed compiler map-file (by using python-script) from `./iar_files/ENTRY_LIST_map.json`
  - `"size_bytes"` - specify checked structure size in bytes
  **Additionally, the user can add other fields here in order to distinguish test cases**
- **`"binary"`** - The binary data file name, whose contents will be transmitted via UART. Corresponding file path is taken from `"binary_cmd_path"` field, see `./config.json`
- **`"struct_check"`** - this section defines a set of validation rules applied to the binary structure received from the target device via UART.
Each entry describes:

  * **`field`** – structure field name. Nested fields are supported using dot notation, for example:

    * `"uart.errorLastOperation"`
    * `"rmap.error"`

  * **`type`** – expected C data type of the field:

    * `uint8`
    * `uint16`
    * `uint32`
    * `uint64`
    * `int8`
    * `int16`
    * `int32`
    * `int64`
    * `float`
    * `double`

  * **`op`** – comparison operator used for validation:

    * `"=="` equal to
    * `"!="` not equal to
    * `">"` greater than
    * `">="` greater than or equal to
    * `"<"` less than
    * `"<="` less than or equal to

  * **`expect`** – expected value used in comparison.

  * **`array`** *(optional)* – if `true`, the specified field is treated as an array and every element is checked against the condition.

  * **`exclude`** *(optional)* – list of array indices excluded from validation. This is useful when some array elements are reserved, contain implementation-specific values, or are expected to change dynamically.

For example:

```json
{
    "field": "rmap.error",
    "array": true,
    "type": "uint16",
    "op": "==",
    "exclude": [0, 22],
    "expect": 0
}
```

This rule means:

* the field `rmap.error` is an array of `uint16`;
* all array elements must be equal to `0`;
* elements with indices `0` and `22` are skipped during validation.

The validation result for each rule is reported independently. If at least one check fails, the test case is considered failed and a detailed diagnostic message is printed, including:

* field name;
* actual value;
* expected value;
* comparison operator;
* array index (for array checks).

The test-case checks the following structure received via UART:
```
typedef __packed struct
{
	uint8_t errorLastOperation; //RMAP_errFlag_en
	uint16_t error[COUNT_ERR_MAX_ELEMENTS_RMAP];
} RMAP_HK_t;

typedef __packed struct
{
	volatile uint8_t errorLastOperation; // UART_operErrCode_enum
	volatile uint16_t error[COUNT_ERR_MAX_ELEMENTS];
} UART_HK_t;

extern UART_HK_t UART_HK;

typedef __packed struct
{
	uint16_t size;
	uint32_t counter;
	UART_HK_t uart;
	RMAP_HK_t rmap;
} Snapshot_HK_t;
```

---

## Reporting  

- **`./logs/test_stand.log`**: contains a report of errors from the Embedded System Test Automation Framework
- **`./logs/test_cases.log`**: contains a report with detailed execution steps.  
- **`./logs/report.html`**: contains a summary report. It is created after execution and includes:  
  - Test case pass/fail status.  
  - Timestamps and error messages.  
  - UART communication statistics.  

---

## Notes (summary) 
- The application uses asynchronous UART communication for real-time testing.

- Incoming data from target may arrive in multiple fragments. The framework accumulates all received packets until the timeout expires, specified in the test case by `"timeout_msec"`
After that, the complete UART message is assembled and processed.

- Test cases are fully described in JSON files and can be extended without modifying the C++ source code.

- Binary UART commands are stored as external `.bin` files. The file name is specified in the `"binary"` field of the test case, while the search directory is configured in `config.json` using `"binary_cmd_path"`.

- When `"protocol_ID": 2` is used, the framework can automatically decode C structures received from the target. Structure descriptions are extracted from:
  - IAR map file (`*.map`);
  - ELF file (`*.out`, `*.elf`);
  - JSON files generated by the Python parsers located in `./python/iar_parser`.

- The files inside `./iar_files/` are generated automatically by:
  - `map_parser.py` – parses IAR map files and extracts symbols, addresses, and sizes;
  - `elf_parser.py` – parses ELF files and exports C structure layouts.

- The framework supports:
  - nested C structures;
  - packed structures;
  - arrays with element exclusion;
  - validation using comparison operators (`==`, `!=`, `>`, `<`, `>=`, `<=`);
  - structure field access using dot notation (for example: `"uart.errorLastOperation"`).

- The application generates:
  - console output;
  - detailed log files in `./logs`;
  - an HTML report summarizing the entire test session.

- The framework is a **console application** (`QT += serialport`, `CONFIG += console`) and does not require GUI components at runtime.

- The project was developed and tested on **Windows 10/11** using:
  - Qt 5.15.2;
  - MinGW 8.1.0 (64-bit);
  - GoogleTest;
  - qmake build system.

- Before running the application, make sure:
  - the UART device is connected;
  - the selected COM port exists and is not occupied by another application;
  - paths in `config.json` point to valid files;
  - the required binary command files exist in the configured `binary_cmd_path`.

- After compilation on Windows, it is recommended to run `windeployqt` to copy all required Qt runtime libraries and plugins to the executable directory.

- The framework can be easily extended with additional UART protocols, new validation rules, custom report generators, and support for other embedded compilers besides IAR.

---