diff --git a/README.md b/README.md new file mode 100644 index 0000000..f01c84c --- /dev/null +++ b/README.md @@ -0,0 +1,665 @@ + +# Embedded System Test Automation Framework + +This is a C++ application designed for testing embedded systems using UART communication. The system allows you to load and execute test cases that interact with the UART interface according to predefined protocols. + +## Table of Contents + +- [Overview](#overview) +- [Key Features](#key-features) +- [Supported Embedded Toolchains](#supported-embedded-toolchains) +- [Project Structure](#project-structure) +- [Requirements](#requirements) + - [Development Environment](#development-environment) + - [Build and Deployment](#build-and-deployment) +- [Configuration](#configuration) + - [Test Stand Settings](#test-stand-settings) + - [Test Plan](#test-plan) +- [Running the Test Stand](#running-the-test-stand) +- [Test Case Syntax](#test-case-syntax-json) + - [Processing Text Data Received via UART](#processing-text-data-received-via-uart) + - [Processing Binary Data Received via UART](#processing-binary-data-received-via-uart) +- [Reporting](#reporting) +- [Notes (summary)](#notes) + +## Overview + +Embedded Systems Test Automation Framework is a console-based C++ application for automated testing of embedded systems through UART communication. + +The framework supports: + +* execution of predefined test plans; +* transmission of binary commands to the target device; +* reception and validation of UART responses (if text format) or parsing of binary C structures directly from the target memory; +* generation of text logs and HTML reports; + +The framework is configured entirely through JSON files and can be easily integrated into automated verification environments. + +## Key Features + +- **Binary Command Support**: Send raw `.bin` payloads directly to the microcontroller. +- **UART I/O Communication**: UART I/O for interacting with microcontrollers. +- **Sequence Validation in text format**: Validates incoming messages by matching specific patterns: + - **Start Message**: Initial handshake/trigger. + - **Sequence**: Ordered sequence of expected intermediate messages. + - **End Message**: Completion criteria for the test case. +- **JSON-Driven Configuration**: Easily define test cases, UART parameters, and test plans in JSON format. +- **Flexible Execution**: + - **Grouping**: Organize tests into suites (e.g., --group=smoke, --group=full). + - **Repeats**: Built-in support for stress testing by repeating cases (using --repeat=N). +- **Advanced Logging**: Generates detailed text logs and visual HTML reports for every run. + +## Supported Embedded Toolchains + +The framework itself is compiler-independent. + +However, automatic binary structure parsing is currently implemented for firmware generated by: + +* IAR Embedded Workbench +* ELF/DWARF compatible toolchains + +The parser extracts: + +* variable addresses; +* structure sizes; +* nested structures; +* array dimensions; +* primitive data types. + +The generated metadata are stored as JSON files inside: + +```text +./iar_files/ +``` + + +## Project Structure + +```text +├── cmd +│   ├── cmd_abort.bin +│   ├── set_interface_params.bin +│   ├── set_settings_NAND_micron_MT29F16G08AJADAWP.bin +├── compiler +│   ├── Makefile +│   ├── Makefile.Debug +│   ├── Makefile.Release +│   ├── data.bin +├── config.json +├── configs +│   ├── cmd +│   ├── test_cases +│   │   ├── test_UART_cmdSetDataInterface.json +│   │   ├── test_abort.json +│   │   ├── test_readSnapshot_HK.json +│   │   ├── test_read_NANDctrlOper.json +│   │   ├── test_set_micron_MT29F16G08AJADAWP.json +│   └── test_plan.json +├── iar_files +│   ├── ENTRY_LIST_map.json +│   ├── NANDctrlOper_t.json +│   └── Snapshot_HK_t.json +├── logs +│   ├── 12_05_2026_15_48_16.log +│   ├── report.html +│   ├── test_cases.log +│   └── test_stand.log +├── python +│   └── iar_parser +│   ├── elf_parser.py +│   ├── map_parser.py +│   ├── requirements.txt +├── reports +├── src +│   ├── core +│   │   ├── cli_parser.cpp +│   │   ├── cli_parser.h +... +│   │   ├── uart.cpp +│   │   └── uart.h +│   ├── embedded_test_stand.pro +│   ├── embedded_test_stand.pro.user +│   ├── gtest_dependency.pri +│   ├── main.cpp +│   └── tests +│   ├── test_runner.cpp +│   ├── uart_fixture.cpp +│   └── uart_fixture.h +└── src.rar +``` + +## Requirements + +### Development Environment +The project was developed and tested on Windows 10/11 using the following software: + +| Component | Version | +|----------|---------| +| Qt | 5.15.2 | +| Qt Creator | 9.x or newer | +| Compiler | MinGW 8.1.0 (64-bit) | +| Debugger | `gdb.exe` | +| Build system | qmake | +| qmake | Qt 5.15.2 | +| mingw32-make | MinGW 8.1.0 | +| Python | 3.9+ | +| GoogleTest | Latest stable version | + +#### Qt Modules + +The following Qt modules are required: + +* QtCore +* QtSerialPort + +#### Python Dependencies + +Python scripts located in: + +```text +python/iar_parser/ +``` + +To install the required dependencies, run the following command: +```bash +pip install -r requirements.txt +``` + +#### GoogleTest + +The project uses GoogleTest sources directly: + +```text +GOOGLETEST_DIR=C:\path\to\googletest +``` + +### Build and Deployment + +The project is built using **qmake**. + +Generate Makefiles: + +```bash +qmake embedded_test_stand.pro +``` + +Build: + +```bash +mingw32-make -j8 +``` + +Clean: + +```bash +mingw32-make clean -j8 +``` + +The build directory used during development: + +```text +./compiler +``` + +After successful compilation, Qt runtime libraries should be deployed using: + +```bash +windeployqt embedded_test_stand.exe +``` + +In Qt Creator the following deployment step is used: + +```text +Command: +windeployqt + +Arguments: +%{buildDir}/release/embedded_test_stand.exe + +Working directory: +%{buildDir}/compiler +``` + +This command automatically copies: + +* QtCore.dll +* Qt5SerialPort.dll +* platform plugins +* required MinGW runtime libraries +* other Qt dependencies + +to the executable directory. + + +## Configuration + +### Test Stand Settings +See `./config.json` file: +```json +{ + "UART": { + "RS485_address": 1, + "COM_port": "COM14", + "COM_baudRate": 921600, + "COM_bits": 8, + "COM_parity": 3, + "COM_stopBits": 1, + "COM_flowControl": false + }, + "Path": { + "MK_elf_file": "C:/Danila/work/sputnik_test/src/sputnik/Debug/c.out", + "MK_map_file": "C:/Danila/work/sputnik_test/src/sputnik/Debug/sputnik.map", + "test_plan_path": "C:/Danila/work/embedded_test_stand/configs/test_plan.json", + "test_cases_path": "C:/Danila/work/embedded_test_stand/configs/test_cases/", + "test_log_path": "C:/Danila/work/embedded_test_stand/logs/", + "stand_report_html_path": "C:/Danila/work/embedded_test_stand/logs/report.html", + "map_parser": "C:/Danila/work/embedded_test_stand/python/iar_parser/map_parser.py", + "elf_parser": "C:/Danila/work/embedded_test_stand/python/iar_parser/elf_parser.py", + "binary_cmd_path": "C:/Danila/work/embedded_test_stand/cmd/" + } +} +``` +- **`UART`** - Define UART hardware parameters +- **`Path`** - Define path settings + - **`MK_elf_file`** - the result of compiling the microcontroller software in the form of en elf-file (needed if using `"protocol_ID": 2` in `test case json file`) + - **`MK_map_file`** - the result of compiling the microcontroller software in the form of map-file + - **`test_plan_path`** - the path with test plan json file + - **`test_cases_path`** - the path with test cases json files + - **`test_log_path`** - the path to which log files will be written + - **`stand_report_html_path`** - the path and filename to which html-report will be written + - **`map_parser`** - the path and filename where python-file is located. This python-script parse map-file (from compiler for microcontroller software) + - **`elf_parser`** - the path and filename where python-file is located. This python-script parse elf-file (from compiler for microcontroller software) + - **`binary_cmd_path`** - the path with binary data will be transmitted to microcontroller through UART + + +### Test Plan +See `./configs/test_plan.json` file. This file define test groups and their associated test cases, for example: +```json +{ + "groups": { + "smoke": [ + "test_UART_cmdSetDataInterface.json", + "test_readSnapshot_HK.json", + "#test_erase_all.json" + ], + + "full": [ + "### set NAND settings ###", + "test_UART_cmdSetDataInterface.json", + "test_start_March_FTE_by_blocks.json" + ] + } +} +``` +- **`"smoke", "full"`** - user defined groups containing sets of test cases +- **`"#test_abort.json"`** - ignored test +- **`"test_readSnapshot_HK.json"`** - example of user test case +- **`"### set NAND settings ###"`** - comments + +## Running the Test Stand +Before starting the application make sure that: + +* the target embedded device is powered on; +* the UART interface is connected correctly; +* the COM port specified in `config.json` exists and is not used by another application; +* all configuration files and binary command files are present; +* ELF and MAP files of the embedded firmware are available if binary structure parsing is used (`protocol_ID = 2`). + + +Execute the built binary with command-line arguments: +```bash +./embedded_test_stand.exe --delay 10 --group full +``` +or +```bash +./embedded_test_stand.exe --delay 10 --group smoke --case test_UART_cmdSetDataInterface +``` +**Available Flags**: +- `--group `: Specify a test group (e.g., `smoke`, `full`). +- `--case `: Run a specific test case (overrides group). +- `--repeat `: Repeat the test case `N` times. +- `--delay`: delay `N` msec between each test cases + +expects the following files to be present: + +```text +config.json +configs/ +├── test_plan.json +└── test_cases/ + ├── test_readSnapshot_HK.json + └── ... +``` + +The application loads: + +* global settings from `config.json`; +* a list of test cases from `test_plan.json`; +* individual test descriptions from `configs/test_cases/*.json`; +* UART command binaries specified in the `"binary"` field of each test case. + +After startup, the application: + +1. Opens the UART port. +2. Sends binary commands to the embedded target. +3. Receives UART responses. +4. Parses received binary structures or text. +5. Validates structure fields according to the `struct_check` rules. +6. Generates console and HTML test reports. + +--- + +## Test Case Syntax +The general view of the test case depends on the type of processed data, received via UART: +- binary data (see below `"protocol_ID": 2`) +- text data (see below `"protocol_ID": 1`) + +### Processing text data, received via UART +Each test case file (for example, `test_start_March_FTE_by_blocks.json`) defines the expected UART sequence: +```json +{ + "meta": { + "name": "start_March-FTE_by_blocks", + "description": "start March-FTE by specified blocks", + "parameters": { + "_comment": "protocol_ID = 1 for text format from UART, see protocol_ID_en", + "protocol_ID": 1, + "version": 1, + "cmd_from_binary_file": true, + "cmd_code": 21, + "mode_March_FTE": 2, + "begin_block": 20, + "end_block": 25 + } + }, + + "binary": "start_March-FTE_by_blocks_20_25.bin", + + "expectations": { + "start": { + "type": "json", + "fields": { + "msgType": 2, + "cmd": 21 + } + }, + + "sequence": [ + { + "type": "regex", + "value": "start march-FTE in block \\d+(, LUN 0, Target 0)?" + } + ], + + "end": { + "type": "contains", + "value": "NAND Flash test stand ready to receive new commands" + } + }, + + "timeout_msec": 35000 +} +``` +- **`"timeout_msec"`** - UART response timeout. Messages may arrive in parts over a specified time interval. After the timeout expires, data reception stops, and the message parts are combined into a final string as they arrive, which is then analyzed in its entirely +- **`"meta"`**: `"parameters"` - user defined test case parameters + - `"protocol_ID"` - defines the type of data received from the UART that needs to be processed + - `"version"` - reserved for future realization + - `"cmd_from_binary_file"` - always `true`, because command data file for transmit via UART is taken form the binary file now, see `"binary"` field below + - `"cmd_code"` - command code (user information) \ + **Additionally, the user can add other fields here in order to distinguish test cases** +- **`"binary"`** - The binary data file name, whose contents will be transmitted via UART. Corresponding file path is taken from `"binary_cmd_path"` field, see `./config.json` +- **`"expectations"`** - Validates incoming messages by matching specific patterns \ + - **`"start"`** Find the initial specified substring at the beginning of the message from UART. + - **`"sequence"`** Find ordered specified sequence of expected intermediate messages from UART. + - **`"end"`** Find specified terminate substring of massage from UART. + - *`"type"`** - specifies the type of message to compare. Possible meanings: `"contains"`, `"regex"`, `"json"` \ + `"contains"` - find the substring, specified in the "value" field in the entire message received via UART \ + `"json"` - find the substring in json-format, specified in the "value" field in the entire message received via UART \ + `"regex"` - find the substring by using regular expression, specified in "value" field in the entire message received via UART + + +An example message from UART for passing this test case: +``` +{"msgType": 2,"cmd": 21} + + NAND Flash manufacture: MICRON + block Endurance (when using internal ECC): 100000 cycles +************************************************************** + + start test March-FTE in NAND Flash + algorithm test: + 1. ERASE BLOCK - erasing one block + 2. READ(~D)n - read all pages of one block 3 times to check for data equality to the 0xFF pattern, incrementing address + 3. PROGRAM(D) - programming pages of one block with pattern data, incrementing address + 4. READ(D)n - read all pages of one block 3 times to check for data equality to written pattern data, incrementing address + 5. READ(D) - repeat read all pages of one block, incrementing address + 6. ERASE BLOCK - erasing one block + 7. READ(~D)n - read all pages of one block 3 times to check for data equality to the 0xFF pattern, incrementing address + 8. PROGRAM(D) - programming pages of one block with pattern data, DECREMENTING address + 9. READ(D)n - read all pages of one block 3 times to check for data equality to written pattern data, DECREMENTING address + 10. READ(D) - repeat read all pages of one block, DECREMENTING address + + March-FTE test will check the following blocks: from: 20 to: 25, LUN: 0, target: 0 + ************************************************************** +start march-FTE in block 20, LUN 0, Target 0 +start march-FTE in block 24, LUN 0, Target 0 +TEST FINISH! +NAND Flash test stand ready to receive new commands +``` + +### Processing binary data, received via UART +Each test case file (for example, `test_readSnapshot_HK.json`) defines the expected structure, received via UART: +```json +{ + "meta": { + "name": "read_Snapshot_HK", + "description": "read Snapshot_HK structure", + "parameters": { + "_comment": "protocol_ID = 2 for binary format data from UART, reading C structure from memory, see protocol_ID_en", + "protocol_ID": 2, + "version": 1, + "cmd_code": 27, + "struct_name": "Snapshot_HK", + "struct_type": "Snapshot_HK_t", + "cmd_from_binary_file": true, + "addr_user_set": true, + "address": "0x08125600", + "size_user_set": true, + "size_bytes": 222 + } + }, + + "binary": "data.bin", + + "struct_check" : [ { + "field": "size", + "type": "uint16", + "op": "==", + "expect": 222 + }, { + "field": "counter", + "type": "uint32", + "op": ">", + "expect": 0 + }, { + "field": "uart.errorLastOperation", + "type": "uint8", + "op": "==", + "expect": 0 + }, { + "field": "rmap.error", + "array": true, + "type": "uint16", + "op": "==", + "exclude": [0, 22], + "expect": 0 + } ], + + "timeout_msec": 100 +} +``` +- **`"timeout_msec"`** - UART response timeout. Messages may arrive in parts over a specified time interval. After the timeout expires, data reception stops, and the message parts are combined into a final string as they arrive, which is then analyzed in its entirely +- **`"meta"`**: `"parameters"` - user defined test case parameters + - `"protocol_ID"` - defines the type of data received from the UART that needs to be processed + - `"version"` - reserved for future realization + - `"cmd_code"` - command code (user information) \ + - `"struct_name"` - a variable having the type of the structure being checked as in the source embedded code (for example: "Snapshot_HK") + - `"struct_type"`: - the name of the type of the structure being checked as in the source embedded code (for example: "Snapshot_HK_t") + - `"cmd_from_binary_file"` - always `true`, because command data file for transmit via UART is taken form the binary file now, see `"binary"` field below + - `"addr_user_set"` - if `true`, means checked variable address is taken from "address" field (see below). If `false` means checked variable address is taken from parsed compiler map-file (by using python-script) from `./iar_files/ENTRY_LIST_map.json` + - `"address"` - checked variable address in hex format, for example: "0x08125600" + - `"size_user_set"` - if `true`, means checked structure size is taken from "size_bytes" filed (see below). If `false` means checked structure size is taken from parsed compiler map-file (by using python-script) from `./iar_files/ENTRY_LIST_map.json` + - `"size_bytes"` - specify checked structure size in bytes + **Additionally, the user can add other fields here in order to distinguish test cases** +- **`"binary"`** - The binary data file name, whose contents will be transmitted via UART. Corresponding file path is taken from `"binary_cmd_path"` field, see `./config.json` +- **`"struct_check"`** - this section defines a set of validation rules applied to the binary structure received from the target device via UART. +Each entry describes: + + * **`field`** – structure field name. Nested fields are supported using dot notation, for example: + + * `"uart.errorLastOperation"` + * `"rmap.error"` + + * **`type`** – expected C data type of the field: + + * `uint8` + * `uint16` + * `uint32` + * `uint64` + * `int8` + * `int16` + * `int32` + * `int64` + * `float` + * `double` + + * **`op`** – comparison operator used for validation: + + * `"=="` equal to + * `"!="` not equal to + * `">"` greater than + * `">="` greater than or equal to + * `"<"` less than + * `"<="` less than or equal to + + * **`expect`** – expected value used in comparison. + + * **`array`** *(optional)* – if `true`, the specified field is treated as an array and every element is checked against the condition. + + * **`exclude`** *(optional)* – list of array indices excluded from validation. This is useful when some array elements are reserved, contain implementation-specific values, or are expected to change dynamically. + +For example: + +```json +{ + "field": "rmap.error", + "array": true, + "type": "uint16", + "op": "==", + "exclude": [0, 22], + "expect": 0 +} +``` + +This rule means: + +* the field `rmap.error` is an array of `uint16`; +* all array elements must be equal to `0`; +* elements with indices `0` and `22` are skipped during validation. + +The validation result for each rule is reported independently. If at least one check fails, the test case is considered failed and a detailed diagnostic message is printed, including: + +* field name; +* actual value; +* expected value; +* comparison operator; +* array index (for array checks). + +The test-case checks the following structure received via UART: +``` +typedef __packed struct +{ + uint8_t errorLastOperation; //RMAP_errFlag_en + uint16_t error[COUNT_ERR_MAX_ELEMENTS_RMAP]; +} RMAP_HK_t; + +typedef __packed struct +{ + volatile uint8_t errorLastOperation; // UART_operErrCode_enum + volatile uint16_t error[COUNT_ERR_MAX_ELEMENTS]; +} UART_HK_t; + +extern UART_HK_t UART_HK; + +typedef __packed struct +{ + uint16_t size; + uint32_t counter; + UART_HK_t uart; + RMAP_HK_t rmap; +} Snapshot_HK_t; +``` + +--- + +## Reporting + +- **`./logs/test_stand.log`**: contains a report of errors from the Embedded System Test Automation Framework +- **`./logs/test_cases.log`**: contains a report with detailed execution steps. +- **`./logs/report.html`**: contains a summary report. It is created after execution and includes: + - Test case pass/fail status. + - Timestamps and error messages. + - UART communication statistics. + +--- + +## Notes (summary) +- The application uses asynchronous UART communication for real-time testing. + +- Incoming data from target may arrive in multiple fragments. The framework accumulates all received packets until the timeout expires, specified in the test case by `"timeout_msec"` +After that, the complete UART message is assembled and processed. + +- Test cases are fully described in JSON files and can be extended without modifying the C++ source code. + +- Binary UART commands are stored as external `.bin` files. The file name is specified in the `"binary"` field of the test case, while the search directory is configured in `config.json` using `"binary_cmd_path"`. + +- When `"protocol_ID": 2` is used, the framework can automatically decode C structures received from the target. Structure descriptions are extracted from: + - IAR map file (`*.map`); + - ELF file (`*.out`, `*.elf`); + - JSON files generated by the Python parsers located in `./python/iar_parser`. + +- The files inside `./iar_files/` are generated automatically by: + - `map_parser.py` – parses IAR map files and extracts symbols, addresses, and sizes; + - `elf_parser.py` – parses ELF files and exports C structure layouts. + +- The framework supports: + - nested C structures; + - packed structures; + - arrays with element exclusion; + - validation using comparison operators (`==`, `!=`, `>`, `<`, `>=`, `<=`); + - structure field access using dot notation (for example: `"uart.errorLastOperation"`). + +- The application generates: + - console output; + - detailed log files in `./logs`; + - an HTML report summarizing the entire test session. + +- The framework is a **console application** (`QT += serialport`, `CONFIG += console`) and does not require GUI components at runtime. + +- The project was developed and tested on **Windows 10/11** using: + - Qt 5.15.2; + - MinGW 8.1.0 (64-bit); + - GoogleTest; + - qmake build system. + +- Before running the application, make sure: + - the UART device is connected; + - the selected COM port exists and is not occupied by another application; + - paths in `config.json` point to valid files; + - the required binary command files exist in the configured `binary_cmd_path`. + +- After compilation on Windows, it is recommended to run `windeployqt` to copy all required Qt runtime libraries and plugins to the executable directory. + +- The framework can be easily extended with additional UART protocols, new validation rules, custom report generators, and support for other embedded compilers besides IAR. + +--- \ No newline at end of file diff --git a/exe_windows_10_11/Qt5Core.dll b/exe_windows_10_11/Qt5Core.dll new file mode 100644 index 0000000..83e4717 Binary files /dev/null and b/exe_windows_10_11/Qt5Core.dll differ diff --git a/exe_windows_10_11/Qt5SerialPort.dll b/exe_windows_10_11/Qt5SerialPort.dll new file mode 100644 index 0000000..f016b4a Binary files /dev/null and b/exe_windows_10_11/Qt5SerialPort.dll differ diff --git a/exe_windows_10_11/libgcc_s_seh-1.dll b/exe_windows_10_11/libgcc_s_seh-1.dll new file mode 100644 index 0000000..abd357d Binary files /dev/null and b/exe_windows_10_11/libgcc_s_seh-1.dll differ diff --git a/exe_windows_10_11/libstdc++-6.dll b/exe_windows_10_11/libstdc++-6.dll new file mode 100644 index 0000000..a7dc1e2 Binary files /dev/null and b/exe_windows_10_11/libstdc++-6.dll differ diff --git a/exe_windows_10_11/libwinpthread-1.dll b/exe_windows_10_11/libwinpthread-1.dll new file mode 100644 index 0000000..500de9d Binary files /dev/null and b/exe_windows_10_11/libwinpthread-1.dll differ