gamkov/Embedded-System-Test-Automation-Framework

Files

Danila 1534d825a4 add readme and exe files for windows

2026-06-17 15:04:11 +03:00

23 KiB

Raw Blame History

Embedded System Test Automation Framework

This is a C++ application designed for testing embedded systems using UART communication. The system allows you to load and execute test cases that interact with the UART interface according to predefined protocols.

Overview
Key Features
Supported Embedded Toolchains
Project Structure
Requirements
- Development Environment
- Build and Deployment
Configuration
- Test Stand Settings
- Test Plan
Running the Test Stand
Test Case Syntax
- Processing Text Data Received via UART
- Processing Binary Data Received via UART
Reporting
Notes (summary)

Overview

Embedded Systems Test Automation Framework is a console-based C++ application for automated testing of embedded systems through UART communication.

The framework supports:

execution of predefined test plans;
transmission of binary commands to the target device;
reception and validation of UART responses (if text format) or parsing of binary C structures directly from the target memory;
generation of text logs and HTML reports;

The framework is configured entirely through JSON files and can be easily integrated into automated verification environments.

Key Features

Binary Command Support: Send raw .bin payloads directly to the microcontroller.
UART I/O Communication: UART I/O for interacting with microcontrollers.
Sequence Validation in text format: Validates incoming messages by matching specific patterns:
- Start Message: Initial handshake/trigger.
- Sequence: Ordered sequence of expected intermediate messages.
- End Message: Completion criteria for the test case.
JSON-Driven Configuration: Easily define test cases, UART parameters, and test plans in JSON format.
Flexible Execution:
- Grouping: Organize tests into suites (e.g., --group=smoke, --group=full).
- Repeats: Built-in support for stress testing by repeating cases (using --repeat=N).
Advanced Logging: Generates detailed text logs and visual HTML reports for every run.

Supported Embedded Toolchains

The framework itself is compiler-independent.

However, automatic binary structure parsing is currently implemented for firmware generated by:

IAR Embedded Workbench
ELF/DWARF compatible toolchains

The parser extracts:

variable addresses;
structure sizes;
nested structures;
array dimensions;
primitive data types.

The generated metadata are stored as JSON files inside:

./iar_files/

Project Structure

├── cmd
│   ├── cmd_abort.bin
│   ├── set_interface_params.bin
│   ├── set_settings_NAND_micron_MT29F16G08AJADAWP.bin
├── compiler
│   ├── Makefile
│   ├── Makefile.Debug
│   ├── Makefile.Release
│   ├── data.bin
├── config.json
├── configs
│   ├── cmd
│   ├── test_cases
│   │   ├── test_UART_cmdSetDataInterface.json
│   │   ├── test_abort.json
│   │   ├── test_readSnapshot_HK.json
│   │   ├── test_read_NANDctrlOper.json
│   │   ├── test_set_micron_MT29F16G08AJADAWP.json
│   └── test_plan.json
├── iar_files
│   ├── ENTRY_LIST_map.json
│   ├── NANDctrlOper_t.json
│   └── Snapshot_HK_t.json
├── logs
│   ├── 12_05_2026_15_48_16.log
│   ├── report.html
│   ├── test_cases.log
│   └── test_stand.log
├── python
│   └── iar_parser
│       ├── elf_parser.py
│       ├── map_parser.py
│       ├── requirements.txt
├── reports
├── src
│   ├── core
│   │   ├── cli_parser.cpp
│   │   ├── cli_parser.h
...
│   │   ├── uart.cpp
│   │   └── uart.h
│   ├── embedded_test_stand.pro
│   ├── embedded_test_stand.pro.user
│   ├── gtest_dependency.pri
│   ├── main.cpp
│   └── tests
│       ├── test_runner.cpp
│       ├── uart_fixture.cpp
│       └── uart_fixture.h
└── src.rar

Requirements

Development Environment

The project was developed and tested on Windows 10/11 using the following software:

Component	Version
Qt	5.15.2
Qt Creator	9.x or newer
Compiler	MinGW 8.1.0 (64-bit)
Debugger	`gdb.exe`
Build system	qmake
qmake	Qt 5.15.2
mingw32-make	MinGW 8.1.0
Python	3.9+
GoogleTest	Latest stable version

Qt Modules

The following Qt modules are required:

QtCore
QtSerialPort

Python Dependencies

Python scripts located in:

python/iar_parser/

To install the required dependencies, run the following command:

pip install -r requirements.txt

GoogleTest

The project uses GoogleTest sources directly:

GOOGLETEST_DIR=C:\path\to\googletest

Build and Deployment

The project is built using qmake.

Generate Makefiles:

qmake embedded_test_stand.pro

Build:

mingw32-make -j8

Clean:

mingw32-make clean -j8

The build directory used during development:

./compiler

After successful compilation, Qt runtime libraries should be deployed using:

windeployqt embedded_test_stand.exe

In Qt Creator the following deployment step is used:

Command:
windeployqt

Arguments:
%{buildDir}/release/embedded_test_stand.exe

Working directory:
%{buildDir}/compiler

This command automatically copies:

QtCore.dll
Qt5SerialPort.dll
platform plugins
required MinGW runtime libraries
other Qt dependencies

to the executable directory.

Configuration

Test Stand Settings

See ./config.json file:

{
  "UART": {
    "RS485_address": 1, 
    "COM_port": "COM14",
    "COM_baudRate": 921600,
    "COM_bits": 8,
    "COM_parity": 3,
    "COM_stopBits": 1,
    "COM_flowControl": false
  },
  "Path": {
    "MK_elf_file": "C:/Danila/work/sputnik_test/src/sputnik/Debug/c.out",
    "MK_map_file": "C:/Danila/work/sputnik_test/src/sputnik/Debug/sputnik.map",
    "test_plan_path": "C:/Danila/work/embedded_test_stand/configs/test_plan.json",
    "test_cases_path": "C:/Danila/work/embedded_test_stand/configs/test_cases/",
    "test_log_path": "C:/Danila/work/embedded_test_stand/logs/",
    "stand_report_html_path": "C:/Danila/work/embedded_test_stand/logs/report.html",
    "map_parser": "C:/Danila/work/embedded_test_stand/python/iar_parser/map_parser.py",
    "elf_parser": "C:/Danila/work/embedded_test_stand/python/iar_parser/elf_parser.py",
    "binary_cmd_path": "C:/Danila/work/embedded_test_stand/cmd/"
  }
}

UART - Define UART hardware parameters
Path - Define path settings
- MK_elf_file - the result of compiling the microcontroller software in the form of en elf-file (needed if using "protocol_ID": 2 in test case json file)
- MK_map_file - the result of compiling the microcontroller software in the form of map-file
- test_plan_path - the path with test plan json file
- test_cases_path - the path with test cases json files
- test_log_path - the path to which log files will be written
- stand_report_html_path - the path and filename to which html-report will be written
- map_parser - the path and filename where python-file is located. This python-script parse map-file (from compiler for microcontroller software)
- elf_parser - the path and filename where python-file is located. This python-script parse elf-file (from compiler for microcontroller software)
- binary_cmd_path - the path with binary data will be transmitted to microcontroller through UART

Test Plan

See ./configs/test_plan.json file. This file define test groups and their associated test cases, for example:

{
  "groups": {
    "smoke": [
      "test_UART_cmdSetDataInterface.json",
      "test_readSnapshot_HK.json",
      "#test_erase_all.json"
    ],

    "full": [
      "### set NAND settings ###",
      "test_UART_cmdSetDataInterface.json",
      "test_start_March_FTE_by_blocks.json"
    ]
  }
}

"smoke", "full" - user defined groups containing sets of test cases
"#test_abort.json" - ignored test
"test_readSnapshot_HK.json" - example of user test case
"### set NAND settings ###" - comments

Running the Test Stand

Before starting the application make sure that:

the target embedded device is powered on;
the UART interface is connected correctly;
the COM port specified in config.json exists and is not used by another application;
all configuration files and binary command files are present;
ELF and MAP files of the embedded firmware are available if binary structure parsing is used (protocol_ID = 2).

Execute the built binary with command-line arguments:

./embedded_test_stand.exe --delay 10 --group full

./embedded_test_stand.exe --delay 10 --group smoke --case test_UART_cmdSetDataInterface

Available Flags:

--group <GROUP_NAME>: Specify a test group (e.g., smoke, full).
--case <TEST_NAME>: Run a specific test case (overrides group).
--repeat <N>: Repeat the test case N times.
--delay: delay N msec between each test cases

expects the following files to be present:

config.json
configs/
├── test_plan.json
└── test_cases/
    ├── test_readSnapshot_HK.json
    └── ...

The application loads:

global settings from config.json;
a list of test cases from test_plan.json;
individual test descriptions from configs/test_cases/*.json;
UART command binaries specified in the "binary" field of each test case.

After startup, the application:

Opens the UART port.
Sends binary commands to the embedded target.
Receives UART responses.
Parses received binary structures or text.
Validates structure fields according to the struct_check rules.
Generates console and HTML test reports.

Test Case Syntax

The general view of the test case depends on the type of processed data, received via UART:

binary data (see below "protocol_ID": 2)
text data (see below "protocol_ID": 1)

Processing text data, received via UART

Each test case file (for example, test_start_March_FTE_by_blocks.json) defines the expected UART sequence:

{
  "meta": {
    "name": "start_March-FTE_by_blocks",
    "description": "start March-FTE by specified blocks",
    "parameters": {
      "_comment": "protocol_ID = 1 for text format from UART, see protocol_ID_en",
      "protocol_ID": 1,
      "version": 1,
      "cmd_from_binary_file": true,
      "cmd_code": 21,
      "mode_March_FTE": 2,
      "begin_block": 20,
      "end_block": 25
    }
  },

  "binary": "start_March-FTE_by_blocks_20_25.bin",

  "expectations": {
    "start": {
      "type": "json",
      "fields": {
        "msgType": 2,
        "cmd": 21
      }
    },

    "sequence": [
      {
        "type": "regex",
        "value": "start march-FTE in block \\d+(, LUN 0, Target 0)?"
      }
    ],

    "end": {
      "type": "contains",
      "value": "NAND Flash test stand ready to receive new commands"
    }
  },

  "timeout_msec": 35000
}

"timeout_msec" - UART response timeout. Messages may arrive in parts over a specified time interval. After the timeout expires, data reception stops, and the message parts are combined into a final string as they arrive, which is then analyzed in its entirely
"meta": "parameters" - user defined test case parameters
- "protocol_ID" - defines the type of data received from the UART that needs to be processed
- "version" - reserved for future realization
- "cmd_from_binary_file" - always true, because command data file for transmit via UART is taken form the binary file now, see "binary" field below
- "cmd_code" - command code (user information)
  Additionally, the user can add other fields here in order to distinguish test cases
"binary" - The binary data file name, whose contents will be transmitted via UART. Corresponding file path is taken from "binary_cmd_path" field, see ./config.json
"expectations" - Validates incoming messages by matching specific patterns \
- "start" Find the initial specified substring at the beginning of the message from UART.
- "sequence" Find ordered specified sequence of expected intermediate messages from UART.
- "end" Find specified terminate substring of massage from UART.
  - "type"* - specifies the type of message to compare. Possible meanings: "contains", "regex", "json"
    "contains" - find the substring, specified in the "value" field in the entire message received via UART
    "json" - find the substring in json-format, specified in the "value" field in the entire message received via UART
    "regex" - find the substring by using regular expression, specified in "value" field in the entire message received via UART

An example message from UART for passing this test case:

{"msgType": 2,"cmd": 21}

 NAND Flash manufacture: MICRON
 block Endurance (when using internal ECC): 100000 cycles
**************************************************************

 start test March-FTE in NAND Flash 
 algorithm test:
 1. ERASE BLOCK - erasing one block
 2. READ(~D)n - read all pages of one block 3 times to check for data equality to the 0xFF pattern, incrementing address
 3. PROGRAM(D) - programming pages of one block with pattern data, incrementing address
 4. READ(D)n - read all pages of one block 3 times to check for data equality to written pattern data, incrementing address
 5. READ(D) - repeat read all pages of one block, incrementing address
 6. ERASE BLOCK - erasing one block
 7. READ(~D)n - read all pages of one block 3 times to check for data equality to the 0xFF pattern, incrementing address
 8. PROGRAM(D) - programming pages of one block with pattern data, DECREMENTING address
 9. READ(D)n - read all pages of one block 3 times to check for data equality to written pattern data, DECREMENTING address
 10. READ(D) - repeat read all pages of one block, DECREMENTING address

 March-FTE test will check the following blocks: from: 20 to: 25, LUN: 0, target: 0
 **************************************************************
start march-FTE in block 20, LUN 0, Target 0
start march-FTE in block 24, LUN 0, Target 0
TEST FINISH!
NAND Flash test stand ready to receive new commands

Processing binary data, received via UART

Each test case file (for example, test_readSnapshot_HK.json) defines the expected structure, received via UART:

{
  "meta": {
    "name": "read_Snapshot_HK",
    "description": "read Snapshot_HK structure",
    "parameters": {
      "_comment": "protocol_ID = 2 for binary format data from UART, reading C structure from memory, see protocol_ID_en",
      "protocol_ID": 2,
      "version": 1,
      "cmd_code": 27,
      "struct_name": "Snapshot_HK",
      "struct_type": "Snapshot_HK_t",
      "cmd_from_binary_file": true,
      "addr_user_set": true,
      "address": "0x08125600",
      "size_user_set": true,
      "size_bytes": 222
    }
  },

  "binary": "data.bin",
  
  "struct_check" : [ {
        "field": "size",
		"type": "uint16",
		"op": "==",
		"expect": 222
    }, {
        "field": "counter",
		"type": "uint32",
		"op": ">",
		"expect": 0
    }, {
        "field": "uart.errorLastOperation",
		"type": "uint8",
		"op": "==",
		"expect": 0
    }, {
        "field": "rmap.error",
		"array": true,
		"type": "uint16",
		"op": "==",
		"exclude": [0, 22],
		"expect": 0
    } ],

  "timeout_msec": 100
}

"timeout_msec" - UART response timeout. Messages may arrive in parts over a specified time interval. After the timeout expires, data reception stops, and the message parts are combined into a final string as they arrive, which is then analyzed in its entirely
"meta": "parameters" - user defined test case parameters
- "protocol_ID" - defines the type of data received from the UART that needs to be processed
- "version" - reserved for future realization
- "cmd_code" - command code (user information) \
- "struct_name" - a variable having the type of the structure being checked as in the source embedded code (for example: "Snapshot_HK")
- "struct_type": - the name of the type of the structure being checked as in the source embedded code (for example: "Snapshot_HK_t")
- "cmd_from_binary_file" - always true, because command data file for transmit via UART is taken form the binary file now, see "binary" field below
- "addr_user_set" - if true, means checked variable address is taken from "address" field (see below). If false means checked variable address is taken from parsed compiler map-file (by using python-script) from ./iar_files/ENTRY_LIST_map.json
- "address" - checked variable address in hex format, for example: "0x08125600"
- "size_user_set" - if true, means checked structure size is taken from "size_bytes" filed (see below). If false means checked structure size is taken from parsed compiler map-file (by using python-script) from ./iar_files/ENTRY_LIST_map.json
- "size_bytes" - specify checked structure size in bytes Additionally, the user can add other fields here in order to distinguish test cases
"binary" - The binary data file name, whose contents will be transmitted via UART. Corresponding file path is taken from "binary_cmd_path" field, see ./config.json
"struct_check" - this section defines a set of validation rules applied to the binary structure received from the target device via UART. Each entry describes:
- field – structure field name. Nested fields are supported using dot notation, for example:
  - "uart.errorLastOperation"
  - "rmap.error"
- type – expected C data type of the field:
  - uint8
  - uint16
  - uint32
  - uint64
  - int8
  - int16
  - int32
  - int64
  - float
  - double
- op – comparison operator used for validation:
  - "==" equal to
  - "!=" not equal to
  - ">" greater than
  - ">=" greater than or equal to
  - "<" less than
  - "<=" less than or equal to
- expect – expected value used in comparison.
- array (optional) – if true, the specified field is treated as an array and every element is checked against the condition.
- exclude (optional) – list of array indices excluded from validation. This is useful when some array elements are reserved, contain implementation-specific values, or are expected to change dynamically.

For example:

{
    "field": "rmap.error",
    "array": true,
    "type": "uint16",
    "op": "==",
    "exclude": [0, 22],
    "expect": 0
}

This rule means:

the field rmap.error is an array of uint16;
all array elements must be equal to 0;
elements with indices 0 and 22 are skipped during validation.

The validation result for each rule is reported independently. If at least one check fails, the test case is considered failed and a detailed diagnostic message is printed, including:

field name;
actual value;
expected value;
comparison operator;
array index (for array checks).

The test-case checks the following structure received via UART:

typedef __packed struct
{
	uint8_t errorLastOperation; //RMAP_errFlag_en
	uint16_t error[COUNT_ERR_MAX_ELEMENTS_RMAP];
} RMAP_HK_t;

typedef __packed struct
{
	volatile uint8_t errorLastOperation; // UART_operErrCode_enum
	volatile uint16_t error[COUNT_ERR_MAX_ELEMENTS];
} UART_HK_t;

extern UART_HK_t UART_HK;

typedef __packed struct
{
	uint16_t size;
	uint32_t counter;
	UART_HK_t uart;
	RMAP_HK_t rmap;
} Snapshot_HK_t;

Reporting

./logs/test_stand.log: contains a report of errors from the Embedded System Test Automation Framework
./logs/test_cases.log: contains a report with detailed execution steps.
./logs/report.html: contains a summary report. It is created after execution and includes:
- Test case pass/fail status.
- Timestamps and error messages.
- UART communication statistics.

Notes (summary)

The application uses asynchronous UART communication for real-time testing.
Incoming data from target may arrive in multiple fragments. The framework accumulates all received packets until the timeout expires, specified in the test case by "timeout_msec" After that, the complete UART message is assembled and processed.
Test cases are fully described in JSON files and can be extended without modifying the C++ source code.
Binary UART commands are stored as external .bin files. The file name is specified in the "binary" field of the test case, while the search directory is configured in config.json using "binary_cmd_path".
When "protocol_ID": 2 is used, the framework can automatically decode C structures received from the target. Structure descriptions are extracted from:
- IAR map file (*.map);
- ELF file (*.out, *.elf);
- JSON files generated by the Python parsers located in ./python/iar_parser.
The files inside ./iar_files/ are generated automatically by:
- map_parser.py – parses IAR map files and extracts symbols, addresses, and sizes;
- elf_parser.py – parses ELF files and exports C structure layouts.
The framework supports:
- nested C structures;
- packed structures;
- arrays with element exclusion;
- validation using comparison operators (==, !=, >, <, >=, <=);
- structure field access using dot notation (for example: "uart.errorLastOperation").
The application generates:
- console output;
- detailed log files in ./logs;
- an HTML report summarizing the entire test session.
The framework is a console application (QT += serialport, CONFIG += console) and does not require GUI components at runtime.
The project was developed and tested on Windows 10/11 using:
- Qt 5.15.2;
- MinGW 8.1.0 (64-bit);
- GoogleTest;
- qmake build system.
Before running the application, make sure:
- the UART device is connected;
- the selected COM port exists and is not occupied by another application;
- paths in config.json point to valid files;
- the required binary command files exist in the configured binary_cmd_path.
After compilation on Windows, it is recommended to run windeployqt to copy all required Qt runtime libraries and plugins to the executable directory.
The framework can be easily extended with additional UART protocols, new validation rules, custom report generators, and support for other embedded compilers besides IAR.

23 KiB Raw Blame History Unescape Escape