gamkov/asotr_csv
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

1. Add public function parse_data_dir for parsing ASOTR MVN data in specified directory and all subdirectories and write data about temperature, temperature set and power to csv files. 2. Correct Readme. 3. Correct linux bash-scripts for unpacking asotr.tar.gz files

Browse Source
This commit is contained in:
Danila Gamkov 2025-01-24 11:42:08 +03:00
parent b738243634
commit d3cf1281dd
4 changed files with 289 additions and 179 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

116
README.markdown
View File

@ -1,29 +1,18 @@
# asotr_csv
The simplest parser for converting data from the ASOTR MVN control channels into the CSV format
The parser for converting data from the ASOTR MVN control channels into the CSV format
## Contents
- Setup
- Using
- Contacts
- **Setup**
- **Using**
- parsing ASOTR MVN data files in specified directory
- parsing all ASOTR MVN data files in specified directory and subdirectories
- **Output asotr_csv data files description**
- **Contacts**
## Setup
Instruction for setup asotr_csv project
1. Clone the repo to your computer:
```
git clone http://heagit.cosmos.ru/gamkov/asotr_csv.git
```
2. Enter the repo and install it:
```
cd <PATH_TO_ASOTR_CSV>
```
3. Install Rust compiler (if you don't have).
1. Install Rust compiler (if you don't have).
Installation on Linux:
```
curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh
@ -33,56 +22,81 @@ Installation on Windows:
Go to address https://www.rust-lang.org/tools/install and follow instructions
For more detailed information you can go to: https://doc.rust-lang.ru/book/ch01-01-installation.html
4. Compile project with Rust:
**Instruction for setup asotr_csv project**
2. Clone the repo to your computer:
```
cargo build
git clone http://heagit.cosmos.ru/gamkov/asotr_csv.git
```
3. Enter the repo and compile it:
```
cd <PATH_TO_ASOTR_CSV>
cargo build --release
```
After this commands you get execution file (asotr_csv) in the following directory: <PATH_TO_ASOTR_CSV>/target/release/
After running this commands you will get an execution file (asotr_csv) in the following directory:
\<PATH_TO_ASOTR_CSV\>/target/release/
## Using
1. Donwload data from science data server
If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95), contact Shtykovsky A.
### parsing ASOTR MVN data files in specified directory
1. Donwload data from science data server to directory \<PATH_TO_ASOTR_DATA\>.
If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95).
For questions about downloading science data contact Shtykovsky A. (a.shtykovsky@cosmos.ru) or Chelovekov I. (chelovekov@cosmos.ru)
2. Copy bash-script in order to unpack ASOTR MVN data:
```
cp <PATH_TO_ASOTR_CSV>/asotr_unzip.sh <PATH_TO_ASOTR_DATA>
```
3. Run asotr_unzip.sh for directory with MVN data, for example:
```
cd <PATH_TO_ASOTR_DATA>
./asotr_unzip.sh ./20241231-001
```
3. Copy asotr_csv parser and asotr_parse.sh script to directory with unpacked ASOTR data:
2. Run linux bash script **asotr_unzip.sh** for directory with MVN data in order to unpack **asotr.tar.gz** archive with ASOTR MVN data, for example:
```
cd <PATH_TO_ASOTR_CSV>
cp asotr_parse.sh <PATH_TO_ASOTR_DATA>/20241231-001/data/asotr
cp target/release/asotr_csv <PATH_TO_ASOTR_DATA>/20241231-001/data/asotr
./asotr_unzip.sh <PATH_TO_ASOTR_DATA>/20241231-001
```
4. Run ASOTR data parsing into csv:
cd <PATH_TO_ASOTR_DATA>
./asotr_parse.sh
**Note**: the script **asotr_unzip.sh** will not work on windows, you will need to unpack the archive **\<PATH_TO_ASOTR_DATA\>/20241231-001/data/asotr.tar.gz** manually or write the corresponding Windows bat-file
3. Run program asotr_csv:
```
cd <PATH_TO_ASOTR_CSV>/target/release/
./asotr_csv -d <PATH_TO_ASOTR_DATA>/20241231-001
```
csv data are ready to use in directory:
PATH_TO_ASOTR_DATA/20241231-001/data
\<PATH_TO_ASOTR_CSV\>/target/release/
**File descriptions:**
asotr1_T.csv - ASOTR1 temperature data in channels 1-6 (in Celsius)
asotr1_P.csv - ASOTR1 power data in channels 1-6 (in %)
asotr1_TSET.csv - ASOTR1 temperature sets in channels 1-6 (in Celsius)
asotr2_T.csv - ASOTR2 temperature data in channels 1-6 (in Celsius)
asotr2_P.csv - ASOTR2 power data in channels 1-6 (in %)
asotr2_TSET.csv - ASOTR2 temperature sets in channels 1-6 (in Celsius)
### parsing all ASOTR MVN data files in specified directory and subdirectories
1. Donwload data from science data server to directory \<PATH_TO_ASOTR_DATA\>.
If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95).
For questions about downloading science data contact Shtykovsky A. (a.shtykovsky@cosmos.ru) or Chelovekov I. (chelovekov@cosmos.ru)
**file data fromat:**
2. Run linux bash script **asotr_all_unzip.sh** for directory with MVN data in order to unpack all **asotr.tar.gz** archives with ASOTR MVN data, for example:
```
cd <PATH_TO_ASOTR_CSV>
./asotr_all_unzip.sh <PATH_TO_ASOTR_DATA>
```
**Note**: the script **asotr_all_unzip.sh** will not work on windows, you will need to unpack the each archive **\<PATH_TO_ASOTR_DATA\>/\<DIRECTORY_WITH_DATA\>/data/asotr.tar.gz** manually or write the corresponding Windows bat-file
3. Run program asotr_csv:
```
cd <PATH_TO_ASOTR_CSV>/target/release/
./asotr_csv -d <PATH_TO_ASOTR_DATA>
```
csv data are ready to use in directory:
\<PATH_TO_ASOTR_CSV\>/target/release/
## Output asotr_csv data files description
**description:**
asotr01_data_T.csv - ASOTR1 temperature data in channels 1-6 (in Celsius)
asotr01_data_P.csv - ASOTR1 power data in channels 1-6 (in %)
asotr01_data_TSET.csv - ASOTR1 temperature sets in channels 1-6 (in Celsius)
asotr02_data_T.csv - ASOTR2 temperature data in channels 1-6 (in Celsius)
asotr02_data_P.csv - ASOTR2 power data in channels 1-6 (in %)
asotr02_data_TSET.csv - ASOTR2 temperature sets in channels 1-6 (in Celsius)
**file data csv fromat:**
column 1: Unix timestamp
column 2: date
column 3: time
columns 4-9 - channels data (power, temperature or temperature set)
columns 4-9 - data from control channels (power, temperature or temperature set)
## Contatcs
For questions about the program, please contact Danila Gamkov, mail: danila_gamkov@cosmos.ru

10
asotr_all_unzip.sh Executable file
View File

@ -0,0 +1,10 @@
#! /bin/bash
if [ $# != 1 ]
then
echo "erorr use $0. Right use this script: "
echo "$0 path"
else
path_=$1
find ${path_} -maxdepth 1 -type d | xargs -I {} ./asotr_unzip.sh {}
fi

8
asotr_parse.sh
View File

@ -1,8 +0,0 @@
#! /bin/bash
ls | grep -P '.*data01.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_T.csv
ls | grep -P '.*data02.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_P.csv
ls | grep -P '.*data06.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_TSET.csv
ls | grep -P '.*data01.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_T.csv
ls | grep -P '.*data02.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_P.csv
ls | grep -P '.*data06.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_TSET.csv

328
src/main.rs
View File

@ -7,13 +7,30 @@ pub mod asotr_data {
use std::time::{SystemTime, UNIX_EPOCH, Duration};
use strum::FromRepr;
use lazy_static::lazy_static;
use std::path::PathBuf;
use regex::Regex;
use walkdir::WalkDir;
lazy_static! {
pub static ref support_dtypes: String =
String::from(".data01.asotr01(02), data02.asotr01(02), data06.asotr01(02)");
pub static ref patterns_fnames_csv_data: Vec<(String, String)> = {
let mut patterns: Vec<(String, String)> = Vec::new();
patterns.push((String::from(".*data01.asotr01"),
String::from("asotr01_data_T.csv")));
patterns.push((String::from(".*data02.asotr01"),
String::from("asotr01_data_P.csv")));
patterns.push((String::from(".*data06.asotr01"),
String::from("asotr01_data_TSET.csv")));
patterns.push((String::from(".*data01.asotr02"),
String::from("asotr02_data_T.csv")));
patterns.push((String::from(".*data02.asotr02"),
String::from("asotr02_data_P.csv")));
patterns.push((String::from(".*data06.asotr02"),
String::from("asotr02_data_TSET.csv")));
patterns
};
}
#[derive(Debug, FromRepr, PartialEq)]
@ -25,122 +42,21 @@ pub mod asotr_data {
struct AsotrDataDesc {
time_s: u64,
time_mks: u32,
datetime: String,
// time_mks: u32,
// datetime: String,
date: String,
time: String,
data_type: AsotrDataType,
kit: u8,
// kit: u8,
}
impl AsotrDataDesc {
pub fn new(time_s: u64, time_mks: u32,
datetime: String, date: String, time: String,
data_type: AsotrDataType, kit: u8) -> AsotrDataDesc {
AsotrDataDesc { time_s, time_mks, datetime, date, time, data_type, kit}
pub fn new(time_s: u64, date: String, time: String,
data_type: AsotrDataType) -> AsotrDataDesc {
AsotrDataDesc { time_s, date, time, data_type }
}
}
fn parse_data_f32(buf: Vec<u8>) -> Result<[f32; 6], String> {
let mut data = &buf[..];
let mut ch: [f32; 6] = [0.0; 6];
for i in 0..6 {
ch[i] = match data.read_f32::<LittleEndian>() {
Ok(val) => val,
Err(msg) => {
return Err(format!("Error parsing file: failed parsing float32 data: {}", msg)); }
}
}
return Ok(ch);
}
fn parse_data_u16(buf: Vec<u8>) -> Result<[u16; 6], String> {
let mut data = &buf[..];
let mut ch: [u16; 6] = [0; 6];
for i in 0..6 {
ch[i] = match data.read_u16::<LittleEndian>() {
Ok(val) => val,
Err(msg) => {
return Err(format!("Error parsing file: failed parsing uint16 data: {}", msg)); }
}
}
return Ok(ch);
}
fn parse_filename(filename_full: String) -> Result<AsotrDataDesc, String> {
let mut filename = String::new();
let msg_prev = format!("Error parsing filename {}:", filename_full);
match filename_full.rfind('/') {
Some(val) => { filename = (filename_full[val+1..filename_full.len()]).to_string(); }
_ => { filename = filename_full.clone(); }
}
if filename.len() != 32 {
return Err(format!("{} unsupported file", msg_prev));
}
let time_unix_ = filename[0..10].parse::<u64>();
let time_unix = match &time_unix_ {
Ok(data) => data,
Err(msg) => {
return Err(format!("{} expected digits in timestamp sec part ({})",
msg_prev, msg));
}
};
let data_type_ = filename[22..24].parse::<u8>();
let data_type_u8 = match &data_type_ {
Ok(data) => data,
Err(msg) => {
return Err(format!("{} expected digits in data type part ({})",
msg_prev, msg));
}
};
if *data_type_u8 == 1 || *data_type_u8 == 2 || *data_type_u8 == 6 { }
else {
return Err(format!("{} parser supports data types: {}",
msg_prev, support_dtypes.to_string()));
}
let data_type = match AsotrDataType::from_repr(*data_type_u8 as usize) {
Some(value) => value,
_ => return Err(format!("{} expected digits in data type part",
msg_prev))
};
let _kit = filename[30..32].parse::<u8>();
let kit = match &_kit {
Ok(data) => data,
Err(msg) => { return Err(format!("{}: expected digits in asotr kit part ({})",
msg_prev, msg)); }
};
let _time_str_mks = filename[11..17].parse::<u32>();
let time_mks = match &_time_str_mks {
Ok(data) => data,
Err(msg) => { return Err(format!("{}: expected digits in timestamp mks part ({})",
msg_prev, msg)); }
};
let time: SystemTime = UNIX_EPOCH + Duration::from_secs(*time_unix);
let date_time = DateTime::<Utc>::from(time);
let date_s = date_time.format("%d.%m.%Y").to_string();
let time_s = date_time.format("%H:%M:%S").to_string();
let head = AsotrDataDesc::new(*time_unix, *time_mks,
format!("{} {}", date_s, time_s),
date_s, time_s, data_type, *kit);
return Ok(head);
}
pub fn read_data(filename_full: String) -> Result<String, String> {
let ch_u16: [u16; 6];
let ch_f32: [f32; 6];
@ -185,11 +101,170 @@ pub mod asotr_data {
}
out.remove(out.len() - 1);
return Ok(out);
}
pub fn parse_data_dir(dir: &str) -> Result<(), String> {
let mut data: Vec<String> = Vec::new();
println!("parse data from directory: {}", dir);
for (pattern, fname) in patterns_fnames_csv_data.iter() {
let files = find_files_regex(dir, pattern)?;
for elem in files {
data.push(read_data(elem)?);
}
data.sort();
println!("save csv data to file: {}", fname);
save_data_csv(data.clone(), fname)?;
data.clear();
}
return Ok(());
}
fn parse_data_f32(buf: Vec<u8>) -> Result<[f32; 6], String> {
let mut data = &buf[..];
let mut ch: [f32; 6] = [0.0; 6];
for i in 0..6 {
ch[i] = match data.read_f32::<LittleEndian>() {
Ok(val) => val,
Err(msg) => {
return Err(format!(
"Error parsing file: failed parsing float32 data: {}", msg)); }
}
}
return Ok(ch);
}
fn parse_data_u16(buf: Vec<u8>) -> Result<[u16; 6], String> {
let mut data = &buf[..];
let mut ch: [u16; 6] = [0; 6];
for i in 0..6 {
ch[i] = match data.read_u16::<LittleEndian>() {
Ok(val) => val,
Err(msg) => {
return Err(format!(
"Error parsing file: failed parsing uint16 data: {}", msg)); }
}
}
return Ok(ch);
}
fn parse_filename(filename_full: String) -> Result<AsotrDataDesc, String> {
let mut filename = String::new();
let msg_prev = format!("Error parsing filename {}:", filename_full);
match filename_full.rfind('/') {
Some(val) => { filename = (filename_full[val+1..filename_full.len()]).to_string(); }
_ => { filename = filename_full.clone(); }
}
if filename.len() != 32 {
return Err(format!("{} unsupported file", msg_prev));
}
let time_unix_ = filename[0..10].parse::<u64>();
let time_unix = match &time_unix_ {
Ok(data) => data,
Err(msg) => {
return Err(format!("{} expected digits in timestamp sec part ({})",
msg_prev, msg));
}
};
let data_type_ = filename[22..24].parse::<u8>();
let data_type_u8 = match &data_type_ {
Ok(data) => data,
Err(msg) => {
return Err(format!("{} expected digits in data type part ({})",
msg_prev, msg));
}
};
if *data_type_u8 == 1 || *data_type_u8 == 2 || *data_type_u8 == 6 { }
else {
return Err(format!("{} parser supports data types: {}",
msg_prev, support_dtypes.to_string()));
}
let data_type = match AsotrDataType::from_repr(*data_type_u8 as usize) {
Some(value) => value,
_ => return Err(format!("{} expected digits in data type part",
msg_prev))
};
// let _kit = filename[30..32].parse::<u8>();
// let kit = match &_kit {
// Ok(data) => data,
// Err(msg) => { return Err(format!("{}: expected digits in asotr kit part ({})",
// msg_prev, msg)); }
// };
// let _time_str_mks = filename[11..17].parse::<u32>();
// let time_mks = match &_time_str_mks {
// Ok(data) => data,
// Err(msg) => { return Err(format!("{}: expected digits in timestamp mks part ({})",
// msg_prev, msg)); }
// };
let time: SystemTime = UNIX_EPOCH + Duration::from_secs(*time_unix);
let date_time = DateTime::<Utc>::from(time);
let date_s = date_time.format("%d.%m.%Y").to_string();
let time_s = date_time.format("%H:%M:%S").to_string();
let head = AsotrDataDesc::new(*time_unix, date_s, time_s, data_type);
return Ok(head);
}
fn find_files_regex(dir: &str, template: &str) -> Result<Vec<String>, String> {
let mut path_vec: Vec<String> = Vec::new();
let regex = match Regex::new(template) {
Ok(val) => val,
Err(msg) => {
return Err(format!("Error create regex template ({}): {}",
template, msg));
}
};
let data = WalkDir::new(dir)
.into_iter()
.filter_map(|e| e.ok())
.filter(|e| regex.is_match(e.file_name().to_str().unwrap()))
.map(|e| e.into_path());
for elem in data {
path_vec.push(elem.display().to_string());
}
if path_vec.len() == 0 {
return Err(format!(
"Error searching files for pattern ({}): files not found in directory {}",
template, dir));
}
return Ok(path_vec);
}
fn save_data_csv(data: Vec<String>, fname: &str) -> Result<(), String> {
match std::fs::write(fname, data.join("\n")){
Ok(f) => f,
Err(msg) => {
return Err(format!("Error write data to csv file {}: {}", fname, msg))
}
}
return Ok(());
}
}
@ -204,22 +279,41 @@ fn main() {
Arg::new("file")
.short('f')
.long("filename")
.help(format!("input ASOTR MVN data ({})", support_dtypes.to_string()))
.required(true),
.help(format!("file with ASOTR MVN data ({})", support_dtypes.to_string()))
)
.arg(
Arg::new("dir")
.short('d')
.long("directory")
.help(format!("directory with ASOTR MVN data"))
)
.get_matches();
let filename: &String = match arguments.get_one::<String>("file") {
Some(fname) => fname,
_ => { println!("Error: wrong argument!"); return; }
};
let s = match read_data(filename.clone()) {
match arguments.get_one::<String>("file") {
Some(fname) => {
let s = match read_data(fname.clone()) {
Ok(elem) => elem,
Err(msg) => { println!("{}", msg); return; }
};
println!("{}", s);
return;
},
_ => { }
};
match arguments.get_one::<String>("dir") {
Some(path) => {
match parse_data_dir(path) {
Ok(_) => {}
Err(msg) => { println!("{}", msg); return; }
}
return;
},
_ => { }
}
println!("Unexpected command. Type --help for more iformation");
}