diff --git a/README.markdown b/README.markdown index 89315da..3d08ea7 100644 --- a/README.markdown +++ b/README.markdown @@ -1,29 +1,18 @@ # asotr_csv -The simplest parser for converting data from the ASOTR MVN control channels into the CSV format +The parser for converting data from the ASOTR MVN control channels into the CSV format ## Contents -- Setup -- Using -- Contacts +- **Setup** +- **Using** +- parsing ASOTR MVN data files in specified directory +- parsing all ASOTR MVN data files in specified directory and subdirectories +- **Output asotr_csv data files description** +- **Contacts** ## Setup - -Instruction for setup asotr_csv project - -1. Clone the repo to your computer: - -``` -git clone http://heagit.cosmos.ru/gamkov/asotr_csv.git -``` - -2. Enter the repo and install it: -``` -cd -``` - -3. Install Rust compiler (if you don't have). +1. Install Rust compiler (if you don't have). Installation on Linux: ``` curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh @@ -33,56 +22,81 @@ Installation on Windows: Go to address https://www.rust-lang.org/tools/install and follow instructions For more detailed information you can go to: https://doc.rust-lang.ru/book/ch01-01-installation.html -4. Compile project with Rust: +**Instruction for setup asotr_csv project** + +2. Clone the repo to your computer: + ``` -cargo build +git clone http://heagit.cosmos.ru/gamkov/asotr_csv.git +``` + +3. Enter the repo and compile it: +``` +cd cargo build --release ``` -After this commands you get execution file (asotr_csv) in the following directory: /target/release/ +After running this commands you will get an execution file (asotr_csv) in the following directory: +\/target/release/ ## Using -1. Donwload data from science data server -If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95), contact Shtykovsky A. +### parsing ASOTR MVN data files in specified directory +1. Donwload data from science data server to directory \. +If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95). +For questions about downloading science data contact Shtykovsky A. (a.shtykovsky@cosmos.ru) or Chelovekov I. (chelovekov@cosmos.ru) -2. Copy bash-script in order to unpack ASOTR MVN data: +2. Run linux bash script **asotr_unzip.sh** for directory with MVN data in order to unpack **asotr.tar.gz** archive with ASOTR MVN data, for example: ``` -cp /asotr_unzip.sh +cd +./asotr_unzip.sh /20241231-001 ``` -3. Run asotr_unzip.sh for directory with MVN data, for example: -``` -cd -./asotr_unzip.sh ./20241231-001 -``` +**Note**: the script **asotr_unzip.sh** will not work on windows, you will need to unpack the archive **\/20241231-001/data/asotr.tar.gz** manually or write the corresponding Windows bat-file -3. Copy asotr_csv parser and asotr_parse.sh script to directory with unpacked ASOTR data: +3. Run program asotr_csv: +``` +cd /target/release/ +./asotr_csv -d /20241231-001 ``` -cd -cp asotr_parse.sh /20241231-001/data/asotr -cp target/release/asotr_csv /20241231-001/data/asotr -``` - -4. Run ASOTR data parsing into csv: -cd -./asotr_parse.sh - csv data are ready to use in directory: -PATH_TO_ASOTR_DATA/20241231-001/data +\/target/release/ -**File descriptions:** -asotr1_T.csv - ASOTR1 temperature data in channels 1-6 (in Celsius) -asotr1_P.csv - ASOTR1 power data in channels 1-6 (in %) -asotr1_TSET.csv - ASOTR1 temperature sets in channels 1-6 (in Celsius) -asotr2_T.csv - ASOTR2 temperature data in channels 1-6 (in Celsius) -asotr2_P.csv - ASOTR2 power data in channels 1-6 (in %) -asotr2_TSET.csv - ASOTR2 temperature sets in channels 1-6 (in Celsius) +### parsing all ASOTR MVN data files in specified directory and subdirectories +1. Donwload data from science data server to directory \. +If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95). +For questions about downloading science data contact Shtykovsky A. (a.shtykovsky@cosmos.ru) or Chelovekov I. (chelovekov@cosmos.ru) -**file data fromat:** +2. Run linux bash script **asotr_all_unzip.sh** for directory with MVN data in order to unpack all **asotr.tar.gz** archives with ASOTR MVN data, for example: +``` +cd +./asotr_all_unzip.sh +``` + +**Note**: the script **asotr_all_unzip.sh** will not work on windows, you will need to unpack the each archive **\/\/data/asotr.tar.gz** manually or write the corresponding Windows bat-file + +3. Run program asotr_csv: +``` +cd /target/release/ +./asotr_csv -d +``` +csv data are ready to use in directory: +\/target/release/ + +## Output asotr_csv data files description +**description:** +asotr01_data_T.csv - ASOTR1 temperature data in channels 1-6 (in Celsius) +asotr01_data_P.csv - ASOTR1 power data in channels 1-6 (in %) +asotr01_data_TSET.csv - ASOTR1 temperature sets in channels 1-6 (in Celsius) +asotr02_data_T.csv - ASOTR2 temperature data in channels 1-6 (in Celsius) +asotr02_data_P.csv - ASOTR2 power data in channels 1-6 (in %) +asotr02_data_TSET.csv - ASOTR2 temperature sets in channels 1-6 (in Celsius) + +**file data csv fromat:** column 1: Unix timestamp column 2: date column 3: time -columns 4-9 - channels data (power, temperature or temperature set) +columns 4-9 - data from control channels (power, temperature or temperature set) + ## Contatcs For questions about the program, please contact Danila Gamkov, mail: danila_gamkov@cosmos.ru diff --git a/asotr_all_unzip.sh b/asotr_all_unzip.sh new file mode 100755 index 0000000..46d0528 --- /dev/null +++ b/asotr_all_unzip.sh @@ -0,0 +1,10 @@ +#! /bin/bash + +if [ $# != 1 ] +then + echo "erorr use $0. Right use this script: " + echo "$0 path" +else + path_=$1 + find ${path_} -maxdepth 1 -type d | xargs -I {} ./asotr_unzip.sh {} +fi diff --git a/asotr_parse.sh b/asotr_parse.sh deleted file mode 100755 index 4e73f93..0000000 --- a/asotr_parse.sh +++ /dev/null @@ -1,8 +0,0 @@ -#! /bin/bash - -ls | grep -P '.*data01.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_T.csv -ls | grep -P '.*data02.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_P.csv -ls | grep -P '.*data06.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_TSET.csv -ls | grep -P '.*data01.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_T.csv -ls | grep -P '.*data02.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_P.csv -ls | grep -P '.*data06.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_TSET.csv diff --git a/src/main.rs b/src/main.rs index a3eb72d..e02c7ae 100644 --- a/src/main.rs +++ b/src/main.rs @@ -7,13 +7,30 @@ pub mod asotr_data { use std::time::{SystemTime, UNIX_EPOCH, Duration}; use strum::FromRepr; use lazy_static::lazy_static; - use std::path::PathBuf; use regex::Regex; use walkdir::WalkDir; lazy_static! { pub static ref support_dtypes: String = String::from(".data01.asotr01(02), data02.asotr01(02), data06.asotr01(02)"); + + pub static ref patterns_fnames_csv_data: Vec<(String, String)> = { + let mut patterns: Vec<(String, String)> = Vec::new(); + patterns.push((String::from(".*data01.asotr01"), + String::from("asotr01_data_T.csv"))); + patterns.push((String::from(".*data02.asotr01"), + String::from("asotr01_data_P.csv"))); + patterns.push((String::from(".*data06.asotr01"), + String::from("asotr01_data_TSET.csv"))); + patterns.push((String::from(".*data01.asotr02"), + String::from("asotr02_data_T.csv"))); + patterns.push((String::from(".*data02.asotr02"), + String::from("asotr02_data_P.csv"))); + patterns.push((String::from(".*data06.asotr02"), + String::from("asotr02_data_TSET.csv"))); + + patterns + }; } #[derive(Debug, FromRepr, PartialEq)] @@ -25,121 +42,20 @@ pub mod asotr_data { struct AsotrDataDesc { time_s: u64, - time_mks: u32, - datetime: String, + // time_mks: u32, + // datetime: String, date: String, time: String, data_type: AsotrDataType, - kit: u8, + // kit: u8, } impl AsotrDataDesc { - pub fn new(time_s: u64, time_mks: u32, - datetime: String, date: String, time: String, - data_type: AsotrDataType, kit: u8) -> AsotrDataDesc { - AsotrDataDesc { time_s, time_mks, datetime, date, time, data_type, kit} + pub fn new(time_s: u64, date: String, time: String, + data_type: AsotrDataType) -> AsotrDataDesc { + AsotrDataDesc { time_s, date, time, data_type } } - } - - fn parse_data_f32(buf: Vec) -> Result<[f32; 6], String> { - let mut data = &buf[..]; - let mut ch: [f32; 6] = [0.0; 6]; - - for i in 0..6 { - ch[i] = match data.read_f32::() { - Ok(val) => val, - Err(msg) => { - return Err(format!("Error parsing file: failed parsing float32 data: {}", msg)); } - } - } - - return Ok(ch); - } - - fn parse_data_u16(buf: Vec) -> Result<[u16; 6], String> { - let mut data = &buf[..]; - let mut ch: [u16; 6] = [0; 6]; - - for i in 0..6 { - ch[i] = match data.read_u16::() { - Ok(val) => val, - Err(msg) => { - return Err(format!("Error parsing file: failed parsing uint16 data: {}", msg)); } - } - } - - return Ok(ch); - } - - fn parse_filename(filename_full: String) -> Result { - let mut filename = String::new(); - - let msg_prev = format!("Error parsing filename {}:", filename_full); - - match filename_full.rfind('/') { - Some(val) => { filename = (filename_full[val+1..filename_full.len()]).to_string(); } - _ => { filename = filename_full.clone(); } - } - - if filename.len() != 32 { - return Err(format!("{} unsupported file", msg_prev)); - } - - let time_unix_ = filename[0..10].parse::(); - let time_unix = match &time_unix_ { - Ok(data) => data, - Err(msg) => { - return Err(format!("{} expected digits in timestamp sec part ({})", - msg_prev, msg)); - } - }; - - let data_type_ = filename[22..24].parse::(); - let data_type_u8 = match &data_type_ { - Ok(data) => data, - Err(msg) => { - return Err(format!("{} expected digits in data type part ({})", - msg_prev, msg)); - } - }; - - if *data_type_u8 == 1 || *data_type_u8 == 2 || *data_type_u8 == 6 { } - else { - return Err(format!("{} parser supports data types: {}", - msg_prev, support_dtypes.to_string())); - } - - let data_type = match AsotrDataType::from_repr(*data_type_u8 as usize) { - Some(value) => value, - _ => return Err(format!("{} expected digits in data type part", - msg_prev)) - }; - - let _kit = filename[30..32].parse::(); - let kit = match &_kit { - Ok(data) => data, - Err(msg) => { return Err(format!("{}: expected digits in asotr kit part ({})", - msg_prev, msg)); } - }; - - let _time_str_mks = filename[11..17].parse::(); - let time_mks = match &_time_str_mks { - Ok(data) => data, - Err(msg) => { return Err(format!("{}: expected digits in timestamp mks part ({})", - msg_prev, msg)); } - }; - - let time: SystemTime = UNIX_EPOCH + Duration::from_secs(*time_unix); - let date_time = DateTime::::from(time); - let date_s = date_time.format("%d.%m.%Y").to_string(); - let time_s = date_time.format("%H:%M:%S").to_string(); - - let head = AsotrDataDesc::new(*time_unix, *time_mks, - format!("{} {}", date_s, time_s), - date_s, time_s, data_type, *kit); - - return Ok(head); - } + } pub fn read_data(filename_full: String) -> Result { let ch_u16: [u16; 6]; @@ -185,11 +101,170 @@ pub mod asotr_data { } out.remove(out.len() - 1); - return Ok(out); } + + pub fn parse_data_dir(dir: &str) -> Result<(), String> { + let mut data: Vec = Vec::new(); + + println!("parse data from directory: {}", dir); + for (pattern, fname) in patterns_fnames_csv_data.iter() { + let files = find_files_regex(dir, pattern)?; + for elem in files { + data.push(read_data(elem)?); + } + data.sort(); + + println!("save csv data to file: {}", fname); + + save_data_csv(data.clone(), fname)?; + data.clear(); + } + + return Ok(()); + } + + fn parse_data_f32(buf: Vec) -> Result<[f32; 6], String> { + let mut data = &buf[..]; + let mut ch: [f32; 6] = [0.0; 6]; + + for i in 0..6 { + ch[i] = match data.read_f32::() { + Ok(val) => val, + Err(msg) => { + return Err(format!( + "Error parsing file: failed parsing float32 data: {}", msg)); } + } + } + + return Ok(ch); + } + + fn parse_data_u16(buf: Vec) -> Result<[u16; 6], String> { + let mut data = &buf[..]; + let mut ch: [u16; 6] = [0; 6]; + + for i in 0..6 { + ch[i] = match data.read_u16::() { + Ok(val) => val, + Err(msg) => { + return Err(format!( + "Error parsing file: failed parsing uint16 data: {}", msg)); } + } + } + + return Ok(ch); + } + + fn parse_filename(filename_full: String) -> Result { + let mut filename = String::new(); + let msg_prev = format!("Error parsing filename {}:", filename_full); + + match filename_full.rfind('/') { + Some(val) => { filename = (filename_full[val+1..filename_full.len()]).to_string(); } + _ => { filename = filename_full.clone(); } + } + + if filename.len() != 32 { + return Err(format!("{} unsupported file", msg_prev)); + } + + let time_unix_ = filename[0..10].parse::(); + let time_unix = match &time_unix_ { + Ok(data) => data, + Err(msg) => { + return Err(format!("{} expected digits in timestamp sec part ({})", + msg_prev, msg)); + } + }; + + let data_type_ = filename[22..24].parse::(); + let data_type_u8 = match &data_type_ { + Ok(data) => data, + Err(msg) => { + return Err(format!("{} expected digits in data type part ({})", + msg_prev, msg)); + } + }; + + if *data_type_u8 == 1 || *data_type_u8 == 2 || *data_type_u8 == 6 { } + else { + return Err(format!("{} parser supports data types: {}", + msg_prev, support_dtypes.to_string())); + } + + let data_type = match AsotrDataType::from_repr(*data_type_u8 as usize) { + Some(value) => value, + _ => return Err(format!("{} expected digits in data type part", + msg_prev)) + }; + + // let _kit = filename[30..32].parse::(); + // let kit = match &_kit { + // Ok(data) => data, + // Err(msg) => { return Err(format!("{}: expected digits in asotr kit part ({})", + // msg_prev, msg)); } + // }; + + // let _time_str_mks = filename[11..17].parse::(); + // let time_mks = match &_time_str_mks { + // Ok(data) => data, + // Err(msg) => { return Err(format!("{}: expected digits in timestamp mks part ({})", + // msg_prev, msg)); } + // }; + + let time: SystemTime = UNIX_EPOCH + Duration::from_secs(*time_unix); + let date_time = DateTime::::from(time); + let date_s = date_time.format("%d.%m.%Y").to_string(); + let time_s = date_time.format("%H:%M:%S").to_string(); + + let head = AsotrDataDesc::new(*time_unix, date_s, time_s, data_type); + + return Ok(head); + } + + fn find_files_regex(dir: &str, template: &str) -> Result, String> { + let mut path_vec: Vec = Vec::new(); + + let regex = match Regex::new(template) { + Ok(val) => val, + Err(msg) => { + return Err(format!("Error create regex template ({}): {}", + template, msg)); + } + }; + + let data = WalkDir::new(dir) + .into_iter() + .filter_map(|e| e.ok()) + .filter(|e| regex.is_match(e.file_name().to_str().unwrap())) + .map(|e| e.into_path()); + + for elem in data { + path_vec.push(elem.display().to_string()); + } + + if path_vec.len() == 0 { + return Err(format!( + "Error searching files for pattern ({}): files not found in directory {}", + template, dir)); + } + + return Ok(path_vec); + } + + fn save_data_csv(data: Vec, fname: &str) -> Result<(), String> { + match std::fs::write(fname, data.join("\n")){ + Ok(f) => f, + Err(msg) => { + return Err(format!("Error write data to csv file {}: {}", fname, msg)) + } + } + + return Ok(()); + } } @@ -204,22 +279,41 @@ fn main() { Arg::new("file") .short('f') .long("filename") - .help(format!("input ASOTR MVN data ({})", support_dtypes.to_string())) - .required(true), + .help(format!("file with ASOTR MVN data ({})", support_dtypes.to_string())) + ) + .arg( + Arg::new("dir") + .short('d') + .long("directory") + .help(format!("directory with ASOTR MVN data")) ) .get_matches(); - let filename: &String = match arguments.get_one::("file") { - Some(fname) => fname, - _ => { println!("Error: wrong argument!"); return; } + match arguments.get_one::("file") { + Some(fname) => { + let s = match read_data(fname.clone()) { + Ok(elem) => elem, + Err(msg) => { println!("{}", msg); return; } + }; + + println!("{}", s); + return; + }, + _ => { } }; - let s = match read_data(filename.clone()) { - Ok(elem) => elem, - Err(msg) => { println!("{}", msg); return; } - }; + match arguments.get_one::("dir") { + Some(path) => { + match parse_data_dir(path) { + Ok(_) => {} + Err(msg) => { println!("{}", msg); return; } + } + return; + }, + _ => { } + } - println!("{}", s); + println!("Unexpected command. Type --help for more iformation"); }