1. Add public function parse_data_dir for parsing ASOTR MVN data in specified directory and all subdirectories and write data about temperature, temperature set and power to csv files. 2. Correct Readme. 3. Correct linux bash-scripts for unpacking asotr.tar.gz files

README.markdown

@@ -1,29 +1,18 @@
 # asotr_csv
 
-The simplest parser for converting data from the ASOTR MVN control channels into the CSV format
+The parser for converting data from the ASOTR MVN control channels into the CSV format
 
 ## Contents
 
-- Setup
+- **Setup**
-- Using
+- **Using**
-- Contacts
+- parsing ASOTR MVN data files in specified directory
+- parsing all ASOTR MVN data files in specified directory and subdirectories
+- **Output asotr_csv data files description**
+- **Contacts**
 
 ## Setup
-
+1. Install Rust compiler (if you don't have).
-Instruction for setup asotr_csv project
-
-1. Clone the repo to your computer:
-
-```
-git clone http://heagit.cosmos.ru/gamkov/asotr_csv.git
-```
-
-2. Enter the repo and install it:
-```
-cd <PATH_TO_ASOTR_CSV>
-```
-
-3. Install Rust compiler (if you don't have).
 Installation on Linux:
 ```
 curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh
@@ -33,56 +22,81 @@ Installation on Windows:
 Go to address https://www.rust-lang.org/tools/install and follow instructions  
 For more detailed information you can go to: https://doc.rust-lang.ru/book/ch01-01-installation.html
 
-4. Compile project with Rust:
+**Instruction for setup asotr_csv project**
+
+2. Clone the repo to your computer:
+
 ```
-cargo build
+git clone http://heagit.cosmos.ru/gamkov/asotr_csv.git
+```
+
+3. Enter the repo and compile it:
+```
+cd <PATH_TO_ASOTR_CSV>
 cargo build --release
 ```
 
-After this commands you get execution file (asotr_csv) in the following directory: <PATH_TO_ASOTR_CSV>/target/release/
+After running this commands you will get an execution file (asotr_csv) in the following directory:  
+\<PATH_TO_ASOTR_CSV\>/target/release/
 
 ## Using
-1. Donwload data from science data server
+### parsing ASOTR MVN data files in specified directory
-If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95), contact Shtykovsky A.
+1. Donwload data from science data server to directory \<PATH_TO_ASOTR_DATA\>.  
+If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95).   
+For questions about downloading science data contact Shtykovsky A. (a.shtykovsky@cosmos.ru) or Chelovekov I. (chelovekov@cosmos.ru)   
 
-2. Copy bash-script in order to unpack ASOTR MVN data:
+2. Run linux bash script **asotr_unzip.sh** for directory with MVN data in order to unpack **asotr.tar.gz** archive with ASOTR MVN data, for example:  
 ```
-cp <PATH_TO_ASOTR_CSV>/asotr_unzip.sh <PATH_TO_ASOTR_DATA>
+cd <PATH_TO_ASOTR_CSV>  
+./asotr_unzip.sh <PATH_TO_ASOTR_DATA>/20241231-001   
 ```
 
-3. Run asotr_unzip.sh for directory with MVN data, for example:
+**Note**: the script **asotr_unzip.sh** will not work on windows, you will need to unpack the archive  **\<PATH_TO_ASOTR_DATA\>/20241231-001/data/asotr.tar.gz** manually or write the corresponding Windows bat-file
-```
-cd <PATH_TO_ASOTR_DATA>
-./asotr_unzip.sh ./20241231-001
-```
 
-3. Copy asotr_csv parser and asotr_parse.sh script to directory with unpacked ASOTR data:
+3. Run program asotr_csv:  
+```
+cd <PATH_TO_ASOTR_CSV>/target/release/  
+./asotr_csv -d <PATH_TO_ASOTR_DATA>/20241231-001  
 ```
-cd <PATH_TO_ASOTR_CSV>
-cp asotr_parse.sh <PATH_TO_ASOTR_DATA>/20241231-001/data/asotr
-cp target/release/asotr_csv <PATH_TO_ASOTR_DATA>/20241231-001/data/asotr
-``` 
-
-4. Run ASOTR data parsing into csv:
-cd <PATH_TO_ASOTR_DATA>
-./asotr_parse.sh
-
 csv data are ready to use in directory:  
-PATH_TO_ASOTR_DATA/20241231-001/data  
+\<PATH_TO_ASOTR_CSV\>/target/release/  
 
-**File descriptions:**  
+### parsing all ASOTR MVN data files in specified directory and subdirectories
-asotr1_T.csv - ASOTR1 temperature data in channels 1-6 (in Celsius)  
+1. Donwload data from science data server to directory \<PATH_TO_ASOTR_DATA\>.  
-asotr1_P.csv - ASOTR1 power data in channels 1-6 (in %)  
+If you don't have MVN data, you might download it from server with science SRG data (IP: 193.232.11.95).   
-asotr1_TSET.csv - ASOTR1 temperature sets in channels 1-6 (in Celsius)  
+For questions about downloading science data contact Shtykovsky A. (a.shtykovsky@cosmos.ru) or Chelovekov I. (chelovekov@cosmos.ru)
-asotr2_T.csv - ASOTR2 temperature data in channels 1-6 (in Celsius)  
-asotr2_P.csv - ASOTR2 power data in channels 1-6 (in %)  
-asotr2_TSET.csv - ASOTR2 temperature sets in channels 1-6 (in Celsius)  
 
-**file data fromat:**  
+2. Run linux bash script **asotr_all_unzip.sh** for directory with MVN data in order to unpack all **asotr.tar.gz** archives with ASOTR MVN data, for example:  
+```
+cd <PATH_TO_ASOTR_CSV>  
+./asotr_all_unzip.sh <PATH_TO_ASOTR_DATA>   
+```
+
+**Note**: the script **asotr_all_unzip.sh** will not work on windows, you will need to unpack the each archive **\<PATH_TO_ASOTR_DATA\>/\<DIRECTORY_WITH_DATA\>/data/asotr.tar.gz** manually or write the corresponding Windows bat-file  
+
+3. Run program asotr_csv:  
+```
+cd <PATH_TO_ASOTR_CSV>/target/release/  
+./asotr_csv -d <PATH_TO_ASOTR_DATA> 
+```
+csv data are ready to use in directory:  
+\<PATH_TO_ASOTR_CSV\>/target/release/  
+
+## Output asotr_csv data files description
+**description:**  
+asotr01_data_T.csv - ASOTR1 temperature data in channels 1-6 (in Celsius)  
+asotr01_data_P.csv - ASOTR1 power data in channels 1-6 (in %)  
+asotr01_data_TSET.csv - ASOTR1 temperature sets in channels 1-6 (in Celsius)  
+asotr02_data_T.csv - ASOTR2 temperature data in channels 1-6 (in Celsius)  
+asotr02_data_P.csv - ASOTR2 power data in channels 1-6 (in %)  
+asotr02_data_TSET.csv - ASOTR2 temperature sets in channels 1-6 (in Celsius)  
+
+**file data csv fromat:**  
 column 1: Unix timestamp  
 column 2: date  
 column 3: time  
-columns 4-9 - channels data (power, temperature or temperature set)  
+columns 4-9 - data from control channels (power, temperature or temperature set)  
+
 
 ## Contatcs
 For questions about the program, please contact Danila Gamkov, mail: danila_gamkov@cosmos.ru

asotr_all_unzip.sh

@@ -0,0 +1,10 @@
+#! /bin/bash
+
+if [ $# != 1 ]
+then
+	echo "erorr use $0. Right use this script: "
+	echo "$0 path"
+else
+	path_=$1
+	find ${path_} -maxdepth 1 -type d | xargs -I {} ./asotr_unzip.sh {}
+fi

asotr_parse.sh

@@ -1,8 +0,0 @@
-#! /bin/bash
-
-ls | grep -P '.*data01.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_T.csv
-ls | grep -P '.*data02.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_P.csv
-ls | grep -P '.*data06.asotr01' | xargs -I {} ./asotr_csv -f {} >> ../asotr1_TSET.csv
-ls | grep -P '.*data01.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_T.csv
-ls | grep -P '.*data02.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_P.csv
-ls | grep -P '.*data06.asotr02' | xargs -I {} ./asotr_csv -f {} >> ../asotr2_TSET.csv

src/main.rs

@@ -7,13 +7,30 @@ pub mod asotr_data {
     use std::time::{SystemTime, UNIX_EPOCH, Duration};
     use strum::FromRepr;
     use lazy_static::lazy_static;
-    use std::path::PathBuf;
     use regex::Regex;
     use walkdir::WalkDir;
 
     lazy_static! {
         pub static ref support_dtypes: String = 
             String::from(".data01.asotr01(02), data02.asotr01(02), data06.asotr01(02)");
+        
+        pub static ref patterns_fnames_csv_data: Vec<(String, String)> = {
+            let mut patterns: Vec<(String, String)> = Vec::new();
+            patterns.push((String::from(".*data01.asotr01"), 
+                           String::from("asotr01_data_T.csv")));
+            patterns.push((String::from(".*data02.asotr01"), 
+                           String::from("asotr01_data_P.csv")));
+            patterns.push((String::from(".*data06.asotr01"), 
+                           String::from("asotr01_data_TSET.csv")));
+            patterns.push((String::from(".*data01.asotr02"), 
+                           String::from("asotr02_data_T.csv")));
+            patterns.push((String::from(".*data02.asotr02"), 
+                           String::from("asotr02_data_P.csv")));
+            patterns.push((String::from(".*data06.asotr02"), 
+                           String::from("asotr02_data_TSET.csv")));
+
+            patterns
+        };
     }
 
     #[derive(Debug, FromRepr, PartialEq)]
@@ -25,121 +42,20 @@ pub mod asotr_data {
 
     struct AsotrDataDesc {
         time_s: u64,
-        time_mks: u32,
+        // time_mks: u32,
-        datetime: String,
+        // datetime: String,
         date: String,
         time: String,
         data_type: AsotrDataType,
-        kit: u8,
+        // kit: u8,
     }
 
     impl AsotrDataDesc {
-        pub fn new(time_s: u64, time_mks: u32, 
+        pub fn new(time_s: u64, date: String, time: String, 
-                   datetime: String, date: String, time: String, 
+                   data_type: AsotrDataType) -> AsotrDataDesc {
-                   data_type: AsotrDataType, kit: u8) -> AsotrDataDesc {
+            AsotrDataDesc { time_s, date, time, data_type }
-            AsotrDataDesc { time_s, time_mks, datetime, date, time, data_type, kit}
         }
-    }
+    } 
-
-    fn parse_data_f32(buf: Vec<u8>) -> Result<[f32; 6], String> {
-        let mut data = &buf[..];
-        let mut ch: [f32; 6] = [0.0; 6];
-
-        for i in 0..6 {
-            ch[i] = match data.read_f32::<LittleEndian>() {
-                Ok(val) => val,
-                Err(msg) => { 
-                    return Err(format!("Error parsing file: failed parsing float32 data: {}", msg)); }
-            }
-        }
-
-        return Ok(ch);
-    }
-
-    fn parse_data_u16(buf: Vec<u8>) -> Result<[u16; 6], String> {
-        let mut data = &buf[..];
-        let mut ch: [u16; 6] = [0; 6];
-
-        for i in 0..6 {
-            ch[i] = match data.read_u16::<LittleEndian>() {
-                Ok(val) => val,
-                Err(msg) => { 
-                    return Err(format!("Error parsing file: failed parsing uint16 data: {}", msg)); }
-            }
-        }
-
-        return Ok(ch);
-    }
-
-    fn parse_filename(filename_full: String) -> Result<AsotrDataDesc, String> {
-        let mut filename = String::new();
-
-        let msg_prev = format!("Error parsing filename {}:", filename_full);
-
-        match filename_full.rfind('/') {
-            Some(val) => { filename = (filename_full[val+1..filename_full.len()]).to_string(); }
-            _ => { filename = filename_full.clone();  }
-        }
-
-        if filename.len() != 32 { 
-            return Err(format!("{} unsupported file", msg_prev)); 
-        }
-
-        let time_unix_ = filename[0..10].parse::<u64>();
-        let time_unix = match &time_unix_ {
-            Ok(data) => data,
-            Err(msg) => { 
-                return Err(format!("{} expected digits in timestamp sec part ({})", 
-                                   msg_prev, msg));  
-            }
-        };
-
-        let data_type_ = filename[22..24].parse::<u8>();
-        let data_type_u8 = match &data_type_ {
-            Ok(data) => data,
-            Err(msg) => { 
-                return Err(format!("{} expected digits in data type part ({})", 
-                                   msg_prev, msg));  
-            }
-        };
-
-        if *data_type_u8 == 1 || *data_type_u8 == 2 || *data_type_u8 == 6 { }
-        else {
-            return Err(format!("{} parser supports data types: {}", 
-                               msg_prev, support_dtypes.to_string()));
-        }
-
-        let data_type = match AsotrDataType::from_repr(*data_type_u8 as usize) {
-            Some(value) => value,
-            _ => return Err(format!("{} expected digits in data type part", 
-                                    msg_prev))
-        };
-
-        let _kit = filename[30..32].parse::<u8>();
-        let kit = match &_kit {
-            Ok(data) => data,
-            Err(msg) => { return Err(format!("{}: expected digits in asotr kit part ({})", 
-                                             msg_prev, msg));  }
-        };
-
-        let _time_str_mks = filename[11..17].parse::<u32>();
-        let time_mks = match &_time_str_mks {
-            Ok(data) => data,
-            Err(msg) => { return Err(format!("{}: expected digits in timestamp mks part ({})", 
-                                             msg_prev, msg));  }
-        };
-
-        let time: SystemTime = UNIX_EPOCH + Duration::from_secs(*time_unix);
-        let date_time = DateTime::<Utc>::from(time);
-        let date_s = date_time.format("%d.%m.%Y").to_string();
-        let time_s = date_time.format("%H:%M:%S").to_string();
-
-        let head = AsotrDataDesc::new(*time_unix, *time_mks, 
-                                      format!("{} {}", date_s, time_s),
-                                      date_s, time_s, data_type, *kit);
-
-        return Ok(head);
-    }
 
     pub fn read_data(filename_full: String) -> Result<String, String> {
         let ch_u16: [u16; 6];
@@ -185,11 +101,170 @@ pub mod asotr_data {
         }
 
         out.remove(out.len() - 1);
-
         return Ok(out);
     }
+ 
+    pub fn parse_data_dir(dir: &str) -> Result<(), String> {
+        let mut data: Vec<String> = Vec::new();
+        
+        println!("parse data from directory: {}", dir);
+        for (pattern, fname) in patterns_fnames_csv_data.iter() {
+            let files = find_files_regex(dir, pattern)?;
 
+            for elem in files {
+                data.push(read_data(elem)?);
+            }
 
+            data.sort();
+            
+            println!("save csv data to file: {}", fname);
+
+            save_data_csv(data.clone(), fname)?;
+            data.clear();
+        }
+        
+        return Ok(());
+    }
+
+    fn parse_data_f32(buf: Vec<u8>) -> Result<[f32; 6], String> {
+        let mut data = &buf[..];
+        let mut ch: [f32; 6] = [0.0; 6];
+
+        for i in 0..6 {
+            ch[i] = match data.read_f32::<LittleEndian>() {
+                Ok(val) => val,
+                Err(msg) => { 
+                    return Err(format!(
+                            "Error parsing file: failed parsing float32 data: {}", msg)); }
+            }
+        }
+
+        return Ok(ch);
+    }
+
+    fn parse_data_u16(buf: Vec<u8>) -> Result<[u16; 6], String> {
+        let mut data = &buf[..];
+        let mut ch: [u16; 6] = [0; 6];
+
+        for i in 0..6 {
+            ch[i] = match data.read_u16::<LittleEndian>() {
+                Ok(val) => val,
+                Err(msg) => { 
+                    return Err(format!(
+                            "Error parsing file: failed parsing uint16 data: {}", msg)); }
+            }
+        }
+
+        return Ok(ch);
+    }
+
+    fn parse_filename(filename_full: String) -> Result<AsotrDataDesc, String> {
+        let mut filename = String::new();
+        let msg_prev = format!("Error parsing filename {}:", filename_full);
+
+        match filename_full.rfind('/') {
+            Some(val) => { filename = (filename_full[val+1..filename_full.len()]).to_string(); }
+            _ => { filename = filename_full.clone();  }
+        }
+
+        if filename.len() != 32 { 
+            return Err(format!("{} unsupported file", msg_prev)); 
+        }
+
+        let time_unix_ = filename[0..10].parse::<u64>();
+        let time_unix = match &time_unix_ {
+            Ok(data) => data,
+            Err(msg) => { 
+                return Err(format!("{} expected digits in timestamp sec part ({})", 
+                                   msg_prev, msg));  
+            }
+        };
+
+        let data_type_ = filename[22..24].parse::<u8>();
+        let data_type_u8 = match &data_type_ {
+            Ok(data) => data,
+            Err(msg) => { 
+                return Err(format!("{} expected digits in data type part ({})", 
+                                   msg_prev, msg));  
+            }
+        };
+
+        if *data_type_u8 == 1 || *data_type_u8 == 2 || *data_type_u8 == 6 { }
+        else {
+            return Err(format!("{} parser supports data types: {}", 
+                               msg_prev, support_dtypes.to_string()));
+        }
+
+        let data_type = match AsotrDataType::from_repr(*data_type_u8 as usize) {
+            Some(value) => value,
+            _ => return Err(format!("{} expected digits in data type part", 
+                                    msg_prev))
+        };
+
+        // let _kit = filename[30..32].parse::<u8>();
+        // let kit = match &_kit {
+        //     Ok(data) => data,
+        //     Err(msg) => { return Err(format!("{}: expected digits in asotr kit part ({})", 
+        //                                      msg_prev, msg));  }
+        // };
+
+        // let _time_str_mks = filename[11..17].parse::<u32>();
+        // let time_mks = match &_time_str_mks {
+        //     Ok(data) => data,
+        //     Err(msg) => { return Err(format!("{}: expected digits in timestamp mks part ({})", 
+        //                                      msg_prev, msg));  }
+        // };
+
+        let time: SystemTime = UNIX_EPOCH + Duration::from_secs(*time_unix);
+        let date_time = DateTime::<Utc>::from(time);
+        let date_s = date_time.format("%d.%m.%Y").to_string();
+        let time_s = date_time.format("%H:%M:%S").to_string();
+
+        let head = AsotrDataDesc::new(*time_unix, date_s, time_s, data_type);
+
+        return Ok(head);
+    }
+
+    fn find_files_regex(dir: &str, template: &str) -> Result<Vec<String>, String> {
+        let mut path_vec: Vec<String> = Vec::new();
+
+        let regex = match Regex::new(template) {
+            Ok(val) => val,
+            Err(msg) => { 
+                return Err(format!("Error create regex template ({}): {}",
+                template, msg)); 
+            }
+        };
+
+        let data = WalkDir::new(dir)
+            .into_iter()
+            .filter_map(|e| e.ok())
+            .filter(|e| regex.is_match(e.file_name().to_str().unwrap()))
+            .map(|e| e.into_path());
+
+        for elem in data {
+            path_vec.push(elem.display().to_string());
+        }
+
+        if path_vec.len() == 0 { 
+            return Err(format!(
+                    "Error searching files for pattern ({}): files not found in directory {}", 
+                    template, dir));  
+        }
+
+        return Ok(path_vec);
+    }
+
+    fn save_data_csv(data: Vec<String>, fname: &str) -> Result<(), String> {
+        match std::fs::write(fname, data.join("\n")){
+            Ok(f) => f,
+            Err(msg) => { 
+                return Err(format!("Error write data to csv file {}: {}", fname, msg)) 
+            }
+        }
+
+        return Ok(());
+    }
 }
 
 
@@ -204,22 +279,41 @@ fn main() {
             Arg::new("file")
             .short('f')
             .long("filename")
-            .help(format!("input ASOTR MVN data ({})", support_dtypes.to_string()))
+            .help(format!("file with ASOTR MVN data ({})", support_dtypes.to_string()))
-            .required(true),
+            )
+        .arg(
+            Arg::new("dir")
+            .short('d')
+            .long("directory")
+            .help(format!("directory with ASOTR MVN data"))
             )
         .get_matches();
 
-    let filename: &String = match arguments.get_one::<String>("file") {
+    match arguments.get_one::<String>("file") {
-        Some(fname) => fname,
+        Some(fname) => {
-        _ => { println!("Error: wrong argument!"); return;  }
+            let s = match read_data(fname.clone()) {
+                Ok(elem) => elem,
+                Err(msg) => { println!("{}", msg); return; } 
+            };
+
+            println!("{}", s);
+            return;
+        },
+        _ => { }
     };
 
-    let s = match read_data(filename.clone()) {
+    match arguments.get_one::<String>("dir") {
-        Ok(elem) => elem,
+        Some(path) => {
-        Err(msg) => { println!("{}", msg); return; }
+            match parse_data_dir(path) {
-    };
+                Ok(_) => {}
+                Err(msg) => { println!("{}", msg); return; } 
+            }
+            return;
+        },
+        _ => { }
+    }
 
-    println!("{}", s);
+    println!("Unexpected command. Type --help for more iformation");
 }