IR_data_access issueshttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues2020-12-05T19:30:01Zhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/21IR.download_heatflux_scene_model_reference and IR.download_scene_model give d...2020-12-05T19:30:01ZDavid BoldIR.download_heatflux_scene_model_reference and IR.download_scene_model give different infoI noticed that the infos from `Pixel_X`/`Pixel_Y` from `IR.download_heatflux_scene_model_reference` do not match the `CAD` plot from `IR.download_scene_model`.
There is a misalignement of a few pixels for the positions of the different ...I noticed that the infos from `Pixel_X`/`Pixel_Y` from `IR.download_heatflux_scene_model_reference` do not match the `CAD` plot from `IR.download_scene_model`.
There is a misalignement of a few pixels for the positions of the different fingers.
Here is a plot of `"CAD"` with Finger 135 set to `0` that shows the misalignment:
![finger_135_overlay](/uploads/4236e603aff2ac411f4826ae1f5a7a57/finger_135_overlay.png)
It does not really affect me, I am just worried that that might impact the analysis, and thus gives wrong results.
This is not an issue with the access routine, but the underlying data, but I am not sure where else to report.Yu GaoYu Gaohttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/20raise errors2020-08-24T13:44:08ZDavid Boldraise errorsIf an error occurs, this is hidden by `try-catch` statements. I could write a wrapper to checks the return code, and raises an error, but this is slower, uglier and I lose what error was initially raised, I want to treat a network issue ...If an error occurs, this is hidden by `try-catch` statements. I could write a wrapper to checks the return code, and raises an error, but this is slower, uglier and I lose what error was initially raised, I want to treat a network issue potentially different then `SIGINT`.
It seems #14 requested the opposite, so I am not sure this change would be welcomed, but I think it is much easier to wrap calls in `try-except` statements, rather then the other way around. I didn't quite understand the use-case of #14 why the call couldn't be wrapped in a `try-except` statement on the caller side.https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/19Can't extract profile from temperature images because of invalid indices in s...2020-07-31T09:23:06ZLukas RudischhauserCan't extract profile from temperature images because of invalid indices in sceneThe images returned by dlir.download_heatflux_by_program for AEF51 have shape 325,1300 (tried for multiple discharges).
The location indices locs for profile 'TM3h_5_5' when applied to the scene from download_heatflux_scene_model_referen...The images returned by dlir.download_heatflux_by_program for AEF51 have shape 325,1300 (tried for multiple discharges).
The location indices locs for profile 'TM3h_5_5' when applied to the scene from download_heatflux_scene_model_reference('AEF51') give X and Y pixel coordinates both greater than 325, resulting in an invalid index error.https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/18default reference time in extract_temperature_profile_from_DL causes download...2020-08-06T13:37:53ZLukas Rudischhauserdefault reference time in extract_temperature_profile_from_DL causes download_heatflux_scene_model_reference to failthe default for reference_time = 0 in extract_temperature_profile_from_DL, which calls download_heatflux_scene_model_reference with timepoint = reference_time causes this function to fail because it checks if timepoint == None. Since thi...the default for reference_time = 0 in extract_temperature_profile_from_DL, which calls download_heatflux_scene_model_reference with timepoint = reference_time causes this function to fail because it checks if timepoint == None. Since this isn't the case, it tries to load the parlog version of nanosecond 0, which is None. None is then parsed as string and put in the url that is called, which obviously fails.
This can be fixed immediately by making the reference_time default = None in extract_temperature_profile_from_DL.
But fundamentally it shows a number of problems:
- check against boolean, not specifically 0, None, len==0, or False to avoid this confusion.
- get_latest_version should return only ever return an int. this saves many checks down the line.
- Similarly if download_heatflux_scene_model_reference encounters an error, why bother with printing it and returning False, {}? Unless we have an alternative cached version or something we are stuck anyway. Instead of checking success further down the line we could have just let the code fail and get the error and stacktrace.https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/17Caching implementation does not work as intended2020-08-06T13:38:56ZLukas RudischhauserCaching implementation does not work as intendedcheck_path_and_make_folders goes into an infinite loop. I suspect this is because os.path.join(os.path.dirname(__file__), 'cache') does not produce a string and the *path.rpartition('/') therefore does not work as intended. The string se...check_path_and_make_folders goes into an infinite loop. I suspect this is because os.path.join(os.path.dirname(__file__), 'cache') does not produce a string and the *path.rpartition('/') therefore does not work as intended. The string seperation will also not work on other operating systems using a different path seperator.
I suggest three changes, in decreasing order of importance:
1. Use os library to handle the traversing of levels in check_path_and_make_folders.
2. Move the construction of the cache filename (from data_folder, save_path and formatted filename) into a seperate function such that there is no duplicate code in extract_heatflux_profile_from_cache and cache_heatflux_by_program.
3. Move options such as cache_folder into a user_settings file each user can modify and load from there.
(4. Make it possible to load already downloaded, longer time window cache files and overwrite shorter time window cache files. Either build by string comparison or include lightweight database.)https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/15Installation issues2020-06-23T11:10:22ZDavid BoldInstallation issuesI tried to install it, and after fixing !47 it fails to find w7xarchive
Where can I find that / get access?I tried to install it, and after fixing !47 it fails to find w7xarchive
Where can I find that / get access?https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/14Lack of exposure time breaks program2020-08-12T08:42:04ZGeorg SchlisioLack of exposure time breaks programWhen running `get_temp_from_raw_by_program` on a program where the script does not find an exposure time, it breaks by raising Warning.
Expected behaviour would be to issue a warning, but returning success=False (as in the commented line...When running `get_temp_from_raw_by_program` on a program where the script does not find an exposure time, it breaks by raising Warning.
Expected behaviour would be to issue a warning, but returning success=False (as in the commented line 2526).
Traceback:
`Traceback (most recent call last):
File "…", line 379, in <module>
status, times, temperatureimages, valid = IR.get_temp_from_raw_by_program("AEF11", lib.format_pid_with_zero(pid), time_window=1, verbose=3, threads=2)
File "…lib\site-packages\IR_data_access\downloadversionIRdata.py", line 2025, in get_temp_from_raw_by_program
cexist, background, LUT,refT, gain, offset, badpixels, t_exp, cfilter, gain_error, offset_error = get_calib_data(portnr, t0=t0, t1=t1, t6=t6, emissivity=emi, T_version=T_version, version=version, back_emissivity=back_emissivity, testmode=testmode, verbose=verbose-1)
File "…lib\site-packages\IR_data_access\downloadversionIRdata.py", line 2525, in get_calib_data
raise Warning("get_calib_data: no exposure time found")
Warning: get_calib_data: no exposure time found`https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/13download_images_by_times create double entries2020-02-19T09:41:16ZHolger Niemanndownload_images_by_times create double entriesdownload_images_by_times create double entries if more than 1s is requested.
The double entries appear in the time vector and the image vector. It is affecting the download_heatflux routines.download_images_by_times create double entries if more than 1s is requested.
The double entries appear in the time vector and the image vector. It is affecting the download_heatflux routines.Holger NiemannHolger Niemannhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/12add wetted area download from archive2020-07-09T15:59:33ZHolger Niemannadd wetted area download from archiveafter implement the upload of the wetted area into the uploader scripts, also add the needed function to the code in addition to the existing function in IR_toolsafter implement the upload of the wetted area into the uploader scripts, also add the needed function to the code in addition to the existing function in IR_toolsHolger NiemannHolger Niemannhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/11Add download of peak heat fluxes2020-07-09T15:59:19ZHolger NiemannAdd download of peak heat fluxesAdd peak heat fluxes to the uploader scripts and implement here in the download section also the needed functionsAdd peak heat fluxes to the uploader scripts and implement here in the download section also the needed functionsHolger NiemannHolger Niemannhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/10change port request functionality to request all ports2020-06-17T14:48:09ZHolger Niemannchange port request functionality to request all portschange port request functionality to request all ports, especially for the auto calculation on the server.
additional parameter "process_all" or similar implementation possiblechange port request functionality to request all ports, especially for the auto calculation on the server.
additional parameter "process_all" or similar implementation possibleHolger NiemannHolger Niemannhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/9Add cache funktionality2020-06-17T14:48:47ZHolger NiemannAdd cache funktionalityadd chaching for heatflux and temperature data.
Lukas provided an example implementationadd chaching for heatflux and temperature data.
Lukas provided an example implementationHolger NiemannHolger Niemannhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/8Add/Improve documentation2020-10-02T08:15:22ZGeorg SchlisioAdd/Improve documentation* Have docstrings for all functions
* Have function overview in Gitlab Wiki
* Add Readme.md for project* Have docstrings for all functions
* Have function overview in Gitlab Wiki
* Add Readme.md for projecthttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/7urllib incorrectly imported2019-08-06T16:01:19ZAlexander Kniepsurllib incorrectly importedin downloadversionIRdata.py, the libraries urllib.request and urllib.error are used, but only urllib is imported. On Python 3.6.1 (latest from DSM), this fails to import the sub-libraries.in downloadversionIRdata.py, the libraries urllib.request and urllib.error are used, but only urllib is imported. On Python 3.6.1 (latest from DSM), this fails to import the sub-libraries.https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/6give_finger_ID does not (and cannot) work when finger is specified2019-08-06T16:01:51ZLukas Rudischhausergive_finger_ID does not (and cannot) work when finger is specified```
from IR_data_access import downloadversionIRdata as dlir
dlir.give_finger_ID("TM3h_5_5") -> (True, 1805)
dlir.give_finger_ID("TM3h_5",5) -> (False, -1)
```
It is clear this will happen because in the else branch (if finger is not N...```
from IR_data_access import downloadversionIRdata as dlir
dlir.give_finger_ID("TM3h_5_5") -> (True, 1805)
dlir.give_finger_ID("TM3h_5",5) -> (False, -1)
```
It is clear this will happen because in the else branch (if finger is not None) 'finger%100' has meaning both when finger is an int and when it's a string.
Also it would be great if there was some error generated that explained why the finger could not be found.https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/5Model data lacking Z information2019-05-15T11:42:58ZSamuel LazersonModel data lacking Z informationWhen I pull the model data for heatflux (`download_heatflux_mapping_reference`) I don't get a Z coordinate. Can this be added? Or am I misinterpreting what I'm pulling?When I pull the model data for heatflux (`download_heatflux_mapping_reference`) I don't get a Z coordinate. Can this be added? Or am I misinterpreting what I'm pulling?Holger NiemannHolger Niemannhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/4extract_heatflux_profile_from_DL always uses timepoint=12019-06-18T07:48:53ZLukas Rudischhauserextract_heatflux_profile_from_DL always uses timepoint=1as in title, this is problematic when the function is called from within another where the expected and desired behaviour is to return mutliple profiles.
Suggested change: Modify the default of timepoint such that the largest available ...as in title, this is problematic when the function is called from within another where the expected and desired behaviour is to return mutliple profiles.
Suggested change: Modify the default of timepoint such that the largest available range of data is returned.https://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/3port instead of portnr2018-11-19T15:26:47ZSamuel Lazersonport instead of portnrThere are various points in the download code where portnr should be used but port is used. And example would be line 1582 in downloadversionIRdata.py. This appears to also be true in the threaded version of the download routines.There are various points in the download code where portnr should be used but port is used. And example would be line 1582 in downloadversionIRdata.py. This appears to also be true in the threaded version of the download routines.Holger NiemannHolger Niemannhttps://gitlab.mpcdf.mpg.de/holn/IR_data_access/-/issues/2Bug in downloadversionIRdata.py2018-11-19T15:25:00ZSamuel LazersonBug in downloadversionIRdata.pyAt line 1299 the call to download_raw_images_by_program should not pass the argument verbose as the routine is not declared to accept that argument.
https://gitlab.mpcdf.mpg.de/holn/IR_data_access/blob/master/downloadversionIRdata.py#L1299At line 1299 the call to download_raw_images_by_program should not pass the argument verbose as the routine is not declared to accept that argument.
https://gitlab.mpcdf.mpg.de/holn/IR_data_access/blob/master/downloadversionIRdata.py#L1299Holger NiemannHolger Niemann