Oscar - export daily "details" needed - power of data analysis / correlation

[PaulNL]

Hello,

I'm Paul and have been a CPAP user now for some years. About two years ago my sleep quality was getting really bad so that I was even not able to work for many months. I tried to improve the situation and used Sleepyhead then to have a look at my data and trying to find correlations between several parameters like the time going to bed, eating in the evening late, weight, stress, state of my nose and more -- comparing to the overall AHI, but also to Cheyne-Stokes and Clear Airway and other relevant Sleeplyhead data.

For that I needed to extract the data on the daily detail tab by hand, because all those valuable informations cannot be exported (except the AHI itself). It took me many hours to type in all the numbers into a spreadsheet table and create plots to see if / where I could possibly find correlations -- in the hope to be able to improve my treatment and / or lifestyle. 

It is a big loss of the potential of the data in Oscar that the export does not allow to export "everything". It would not be such a large change in the code to export all the data available in Oscar, but it would tremendously improve the usefulness of the data.

I know the statistics options of Oscar, but they are and will always be too limited to cover the different personal needs. For example now it's not possible read out a moving average, which could be highly beneficial to understand the data and the overall trend better. I really miss to be able to use the data to a higher potential.

2 1/2 years ago, when I plotted so many different scatter plots, trying to find potential correlations, it got clear to me one day that the last months, while I lost a few kg (from 95 to 91kg) my sleep has gotten worse instead of better. This was not easily visible in the Sleepyhead / Oscar report, but the scatter plot showed very clear that there was a negative correlation of the weight loss to the increasing AHI value. So I thought that possibly my sleep has gotten worse (with higher AHI values) not despite myself loosing weight, but because of loosing weight. At that time my pressure was at 5.6. I decided to try to lower the pressure, because I already knew that my body can react sensitive on "too much" pressure (that was with values short of 8 when I noted that sensitivity, but also using EPR making my sleep much worse). So against the general recommendation of the doctor I just tried to lower the pressure to 4.8 / 5.  My average AHI values came down to the half immediately as soon I made the change and stayed there for over half a year. So that was -- at that time -- for sure a good decision. The doctor was surprised to see that the lower pressure worked better, but there was of course no discussion, because the results spoke for themself.

I'm know in a different phase of life, many things changed and unfortunately my AHI values are much higher again. I am now again trying to find out how I can improve my treatment and / or lifestyle. I also need to assess the influence of additional treatments like a small surgery to my nose. It is clearly seen that this had a significant impact on my sleep quality (AHI). Although my values are now jumping up and down, without myself being able to find the pattern yet. 

I again would need to have access to the detailed sleep data, but it is hard for me at that moment to find the time and energy to manually type in the detail data of more than 2 years into the spreadsheet. 

I wrote an e-mail to Arie about a year ago and asking for the option to export all the OSCAR data. Unfortunately he did not understand the huge potential benefit when the data can be used outside Oscar and therefore did not support my wish. I strongly hope that my wish can be made true -- hopefully in the coming 1.3 release already. 


I'm very computer savvy -- have programming experience (mostly Matlab, but also a little bit of Java - which is now rusty) and have beta tested several programs before. So I surely could also support in that way here as well. I'm also willing to create a documentation and / or video explaining the use the data and can provide some example Google or Excel spreadsheets which can demonstrate how to use the data in the search for potential correlations and how that can benefit to understand the situation better. I have worked in a research institute for around 8 years, where we often needed to find correlations between data from psycho-physical experiments to various measurement data. 

As a first quick step it can be helpful to create a scatter plot and plot a best-fit line to the data and determine the correlation coefficient. That's not really hard to do, but can often already tell quite a bit or help point to directions where to investigate further.

Looking forward to hear your input on these thoughts.


Best regards, Paul

[GuyScharf]

What data do you need that is not in the current File/Export Data feature? If there are data values missing, we can look at adding them.

Are you asking for the detailed flow data showing every variation in every breath? Are you wanting to correlate data within a day or between days?

If you need detailed data from within a day, there may be other ways (than OSCAR) to get that, such as directly from the EDF files produced by the CPAP machine.

[PaulNL]

Thanks for the quick answer. In general all data can be valuable. But especially the statistics which are shown on the Daily-Details tab are interesting and pretty easy accessible (Cheyne Stokes %, Large Leak, Clear Airway...., Machine Settings, Sessions.... But also the Daily-Events-tab data would could be useful to have.

I start by looking at the plots in Oscar (Daily or Statistics) and then from there I try to find possible patterns related to data -- mainly data outside of Oscar, like

* weight
* how stuffed was my nose in the morning (so likely also during the night), right side / left side / both sides of the nose?
* feeling well or not (in my case also headaches)
* determine trends which are hidden in the Statistics tab, like explained above, for example to be able to plot moving averages of AHI or other values, calculate median or other percentiles for own defined time-spans and so on...

but also being able to find trends (CA/AHI ratio is in my case interesting for example) is a big plus when having that data available in an export table.

So I was now mainly looking for ways to correlate data between days. But one thing I also would like to try to investigate is when in the night I stop breathing. I have the impression that there is a trend that this often is in the first half of the night (first one or two sleep cycles I guess). This is something I cannot quantify at the moment with the data which is exported, or can I?

One additional idea I had which could also be a nice "add-on" is using the feature of the S10 (or similar) that one can use a lower pressure to fall easier to sleep and the machine then switches to the target pressure as soon the machine detects regular breathing (which starts when one was fallen asleep). I personally do not need / use this feature to help me fall asleep, _but_ anyhow just set the pressure 0.2 below the "correct" pressure. Then OSCAR shows me that the pressure rose to the target pressure after some time. This time will very closely correlate to the time needed to fall asleep. I can now have a look on the plots each day and manually check how long it took to start sleeping. Would be great if OSCAR could report that time (for each session of course) automatically. People having problems to fall asleep can possibly get valuable insights from that. This time could also be used to correct the total sleep time.

[GuyScharf]

But one thing I also would like to try to investigate is when in the night I stop breathing. I have the impression that there is a trend that this often is in the first half of the night (first one or two sleep cycles I guess). This is something I cannot quantify at the moment with the data which is exported, or can I?

Have you examined File/Export and its Details option?

[PaulNL]

Have you examined File/Export and its Details option?

I did 2 1/2 years ago. It seems I can solve those questions with the detail export. I'll have a look into that. Will need some coding and not be a quick plot in a spreadsheet program. 

BTW, I also used the "normal" export function and added then to those exported tables the data I was missing by hand.

[PaulNL]

What data do you need that is not in the current File/Export Data feature? If there are data values missing, we can look at adding them.
[...]
If you need detailed data from within a day, there may be other ways (than OSCAR) to get that, such as directly from the EDF files produced by the CPAP machine.

I would like to have access / export of the Details-tab on in the Daily-tab first and foremost. Access to the data on the Events-tab (also on the Daily-tab) would be a plus.

When I understand it right the EDF data is the data from the machine, right? Is the information from the two mentioned tabs stored from the machine itself already and thus could be read out from the EDF file as well or are they calculated / generated from Oscar?

Currently there is indeed no chance to get the data I want from Oscar, correct? When exporting this data from Oscar is not wanted / possible right now, would it be an option to read out  the data from the files? Which files would those be? Could that be done relatively easy with a Python or whatever script?

[GuyScharf]

I would like to have access / export of the Details-tab on in the Daily-tab first and foremost. Access to the data on the Events-tab (also on the Daily-tab) would be a plus.

When I understand it right the EDF data is the data from the machine, right? Is the information from the two mentioned tabs stored from the machine itself already and thus could be read out from the EDF file as well or are they calculated / generated from Oscar?

Currently there is indeed no chance to get the data I want from Oscar, correct? When exporting this data from Oscar is not wanted / possible right now, would it be an option to read out  the data from the files? Which files would those be? Could that be done relatively easy with a Python or whatever script?

Did you look at the Details export CSV? Data from the Events tab is there, as well as pressure data and others.

Yes, the EDF files are what ResMed machines produce (other brands use different file formats reporting data). On ResMed, all detail comes from EDF files. Depending on the machine model and brand, some of the graph data may be calculated vs come from the CPAP's data files.

There is a program available on the web named EDF Browser. It allows you to examine EDF files and (I think) export them.

If you want to read OSCAR's data files, you would need to look at the OSCAR source code to see how they are organized. You can find a link to the source code on the OSCAR download page.

[PaulNL]

Hello Guy,

thanks for the answer.

Yes, I did look at the details report -- it did not seem to provide me the same information like the "Session-tab" does. I was looking for Clear-Airway, Obstructive and Hypopnea-Events but found a file full of mostly EPG-Events and also Pressure- and EPAP-Events. A few random looking Clear Airway or Obstructive events where also noted, but I did not see a pattern when those show up. So that file did not make sense to me - especially the thousands of "EPG-Events". BTW, the EPG-Events have mostly a value of zero or close to zero. 

After your question I have now taken a third look and compared an actual day and times to find that in -- what seemed to me as not useful data -- hidden there in all the thousands of "Non-Events" (Pressure, EPAP and FLG - whatever the last one means) I can find the real non-breathing events I was looking for indeed and that they seem to be complete. I guess I can just ignore the main bulk of data then. From that I can go on, combine the output with that of the summary and daily summary exports, and will have most of the information which I wanted to get access too. :-) Thanks!

Just Cheyne-Stokes "events" or better said the time which was marked as Cheyne-Stokes seem not to be in those export-files, but that is much less relevant for me than the other data.

When there is no description how to read Oscar files (and what they are), and I would need to dive into the source code of such a large project, it will not be reasonable for me to try to read the files with my current time, knowledge and energy constraints.

Regards, Paul

P.S.: Has no one so far made a program to combine the export-files, allowing to use them for data analysis? It seems that much relevant information is available already through Oscar, but somehow "hidden" and not easy enough accessible to be utilized by (likely) the majority of CPAP users!? I hope I can use the data in the way I want. If I succeed that could potentially also become a stepping stone for others to have a deeper look into their data, helping to optimize the therapy and / or lifestyle.

[SarcasticDave94]

I guess I've missed something. For me to view my detailed machine data, I've just viewed the Daily Details tab in OSCAR. And I'm using higher level Apnea devices than most others due to high Central Apnea and COPD. I find this level of data is enough information to Titrate the ASV when I had it, and then a short trial of ST-A.

What info is missing from your Daily Details tab from your CPAP usage?

[PaulNL]

Hi Dave,

the summary to your "question" is: my goal is finding mid- and long term trends and correlations to whatever to potentially learn what to do to improve my CPAP or medical treatment or lifestyle.

Whatever can be very personal. For me it is weight, bed-time, sleeping location, state of my nose, used pillow...

Oscar is already incredibly useful the way it is. What I am trying to achieve is to be able to get the data out of Oscar to allow me to take different views on the data -- which is not possible in Oscar itself. You can look at the Overview or Daily tab. Both can be very useful like they are. But my nights vary wildly. That makes it hard to find a pattern and potential influences on my sleeping quality. So even when using Oscar there is a chance one misses insights one could get when one can view data in a way that matches to the personal situation and also include external data (my whatever)

With that approach my first success was, that only by plotting the data and being able to plot trend lines as well, it got clear to me that what I thought was good (loosing weight) at the same time worked against me and was (with the unchanged, already quite low pressure of 5.6) the _cause_ for more apneas. My doctor would have never advised to lower the pressure even more, but it turned out to be the right thing (at that time -- which proved to be good for more than a year). But situations change...

Right now I have a situation I would like to "research". Most weeks I sleep one day not in my own bed, but in a hotel (the same btw). I have the impression that those nights I have more apneas. But it's hard to tell, because my values can jump from night to the next night -- and I also had good nights in the hotel and not only bad ones. When I have the data outside Oscar I can create a file containing the hotel-night-dates and compare that to the average (or likely better the median) of 6 nights (3 before and after the hotel night). By that I have a real chance to see over many weeks if my impression that I sleep less good in that hotel can be quantified. When this suspicion would be confirmed I can try to find out what causes the worse sleep. I guess for example the influence of pillow and mattress could be an explanation. This is knowledge I can then try to make use of.

I am as well struggling with a stuffy nose -- sometimes the nose itself and sometimes more "inside" where a nasal douche does not really help. I can quantify the stuffiness and the location of the stuffiness of my nose and also correlate that to the Oscar numbers and see if / where a correlation is. Again information which could be useful to optimize my CPAP or medical treatment. I see a strong change in my sleeping quality now after a nose surgery. Unfortunately not only to the better, but the variations between the nights now seem to be even larger. Again hard to quantify by looking at the statistics tab alone, but easy when I have access to the data itself and can calculate quantiles, moving averages, standard deviation and so on.

Those are two examples why access to the data can be helpful. And when one would have a table (CSV, Excel) with the exported Details and Session-tab data (best without "data gibberish") it's pretty straight-forward to include personal data and create some scatter plots and include some trend lines. 

P.S.: 
When you're familiar with Excel: having the data available in a table allows to "play" with it, like you can create quickly many different views on the data, in an pivot-table for example.