Hello Guest, Welcome to Apnea Board !
As a guest, you are limited to certain areas of the board and there are some features you can't use.
To post a message, you must create a free account using a valid email address.

or Create an Account


New Posts   Today's Posts

Dreem 2 to OSCAR importing help request
#11
RE: Dreem 2 to OSCAR importing help request
Thank you James,

I am using W10 and found that OSCAR 1.1.0 would not work either (along with the 1.2.0 versions, whether 32 or 64 bit or on either of two 64-bit laptops). Will try fresh OSCAR and 1.1.0 and profiles once more tomorrow, before moving on. 

I'm at a loss now, can't think of anything else to try at this stage after weeks of messing with it changing one suspect variable/item at a time. I can just do a straightforward Excel spreadsheet, to convert the Dreem data, as done for the 3-axis accelerometer. Mystified, I'd be curious to know if any other Dreem user has downloaded the current version and been successful with it. 

The repeated finding (er, not finding) still is  OSCAR/W10 not finding (or not making) that same 001 file series just as is mentioned above and as should be found in the Events subfolder folder of the Dreem profile folder. A separate Events subfolder is never created but debug lists the Events subfolder that is strangely in the same path as the Summary subfolder. The summary files are created and are in their subfolder--giving off, I suspect, the false message that Dreem has been imported or that Dreem is already aboard if you haven't purged Dreem from the previous failed trial.

A search for the "file not found" file  (file with 001 ending) comes up blank, but the similar named file (000 ending) is found and is in the Summaries, as mentioned. 

I did get one variation of debug error for one of the trials. The debug listed the date and time part of the A column cell for each session listed and it reported a warning that included a UTC time (the same -8 or -28800 seconds). After checking my profile time zone I saw I had the right -8 GMT zone, but had not selected the correct country. After that, I still got the usual problem. But it reminded me that for the first few nights that the Import worked for me (before the time change) the session dates strangely had a calendar date one day earlier than the earlier of the two dates OSCAR always displays in the sidebar (that OSCAR date for the PM when more sensible sleepers go to bed and the corresponding date the next day AM). Note: I usually go to bed around 0200, the "next day". Upshot: I tried yet a day earlier and later than I knew to be correct: no help.

Just to touch on one related matter and leave a crumb of it along the tortuous trail, too:  one of my sleep sessions had about a 12 minute time difference between the AirCurve and Dreem 2 device startup times: Dreem started close to 12 minutes late. To closely synchronize startups I inserted and added that much WAKE time (24 row-columns full of 30 sec intervals) at the B column of that data row and added that amount to the Sleep Onset time.  That made no difference for my missing file problem and, in the absence of waveform creation calcs taking place, it is not a sure indication the technique can be applied to compensate inadvertent startup time differences or synchronize that way. This raises the question of whether calcs are run in the Import execution that will balk or fail if entries don't add up precisely. It may be that text is just text at that point and its a matter of time staying within range and limits of formatting.

I'll revisit the OSCAR 1.1.0 one more time tomorrow. If that and a sleep-cleared mind won't cut through it all tomorrow, its back to Excel as I await the next tweak of OSCAR. I'm not saying its relevant here, but the time-synch related troubles some people are having with other files not loading sure underscores the struggles many of us have with date and time handling of computers. Time interference is a booby trap in Excel too. Often the expedient thing is to use variations of  yyyymmddd and hhmmss format and never use Excel's compulsive standard date and time formatting until you must when all work is nearly finished. Excel will make a date or time out of anything that looks like it might be intended.
I have no particular qualifications or expertise with respect to the apnea/cpap/sleep related content of my posts beyond my own user experiences and what I've learned from others on this site. Each of us bears the burden of evaluating the validity and applicability of what we read here before acting on it.  

Of my 3 once-needed, helpful, and adjunctive devices I have listed, only the accelerometer remains operative (but now idle). My second CMS50I died, too, of old age and the so-so Dreem 2 needs head-positioning band repair--if, indeed, Dreem even supports use of it now.



 
Post Reply Post Reply
#12
RE: Dreem 2 to OSCAR importing help request
I am interested at looking at your problem.  If you don't mind providing your Dreem data, I will see if I can be of some help.  Based on my understanding, there isn't any personal information attached to the raw data.  

Here is my Dropbox upload folder, Red's Upload Folder

- Red
Crimson Nape
Apnea Board Moderator
www.ApneaBoard.com
___________________________________
Useful Links -or- When All Else Fails:
The Guide to Understanding OSCAR
OSCAR Chart Organization
Attaching Images and Files on Apnea Board
Apnea Helpful Tips

INFORMATION ON APNEA BOARD FORUMS OR ON APNEABOARD.COM SHOULD NOT BE CONSIDERED AS MEDICAL ADVICE. ALWAYS SEEK THE ADVICE OF A PHYSICIAN BEFORE SEEKING TREATMENT FOR MEDICAL CONDITIONS, INCLUDING SLEEP APNEA. INFORMATION POSTED ON THE APNEA BOARD WEB SITE AND FORUMS ARE PERSONAL OPINION ONLY AND NOT NECESSARILY A STATEMENT OF FACT.
Post Reply Post Reply
#13
RE: Dreem 2 to OSCAR importing help request
I just finished looking at importing your Dreem data, without incident.  Not knowing what should be reported with a Dreem, all I'm receiving is "Sleep Stage".  

Your data was downloaded to a newly created directory, named Dreem, that I located in my C:\Users\CN\Downloads\ directory.   I then created a new account named "Dreem Tester" in OSCAR and then imported my CPAP data and then your Dreem data.  I didn't experience any problems.   My system is a Windows 7 Pro.

EDIT:
If I use the original file it will import but just list the Sleep Stages.  If I use the Data ⇾ Advanced ⇾ Purge All Machine Data ⇾ Dreem, I can then reimport it.  Again only the Sleep Stages appear.  I now have noticed that the more important parameters are delimited with a semicolon and all the informational parameters seem to be delimited with commas housed within square brackets ("[]").  I'm continuing to look.
Crimson Nape
Apnea Board Moderator
www.ApneaBoard.com
___________________________________
Useful Links -or- When All Else Fails:
The Guide to Understanding OSCAR
OSCAR Chart Organization
Attaching Images and Files on Apnea Board
Apnea Helpful Tips

INFORMATION ON APNEA BOARD FORUMS OR ON APNEABOARD.COM SHOULD NOT BE CONSIDERED AS MEDICAL ADVICE. ALWAYS SEEK THE ADVICE OF A PHYSICIAN BEFORE SEEKING TREATMENT FOR MEDICAL CONDITIONS, INCLUDING SLEEP APNEA. INFORMATION POSTED ON THE APNEA BOARD WEB SITE AND FORUMS ARE PERSONAL OPINION ONLY AND NOT NECESSARILY A STATEMENT OF FACT.
Post Reply Post Reply
#14
RE: Dreem 2 to OSCAR importing help request
Red ; as a delimiter in a CSV is often used in countries in europe, I wonder if the locale of the windows machine affects the CSV output ? Some locales have a different separator eg semicolon rather than comma
Post Reply Post Reply
#15
RE: Dreem 2 to OSCAR importing help request
Good thought, jaswilliams!

When looking at your data in a spreadsheet, I have noticed that the 18th entry with a time span of, 2020-11-07T01:00:03-08:00 to 2020-11-07T06:30:00-08:00, does not produce a new row. It continues from the previous entry. This is the only entry to do this. Running this file through a hex viewer shows that the preceding entry (the 17th) is missing an 0x0A character (Newline) to designate the end of an entry.

Basically your file is corrupted.
Crimson Nape
Apnea Board Moderator
www.ApneaBoard.com
___________________________________
Useful Links -or- When All Else Fails:
The Guide to Understanding OSCAR
OSCAR Chart Organization
Attaching Images and Files on Apnea Board
Apnea Helpful Tips

INFORMATION ON APNEA BOARD FORUMS OR ON APNEABOARD.COM SHOULD NOT BE CONSIDERED AS MEDICAL ADVICE. ALWAYS SEEK THE ADVICE OF A PHYSICIAN BEFORE SEEKING TREATMENT FOR MEDICAL CONDITIONS, INCLUDING SLEEP APNEA. INFORMATION POSTED ON THE APNEA BOARD WEB SITE AND FORUMS ARE PERSONAL OPINION ONLY AND NOT NECESSARILY A STATEMENT OF FACT.
Post Reply Post Reply
#16
RE: Dreem 2 to OSCAR importing help request
Crimson Nape and jameswilson,

Thank you for fast responsive answers. I just read them and have been looking at them. With word about the missing end-of-row indicator I, ham-fisted probably, simply tried copying the first "WAKE]" cell ending into endings of a block of four data rows and then attempted an import, with the same failed result. I suppose it is possible I or my systems began corrupting the data files.

In prepping to send what I did to you, CN, I parsed and compared earliest and latest data sheet column headers and found no differences in column headings in that check.

From the initial to most recent Dreem 2 (D2) CSV data sheet the ";" is the header row delimiter and, from your comments and as I recall other experience, Excel does/may convert "," delimited data in CSV files by parsing the individual data string value-comma pairs into separate single columns. I saw it automatically repeat a parse of a ";"separated string after I had parsed once earlier in the same workbook-even though I did not use the Excel tool to parse that second time. 

I believe that, aside from the header row, only start and end date and time and the listing of 30-second stage values is all that is now importable. The first cell of the row does have other sleep, heart, position, etc. metric summary values. (Re parsing: that row and the contents of the A-column of a data row were the two items I parsed using the Excel tool the first time, but not the second time when it was done automatically.)

This excerpt from a OSCAR debug  .txt log of debug notes may reveal either the missing end of line or row character or some other problem. What might the repeated string "@  0 unknown sleep stage" indicate to you? This arose in one trial import cycle, but is new to my eye.

   

RELATED BUT OFF-POINT DREEM QUESTION:

While on the Dreem topic and optimistic you tech adept members will put your finger on my Dreem Import problem,  I continue trying to learn how to assess and exploit whatever the D2 sleep sensor offers. To that end  and in passing:
  • Can you or anyone else answer whether OSCAR's import execution does anything with a sleep stage-name-variable value (e.g., WAKE, Light, REM, DEEP or their equivalents 1, 3, 2, 4) other than display each one as an inert datum, versus it being utilized as an independent variable in some calculation? Knowing that could eliminate need for blind trials to find out.


  • It would be helpful if, for present uses, I could add or remove from a sleep session data row's beginning either "WAKE" or "N/A" values for the purpose of synchronizing the D2 data  with AirCurve data logged time. Along with +/- padding adjustments I would adjust the start time of the D2 accordingly. The D2 data export's summary of WAKE time would likely be affected, but I wouldn't care, even if the sleep efficiency percentage were affected. (I've not used the clock drift compensator facility in the OSCAR Preferences, but have seen there may be problems with it. In my earliest use of the D2 there was an extended period when it was recording WAKE time before my start of the AirCurve. In any case, a synch must be done for each sleep session unless crude results are OK. The D2 has accurate time, but not my other 3 logger devices.


  • It's important to me because I'm more and more tardy following up in another thread I started some time ago about our learning how to extract useful info from (those unflagged) flow limit deformed peaks of inspiratory flow rate waves. It's important to do the most accurate time synchronizations I can attain for AirCurve, accelerometer, D2 and oximeter: on one hand, look at timing of the first body-shift motion waves vs. the rise of the first waves of bursts in FR that are arousal (WAKING) suspects; on the other hand, look at the described preceding relationship vs.  the timing of a  sleep stage change from sleep to WAKE or to a lighter sleep stage. 

    I want to see if there is any significantly regular pattern that can discerned, night to night. Unfortunately, as I knew before buying, the D2 is extremely limited; it will not detect sleep stage embedded micro-arousals ("MA") [<30 sec., I think, any of those arousals being equal to, or a multiple of, the width of a D2 sample window]. For the always tired UARS sufferers with low AHI, the D2 would offer little benefit because it would mask MA. But there is hope of it revealing at least a strongly suggestive association of sleep stage degradations following closely after a MA. As I understand, WatchPAT--the 200 model is $2700--detection of arousals is second to competently interpreted EEGs, but I'm hoping one of us will be able to discover a useful way for us to score and recognize our arousals using our lower cost tools and, after a time, trained eyes.  

Thank you again for your help. 

2SB
I have no particular qualifications or expertise with respect to the apnea/cpap/sleep related content of my posts beyond my own user experiences and what I've learned from others on this site. Each of us bears the burden of evaluating the validity and applicability of what we read here before acting on it.  

Of my 3 once-needed, helpful, and adjunctive devices I have listed, only the accelerometer remains operative (but now idle). My second CMS50I died, too, of old age and the so-so Dreem 2 needs head-positioning band repair--if, indeed, Dreem even supports use of it now.



 
Post Reply Post Reply
#17
RE: Dreem 2 to OSCAR importing help request
Bingo! Finally a successful import. I decided to go to true CSV, not use the "proactive", special/"helpful" parsing of Excel--Excel 10 here--version of CSV. I simply opened the raw Dreem export file that came from Dreem using Notepad++, saved that file and then imported it with my most recent seven nights of data into OSCAR. All without a hitch. 

Crimson Nape and jameswilliam's helpful bits about end of row markers and CSV, etc., somehow reminded me eventually of Gulf Data Concepts warning not to do editing of their accelerometer's config and time clock setting using Notepad because of an end of row (or line) marker peculiarity: their users are to use WordPad or Notepad. I don't remember if Notepad lacked or added an item Notepad++ and Wordpad handled differently.

In hindsight, it makes sense that dates and times would be peeled off data rows, as they were, with OSCAR loading going fine until its loader encountered the very first sleep value datum which was presented to OSCAR in the special Excel columnar-parsed CSV format (it was a columnar parse versus the ","-comma delimited parse format) . The attachment in my previous post shows that OSCAR immediately saw an unrecognizable sleep stage code in the very first cell of sleep stage data for each sleep session being imported. 

The imported (OSCAR-accepted) dates and time strings would trigger, for each sleep session, an entry into the Dreem profile Summaries folder and give false indication of a successful import.  

OSCAR's idiopathic balk, after working a number of times for me, is puzzling. I wonder if CSV files from the Open Office version of Excel would be treated similarly by OSCAR.

Since it might help someone else, I'll wave more yellow flags re quirks of all CSV files, although memory of this matter has faded some: 

I can now remember, and should have earlier, getting Open Office because of some earlier problem with Excel CSV and finding that there are several versions of CSV files. In learning to use Somnopose for position and motion display, there was a requirement for 2 decimal point format for data values. It made me crazy for a time as I would paste 2-digit things into the Excel CSV file for OSCAR import only to see "1.00" turn into "1" on screen. I had no end of trouble with sometimes-required ".00" values for a time and just began to force ".01" values which would present no problem. Eventually I learned how to get the job done, only to find out late in the game that the digit absences I saw were not passed on from the CSV file to the OSCAR importer; the latter would "see" actual table structure compliance with the requirement for 2-digit accuracy.

Thank you again, CN and James.

2SB
I have no particular qualifications or expertise with respect to the apnea/cpap/sleep related content of my posts beyond my own user experiences and what I've learned from others on this site. Each of us bears the burden of evaluating the validity and applicability of what we read here before acting on it.  

Of my 3 once-needed, helpful, and adjunctive devices I have listed, only the accelerometer remains operative (but now idle). My second CMS50I died, too, of old age and the so-so Dreem 2 needs head-positioning band repair--if, indeed, Dreem even supports use of it now.



 
Post Reply Post Reply
#18
RE: Dreem 2 to OSCAR importing help request
Mystery solved.


CSV files from some sources--possibly even an Excel spreadsheet program--are sometimes modified in an invisible and troubling way by Excel's mere opening of them after their first save. Key points reporting on that and a workaround for one trouble with them follows in the bold font below. The attachment shows the appearance of a typical Dreem 2 data sheet. It is unchanged to the eye upon being created and saved and then being opened afterward. But the latter CSV differs from the first CSV in some hidden way. I have looked for an Excel preference or option setting that would prevent the problem or switch off "friendly" help. Did not find one.


As requested, here is an attachment with an excerpt from a Dreem 2 CSV data export. It is typical in all respects, but a lot of the original actual data has been deleted because it is redundant here where only key elements need to be illustrated. The sheet was opened by Excel 2010. It was a long struggle trying to repeat my early successes that soon turned to sudden unexplained failure in my uploading of Dreem 2 data to OSCAR. Here is the best I know and can remember from so many failed attempts, as are partly documented in this thread. Crimson Nape and jameswilliams touched on the possibility of some quirk or feature of Excel CSV files, or corruption of my file, as possibly causing the problem. That help led to solving the problem.


One can import the never opened, raw form of the CSV file from the Dreem 2 export, "dreem_export.csv". But my experience is that such files will not fully import after the file has once been opened by Excel. In my case, such post-opening imports would be reported as being successfully imported by a fly out message from OSCAR (or Windows 10, whichever it is) . But no detailed waveform file(s) were produced. Entries for the action had been made in the OSCAR Dreem Profile's Summaries sub folder. No Events subfolder was created and, accordingly, no waveform files were in it as there should have been.

The breakthrough to importing again came when I decided to use NotePad++ to open and save the raw, never opened Excel CSV file before importing it with OSCAR. Bingo. No problem at all. I then checked to see if a never opened Excel data spreadsheet would import. It did.


Wow, all the wasted time! Excel's (frequently interfering) "pro active" nature guesses (like spell checkers) at what we need or would find helpful. It does that, for another example, with anything that might resemble a date or time--making all such into a date or time--but, unlike what I reported above, at least one can see the fouling at the time. (I finally resorted to never entering dates and time nor anything looking like them --that use digits along with hyphens or colon separators--unless I must, preferring instead to use yyyymmdd or hhmmss for dates and time references; those sort most easily, too, in most cases.) 


   


Upon my opening of the data file Excel it recast it and presented all the once-truly-comma-separated data in the more eye pleasing, neat separate columns as shown. Apparently the Dreem import feature of OSCAR "wants to see" the data as it was truly comma separated in its first saved form. You may reasonably ask why I opened rather than import the file as-is from the beginning onward. Among reasons was the fact my Dreem data bridged across the time change that had not been applied to my AirCurve's clock. Further, as a new user, I wasted a lot of various lengths of WAKE time at the starts of the earlier sleep sessions. I wanted to ignore those from the listed data for all my sessions.
I have no particular qualifications or expertise with respect to the apnea/cpap/sleep related content of my posts beyond my own user experiences and what I've learned from others on this site. Each of us bears the burden of evaluating the validity and applicability of what we read here before acting on it.  

Of my 3 once-needed, helpful, and adjunctive devices I have listed, only the accelerometer remains operative (but now idle). My second CMS50I died, too, of old age and the so-so Dreem 2 needs head-positioning band repair--if, indeed, Dreem even supports use of it now.



 
Post Reply Post Reply
#19
RE: Dreem 2 to OSCAR importing help request
The conclusion of this matter with this "Thank you" to those who posted suggestions:

1. If it is necessary to open a Dreem headband's freshly generated dreem_export.csv file to edit it  before OSCAR import, then the free program Notepad++, not NotePad, is one that can be used successfully to apply edits if [a] done correctly (i.e., without changing the data structure), [b] the edited file is saved from Notepad++ to a CSV file and [c] that same csv file is then imported by OSCAR.

2. CSV files generated by programs are not all alike nor equally application friendly. I'll start a separate thread where I'll post many links that address CSV difficulties. Please comment there on that topic and on other intrinsic (but hidden) problems in the general use software we often use. Posts there of how to solve or work around those problems would likely be helpful to many. OEM manufacturers, agencies and other data users who use CSV files as a vehicle for data transfer have long struggled with their differences--little did I know after using Excel and its CSV files for so many years.

3. My problem was solved, as above, by Crimson Nape seeing and telling me that Excel--possibly all Microsoft data vehicles, if I correctly recall my readings at many links--ends a data line with a CR and a LF combination that is inseparable (ASCII 13 & ASCII 10), not just a LF (ASCII 10). Accordingly, with reference to the OP and my other posts in this thread, once I opened a fresh dreem_export.csv file with Excel it became blighted with the ASCII combo and no data lines could be read by OSCAR. 

I had imported a few of the early fresh CSV files without problems, but when I began opening them to edit and synchronize my disparate newbie-startup dead times better, OSCAR would not import them later because of that CR (ASCII 13) effect.

I should have remembered a precaution, which I little understood but complied with then, that came with my Gulf Data Concepts X2-02 accelerometer. It requires users to provide the device a short configuration file, with my selections of device settings, and to provide a time.txt file to set the time for the onboard RTC clock. The two short files' syntax must be exact and it came with the warning to use use NotePad++ or WordPad, not Microsoft's NotePad,  because of a line ending problem. 

A post-drafting mystery: the copyright symbol in my first paragraph above will not go away unless I replace it, say, with another letter, like the "d" I tried inside parenthesis. I cannot avoid showing that symbol and will not continue trying to type the intended letter, which does reside inside the symbol's circle instead of inside of parentheses matching the (a) and (b) markers. In some unnoticed typo move I must have hit some magic code identity the circled character now triggers when it is isolated or put inside parenthesis. Thinking-about

Moderator Note: Parentheses "()" were replaced with brackets "[]"
I have no particular qualifications or expertise with respect to the apnea/cpap/sleep related content of my posts beyond my own user experiences and what I've learned from others on this site. Each of us bears the burden of evaluating the validity and applicability of what we read here before acting on it.  

Of my 3 once-needed, helpful, and adjunctive devices I have listed, only the accelerometer remains operative (but now idle). My second CMS50I died, too, of old age and the so-so Dreem 2 needs head-positioning band repair--if, indeed, Dreem even supports use of it now.



 
Post Reply Post Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Trouble importing data to Oscar bubbafoxus 2 87 03-22-2024, 05:40 PM
Last Post: bubbafoxus
  Importing Oximetry data billmelk 1 134 03-10-2024, 02:27 PM
Last Post: Crimson Nape
Question Importing Data Only After A Certain Date ResMed AirCurve 10 somewherein413 4 335 02-14-2024, 05:40 PM
Last Post: Crimson Nape
  OSCAR crash importing Well O2 Max data RobertFz 7 427 02-01-2024, 10:04 PM
Last Post: LoudSnorer
  Request Copy of Resmed Airsense 10 SDCard core files gedm 6 484 12-14-2023, 07:04 PM
Last Post: gedm
  Oscar importing csv files + O2 Insight Pro import eflat 1 363 10-24-2023, 10:10 PM
Last Post: Crimson Nape
  [Feature Request] EPR mode in overview pressure graph tossinandturnin 8 853 07-24-2023, 06:24 AM
Last Post: tossinandturnin


New Posts   Today's Posts


About Apnea Board

Apnea Board is an educational web site designed to empower Sleep Apnea patients.