August 27, 2012 at 12:39 pm #1402
Hi, I’m having trouble processing ASCII data with EddyPro4. I have already processed files before with no problem but during my last runs I’ve run into the problem, “operating system error, out of memory”. I’m processing data from folders that contain one month’s data, broken up into day long files of 1440 minutes. The software processes all the days with no problem but fails with the above error message when writing the final (full output) file. I’m writing to an external hard drive with 340Gb of memory free and using a laptop with an i5 processor, Windows 7 premium and its own hard drive free memory space of over 300Gb so I don’t understand the out of memory message. I’ve tried writing to different folder locations with the same problem. I have both 10hz and 20hz data and have the same problem with both. I have already processed these files before (as a trial with different processing options) with no problems. Thanks for your help.August 27, 2012 at 9:11 pm #1630
the problem you encountered isn’t related to HD space, rather to RAM availability. Due to an excessive RAM requirement, EddyPro may sometimes crash when the system’s resources are not sufficient (e.g. if the computer is running multiple applications) or when EddyPro asks the OS an amount of RAM larger than the maximum allowed, which I believe is 2 GB for the latest Windows versions.
This is a known problem that we are addressing in view of the next release. Some users who encountered the same issue were able to get rid of it by simply restarting the computer and avoiding running other major application while running EddyPro. However, the effectiveness of this solution will depend on your computer capabilities and on the settings of EddyPro, or the combination of both.
Hope this helps.
GerardoAugust 28, 2012 at 10:40 am #1631
Hi Gerardo, thanks for the reply. I tried what you said, made sure nothing else was running on my computer and tried processing one month’s data with output set to minimal in advanced settings but still no luck. All processing runs fine right through all files and up to the point where EddyPro is writing the final “full output” file then crashes with the message, “Operating system error, not enough space, out of memory”. My laptop has 6Gb of RAM and has previously run this data with no problem so just can’t see what’s going wrong now. This is very frustrating, I’ve been processing these files before without any problems but now cannot process any of my more recent data. I’ve tried running subperiods, it runs the first ten days’ of data ok, then runs the next ten days’ ok but then crashes with the same problem on the last few days. Even when I try to run to the end of the month two days at a time. I’ll try installing EddyPro onto a different computer and see if I have the same problems. Thanks again for your help.August 28, 2012 at 11:32 am #1632
Well, this sounds odd, indeed. It seems related to the allocation of a too-large array when it’s the time to write the full output file.
Please try the following:
1. If you selected the option “Build continuous dataset”, try and run a few days without that option.
2. Make sure there is no raw file with a timestamp that is – for any reason – very far away from those of the main dataset.
Using the option “Build continuous dataset”, EddyPro will attempt to allocate an array as long as the whole dataset you’re processing, in a “timestamp sense”. For example, if you want to process only 2 raw files which, for any reasons, have timestamps of 1 Jan 2000 and 1 Jan 2012, EddyPro will try to allocate a 12-years array, and this may cause that problem.
If this doesn’t help,and if you can consistently replicate the problem with just 2 days worth of raw files, you may try to make those files available to me (e.g. FTP, dropbox..) so that I can investigate the problem.
GerardoAugust 28, 2012 at 8:51 pm #1633
I’m getting the same error. I first received it when running a large dataset of TOB1 files. I was attempting to process data from Feb to the beginning of Aug. I got the error on the July 1 timestamp. So, I’ve tried running July 1 thru beginning of August on a new machine (an Intel Core 2 Duo 3.2GHz, 8 GB RAM, windows 7 64bit) and it’s still not working. The error pops up immediately. I’ve triple checked that no other processes are running on the computer. Any additional thoughts on this topic?? Should I go back to using a previous version of EddyPro?? Thanks, RebeccaAugust 29, 2012 at 8:14 am #1634
Hi Gerardo, I’ve tracked the problem down to one single day file May 30, I can process all the days in the month previous and the following days but EddyPro seems to be tripping up over this one for some reason, it actually starts processing this file but only gets to 01:30 before returning the memory error message. Rebekah it would be interesting if you skip your July 1 and see if your other files will process ok. I’ve had a look in my file and can’t see anything obvious, the timestamps seem to be in order. I have been using “set minimal” output with the “build continuous dataset” box ticked. This file is only at 10hz so is only 110kb, Gerardo, do you have an email address I can use to send it to you? I’ve subsequently processed two months’ worth of 20hz data with no problem so definitely not lacking processing power. Thanks, JonAugust 29, 2012 at 10:02 am #1635
Hello Rebecca and Jon,
Jon: from your original messages I did not guess that the problem could be related to an individual raw file, because you mentioned that the crash was occurring during the creation of the full output file, at the end of the run, when all raw files were already dealt with. So I am still puzzled how you could fix the problem by eliminating one raw file.. 🙂
I was reported of the “out of memory” problem also in relation to the calculation of the random uncertainty (and I don’t exclude it could also arise somewhere else). In these cases, I would expect the software to crash while processing a specific problematic file (and not at the end of the run). This might be Rebecca’s case, and Jon’s suggestion to try and eliminate the July 1 file is valid.
I already fixed this problem in relation to the random uncertainty calculation. This fix will be available in the next release.
If you manage to individuate a specific problematic raw file, please send it to me along with the “.eddypro” and “.metadata” files at <firstname.lastname@example.org>, so that I can investigate the origin of the problem.
Also, if you urgently need to process your files but you’re stuck by this “out of memory” problem, please contact us by email and we’ll figure out a solution.
PS: I want to point out that any “out of memory” crash *is* an EddyPro problem that we need to address. In my previous posts I didn’t mean to say that you should use more powerful computers, rather than I need to reduce EddyPro’s RAM demand.August 29, 2012 at 3:33 pm #1636
Hey guys. Hopefully I’m not jinxing my processing but I eliminated my first file and things seem to be working. Good idea Jon!August 30, 2012 at 8:34 am #1638
Well i’m even more confused now, I went back and ran what I thought was the problem file as a new project to produce some fresh processing files for you Gerardo and it ran fine, I then ran the entire month through and that worked fine! I then went back to my original project file and ran it, this one did crash with the memory problem so it must be some difference in the way I’ve set up the project processing options though as far as I can tell they are both set up the same and I’m using the same metadata file for both. Obviously this is great because I’ve now processed the data but I’m still unclear what I did wrong which is frustrating. I’ve emailed some dropbox links to you for the files for the problem processing. I notice the “full output file” is written to the folder but still stored as a .tmp file, it doesn’t get as far as being saved into the .csv file which you would expect. And always stops at the same place in the same file i.e. 01:30 May 31.
As I say there’s no great urgency as I have now managed to process this lot but would love to know what the problem was for future reference.Rebecca perhaps you might like to try yours as a new project and see if it makes a difference?
You must be logged in to reply to this topic.