Search/Replace List memory issue

Get help with installation and running here.

Moderators: DataMystic Support, Moderators, DataMystic Support, Moderators, DataMystic Support, Moderators

Post Reply
alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Search/Replace List memory issue

Post by alnico »

If a run a Search/Replace List of 10,000 lines (3.5Mb) against 3 (100 line 45kb) input files...problems occur...

Immediately TextPipe will hang and the header bar may or may not say "Not Responding".
After about 4 minutes TextPipe will come back to life and throw errors:

"Out of memory"
"Not enough storage available to process this command"

I always get the first error but not always the second one.

I have tested this on XP and Win7(64bit) with the same results.

I have never run a search replace list this big before...but am planning on it being much larger (50x larger).

Is something buggy here or are there limitations that I need to be aware of?

Running TextPipe 8.8
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Post by DataMystic Support »

We did a lot of performance work on that filter for one user - so it should be fine, however, it may need to check for user input periodically.

Coulld you email me sample data?
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Post by DataMystic Support »

We can fix the unresponsiveness easily, but I need to get an idea of what you are doing.

Sample files will be really helpful - especially to see what kind of search/replace you are doing, if you are sorting the list (ie longest first) etc.
alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Re: Search/Replace List memory issue

Post by alnico »

Attached is a filter with three input files and a tab deliminated Replace list.

If a smaller Replace list is used...it works fine; otherwise TextPipe throws errors or simply stops.

However, I have since figured out a more efficient way of getting what I need here...so fixing this is not an issue; but maybe you will still find it useful 'IF' there is indeed a bug...(maybe it is simply to large for my system resources).

Thanks,

Brent

===UPDATE===============================

I had to remove about 10,000 lines from the Replace List tab file (included in the zip archive)...as the file upload is limited to 256kb.
I just ran the filter...and to my surprise it does work with a 70,000 Replace List...so initially I guess I went over the limit ;-)
Last edited by alnico on Thu Apr 14, 2011 11:43 pm, edited 1 time in total.
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Post by DataMystic Support »

Thanks Brent,

70,000 replaces is a lot, but if you check the simultaneous replace it will combine them into a much smaller number of patterns which uses less memory.

Could you email me the original data (not upload?) The forum limit is designed for filters rather than sample data.
alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Re: Search/Replace List memory issue

Post by alnico »

Simon, I emailed you the original data to test.

Simultaneous replace does not work as they contain patterns; not 'exact' matches.

But again, I figured out a more efficient solution...

Thanks,
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Search/Replace List memory issue

Post by DataMystic Support »

You need to check the 'Simultaneous Search' option - otherwise it creates one filter for every single search/replace row.
It only took 5 seconds on my machine for > 65535 search/replaces in your file.
Post Reply