Search/Replace List memory issue

[alnico]

If a run a Search/Replace List of 10,000 lines (3.5Mb) against 3 (100 line 45kb) input files...problems occur...

Immediately TextPipe will hang and the header bar may or may not say "Not Responding".
After about 4 minutes TextPipe will come back to life and throw errors:

"Out of memory"
"Not enough storage available to process this command"

I always get the first error but not always the second one.

I have tested this on XP and Win7(64bit) with the same results.

I have never run a search replace list this big before...but am planning on it being much larger (50x larger).

Is something buggy here or are there limitations that I need to be aware of?

Running TextPipe 8.8

[DataMystic Support]

We did a lot of performance work on that filter for one user - so it should be fine, however, it may need to check for user input periodically.

Coulld you email me sample data?

[DataMystic Support]

We can fix the unresponsiveness easily, but I need to get an idea of what you are doing.

Sample files will be really helpful - especially to see what kind of search/replace you are doing, if you are sorting the list (ie longest first) etc.

[alnico]

Attached is a filter with three input files and a tab deliminated Replace list.

If a smaller Replace list is used...it works fine; otherwise TextPipe throws errors or simply stops.

However, I have since figured out a more efficient way of getting what I need here...so fixing this is not an issue; but maybe you will still find it useful 'IF' there is indeed a bug...(maybe it is simply to large for my system resources).

Thanks,

Brent

===UPDATE===============================

I had to remove about 10,000 lines from the Replace List tab file (included in the zip archive)...as the file upload is limited to 256kb.
I just ran the filter...and to my surprise it does work with a 70,000 Replace List...so initially I guess I went over the limit

[DataMystic Support]

Thanks Brent,

70,000 replaces is a lot, but if you check the simultaneous replace it will combine them into a much smaller number of patterns which uses less memory.

Could you email me the original data (not upload?) The forum limit is designed for filters rather than sample data.

[alnico]

Simon, I emailed you the original data to test.

Simultaneous replace does not work as they contain patterns; not 'exact' matches.

But again, I figured out a more efficient solution...

Thanks,

[DataMystic Support]

You need to check the 'Simultaneous Search' option - otherwise it creates one filter for every single search/replace row.
It only took 5 seconds on my machine for > 65535 search/replaces in your file.