DataMystic

Posted: **Thu Jul 07, 2011 8:33 pm**

My Toshiba Satellite C650 has a Pentium® Dual-Core CPU, running Windows 7 64-bit.

Nothing in the TextPipe help file mentions multi-core processors, yet there must be a strong probability that TextPipe uses multi-threading, else why would there be a need for a Restrict to each line in turn filter?

See http://en.wikipedia.org/wiki/Multi-core_processor

I'm beginning to wonder whether some of the recent problems I've encountered are caused by TextPipe's software algorithms and how these operate in a dual-core system.

TextPipe originated in the days when personal computers were generally single core.

The only reference in the help file to multiple CPUs is in the page for Switch CPU.

Found under Filters\Special (Server Edition Only)

This filter causes a logical break in the processing from one CPU to another. It allows processing load to be distributed across multiple CPUs.

For example, if a filter list involved data preparation, sorting, and then output generation, then it makes sense to have the sorting on one CPU alone, as sorting is very CPU intensive.

Start by placing the splitter roughly in the middle of the filter list – before or after a sort typically. Then try moving it up and down to gauge performance.

Note: The numeric field of this filter is currently unused.

Each time you use this filter, a new logical processing thread is created. Windows takes care of distributing this thread to the CPU with the least load.

Please comment.

Posted: **Thu Jul 07, 2011 9:07 pm**

Hmm, I was sure there was a 'How TextPipe works' in the help, but I couldn't find it. Time for one soon.

Your recent problems relate more to assumptions about UTF-8, and also not understanding subfilters. I will address the use of Capture filters as subfilters in the other thread.

It doesn't use multi-threading, except in the Server edition, where processing can be explicitly split amongst CPUs.

Posted: **Thu Jul 07, 2011 9:10 pm**

The following observations relate to the issue I encountered when using the Capture text to a variable filter.
I decided to see what happens if I made a separate filter to do the task I was previously trying within a T-filter.
Here are the results I obtained.

Issue analysis

When I didn't use
Restrict to each line in turn
the captured variable @BookChapter has
147 values instead of 1190 values.

e.g. First value is GENESE 6 when it should be GENESE 1
Yet, Break on value change works as expected.

It all works OK when everything is a subfilter of
Restrict to each line in turn

For complex filters this is not always possible,
as some of the earlier filters may need
to process patterns that span more than one line!

Furthermore, the simplicity of this test filter meant that it was
much faster than when I was using a T-filter within my complex one.

Being speedier, when I don't use
Restrict to each line in turn
the number of unique values captured is far fewer
than for the slower more complex one.
147 compared to 388.

Ergo: All to do with multi-threading and dual-core.

The total number of captures probably also depends on what other
applications are open in the background.

This pitfall could well cause other users to stumble.
It therefore needs more careful explanation in the Help file, followed by some serious research by the programming team at DataMystic.

The advent of Dual Core processors in personal computers (not just servers within a network envirnment)
means that what could earlier be taken for granted for the environment most single users work in is no longer universally true.

A big rethink is therefore urgently overdue.

Posted: **Thu Jul 07, 2011 9:14 pm**

My reply was written before yours, Simon.

Posted: **Thu Jul 07, 2011 9:22 pm**

It's not an issue David.

Without the Restrict to each Line filter, TP sends a large chunk of text (multiple lines) to the search/replace, and only the final replacement hits the Capture Filter.
With the Restrict to each Line filter, TP sends at most one line to the search/replace, and hence every single match hits the Capture Filter.

Posted: **Thu Jul 07, 2011 10:18 pm**

That's how I understood it was supposed to work, but something odd happens when there is a T-filter involved as well, along with its own secondary output filter.
So I still think there's something about how TextPipe works that is incompletely explained.

David

Posted: **Fri Jul 08, 2011 11:58 am**

Ok. If you could trim back your complex filter to just the section of interest (by deleting subsequent filters, and generating intermediate files for the preceding filters) then I would love to be able to explain how this works to you!

DataMystic

Dual core processors?

Dual core processors?

Re: Dual core processors?

Re: Dual core processors?

Re: Dual core processors?

Re: Dual core processors?

Re: Dual core processors?

Re: Dual core processors?