DataMystic

Is it possible for TextPipe to remove duplicate lines without having to sort all the lines first?

If sorted, the original order of lines has been crashed. This is usually not desirable.

We've added a new filter to the next release - 'etext\remove duplicates without sorting.fll'.

Here is the filter export:

|--Comment...
| Add line numbers which we will use to later restore the original line order
|
|--Add line numbers
| Start line 0
| Increment 1
| [ ] Don't increment on blank lines
| [ ] Don't number blank lines
| Format %7.7d
| [ ] Don't reset on end of file
|
|--Comment...
| Compare lines (ignoring line numbers) and discard duplicates
|
|--Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50
|
|--Comment...
| Return the lines to the original order
|
|--Ascending numeric sort, length 7
|
|--Comment...
| Now remove the line numbers
|
|--Remove column:8 columns starting at column 1
|

Compare lines (ignoring line numbers) and discard duplicates

How can I ignore line numbers when comparing lines?

In

Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50

set the Start Column (here set to 9).