Remove duplicate lines without sorting?

[lngzlz]

Is it possible for TextPipe to remove duplicate lines without having to sort all the lines first?

If sorted, the original order of lines has been crashed. This is usually not desirable.

[DataMystic Support]

We've added a new filter to the next release - 'etext\remove duplicates without sorting.fll'.

Here is the filter export:

|--Comment...
| Add line numbers which we will use to later restore the original line order
|
|--Add line numbers
| Start line 0
| Increment 1
| [ ] Don't increment on blank lines
| [ ] Don't number blank lines
| Format %7.7d
| [ ] Don't reset on end of file
|
|--Comment...
| Compare lines (ignoring line numbers) and discard duplicates
|
|--Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50
|
|--Comment...
| Return the lines to the original order
|
|--Ascending numeric sort, length 7
|
|--Comment...
| Now remove the line numbers
|
|--Remove column:8 columns starting at column 1
|

[lngzlz]

Compare lines (ignoring line numbers) and discard duplicates

How can I ignore line numbers when comparing lines?

[DataMystic Support]

In

Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50

set the Start Column (here set to 9).