Delete duplicate lines in a delimeted file

[leorr11]

Hi all,

I have a files with pipe delimited fields.
I`d like to delete lines which are duplicated in one field only.

I`ve been trying in several ways without result.

It seems that "delete duplicate lines" and "count duplicate lines" don`t work as a subfilter of "restrict fields"

[DataMystic Support]

No, as each text fragment passed to Delete duplicate lines is considered a new 'file'.

Are the lines with duplicate fields next to each other? If so, use an EasyPattern like this to identify them:

Code: Select all

[ capture( pipefield), pipe, capture( pipefield), pipe, capture( pipefield), pipe, cr, lf
  group1, pipe ]

This will find lines where field1 is repeated on the next line.

This next EasyPattern finds lines where field 3 is repeated on the next line:

Code: Select all

[ capture( pipefield), pipe, capture( pipefield), pipe, capture( pipefield), pipe, cr, lf
  capture( pipefield), pipe, capture( pipefield), pipe, group3, pipe ]

[leorr11]

thanks, it works

[JamesB]

Hi all,

I have a files with pipe delimited fields.
I`d like to delete lines which are duplicated in one field only.

I`ve been trying in several ways without result.

It seems that "delete duplicate lines" and "count duplicate lines" don`t work as a subfilter of "restrict fields"

I have a similar question, is there a way to delete duplicate lines but leave the original line? When I do this it seems to remove even the original.

[DataMystic Support]

Which version of TP do you have?

Is Filter Library\Remove\Duplicate lines your only filter?