Delete duplicate lines in a delimeted file

Get help with installation and running here.

Moderators: DataMystic Support, Moderators, DataMystic Support, Moderators, DataMystic Support, Moderators

Post Reply
leorr11
Posts: 2
Joined: Sun May 15, 2016 5:15 am

Delete duplicate lines in a delimeted file

Post by leorr11 »

Hi all,

I have a files with pipe delimited fields.
I`d like to delete lines which are duplicated in one field only.

I`ve been trying in several ways without result.

It seems that "delete duplicate lines" and "count duplicate lines" don`t work as a subfilter of "restrict fields"
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Delete duplicate lines in a delimeted file

Post by DataMystic Support »

No, as each text fragment passed to Delete duplicate lines is considered a new 'file'.

Are the lines with duplicate fields next to each other? If so, use an EasyPattern like this to identify them:

Code: Select all

[ capture( pipefield), pipe, capture( pipefield), pipe, capture( pipefield), pipe, cr, lf
  group1, pipe ]
This will find lines where field1 is repeated on the next line.

This next EasyPattern finds lines where field 3 is repeated on the next line:

Code: Select all

[ capture( pipefield), pipe, capture( pipefield), pipe, capture( pipefield), pipe, cr, lf
  capture( pipefield), pipe, capture( pipefield), pipe, group3, pipe ]
leorr11
Posts: 2
Joined: Sun May 15, 2016 5:15 am

Re: Delete duplicate lines in a delimeted file

Post by leorr11 »

thanks, it works
JamesB
Posts: 1
Joined: Sun Aug 21, 2016 8:31 pm

Re: Delete duplicate lines in a delimeted file

Post by JamesB »

leorr11 wrote:Hi all,

I have a files with pipe delimited fields.
I`d like to delete lines which are duplicated in one field only.

I`ve been trying in several ways without result.

It seems that "delete duplicate lines" and "count duplicate lines" don`t work as a subfilter of "restrict fields"
I have a similar question, is there a way to delete duplicate lines but leave the original line? When I do this it seems to remove even the original.
Last edited by JamesB on Fri Apr 07, 2017 11:57 pm, edited 1 time in total.
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Delete duplicate lines in a delimeted file

Post by DataMystic Support »

Which version of TP do you have?

Is Filter Library\Remove\Duplicate lines your only filter?
Post Reply