conversion from unknown encoding to utf-8

[Richar007]

Is it possible to create scripting using textpipe to convert from unknown encoding to utf-8?
I'm trying to process batches of .sql files (of unknown encoding) which are then concatenated to be one big sql file.
So far, all my investigation suggests the input files need to be the same encoding and I need to use that as a parameter for the conversion - can anyone do better?!
Thanks.

[DataMystic Support]

If the input files are only utf-16, utf-32 and utf-8, then you can use Filters\Restrict\Restrict to Unicode UTF16 files and
Filters\Restrict\Restrict to Unicode UTF32 files
to force conversion to utf-8.

Aside from that, there are heuristic approaches to detecting an file encoding, but as yet not included in TextPipe.

[Richar007]

I've managed to find a new tool that will process files with any encoding and change all files into UTF-8, and I've got a textpipe filter working to join these into one long file with various headers and footers, but I'd like to run it at the command line and I cannot find the command line command for "Remove BOM (Byte Order Mark)" - am I being daft or does it not exist?

Thanks in advance for any help.

[DataMystic Support]

Quite a few filters cannot be added from the command line - but you can save it to a .fll filter file, and then add filters together on the command line.

Which program did you use to convert files?