Remove Duplicate Lines - Keep First or Last, See What Was Removed

Every line gets a comparison key: the line itself, trimmed if you ask, and lower-cased if you ask. The tool counts the keys in one pass, then reads the lines again and keeps the copy you chose. There is no sorting, so the original order is kept, and the work grows in step with the size of the text, so 200,000 lines take a fraction of a second. The Unix uniq command only finds repeats that sit next to each other (the GNU manual says repeated lines are detected only if they are adjacent). This tool compares every line with every other line, wherever they are, so the text does not need sorting first.

There are three views. 'Remove duplicates' keeps one copy of every line. 'Only the duplicated lines' shows each line that appears more than once, one time each, like uniq -d but across the whole text. 'Only lines that appear once' shows the lines that never repeat, like uniq -u. 'Which copy stays' decides whether the first or the last copy of a repeated line is kept. With the last copy, the lines are in the order of their last appearance.

'Ignore upper and lower case' compares lower-cased lines, so Apple and apple match, and the line you see is still the one you typed. It uses simple lower-casing, so ß and SS stay different. 'Trim spaces' removes spaces, tabs, non-breaking spaces and similar characters at both ends of every line before comparing, and the lines in the result are trimmed too. Blank lines can be left alone (the default, which keeps the gaps between paragraphs), treated like any other line (identical blank lines are reduced to one), or removed. A line that holds only spaces counts as blank.

The headline counts extra copies: a line that appears three times counts as two removed. Under the result, 'Which lines were duplicates' lists each repeated line with how many times it appeared and on which lines, so you can see what was dropped and check it. If two lines look the same but are not matched, the cause is usually a trailing space, a tab or a non-breaking space (turn on trimming), a difference in capitals (turn on ignore case), or an accented letter written as a letter plus a combining accent, which is a different sequence of characters from the single accented letter.