Linux Shell Scripting Tutorial (LSST) v1.05r3
Prev
Chapter 5: Essential Utilities for Power User
Next

Removing duplicate lines using uniq utility

Create text file personame as follows:

personame

Hello I am vivek
12333
12333
welcome
to
sai computer academy, a'bad.
what still I remeber that name.
oaky! how are u luser?
what still I remeber that name.


After creating file, issue following command at shell prompt
$ uniq personame
Hello I am vivek
12333
welcome
to
sai computer academy, a'bad.
what still I remeber that name.
oaky! how are u luser?
what still I remeber that name.

Above command prints those lines which are unique. For e.g. our original file contains 12333 twice, so additional copies of 12333 are deleted. But if you examine output of uniq, you will notice that 12333 is gone (Duplicate), and "what still I remeber that name" remains as its. Because the uniq utility compare only adjacent lines, duplicate lines must be next to each other in the file. To solve this problem you can use command as follows
$ sort personame | uniq

General Syntax of uniq utility:
Syntax:
uniq {file-name}


Prev
Home
Next
sed utility - Editing file without using editor
Up
Finding matching pattern using grep utility