the distinct operator is used to remove redundant (duplicate) tuples from a relation.

syntax

given below is the syntax of the distinct operator.

grunt> relation_name2 = distinct relatin_name1;

example

assume that we have a file named student_details.txt in the hdfs directory /pig_data/ as shown below.

student_details.txt

001,rajiv,reddy,9848022337,hyderabad
002,siddarth,battacharya,9848022338,kolkata 
002,siddarth,battacharya,9848022338,kolkata 
003,rajesh,khanna,9848022339,delhi 
003,rajesh,khanna,9848022339,delhi 
004,preethi,agarwal,9848022330,pune 
005,trupthi,mohanthy,9848022336,bhuwaneshwar
006,archana,mishra,9848022335,chennai 
006,archana,mishra,9848022335,chennai

and we have loaded this file into pig with the relation name student_details as shown below.

grunt> student_details = load 'hdfs://localhost:9000/pig_data/student_details.txt' using pigstorage(',') 
   as (id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray);

let us now remove the redundant (duplicate) tuples from the relation named student_details using the distinct operator, and store it as another relation named distinct_data as shown below.

grunt> distinct_data = distinct student_details;

verification

verify the relation distinct_data using the dump operator as shown below.

grunt> dump distinct_data;

output

it will produce the following output, displaying the contents of the relation distinct_data as follows.

(1,rajiv,reddy,9848022337,hyderabad)
(2,siddarth,battacharya,9848022338,kolkata) 
(3,rajesh,khanna,9848022339,delhi) 
(4,preethi,agarwal,9848022330,pune) 
(5,trupthi,mohanthy,9848022336,bhuwaneshwar)
(6,archana,mishra,9848022335,chennai)