Untitled

EXPERIMENT-3: 
File management tasks & Basic linux commands
(i) Creating a directory in HDFS 
(ii) Moving forth and back to directories
(iii) Listing directory contents 
(iv) Uploading and downloading a file in HDFS 
(v) Checking the contents of the file
(vi) Copying and moving files 
(vii) Copying and moving files between local to HDFS environment
(viii) Removing files and paths 
(ix) Displaying few lines of a file 
(x) Display the aggregate length of a file 
(xi) Checking the permissions of a file 
(xii) Zipping and unzipping the files with & without permission pasting it to a location 
(xiii) Copy, Paste commands
Solution
(i) Creating a Directory in HDFS
To create a directory in HDFS, use the following command:
hadoop fs -mkdir /user/hadoop/new_directory
Example:
hadoop fs -mkdir /user/hadoop/mydir
Checking the directory with `ls`:
hadoop fs -ls /user/hadoop/
Output:
drwxr-xr-x - hadoop supergroup 0 2024-12-13 10:20 /user/hadoop/mydir
(ii) Moving Forth and Back to Directories
To move between directories in HDFS, use the `cd` command:
hadoop fs -cd /user/hadoop/mydir
(iii) Listing Directory Contents
To list the contents of a directory in HDFS, use:
hadoop fs -ls /user/hadoop/mydir
Example:
hadoop fs -ls /user/hadoop/
Output:
drwxr-xr-x - hadoop supergroup 0 2024-12-13 10:20 /user/hadoop/mydir
-rw-r--r-- 3 hadoop supergroup 234 2024-12-13 10:25 /user/hadoop/myfile.txt
(iv) Uploading and Downloading a File in HDFS
To upload a file from local to HDFS:
hadoop fs -put /localpath/myfile.txt /user/hadoop/
BIG DATA: SPARK

To download a file from HDFS to local:

hadoop fs -get /user/hadoop/myfile.txt /localpath/
(v) Checking the Contents of the File
To check the contents of a file in HDFS:
hadoop fs -cat /user/hadoop/myfile.txt
Example Output:
This is the content of the file in HDFS.
(vi) Copying and Moving Files
To copy a file in HDFS:
hadoop fs -cp /user/hadoop/myfile.txt /user/hadoop/backup/myfile.txt
To move a file in HDFS:
hadoop fs -mv /user/hadoop/myfile.txt /user/hadoop/backup/myfile.txt
(vii) Copying and Moving Files Between Local and HDFS
To copy a file from local to HDFS:
hadoop fs -copyFromLocal /localpath/myfile.txt /user/hadoop/
To move a file from local to HDFS:
hadoop fs -moveFromLocal /localpath/myfile.txt /user/hadoop/
To copy a file from HDFS to local:
hadoop fs -copyToLocal /user/hadoop/myfile.txt /localpath/
(viii) Removing Files and Paths
To remove a file or directory in HDFS:
hadoop fs -rm /user/hadoop/myfile.txt
hadoop fs -rm -r /user/hadoop/mydir
(ix) Displaying Few Lines of a File
To display the first few lines of a file in HDFS:
hadoop fs -head /user/hadoop/myfile.txt
Example Output:
This is the first few lines of the file.
To display the last few lines of a file:
hadoop fs -tail /user/hadoop/myfile.txt
Example Output:
This is the last few lines of the file.
(x) Display the Aggregate Length of a File
To get the length of a file in HDFS:
hadoop fs -du -s /user/hadoop/myfile.txt
/user/hadoop/myfile.txt

(xi) Checking the Permissions of a File

To check the permissions of a file:
hadoop fs -ls /user/hadoop/myfile.txt
Example Output:
-rw-r--r-- 3 hadoop supergroup 234 2024-12-13 10:25 /user/hadoop/myfile.txt
(xii) Zipping and Unzipping Files
To zip a file:
zip myfile.zip myfile.txt
To unzip a file:
unzip myfile.zip
For zipping/unzipping files in HDFS:
1. Upload the zip file to HDFS:
hadoop fs -put myfile.zip /user/hadoop/
2. Then unzip it using:
hadoop fs -cat /user/hadoop/myfile.zip | unzip -
(xiii) Copy, Paste Commands
Copy command: `cp` in Linux and `hadoop fs -cp` in HDFS.
Paste command: `mv` in Linux and `hadoop fs -mv` in HDFS