Saturday, July 20, 2019

Hadoop 1.2.1 Installation and Configuration on Multiple Nodes

There are few changes one has to make for multi-node setup from single node. First you need to complete single node setup up to DFS formatting step.

Mainly there are five steps to follow for multi-node setup from single node:

STEPS:
  1. SSH COPY ID to all nodes
  2. Configure masters and slaves
  3. Configure CORE-SITE.XML and MAPRED-SITE.XML
  4. Format DFS
  5. START-ALL.SH
Now I am going to explain this steps in detail:




Step-1 SSH COPY ID to all nodes:
From NAME NODE, We need to generate SSH KEY and distribute it to all the SLAVE NODES and also SECONDARY NAME NODE (if any)
Command:
ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@datanode1
here "hadoop" is an user name and "datanode1" is a system name, which you need to change according to your setup.
COPY FINGERPRINT : GIVE YES
Do the same for all DATA NODES and for SECONDARY NAME NODE (if any)
Check whether it is successfully copied or not
ssh datanode1
it should not ask for password
eg.
ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@datanode1
ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@datanode2
ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@datanode3
 



Step-2 Configure masters and slaves:
We need to do it on NAME NODE alone (not on the DATA NODES and SECONDARY NAME NODE)
Go to NAME NODE
Command:
cd /usr/local/hadoop/conf
Find the two files: masters, and slaves
Masters for NAME NODE and SECONDARY NAME NODE
Slaves for DATA NODES
Command:
sudo nano /usr/local/hadoop/conf/masters
By default it contains 'localhost', Change it to the name of NAME NODE (i.e. namenode in my case)
Ctrl + o to save
Enter
Ctrl + x to exit
sudo nano /usr/local/hadoop/conf/slaves
By default it contains 'localhost', Change it to contain names of all DATA NODES one per line, in my case
datanode1
datanode2
datanode3
Ctrl + o to save
Enter
Ctrl + x to exit




Step-3 Configure CORE-SITE.XML and MAPRED-SITE.XML
go to SLAVES/SECONDARY NAME NODE and we need to make them point to the master
Command:
sudo nano /usr/local/hadoop/conf/core-site.xml
Check whether it is pointing to NAME NODE (i.e. namenode in my case) in 'FS.DEFAULT.NAME', if it is pointing to localhost:10001, update localhost with namenode
Ctrl + o to save
Enter
Ctrl + x to exit
Same way for MAPRED-SITE.XML
Command:
sudo nano /usr/local/hadoop/conf/mapred-site.xml
Check whether it is pointing to the JOB TRACKER / NAME NODE (i.e. namenode, in my case)
If it is 'localhost:10002', update it as 'namenode:10002'
Remove LOCAL HOST entries from /ETC/HOSTS file
Command:
sudo nano /etc/hosts
remove localhost and entries for 127.0.0.1




Step-4 Format DFS:
If converting the existing single node installation then you must delete the /USR/LOCAL/HADOOP/TMP and then create it again in all the nodes and then format it from NAME NODE alone. skip up to formatting steps if you haven't formatted your HDFS with single node setup.
Command:
To remove directory:
sudo rm -r /usr/local/hadoop/tmp
Create tmp directory
sudo mkdir /usr/local/hadoop/tmp
Changing ownership of tmp as well as hadoop directory
sudo chown hadoop /usr/local/hadoop/tmp
sudo chown hadoop /usr/local/hadoop
Format NAME NODE
hadoop namenode -format
Check for 'name node successfully formatted' message




Step-5 START-ALL.SH
To start hadoop cluster with multi-node, we have to run this command from NAME NODE and it starts respective services on all NODES
Command:
start-all.sh
jps
check each system separately to find specific JVMs running on them
Check number of live nodes in web GUI (it will take few minuets)
stop-all.sh

For any queries you can write in a comment or mail me at: “brijeshbmehta@gmail.com”

Courtesy: Mr. Anand Kumar, NIT, Trichy




Saturday, July 13, 2019

Book Review: "Dhirubhaism (Gujarati)"

Image Source: https://rukminim1.flixcart.com/image/832/832/book/8/7/8/dhirubhaism-original-imadrkjptnbfnfh9.jpeg?q=70




Dhirubhaism is a story of the work ethics Dhirubhai Ambani believes in. How a small town boy become the most successful businessman in India. He believe in some ideology and he follows it through out his life and most of the people around him believe that those ideologies has make him successful. The author Mr. A. G. Krishnamurthy is one the closest friend to Dhirubhai. In this book he discuss the ideologies of Dhirubhai and his success story due to those ideology. It is worth reading for all the entrepreneurs and the one dream of it.





Friday, July 5, 2019

Book Review: "EM and the big HOOM"

Image Source: https://images-na.ssl-images-amazon.com/images/I/51omE0F1hPL._SX326_BO1,204,203,200_.jpg

The book was gifted by one of my colleague as a farewell gift. I love reading books but this is something different! It is a story of a family 'EM' the mother, 'HOOM' the father, a son (who tells the story), and a daughter. Story is more focused on EM than HOOM. The author added all the possible feelings in the story. EM is a mentally unstable character but her family loves and support her a lot. It is very difficult to take care of a person with suicidal tendency but the family stood strong. The story starts with EM and HOOM's relationship and end with EM's death but every moment of it is very well written in the book. I really like the book and I recommend it to the serious story lovers.

Wisdoms:

  1. I used to wonder: What must it mean for a lower middle-class woman to tell a strange about her sexual history and her fantasy life? Does she understand the free association that is sometimes used, or why the psychiatric social worker wants to know so much about her childhood? Those who have some experience with homeopathy may not be shaken or shamed by the bizarreness of the question, but which Indian woman will talk about masturbation? And what can mental health mean in a nation that wants an injection to put it back on its feet the next morning.
  2. God listens to the prayers of mental because they are touched by his hands.
  3. I believe in Jesus Christ and the Buddha and Krishna and Allah because you can believe in anything if you look straight at the message:
    • Love one another? Good idea
    • Detach yourself? Good idea
    • Do your duty? Good idea
    • Submit to the will of God and go with the flow? Good idea
  4. If anyone ever does you a favor, you cannot forget it. You must always credit them, especially in public, especially to those they love and those who love them. You must pay your debts, even those that you can never fully repay. Anything less makes you less.
  5. It occurred to me then that the mad in India are not the mentally ill, they are, simply mad. They have no other identity. Here, everyone was mad. They had lost their hair so that the institution could keep them free of lice. They had lost their clothes because their families had abandoned them, and they had lost their lives because they had lost their families. They were now free in a bizarre sort of way. They were also alone except for the shoulder in front and the touch of the fingers of the person following behind.
  6. When you're a child, cast the runes. When you're an adult, ask an expert.



Saturday, June 29, 2019

Book Review: "Albert Einstein: A Short Biography"

Image Source: https://rukminim1.flixcart.com/image/704/704/book/1/1/3/albert-einstein-a-short-biography-original-imadgbscjvchzcxn.jpeg?q=70





Top post on IndiBlogger, the biggest community of Indian Bloggers


It is one of most inspiring biography. We all know that Albert Einstein was a great scientist but very few know how he became one. This book shows the journey of Einstein from childhood to the great scientist in a very short but interesting way. Einstein face many problems in his life but his never give up attitude keeps him in the hunt.




Saturday, June 22, 2019

Book Review: "Anything for You, Ma'am"

Image Source: https://images-na.ssl-images-amazon.com/images/I/815MQHTyAfL.jpg


One more attempt of an IITian to be an author. It is a simple love story with humor and suffering. It came in the era of Chetan Bhagat (CB), however, it doesn't have a vulgarity and sex which is always there in CB books. It is worth spending your free time to read this innocent love story.





Tuesday, June 18, 2019

Book Review: "The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life"

Image Source: https://upload.wikimedia.org/wikipedia/en/thumb/b/bd/The_Subtle_Art_of_Not_Giving_a_F%2Ack_by_Mark_Manson_-_Book_Cover.png/220px-The_Subtle_Art_of_Not_Giving_a_F%2Ack_by_Mark_Manson_-_Book_Cover.png


Last month when I visited crossword in Pune, I find this book. From title itself it looks strange to me and may be to many of you. When I read its short summery, I found it interesting. Therefore, I bought it and It is worth buying, for me at least. I consider it to be one of the best motivational book I have ever read. However, I can't compare it with the books from Robin Sharma. In starting, one might get bored with more use of " f " word but after first chapter, it is interesting. The real life examples and wisdom from other books and authors make it easy to understand. It is more of a casual writing than a typical motivational writing but still there are many hidden messages in the book.

Wisdom:

  • The desire for more positive experience is itself a negative experience. And paradoxically the acceptance of one's negative experience is itself a positive experience.
  • Not giving a fuck does not mean being indifferent; it means being comfortable with being different.
  • To not give a fuck about adversity you must first give a fuck about something more important than adversity.
  • Whether you realize it or not, you are always choosing what to give fuck about.
  • Life itself is a form of suffering. The rich suffer because of their riches. The poor suffer because of their poverty. People without family suffer because they have no family. People with family suffer because of their family. People who pursue worldly pleasures suffer because of their worldly pleasures. People who abstain from worldly pleasures suffer because of their abstention.
  • We suffer for the simple reason that suffering is biologically useful. It is nature's preferred agent for inspiring change.
  • Life is essentially an endless series of problems. The solution to one problem is merely the creation of the next one.
  • Don't hope for a life without problems. Instead, hope for a life full of good problems.
  • Happiness comes from solving problems.
  • Problems never stop; they merely get exchanged and/or upgraded.
  • Happiness requires struggle. It grows from problems.
  • The deeper the pain, the more helpless we feel against our problems, and the more entitlement we adopt to compensate for those problems.
  • There is no such thing as a personal problem. If you have got a problem, chances are millions of other people have had it in the past, have it now, and are going to have it in the future.
  • Self-awareness is like an onion. There are multiple layers to it, and the more you peel them back, the more likely you are going to start crying at inappropriate times.
  • Pleasure is not the cause of happiness; rather, it is the effect. If you get the other stuff right, then pleasure will naturally occur as a by-product.
  • Sometimes life sucks, and the healthiest thing you can do is admit it.
  • Often the only difference between a problem being painful of being powerful is a sense that we choose it, and that we are responsible for it.
  • I'm always wrong about everything, over and over and over again, and that's why my life improves.
  • The more you try to be certain about something, the more uncertain and insecure you will feel.
  • The more you embrace being uncertain and not knowing, the more comfortable you will feel in knowing what you don't know.
  • The more something threaten your identity, the more you will avoid it.
  • It is the mark of an educated mind to be able to entertain a thought without accepting it.
  • If it feels like it's you versus the world, chances are it's really just you versus yourself.
  • Improvement at anything is based on thousands of tiny failures, and the magnitude of your success is based on how many times you've failed at something.
  • Don't just sit there. Do something. The answers will follow.
  • The fear of death follows from the fear of life. A man who lives fully is prepared to die at any time.
  • We're all going to die, all of us. What a circus! That alone should make us love each other, but it doesn't. We are terrorized and flattened by life's trivialities; we are eaten up by nothing.