What Is Backpropagation

What Is Backpropagation Backpropagation Algorithm For Training A Neural Network Back Home Categories Online Courses Mock Interviews Webinars NEW Community Write for Us Categories Artificial Intelligence AI vs Machine Learning vs Deep LearningMachine Learning AlgorithmsArtificial Intelligence TutorialWhat is Deep LearningDeep Learning TutorialInstall TensorFlowDeep Learning with PythonBackpropagationTensorFlow TutorialConvolutional Neural Network TutorialVIEW ALL BI and Visualization What is TableauTableau TutorialTableau Interview QuestionsWhat is InformaticaInformatica Interview QuestionsPower BI TutorialPower BI Interview QuestionsOLTP vs OLAPQlikView TutorialAdvanced Excel Formulas TutorialVIEW ALL Big Data What is HadoopHadoop ArchitectureHadoop TutorialHadoop Interview QuestionsHadoop EcosystemData Science vs Big Data vs Data AnalyticsWhat is Big DataMapReduce TutorialPig TutorialSpark TutorialSpark Interview QuestionsBig Data TutorialHive TutorialVIEW ALL Blockchain Blockchain TutorialWhat is BlockchainHyperledger FabricWhat Is EthereumEthereum TutorialB lockchain ApplicationsSolidity TutorialBlockchain ProgrammingHow Blockchain WorksVIEW ALL Cloud Computing What is AWSAWS TutorialAWS CertificationAzure Interview QuestionsAzure TutorialWhat Is Cloud ComputingWhat Is SalesforceIoT TutorialSalesforce TutorialSalesforce Interview QuestionsVIEW ALL Cyber Security Cloud SecurityWhat is CryptographyNmap TutorialSQL Injection AttacksHow To Install Kali LinuxHow to become an Ethical Hacker?Footprinting in Ethical HackingNetwork Scanning for Ethical HackingARP SpoofingApplication SecurityVIEW ALL Data Science Python Pandas TutorialWhat is Machine LearningMachine Learning TutorialMachine Learning ProjectsMachine Learning Interview QuestionsWhat Is Data ScienceSAS TutorialR TutorialData Science ProjectsHow to become a data scientistData Science Interview QuestionsData Scientist SalaryVIEW ALL Data Warehousing and ETL What is Data WarehouseDimension Table in Data WarehousingData Warehousing Interview QuestionsData warehouse architectureTalend T utorialTalend ETL ToolTalend Interview QuestionsFact Table and its TypesInformatica TransformationsInformatica TutorialVIEW ALL Databases What is MySQLMySQL Data TypesSQL JoinsSQL Data TypesWhat is MongoDBMongoDB Interview QuestionsMySQL TutorialSQL Interview QuestionsSQL CommandsMySQL Interview QuestionsVIEW ALL DevOps What is DevOpsDevOps vs AgileDevOps ToolsDevOps TutorialHow To Become A DevOps EngineerDevOps Interview QuestionsWhat Is DockerDocker TutorialDocker Interview QuestionsWhat Is ChefWhat Is KubernetesKubernetes TutorialVIEW ALL Front End Web Development What is JavaScript â€" All You Need To Know About JavaScriptJavaScript TutorialJavaScript Interview QuestionsJavaScript FrameworksAngular TutorialAngular Interview QuestionsWhat is REST API?React TutorialReact vs AngularjQuery TutorialNode TutorialReact Interview QuestionsVIEW ALL Mobile Development Android TutorialAndroid Interview QuestionsAndroid ArchitectureAndroid SQLite DatabaseProgramming Algori... TensorFlow Ce rtification Training (41 Blogs) Become a Certified Professional AWS Global Infrastructure Introduction to Artificial Intelligence Artificial Intelligence â€" What It Is And How Is It Useful?Artificial Intelligence Tutorial : All you need to know about AIArtificial Intelligence Algorithms: All you need to knowTypes Of Artificial Intelligence You Should KnowWhat are the Advantages and Disadvantages of Artificial Intelligence?AI Applications: Top 10 Real World Artificial Intelligence ApplicationsImplementing Artificial Intelligence In HealthcareTop 10 Benefits Of Artificial IntelligenceHow to Become an Artificial Intelligence Engineer? A Roadmap to the FutureTop 12 Artificial Intelligence Tools Frameworks you need to knowAI vs Machine Learning vs Deep LearningA Comprehensive Guide To Artificial Intelligence With Python Introduction to Deep Learning What is Deep Learning? Getting Started With Deep LearningDeep Learning with Python : Beginners Guide to Deep LearningWhat Is A Neur al Network? Introduction To Artificial Neural NetworksDeep Learning Tutorial : Artificial Intelligence Using Deep LearningPyTorch vs TensorFlow: Which Is The Better Framework? Neural Networks Deep Learning : Perceptron Learning AlgorithmNeural Network Tutorial â€" Multi Layer PerceptronBackpropagation â€" Algorithm For Training A Neural Network Tensorflow A Step By Step Guide to Install TensorFlowTensorFlow Tutorial â€" Deep Learning Using TensorFlowConvolutional Neural Network Tutorial (CNN) â€" Developing An Image Classifier In Python Using TensorFlowCapsule Neural Networks â€" Set of Nested Neural LayersObject Detection Tutorial in TensorFlow: Real-Time Object DetectionTensorFlow Image Classification : All you need to know about Building ClassifiersRecurrent Neural Networks (RNN) Tutorial | Analyzing Sequential Data Using TensorFlow In Python Dimensionality Reduction Autoencoders Tutorial : A Beginner's Guide to AutoencodersRestricted Boltzmann Machine Tutorial â€" Introduc tion to Deep Learning Concepts Most Frequently Asked Artificial Intelligence Interview Questions Artificial Intelligence Topics CoveredMachine Learning with Mahout (7 Blogs)TensorFlow Certification Training (39 Blogs)Artificial Intelligence and Machine Learning (19 Blogs)SEE MORE Backpropagation Algorithm For Training A Neural Network Last updated on Apr 24,2020 61.5K Views Saurabh Saurabh is a technology enthusiast working as a Research Analyst at Edureka.... Saurabh is a technology enthusiast working as a Research Analyst at Edureka. His areas of interest are - DevOps, Artificial Intelligence, Big Data and...7 Comments Bookmark 3 / 3 Blog from Neural Networks Become a Certified Professional Backpropagation:Backpropagation is a supervised learning algorithm, for training Multi-layer Perceptrons (Artificial Neural Networks).I would recommend you to check out the following Deep Learning Certification blogs too:What is Deep Learning?Deep Learning TutorialTensorFlow TutorialNe ural Network TutorialBut, some of you might be wondering why we need to train a Neural Network or what exactly is the meaning of training.Why We Need Backpropagation?While designing a Neural Network, in the beginning, we initialize weights with some random values or any variable for that fact.Now obviously, we are not superhuman.So, its not necessary that whatever weight values we have selected will be correct, or it fits our model the best.Okay, fine, we have selected some weight values in the beginning, but our model output is way different than our actual output i.e. the error value is huge.Now, how will you reduce the error?Basically, what we need to do, weneed to somehow explain the model to change the parameters (weights), such that error becomes minimum.Lets put it in an another way, we need to train our model.One way to train our model is called as Backpropagation. Consider the diagram below:Let me summarize the steps for you:Calculate the error How far is your model output from the actual output.Minimum Error Check whether the error is minimized or not.Update the parameters If the error is huge then, update the parameters (weights and biases). After that again check the error. Repeat the process until the error becomes minimum.Model is ready to make a prediction Once the error becomes minimum, you can feed some inputs to your model and it will produce the output.I am pretty sure, now you know, why we need Backpropagation or why and what is the meaning of training a model.Now is the correct time to understand what is Backpropagation.What is Backpropagation?The Backpropagation algorithm looks for the minimum value of the error function in weight space using a technique called the delta rule or gradient descent. The weights that minimize the error function is then considered to be a solution to the learning problem.Lets understand how it works with an example:You have a dataset, which has labels.Consider the below table:InputDesired Output001224Now th e output of your model when W value is 3:InputDesired OutputModel output (W=3)000123246Notice the difference between the actual output and the desired output:InputDesired OutputModel output (W=3)Absolute ErrorSquare Error000001231124624Lets change the value of W. Notice the error when W = 4InputDesired OutputModel output (W=3)Absolute ErrorSquare ErrorModel output (W=4)Square Error0000000123114424624816Now if you notice, when we increase the value of W the error has increased. So, obviously there is no point in increasing the value of W further. But, what happens if I decrease the value of W? Consider the table below:InputDesired OutputModel output (W=3)Absolute ErrorSquare ErrorModel output (W=2)Square Error000000012324302462440Now, what we did here: We first initialized some random value to W and propagated forward.Then, we noticed that there is some error. To reduce that error, we propagated backwards and increased the value of W.After that, also we noticed that the error has inc reased. We came to know that, we cant increase the W value.So, we again propagated backwards and we decreased W value.Now, we noticed that the error has reduced.So, we are trying to get the value of weight such that the error becomes minimum. Basically, we need to figure out whether we need to increase or decrease the weight value. Once we know that, we keep on updating the weight value in that direction until error becomes minimum. You might reach a point, where if you further update the weight, the error will increase. At that time you need to stop, and that is your final weight value.Consider the graph below:We need to reach the Global Loss Minimum.This is nothing but Backpropagation.Lets now understand the math behind Backpropagation.How Backpropagation Works?Consider the below Neural Network:The above network contains the following:two inputstwo hidden neuronstwo output neuronstwo biasesBelow are the steps involved in Backpropagation:Step 1: Forward PropagationStep 2:Backward PropagationStep 3: Putting all the values together and calculating the updated weight valueStep 1: Forward PropagationWe will start by propagating forward.We will repeat this process for the output layer neurons, using the output from the hidden layer neurons as inputs.Now, lets see what is the value of the error:Step 2:Backward PropagationNow, we will propagate backwards. This way we will try to reduce the error by changing the values of weights and biases.Consider W5, we will calculate the rate of change of error w.r.t change in weight W5.Since we are propagating backwards, first thing we need to do is, calculate the change in total errors w.r.t the output O1 and O2.Now, we will propagate further backwards and calculate the change in output O1 w.r.t to its total net input.Lets see nowhow much does the total net input of O1 changesw.r.t W5?Step 3: Putting all the values together and calculating the updated weight valueNow, lets put all the values together:Lets calculate the up dated value of W5:Similarly, we can calculate the other weight values as well. After that we will again propagate forward and calculate the output. Again, we will calculate the error. If the error is minimum we will stop right there, else we will again propagate backwards and update the weight values. This process will keep on repeating until error becomes minimum.Conclusion:Well, if I have to conclude Backpropagation, the best option is to write pseudo code for the same.Backpropagation Algorithm:initialize network weights (often small random values) do forEach training example named ex prediction = neural-net-output(network, ex) // forward pass actual = teacher-output(ex) compute error (prediction - actual) at the output units compute{displaystyle Delta w_{h}}for all weights from hidden layer to output layer // backward pass compute{displaystyle Delta w_{i}}for all weights from input layer to hidden layer // backward pass continued update network weights // input layer not modified by error estimate until all examples classified correctly or another stopping criterion satisfied return the network I hope you have enjoyed reading this blog on Backpropagation, check out theDeep Learning with TensorFlow Trainingby Edureka,a trusted online learning companywith a network of more than250,000satisfied learnersspread acrossthe globe. The Edureka Deep Learning with TensorFlow Certification Training coursehelps learners becomeexpert in training and optimizing basic and convolutional neural networks using real time projects and assignments along with concepts such as SoftMax function, Auto-encoder Neural Networks, Restricted Boltzmann Machine (RBM).Got a question for us? Please mention it in the comments section and we will get back to you.Recommended videos for you What Is Deep Learning Deep Learning Simplified Watch Now Introduction to Mahout Watch Now Deep Learning Tutorial Deep Learning With TensorFlow Watch NowRecommended blogs for you Artificial Intelligence What It Is And How Is It Useful? Read Article What is the A* Algorithm and How does it work? Read Article Machine Learning Engineer Salary : How Much Does an ML Engineer Earn? Read Article What Is A Neural Network? Introduction To Artificial Neural Networks Read Article SQL For Data Science: One stop Solution for Beginners Read Article A Step By Step Guide to Install TensorFlow Read Article Types Of Artificial Intelligence You Should Know Read Article What is Cognitive AI? Is It the Future? Read Article How to Become an Artificial Intelligence Engineer? A Roadmap to the Future Read Article Theano vs TensorFlow : A Quick Comparision of Frameworks Read Article Deep Learning : Perceptron Learning Algorithm Read Article Top 10 Skills to Become a Machine Learning Engineer Read Article PyTorch Tutorial Implementing Deep Neural Networks Using PyTorch Read Article Supervised Learning In Apache Mahout Read Article TensorFlow Tutorial Deep Learning Using TensorFlow Read Article All You Need To Know About The Breadth First Search Algorithm Read Article Implementing Artificial Intelligence In Healthcare Read Art icle Pattern Recognition : How is it different from Machine Learning Read Article What is Production System in Artificial Intelligence? Read Article Understanding Distance Measures in Mahout Read Article Comments 7 Comments Trending Courses in Artificial Intelligence AI Deep Learning with TensorFlow18k Enrolled LearnersWeekendLive Class Reviews 5 (7000)