References
Amini, A. (2018, February 01). MIT 6.S191: Introduction to deep learning. Retrieved January 6, 2019, from https://www.youtube.com/watch?v=JN6H4rQvwgY
Dietterich, T. (1995). Overfitting and under computing in machine learning. ACM Computing Surveys,27(3), 326-327. doi:10.1145/212094.212114
Gendler, A. (2016, April 25). The Turing Test: Can a computer pass for a human? Retrieved September 11, 2018, from https://www.youtube.com/watch?v=3wLqsRLvV-c
Glorot, X., & Bengio, Y. (2018). Understanding the difficulty of training deep feedforward Neural Networks. History Studies International Journal of History,10(7), 241-264. doi:10.9737/hist.2018.658
Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning. Reading, MA: Addison-Wesley.
Gregg, B. (2013). Table 2.2. In systems performance: Enterprise and the cloud (1st ed.). Prentice Hall.
Grimson, E. (2017, May 19). 11. Introduction to machine learning. Retrieved November 27, 2018, from https://www.youtube.com/watch?v=h0e2HAPTGF4
Larochelle, H. (2016, October 12). The deep end of deep learning. Retrieved December 28, 2018, from https://www.youtube.com/watch?v=dz_jeuWx3j0
LeBlanc, A. (2015, January 12). Artificial intelligence and the future. Retrieved December 18, 2018, from https://www.youtube.com/watch?v=xH_B5xh42xc
Leedy, P. D., & Ormrod, J. E. (2016). Practical Research(11th ed.). Pearson Education.
Loy, J. (2018, May 14). How to build your own Neural Network from scratch in python. Towards Data Science. Retrieved October 2, 2018, from https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6
Mallawaarachchi, V. (2017). Introduction to Genetic Algorithms?. Towards Data Science. Retrieved January 19, 2019, from https://towardsdatascience.com/introduction-to-genetic-algorithms-including-example-code-e396e98d8bf3.
Mansor, M. A., & Sathasivam, S. (2016). Activation function comparison in neural-symbolic integration. American Institute of Physics. doi:10.1063/1.4954526
McCaffrey, J. D. (2014). Neural Networks Using C# Succinctly. Retrieved January 16, 2019, from https://www.syncfusion.com/ebooks/neuralnetworks/neural-networks
Reptile [Def. 2]. (n.d.). In Merriam Webster. Retrieved March 3, 2019, from https://www.merriam-webster.com/dictionary/reptile
Mitchell, M. (2018, March 12). How we can build AI to help humans, not hurt us. Retrieved September 12, 2018, from https://www.youtube.com/watch?v=twWkGt33X_k
Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Searching for activation functions. ArXiv. Retrieved December 12, 2018, from https://arxiv.org/pdf/1710.05941.pdf.
Raval, S. (2017, May 26). Which activation function should I use? Retrieved December 19, 2018, from https://www.youtube.com/watch?v=-7scQpJT7uo
Raval, S. (2017, January 13). Intro to deep learning #1. Retrieved January 2, 2019, from https://www.youtube.com/watch?v=vOppzHpvTiQ
Raval, S. (2017, January 20). Intro to deep learning #2. Retrieved January 3, 2019, from https://www.youtube.com/watch?v=p69khggr1Jo
Raval, S. (2018, July 08). Backpropagation explained. Retrieved January 5, 2019, from https://www.youtube.com/watch?v=FaHHWdsIYQg
Reed, R., & Marks, R. J. (1999). Learning Rate and Momentum. Neural Smithing. doi:10.7551/mitpress/4937.003.0007
Richter, J. (2012). CLR via c#(4th ed.). Microsoft Press.
Rosenblatt, Frank (1962). Principles of neurodynamics. New York: Spartan. Cf. Rumelhart, D.E., J. L. McClelland and the PDP Research Group (1986). Parallel Distributed Processing vol. 1&2. Cambridge: MIT.
Sanderson, G. (2017, November 03). Backpropagation calculus: Deep learning, chapter 4. Retrieved October 3, 2018, from https://www.youtube.com/watch?v=tIeHLnjs5U8
Sanderson, G. (2017, October 05). But what is a Neural Network? Deep learning, chapter 1. Retrieved September 24, 2018, from https://www.youtube.com/watch?v=aircAruvnKk
Sanderson, G. (2017, October 16). Gradient descent, how Neural Networks learn: Deep learning, chapter 2. Retrieved September 24, 2018, from https://www.youtube.com/watch?v=IHZwWFHWa-w
Sanderson, G. (2017, November 03). What is backpropagation really doing? Deep learning, chapter 3. Retrieved September 25, 2018, from https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=24s&list=WL&index=7
SethBling. (2015, June 13). MarI/O: Machine learning for video games. Retrieved October 31, 2018, from https://www.youtube.com/watch?v=qv6UVOQ0F44
Shah, D. (2017). Activation functions. Towards Data Science. Retrieved January 2, 2019, from https://towardsdatascience.com/activation-functions-in-neural-networks-58115cda9c96.
Skeet, J. (2014). C# in depth(3rd ed.). Shelter Island, NY: Manning.
Strauss, A., & Corbin, J. (1994). Grounded theory methodology. Handbook of Qualitative Research, 17, 273-285. Retrieved November 29, 2018, from http://www.depts.ttu.edu/education/our-people/Faculty/additional_pages/duemer/epsy_5382_class_materials/Grounded-theory-methodology.pdf
Tyka, M. (2015, December 07). The art of Neural Networks. Retrieved January 2, 2019, from https://www.youtube.com/watch?v=0qVOUD76JOg
Winston, P. H. (2010, Fall). Lecture 6: Search: Games, minimax, and alpha-beta. Retrieved October 9, 2018, from https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/lecture-videos/lecture-6-search-games-minimax-and-alpha-beta
Zell, A. (1997). Simulation neuronaler Netze. Bonn: Addison-Wesley.
Appendix A
Glossary of Terms and Definitions
| Term | Definition |
| Activation Function | A function that is used in all neurons within a Neural Network to constrain the output of each neuron into a range that is useful to the programmer (McCaffrey, 2014; Raval, “Which activation function should I use?” 2017; Shah, 2017). |
| Computer Logic Gate | An electronic component that performs a logical function. |
| Data Structure | A method of management/organization of data in software that allows for efficient access and modification. |
| Dendrite | A connection between two neurons within a Neural Network. Each dendrite has a weight which is used in the calculation of a neuron’s value (McCaffrey, 2014; Sanderson, “What is backpropagation really doing? Deep learning, chapter 3” 2017; Sanderson, “Backpropagation calculus: Deep learning, chapter 4” 2017). |
| Error | The Error refers to the overall deviation from the desired outputs of the Neural Network. The error is calculated as the Neural Network’s average squared difference from the Neural Network prediction value and corresponding correct value. |
| Epoch/Generation | Every time a Neural Network revisits each of the given test data points when training with a learning algorithm. |
| Feed Forward | The process in which a Neural Network passes values through its neurons to compute the overall output of the Neural Network (Zell, 1997). |
| Fitness Rating | The rating that is given to the specific Neural Network that measures how well the Neural Network completed the given task. |
| Fitness Calculation | A calculation that the Genetics Learning Algorithm performs to find each Neural Network’s fitness rating. In this research project specifically, the fitness ratings were calculated as the Neural Network’s error or deviation from the desired output(s). |
| Learning algorithms | Computational formulas used in machine learning to help the technology imitate the human learning process (Grimson, 2017). |
| Learning/Training Process | The process in which a Neural Network’s dendrites are manipulated by a Learning Algorithm. This process is what “teaches” the Neural Network allowing it to successfully complete tasks with minimal error. |
| Learning Rate | The constant value in which the Backpropagation Learning Algorithm uses to limit its dendrite weight updates lowering the magnitude of its steps towards a lower error (Sanderson, “Backpropagation calculus: Deep learning, chapter 4” 2017). |
| Machine Learning | A method of data analysis that uses computers to identify patterns and make decisions with minimal human intervention (Grimson, 2017). |
| Momentum Rate | The multiplier used to limit the influence of the previous dendrite weight updates on the new dendrite weight update. |
| Neural Network | A data structure that acts as a mathematical function mapping specific inputs to corresponding outputs (Loy, 2018; McCaffrey, 2014; Zell, 1997). |
| Overfitting | The process in machine learning where a learning algorithm too closely fits a limited set of data points hindering it from understanding the underlying pattern and thus from predicting future points in the series (Dietterich, 1995). |
| Plateau | A period of time in which the error of a learning algorithm is stagnant despite continuous training. |
| Training Data | Data that the learning algorithm uses to train the Neural Network. |
| Supervised Data | Data that is labeled with the correct output that the Neural Network is set to predict. |
| Weight Update | A weight update is a value calculated by the Backpropagation Learning Algorithm which identifies the direction and distance each dendrite should follow to minimize the Neural Network’s error. |
Appendix B
XOR Test Results
| Backpropagation Learning Algorithm | Genetics Learning Algorithm | |||||
| Test Number | Epoch Count | Time (Milliseconds) | Test Number | Generation Count | Time (Milliseconds) | |
| 1 | 3656 | 249.6213 | 1 | 209 | 305.5349 | |
| 2 | 5582 | 284.6728 | 2 | 432 | 620.5403 | |
| 3 | 2819 | 171.0377 | 3 | 102 | 214.5134 | |
| 4 | 3387 | 251.8245 | 4 | 164 | 318.0572 | |
| 5 | 9455 | 530.5284 | 5 | 568 | 996.9213 | |
| 6 | 6509 | 401.9979 | 6 | 766 | 1286.4294 | |
| 7 | 3153 | 223.0116 | 7 | 118 | 236.9378 | |
| 8 | 3261 | 163.9362 | 8 | 227 | 340.9875 | |
| 9 | 6003 | 301.9228 | 9 | 162 | 251.1158 | |
| 10 | 4557 | 233.7867 | 10 | 118 | 185.4179 | |
| 11 | 10691 | 549.0612 | 11 | 177 | 279.5639 | |
| 12 | 6955 | 369.4169 | 12 | 367 | 612.7986 | |
| 13 | 4015 | 209.9101 | 13 | 134 | 221.8533 | |
| 14 | 5147 | 280.4362 | 14 | 200 | 293.5432 | |
| 15 | 7369 | 399.9776 | 15 | 174 | 278.4152 | |
| 16 | 5228 | 296.4309 | 16 | 207 | 330.7115 | |
| 17 | 3418 | 181.2077 | 17 | 120 | 177.5634 | |
| 18 | 12543 | 647.6394 | 18 | 260 | 376.7631 | |
| 19 | 11302 | 600.0225 | 19 | 134 | 203.6898 | |
| 20 | 3614 | 191.3684 | 20 | 127 | 189.2223 | |
| 21 | 9749 | 458.3669 | 21 | 228 | 335.4027 | |
| 22 | 4957 | 269.0165 | 22 | 117 | 185.0294 | |
| 23 | 6783 | 353.1238 | 23 | 195 | 292.0773 | |
| 24 | 7511 | 407.3662 | 24 | 121 | 192.3093 | |
| 25 | 8032 | 454.6105 | 25 | 126 | 223.1806 | |
| 26 | 8198 | 456.6348 | 26 | 141 | 227.1523 | |
| 27 | 5656 | 294.3335 | 27 | 101 | 169.9704 | |
| 28 | 5165 | 262.7309 | 28 | 1435 | 1863.1769 | |
| 29 | 5968 | 313.0351 | 29 | 139 | 231.7074 | |
| 30 | 4831 | 235.9236 | 30 | 106 | 214.5134 | |
| Epoch Count | Time (Milliseconds) | Generation Count | Time (Milliseconds) | |||
| Minimum: | 2819 | 163.9362 | Minimum: | 101 | 169.9704 | |
| 1st Quartile: | 4150.5 | 239.348025 | 1st Quartile: | 122.25 | 214.5134 | |
| Median: | 5619 | 295.3822 | Median: | 163 | 264.7655 | |
| 3rd Quartile: | 7475.5 | 406.024125 | 3rd Quartile: | 222 | 334.2299 | |
| Maximum: | 12543 | 647.6394 | Maximum: | 1435 | 1863.1769 |
Appendix C
Reptile Classification Test Results
| Training Time (in milliseconds) | Backpropagation Prediction Percent | Genetics Prediction Percent |
| 0 | 61.54 | 53.85 |
| 15 | 46.15 | 61.54 |
| 30 | 92.31 | 61.54 |
| 45 | 92.31 | 61.54 |
| 60 | 92.31 | 61.54 |
| 75 | 100 | 61.54 |
| 90 | 100 | 61.54 |
| 105 | 100 | 61.54 |
| 120 | 100 | 61.54 |
| 135 | 100 | 61.54 |
| 150 | 100 | 69.23 |
| 165 | 100 | 69.23 |
| 180 | 100 | 61.54 |
| 195 | 100 | 61.54 |
| 210 | 100 | 61.54 |
| 225 | 100 | 92.31 |
| 240 | 100 | 92.31 |
| 255 | 100 | 92.31 |
| 270 | 100 | 100 |
| 285 | 100 | 100 |
| 300 | 100 | 100 |
| 315 | 100 | 92.31 |
| 330 | 100 | 92.31 |
| 345 | 100 | 100 |
| 360 | 100 | 100 |
| 375 | 100 | 100 |
| 390 | 100 | 100 |
| 405 | 100 | 100 |
| 420 | 100 | 100 |
Appendix D
Activation Function Definitions
| Sigmoid Function |  |  |
| Soft Sign Function |  |  |
Appendix E
Project Source Code
The project source code written to conduct the research has been publicly shared through Github for other researchers to access. The following link has the code which was created under Amitai’s Github account.
