Preface
In machine learning, convolutional neural networks are deep feedforward artificial neural networks that have been successfully applied to image recognition. At present, many license plate number recognition, face recognition, etc. use convolutional neural networks. It can be said that convolutional neural networks have achieved great success in image recognition. There are many open source deep learning frameworks, such as caffe, tensorflow, torch, etc. These deep learning frameworks include the implementation of complete convolutional neural networks. So, why do we still have to write convolutional neural networks ourselves? It is better to use these open source deep learning frameworks directly, which are fast and easy to deal with, have good performance and few bugs. Yes, if you just use convolutional neural networks to do some applications and don’t care about its working principle, then you don’t have to work hard to write convolutional neural networks. But if you want to fully master the working principle of convolutional neural networks, the ancients said: What you get on paper is always shallow, and you must practice awareness at this time. Therefore, it is very necessary for you to implement the convolutional neural network yourself to deepen your understanding of it.
What is CupCnn
CupCnn is a convolutional neural network written in Java. In addition to work, I realized it in order to deepen my understanding of convolutional neural networks. It is simple enough and performs well, making it very suitable for beginners to refer to. Its source code can be downloaded from github: CupCnn
You don't have to worry about the limitations of its protocol or something. You can use it to do anything and modify it arbitrarily. If it can help you, I hope it can give you a star! ! !
^-^^-^^-^^-^
Design ideas
I hope it is a neural network that is simple enough to help beginners learn. So I didn't implement those concurrent acceleration stuff, which guarantees the introductory nature of the code. When designing, I divided the convolutional neural network into four modules: Network (layer blob loss active), which can be seen from the package name. layer, loss, and active all have a base class, and the programming of the entire neural network is oriented towards the base class. Network is the center that integrates these four modules, coordinates and dispatches resources. Each layer will have an instance of the Network, so that various data can be easily obtained through the Network, such as obtaining the output of each layer, diff, etc.
The design block diagram is as follows:
Saving parameters is very simple for Java. Implementing the Serializable interface can quickly implement the serialization and deserialization of parameters. CupCnn only implements the Serializable interface for Blob and BlobParams in the data directory, and all parameters are implemented by these two.
Current performance
Fully connected neural network
Currently, on the mnist dataset, a fully connected neural network (full connection (100) + full connection (30) + full connection (10) + softmax) is trained with 30 epoes, with an accuracy rate of 96.76
Convolutional neural network
Convolutional neural network (6 features) + maximum pooling + convolution (6 features) + full connection (512) + full connection (30) + full connection (10) + softmax), when the learning rate is 0.2, 30 epoes are trained, the accuracy rate is 97.79. I believe that after further parameter tuning, the accuracy rate can be higher under sufficient training.
The training snapshot of the convolutional neural network is as follows:
begin train
epoe: 0 lossValue: 2.3019369891560455 lr: 0.2 accuracy is 0.13
epoe: 0 lossValue: 2.0722489482105195 lr: 0.2 accuracy is 0.44
epoe: 0 lossValue: 1.2423286194012682 lr: 0.2 accuracy is 0.72
epoe: 0 lossValue: 0.7860529560675255 lr: 0.2 accuracy is 0.79
epoe: 0 lossValue: 0.6272194196176664 lr: 0.2 accuracy is 0.87
epoe: 0 lossValue: 0.5240051326725808 lr: 0.2 accuracy is 0.84
epoe: 0 lossValue: 0.27637563581928026 lr: 0.2 accuracy is 0.95
epoe: 0 lossValue: 0.35585388987055083 lr: 0.2 accuracy is 0.92
epoe: 0 lossValue: 0.441971528417802 lr: 0.2 accuracy is 0.92
epoe: 0 lossValue: 0.25637710325999674 lr: 0.2 accuracy is 0.95
epoe: 0 lossValue: 0.39872273532502 lr: 0.2 accuracy is 0.9
epoe: 1 lossValue: 0.264085484522027 lr: 0.160000000000000000003 accuracy is 0.91
epoe: 1 lossValue: 0.22754066024803088 lr: 0.160000000000000000003 accuracy is 0.96
epoe: 1 lossValue: 0.30256420975577103 lr: 0.160000000000000000003 accuracy is 0.96
epoe: 1 lossValue: 0.18149648622985948 lr: 0.16000000000000000003 accuracy is 0.99
epoe: 1 lossValue: 0.177239938748327 lr: 0.160000000000000000003 accuracy is 0.96
epoe: 1 lossValue: 0.15041993009777443 lr: 0.160000000000000000003 accuracy is 0.98
epoe: 1 lossValue: 0.10759545752665524 lr: 0.1600000000000000000003 accuracy is 1.0
The use of CupCnn
At present, CupCnn implements tests on mnist dataset. Under src/test, MnistTest is the entrance to the main function, and the specific neural network is built in the MnistNetwork class. In the MnistNetwork class, buildConvNetwork and buildFcNetwork respectively implement the construction of convolutional neural networks and the construction of fully connected neural networks. Thanks to Java's good cross-platform properties, after you download the source code of CupCnn, use eclipse to open the project, and then run it directly, you should be able to start training and testing on the mnist dataset.
Building a neural network
public void buildNetwork(){ //First build the neural network object and set the parameters network = new Network(); network.setBatch(100); network.setLoss(new LogLikeHoodLoss()); //network.setLoss(new CrossEntropyLoss()); optimizer = new SGDOptimizer(0.2); network.setOptimizer(optimizer); //buildFcNetwork(); buildConvNetwork(); network.prepare(); } The setBatch() function sets how many pictures are in a batch.
setLoss() sets the loss function to use. CupCnn implements the cross entropy loss function and the log-likelihood loss function.
setOptimizer() should be used to set the optimizer. CupCnn only implements the SGD optimizer. If you implement a better optimizer and are willing to submit it to CupCnn, I would like to deeply welcome it.
Build a fully connected neural network
private void buildFcNetwork(){ //Add network layer to the network InputLayer layer1 = new InputLayer(network,new BlobParams(network.getBatch(),1,28,28)); network.addLayer(layer1); FullConnectionLayer layer2 = new FullConnectionLayer(network,new BlobParams(network.getBatch(),784,1,1)); layer2.setActivationFunc(new ReluActivationFunc()); network.addLayer(layer2); FullConnectionLayer layer3 = new FullConnectionLayer(network,new BlobParams(network.getBatch(),100,1,1)); layer3.setActivationFunc(new ReluActivationFunc()); network.addLayer(layer3); FullConnectionLayer layer4 = new FullConnectionLayer(network,new BlobParams(network.getBatch(),30,1,1)); layer4.setActivationFunc(new SigmodActivationFunc()); network.addLayer(layer4); FullConnectionLayer layer5 = new FullConnectionLayer(network,new BlobParams(network.getBatch(),10,1,1)); layer5.setActivationFunc(new ReluActivationFunc()); network.addLayer(layer5); SoftMaxLayer sflayer = new SoftMaxLayer(network,new BlobParams(network.getBatch(),10,1,1)); network.addLayer(sflayer); }As shown in the above code, each layer needs a network, which is an instance of the Network. The Network is the global administrator and the dispatcher of resources. With the reference of the Network, we can easily obtain the output data, output errors, etc. of each layer. In addition, each layer needs a parameter that specifies the size of the current layer's output data block, which tells a certain layer how much data you need to output. For example, the last layer of a neural network is SoftMaxLayer, which number needs to be output. This number is represented by a vector of length 10, such as the number 7, so SoftMaxLayer should output the value of the 8th element to be 1 and the value of other elements to be 0. The convolution layer and the pooling layer require more parameters because they both have a kernel. For the convolution layer, it is called a convolution kernel. The stride of each direction of the convolution layer is 1, which is still room for improvement. For the pooling layer, in addition to passing in the parameters of the pooling core, you also need to passing in the horizontal and vertical steps, which is necessary.
Training and testing
After building a neural network, you need to call the network.prepare() method, which will create output data blocks and error data blocks based on the data parameters of each layer. Therefore, the call to this method is necessary.
public void train(List<DigitImage> imgList,int epoes){ System.out.println("begin train"); int batch = network.getBatch(); double loclaLr = optimizer.getLr(); for(int e=0;e<epoes;e++){ Collections.shuffle(imgList); for(int i=0;i<imgList.size()-batch;i+=batch){ List<Blob> inputAndLabel = buildBlobByImageList(imgList,i,batch,1,28,28); double lossValue = network.train(inputAndLabel.get(0), inputAndLabel.get(1)); if(i>batch && i/batch%50==0){ System.out.print("epoe: "+e+" lossValue: "+lossValue+" "+" lr: "+optimizer.getLr()+" "); testInner(inputAndLabel.get(0), inputAndLabel.get(1)); } } if(loclaLr>0.001){loclaLr*=0.8; optimizer.setLr(loclaLr); } } } public void test(List<DigitImage> imgList){ System.out.println("begin test"); int batch = network.getBatch(); int correctCount = 0; int i = 0; for(i=0;i<imgList.size()-batch;i+=batch){ List<Blob> inputAndLabel = buildBlobByImageList(imgList,i,batch,1,28,28); Blob output = network.predict(inputAndLabel.get(0)); int[] calOutLabels = getBatchOutputLabel(output.getData()); int[] realLabels = getBatchOutputLabel(inputAndLabel.get(1).getData()); for(int kk=0;kk<calOutLabels.length;kk++){ if(calOutLabels[kk] == realLabels[kk]){ correctCount++; } } } double accuracy = correctCount/(1.0*i+batch); System.out.println("test accuracy is "+accuracy+" correctCount "+correctCount); }As mentioned above, you can train by calling the Network's train, and you can test by calling the Network's predict method.
Save and load parameters
public void saveModel(String name){ network.saveModel(name); } public void loadModel(String name){ network = new Network(); network.loadModel(name); network.prepare(); }Calling the saveModel and loadModel of Network can respectively save and load parameters. You only need to pass in a file name. When we create a neural network through saved parameters, we need to first new a Network, then call the loadModel of this network to load the saved parameters, and then don't forget to call the prepare method to create output data blocks and error data blocks for each layer.
Current completion status and future plans
Currently, the layers implemented include: full connection, convolution, maximum pooling layer, average pooling layer, and softmax layer. The activation functions implemented are: sigmod, tanh, relu.
The implemented loss functions are: cross entropy, log-likelihood. The optimization implemented is: SGD. The parameters can already save and load. Next, the dropout layer will be added, and the examples on cifar-10 will be added.
In addition, I will write some articles to review my thoughts and questions during the process of writing CupCnn for reference by beginners. Please take a detour. Those who are interested can continue to pay attention. I hope it will be helpful to everyone's learning and I hope everyone will support Wulin.com more.