caffe document

Posted on   |   In

This document may be a little massi.. this is just served as an reference as I am current doing some experiment on caffe and wants to mark down something. I may make it tidy when I think it is time to :)

official caffe tour

  • nets, layers and blobs
  • forward/backwards
  • loss
  • solver
  • layer catalogue
  • interfaces
  • data

nets, layers and blobs

layer

  • layer categories: http://caffe.berkeleyvision.org/tutorial/layers.html
  • each layer have 3 computation:
    • setup
      • initialization of the layer and its connections is done at model initialization.
    • forward: bottom -(input)-> … -(output)-> top
    • backward: top -(gradient wrt output)-> …compute gradient wrt parameters.. -(gradient)-> bottom
      • takes the gradient with respect to the output, and computes the gradients with respect to the parameters and to the inputs

alt text

definition

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// 前四行-层属性
name:"conv1"
type:CONVOLUTION
// the link structure
bottom:"data"
top:"conv1"

// parameter of the layer
convolution_param{
    num_output:<span>20
    kernel_size:5
    stride:1
    weight_filler{
        type: "<span style="color: #c0504d;">xavier</span>"
    }
}

net

Blob

  • defined in caffe_pb2.BlobProto
  • attribute:
    • data; diff; num; channels; height; width
  • 4 dimension array
    • data: number/N(batch size of the data), channel/K, height, width
    • convolution weight: output, input, height, wifth
    • convoluion bias: output 1 1 1

datum

  • defined in caffe_pb2.BlobProto
  • attribute: shape, channels, height, width, data,
  • 3 dimensional array can be converted to datum by io.array_to_datum

#structure of caffe#
alt text

1.预处理图像的leveldb构建
输入：一批图像和label （2和3）
输出：leveldb （4）
指令里包含如下信息：
conver_imageset （构建leveldb的可运行程序）
train/ （此目录放处理的jpg或者其他格式的图像)
label.txt (图像文件名及其label信息)
输出的leveldb文件夹的名字
CPU/GPU (指定是在cpu上还是在gpu上运行code)
CNN网络配置文件

  1. network prototxt
    Imagenet_solver.prototxt （包含全局参数的配置的文件）
    Imagenet.prototxt （包含训练网络的配置的文件）
    Imagenet_val.prototxt （包含测试网络的配置文件）
    refer: http://www.csdn.net/article/2015-01-22/2823663

train your own model on your own data

  1. data preparation:
    1. The training and validation input are described in train.txt and val.txt as text listing all the files and their labels
    2. create the leveldbs with examples/imagenet/create_imagenet.sh
  2. Compute Image Mean:
    The model requires us to subtract the image mean from each image, so we have to compute the mean. tools/compute_image_mean.cpp
    the mean computation can be carried out as:
    ./examples/imagenet/make_imagenet_mean.sh
    make data/ilsvrc12/imagenet_mean.binaryproto
  3. Model Definition
    models/bvlc_reference_caffenet/train_val.prototxt
    noticed that the train and test network are both defined in this file and only the input layers and one output layer are different:
    for training, data input: `examples/imagenet/ilsvrc12_train_leveldb`
for testing, data input: `examples/imagenet/ilsvrc12_val_leveldb`
    lay out a protocol buffer for running the solver
    in `models/bvlc_reference_caffenet/solver.prototxt`.

file system

leveldb

  1. leveldb中数据按key/value形式存储。
  2. leveldb的使用介绍： http://zh.wikipedia.org/wiki/LevelDB
    3.将图像以及标注信息，先经过Datum序列化成字符串，再存到leveldb数据库中。见tools/convert_imageset.cpp
  3. leveldb学习资料： http://blog.csdn.net/hzqhbc/article/details/9367327

.py file

\python\caffe\classifier.py:

1
2


caffe.Net is the central interface for loading, configuring, and running models. caffe.Classsifier and caffe.Detector provide convenience interfaces for common tasks.
caffe.SGDSolver exposes the solving interface.
caffe.io handles input / output with preprocessing and protocol buffers.
caffe.draw visualizes network architectures.
Caffe blobs are exposed as numpy ndarrays for ease-of-use and efficiency.

callable python file:
classify.py

- argument:
    input image,directory or npy 
    output npy filename
    model-def file, default ../models/bvlc_reference_caffenet/deploy.prototxt
    pretrain model -- trained model weight file
         default ../models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel
    --gpu switch to gpu mode 
    --center only, switch for prediction from center crop alone instead of averaging prediction across crops
    -- image dim, default 256,256
    mean_file default caffe/imagenet/ilsvrc_2012_mean.npy
    --input scale, multiplly input features by this scale to finish preprocessing
    --raw scale, multiply raw input by this scale before preprocessing
    --channel_swap RGB-BGR since BGR is the caffe default by way of opencv
    --ext default jpg, image file to take as input
    --print-result, weite ourpur text or serializing to a file
- call classifier with
<!--2-->

- Load numpy array (.npy), directory glob (*.jpg), or image file.

1
inputs = [caffe.io.load_image(args.input_file)]
- classify and store

1
2
predictions = classifier.predict(inputs, not args.center_only)
np.save(args.output_file, predictions)

for .proto file:

  1. write a .proto file
  2. compile with protoc
  3. command:
    1
    2
    3
    4
    protoc--proto_path=IMPORT_PATH --cpp_out=DST_DIR --java_out=DST_DIR--python_out=DST_DIR path/to/file.proto

    //
    --proto_path 也可以简写成-I 是.proto所在的路径 输出路径： --cpp_out 要生成C++可用的头文件，分别是***.pb.h（包含申明类）***.pb.cc（包含可执行类），使用的时候只要include "***.pb.h" --java_out 生成java可用的头文件 --python_out 生成python可用的头文件，**_pb2.py，使用的时候import**_pb2.py即可 最后一个参数就是你的.proto文件完整路径。

type of prototxt

1
2
// examples/feature_extraction/imaget_val.prototxt
define the layers

http://htmlpreview.github.io/?https://github.com/google/leveldb/blob/master/doc/index.html
documentation of leveldb
try to run lmdb2mat but say db.Get(‘$d’ $(im_indx)) not exist?

how to run plt in ssh? use xwin
http://www.arsc.edu/arsc/knowledge-base/ssh-and-x11-forwarding-us/index.xml

run 00classification, need:

.caffemodel file

read weights and biases for a caffemodel given the prototxt would be to just load the network in Python and read the weights.

1
2
import caffe
net = caffe.Net(<prototxt-file>,<model-file>,<phase>);

and access the params from net.params

READ DEPLOTTXT

type of layer

http://cs231n.github.io/convolutional-networks/
QQQQQ: why need mean of image

  • data

  • conv1

    • Just like normal Hidden Layer
    • Connect neurons to the input in a local receptive field
    • All neurons in a single depth slice share weights

  • relu

  • norm
  • pool
  • fc: fully connect
  • drop

Read the code! Two most important files:
http://vision.stanford.edu/teaching/cs231n/slides/caffe_tutorial.pdf
● caffe/python/caffe/_caffe.cpp:
○ Exports Blob, Layer, Net, and Solver classes
● caffe/python/caffe/pycaffe.py
○ Adds extra methods to Net class

name: “AlexNet”

input: “data”
input_shape {
dim: 10
dim: 3
dim: 227
dim: 227
}

layer {
name: “conv1”
type: “Convolution”
bottom: “data”
top: “conv1”
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 2
decay_mult: 0
}
convolution_param {
num_output: 96
kernel_size: 11
stride: 4
}
}
layer {
name: “relu1”
type: “ReLU”
bottom: “conv1”
top: “conv1”
}
layer {
name: “norm1”
type: “LRN”
bottom: “conv1”
top: “norm1”
lrn_param {
local_size: 5
alpha: 0.0001
beta: 0.75
}
}
layer {
name: “pool1”
type: “Pooling”
bottom: “norm1”
top: “pool1”
pooling_param {
pool: MAX
kernel_size: 3
stride: 2
}
}
layer {
name: “pool2”
type: “Pooling”
bottom: “norm2”
top: “pool2”
pooling_param {
pool: MAX
kernel_size: 3
stride: 2
}
}
…
layer {
name: “fc7”
type: “InnerProduct”
bottom: “fc6”
top: “fc7”
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 2
decay_mult: 0
}
inner_product_param {
num_output: 4096
}
}
…
param {
lr_mult: 2
decay_mult: 0
}
inner_product_param {
num_output: 1000
}
}
layer {
name: “prob”
type: “Softmax”
bottom: “fc8”
top: “prob”
}

pipline for python

\python\caffe\io.py:

  • conversion between datum, narray and proto(blob)
  • preprocessing
  • image IO

\python\caffe\net_spec.py

\python\caffe\draw.py

\examples\pycaffe\caffenet.py

  • def conv_relu, fc_relu, max_pool, caffenet(the net itself), make_net

  • preprocessing

    • \python\caffe\io.py
      • class Transformer,
      • oversample

how to build a new layer:

  • Add a class declaration for your layer to the appropriate one of common_layers.hpp, data_layers.hpp,loss_layers.hpp, neuron_layers.hpp, or vision_layers.hpp. Include an inline implementation of type and the Blobs() methods to specify blob number requirements. Omit the _gpu declarations if you’ll only be implementing CPU code.
  • Implement your layer in layers/your_layer.cpp.
  • SetUp for initialization: reading parameters, allocating buffers, etc.
    Forward_cpu for the function your layer computes
    Backward_cpu for its gradient
    (Optional) Implement the GPU versions Forward_gpu and Backward_gpu in layers/your_layer.cu.
  • Add your layer to proto/caffe.proto, updating the next available ID. Also declare parameters, if needed, in this file.
  • Make your layer createable by adding it to layer_factory.cpp.
  • Write tests in test/test_your_layer.cpp. Use test/test_gradient_check_util.hpp to check that your Forward and Backward implementations are in numerical agreement.

include/caffe/vision_layers.hpp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#include "caffe/blob.hpp"
#include "caffe/common.hpp"
#include "caffe/common_layers.hpp"
#include "caffe/data_layers.hpp"
#include "caffe/layer.hpp"
#include "caffe/loss_layers.hpp"
#include "caffe/neuron_layers.hpp"
#include "caffe/proto/caffe.pb.h"

class BaseConvolutionLayer: public Layer<Dtype>
class ConvolutionLayer: public BaseConvolutionLayer<Dtype>
class DeconvolutionLayer : public BaseConvolutionLayer<Dtype>
class Im2colLayer : public Layer<Dtype>
// normalization
class LRNLayer : public Layer<Dtype>
class PoolingLayer : public Layer<Dtype>
class CuDNNPoolingLayer : public PoolingLayer<Dtype>
class CropLayer : public Layer<Dtype>
class SPPLayer : public Layer<Dtype>

2015/11/1
add new layer: crop layer

  1. add Crop_layers.cpp to source/caffe/layers
  2. add CROP ID 40 to caffe.proto
  3. add class croplayer to visual_layer.hpp
  4. 1
    2
    /home/xizeng/proj/caff/include/caffe/vision_layers.hpp:536:18: error: LayerParameter_LayerType▒ does not name a type
       virtual inline LayerParameter_LayerType type() const {

object netParameter

  • define in caffe.proto
  • arrtri: LayerParameter, input, input_dim, force_backward, NetState

blob.hpp

1
2
3
4
#include "caffe/common.hpp"
#include "caffe/proto/caffe.pb.h"
#include "caffe/syncedmem.hpp"
#include "caffe/util/math_functions.hpp"

attribute:
data
diff
count
capacity

python: module, class and namespace, and library

module

A module is simply a file containing Python code. This code can be in the form of Python classes, functions, or just a list of names. Each module gets it’s own global namespaces.

1
2
3
4
5
6
7
module1 called Integer 
module2 called FloatingPoint 
both have function named add(). 
// Once you import the module into your script, 
// you can access the names by prefixing them with the module name: 
FloatingPoint.add() 
Integer.add().

import

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#1
import SomeModule
//call by 
SomeModule.function()

#2
from SomeModule import SomeName
//no longer have to use a prefix
// in Interger.py define function add()
from Interger import add
a = add(1,2)
// but if your program has an add() function
// you’ll lose access to the Integer’s add() function.

#3 
from SomeModule import *

#4
import SomePackage
import scipy
print scipy.pi + 'from the scipy package'

import math
print math.pi + 'from the math module'

namespace

  • namespaces are just containers for mapping names to objects
    1
    2
    a_namespace = {'name_a':object_1, 'name_b':object_2, ...}
    b_namespace = {'name_a':object_3, 'name_b':object_4, ...}

Namespaces are also searched for names inside out. This means that if there is a certain name declared in the module’s global namespace, you can reuse the name inside a function while being certain that any other function will get the global name.

classes

Classes and namespaces have special interactions. The only way for a class’ methods to access it’s own variables or functions (as names) is to use a reference to itself.

the first argument of a method must be a ‘self’ parameter, if it to access other class attributes

the module has a global namespace, the class itself does not

below is different from other object-oriented languages
can def multi class in one modules

1
2
3
4
5
6
7
8
9
10
// in same python/modlues file / in same namespace
global data1;
global data2;

class Fruit{
  // define a function in the class and call the class object itself
  def add(self,color){
    ....
  }
}

library

a very good command for exploring library

1
2
3
4
5
6
import urllib // a library
dir(urllib) 
// shows the modules in the library(ie the files in the urllib folder)

help(urllib.urlopen) 
// show the detail of the modules

caffe

in caffe\__init__.py

  • import the function or attributes defined in other file in the same directory

log for bugs:

1
2
I0706 11:00:31.481590 8842 data_transformer.cpp:22] Loading mean file from: data/ilsvrc12/imagenet_mean.binaryproto
F0706 11:00:31.485407 8842 db_lmdb.hpp:13] Check failed: mdb_status == 0 (2 vs. 0) No such file or directory

原因是create_cifar10的时候设置了dbtype是leveldb但是train_text_prototxt(定义net)的时候用的是lmdb所以找不到train_imdb file