A Quick Note Comparing Two Implementations of Neural Networks in C++

Background

MLPack-ANN and Tiny-cnn are two good implementations of neural networks. Their design philosophy is different. This blog compares the two implementations from the point of view of executing a neural network on an embedded device and concludes that although MLPack-ANN may be expected to be faster, the flexibility that Tiny-cnn offers makes it a better model.

Design

MLPack-ANN implements neural networks in C++ using tuples. A tuple is an immutable ordered collection of objects whose states may change. Since the layers of a neural network are ordered, a tuple is a good way to represent a neural network. In C++ a tuple is a template and thus MLPack-ANN uses static polymorphism to configure a neural network.

Speed

Tiny-cnn uses a decorator pattern (Gamma, Helm, Johnson & Vlissides 1995, pp 175-184), implemented using virtual functions (dynamic polymorphism) to achieve the same result. One would expect that static polymorphism would be faster, although I am not aware of any empirical data to make that claim in this case. However, given that most of the time is spent performing matrix multiplications, both in the learning phase and execution phase the difference in run times is unlikely to be significant. Based on my experience, I have reason to believe that Tiny-cnn is in fact faster for training on MNist data.

Flexibility

The use of templates means that the neural network model has to be decided at compile time. The use of dynamic polymorphism allows the model to be decided at run time. For example the number of layers can be set in a configuration file. This is important because one would train the network using various models on a large machine on the cloud perhaps, and then pass the result to a device that can use the given model. For example a device could react to the sound of a crying baby and ignore other sounds.

Tiny-cnn configures the distance metric and the optimisation function at compile time beacuse they are both template parameters. Changing from static polymorphism to dynamic polymorphism in this case should not be difficult

Conclusion

Although virtual functions involve some overhead, the flexibility it offers far outweighs the cost in this case, because it allows to the model to be changed significantly at runtme.

References

  1. Erich Gamma, Richard Helm, Ralph Johnson & John Vlissides IN Design Patterns: Elements of Reusable Object Oriented Software, Addison Wesley 1995
Advertisements

About The Sunday Programmer

Joe is an experienced C++/C# developer on Windows. Currently looking out for an opening in C/C++ on Windows or Linux.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s