3. TensorFlow Lite とは?
TensorFlow Lite は、TensorFlowのモデルを
デバイス上(mobile / embedded / IoT)での推論を可能にする、
オープンソースのディープ ラーニング フレームワーク
デバイス上で推論すると、こんなにいいことが!
● レイテンシー :サーバーにデータを送らないので!
● プライバシー :データをデバイスに残す必要がない!
● コネクティビティ :インターネットに接続しなくていい!
● 電力消費 :ネットワーク接続は電気食い
TensorFlow Lite for Microcontrollers のお話は、今回しません
4. TensorFlow Lite を利用するステップ
● モデルの選択:学習する・学習済み
○ 学習はクラウド : Google Colaboratory (GPU, TPUも使える)
○ 学習済みモデルもたんまり
● 変換:TensorFlow => TensorFlow Lite
○ TensorFlow Lite Converter
● 最適化
○ 量子化 (8bit int, 16bit float), delegate
● デプロイ:専用の推論エンジン
○ TensorFlow Lite Interpreter
TensorFlow Lite guide を読めばいい!(一部は日本語に訳されているよ)
5. TensorFlow Lite Delegates
Delegate API は、まだ experimental なので、今後変更有
Delegate は、TensorFlow Liteのグラフの一部/全体をCPU以外の Executor
に委譲 (delegate) するもの
● GPU :OpenGL ES v3.1 / OpenCL / Metal (iOS)
● AIアクセラレータ :Google Edge TPU / Google Pixel Neural Core
● Android NN APIs :v1.0 (8.1) / v1.1 (9.0) / v1.2 (10.0)
● Flex :TensorFlow Select
● その他 :Arm NN (PR#33436)
6. TensorFlow Lite 2019 Roadmap
https://www.tensorflow.org/lite/guide/roadmap
● Updated NN API support
Full support for new Android Q NN API features, ops and types
● GPU backend optimizations
OpenCL and Vulkan support on Android
Metal and Objective-C CocoaPods for Metal acceleration
● Hexagon DSP backend
Initial release of DSP acceleration for pre-Android P devices
7. TensorFlow LiteDelegate documentation
https://www.tensorflow.org/lite/performance/delegates
Run inference with TensorFlow Lite in Python
https://coral.withgoogle.com/docs/edgetpu/tflite-python/
Python quickstart (tflite_runtime というのがある)
https://www.tensorflow.org/lite/guide/python
Edge TPU inferencing overview
https://coral.withgoogle.com/docs/edgetpu/inference/
Run inference with TensorFlow Lite in C++ (Edge TPU)
https://coral.withgoogle.com/docs/edgetpu/tflite-cpp/
13. もうひとつのModifyGraphWithDelegate
// Owning handle to a TfLiteDelegate instance.
using TfLiteDelegatePtr = std::unique_ptr<TfLiteDelegate, void (*)(TfLiteDelegate*)>;
/// Same as ModifyGraphWithDelegate except this interpreter takes
/// ownership of the provided delegate. Be sure to construct the unique_ptr
/// with a suitable destruction function.
/// WARNING: This is an experimental API and subject to change.
TfLiteStatus ModifyGraphWithDelegate(TfLiteDelegatePtr delegate);
14. Android NN APIs Delegate
// モデルの読み込み、推論エンジンへのモデルのロードまでは同じ
auto* delegate = NnApiDelegate();
interpreter->ModifyGraphWithDelegate(delegate);
// 推論エンジンの実行も同じ
// DEPRECATED: Please use StatefulNnApiDelegate class instead.
TfLiteDelegate* NnApiDelegate() {
static StatefulNnApiDelegate* delegate = new StatefulNnApiDelegate();
return delegate;
}
24. lite.experimental.load_delegate
@_tf_export('lite.experimental.load_delegate')
def load_delegate(library, options=None):
"""Returns loaded Delegate object.
Args:
library: Name of shared library containing the
[TfLiteDelegate](https://www.tensorflow.org/lite/performance/delegates).
options: Dictionary of options that are required to load the delegate. All
keys and values in the dictionary should be convertible to str. Consult
the documentation of the specific delegate for required and legal options.
(default None)
Returns:
Delegate object.
Raises:
ValueError: Delegate failed to load.
RuntimeError: If delegate loading is used on unsupported platform.
"""
25. lite.experimental.load_delegate
# TODO(b/137299813): Fix darwin support for delegates.
if sys.platform == 'darwin':
raise RuntimeError('Dynamic loading of delegates on Darwin not supported.')
try:
delegate = Delegate(library, options)
except ValueError as e:
raise ValueError('Failed to load delegate from {}n{}'.format(
library, str(e)))
return delegate
26. Delegate クラス
class Delegate(object):
"""Python wrapper class to manage TfLiteDelegate objects.
The shared library is expected to have two functions:
TfLiteDelegate* tflite_plugin_create_delegate(
char**, char**, size_t, void (*report_error)(const char *))
void tflite_plugin_destroy_delegate(TfLiteDelegate*)
The first one creates a delegate object. It may return NULL to indicate an
error (with a suitable error message reported by calling report_error()).
The second one destroys delegate object and must be called for every
created delegate object. Passing NULL as argument value is allowed, i.e.
tflite_plugin_destroy_delegate(tflite_plugin_create_delegate(...))
always works.
"""
28. lite.Interpreter
# Each delegate is a wrapper that owns the delegates that have been loaded
# as plugins. The interpreter wrapper will be using them, but we need to
# hold them in a list so that the lifetime is preserved at least as long as
# the interpreter wrapper.
self._delegates = []
if experimental_delegates:
self._delegates = experimental_delegates
for delegate in self._delegates:
self._interpreter.ModifyGraphWithDelegate(
delegate._get_native_delegate_pointer()) # pylint: disable=protected-access
Edge TPU CompilerがUpdate
https://vengineer.hatenablog.com/entry/71953852