Instruction following using Databricks Dolly 2.0 and OpenVINO
=============================================================

The instruction following is one of the cornerstones of the current
generation of large language models(LLMs). Reinforcement learning with
human preferences (`RLHF <https://arxiv.org/abs/1909.08593>`__) and
techniques such as `InstructGPT <https://arxiv.org/abs/2203.02155>`__
has been the core foundation of breakthroughs such as ChatGPT and GPT-4.
However, these powerful models remain hidden behind APIs and we know
very little about their underlying architecture. Instruction-following
models are capable of generating text in response to prompts and are
often used for tasks like writing assistance, chatbots, and content
generation. Many users now interact with these models regularly and even
use them for work but the majority of such models remain closed-source
and require massive amounts of computational resources to experiment
with.

`Dolly
2.0 <https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm>`__
is the first open-source, instruction-following LLM fine-tuned by
Databricks on a transparent and freely available dataset that is also
open-sourced to use for commercial purposes. That means Dolly 2.0 is
available for commercial applications without the need to pay for API
access or share data with third parties. Dolly 2.0 exhibits similar
characteristics so ChatGPT despite being much smaller.

In this tutorial, we consider how to run an instruction-following text
generation pipeline using Dolly 2.0 and OpenVINO. We will use a
pre-trained model from the `Hugging Face
Transformers <https://huggingface.co/docs/transformers/index>`__
library. To simplify the user experience, the `Hugging Face Optimum
Intel <https://huggingface.co/docs/optimum/intel/index>`__ library is
used to convert the models to OpenVINO™ IR format.

The tutorial consists of the following steps:

-  Install prerequisites
-  Download and convert the model from a public source using the
   `OpenVINO integration with Hugging Face
   Optimum <https://huggingface.co/blog/openvino>`__.
-  Compress model weights to INT8 with `OpenVINO
   NNCF <https://github.com/openvinotoolkit/nncf>`__
-  Create an instruction-following inference pipeline
-  Run instruction-following pipeline

About Dolly 2.0
---------------

Dolly 2.0 is an instruction-following large language model trained on
the Databricks machine-learning platform that is licensed for commercial
use. It is based on `Pythia <https://github.com/EleutherAI/pythia>`__
and is trained on ~15k instruction/response fine-tuning records
generated by Databricks employees in various capability domains,
including brainstorming, classification, closed QA, generation,
information extraction, open QA, and summarization. Dolly 2.0 works by
processing natural language instructions and generating responses that
follow the given instructions. It can be used for a wide range of
applications, including closed question-answering, summarization, and
generation.

The model training process was inspired by
`InstructGPT <https://arxiv.org/abs/2203.02155>`__. To train InstructGPT
models, the core technique is reinforcement learning from human feedback
(RLHF), This technique uses human preferences as a reward signal to
fine-tune models, which is important as the safety and alignment
problems required to be solved are complex and subjective, and aren’t
fully captured by simple automatic metrics. More details about the
InstructGPT approach can be found in OpenAI `blog
post <https://openai.com/research/instruction-following>`__ The
breakthrough discovered with InstructGPT is that language models don’t
need larger and larger training sets. By using human-evaluated
question-and-answer training, authors were able to train a better
language model using one hundred times fewer parameters than the
previous model. Databricks used a similar approach to create a prompt
and response dataset called they call
`databricks-dolly-15k <https://huggingface.co/datasets/databricks/databricks-dolly-15k>`__,
a corpus of more than 15,000 records generated by thousands of
Databricks employees to enable large language models to exhibit the
magical interactivity of InstructGPT. More details about the model and
dataset can be found in `Databricks blog
post <https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm>`__
and `repo <https://github.com/databrickslabs/dolly>`__

**Table of contents:**


-  `Prerequisites <#prerequisites>`__
-  `Convert model using Optimum-CLI
   tool <#convert-model-using-optimum-cli-tool>`__
-  `Compress model weights <#compress-model-weights>`__

   -  `Weights Compression using
      Optimum-CLI <#weights-compression-using-optimum-cli>`__

-  `Select model variant and inference
   device <#select-model-variant-and-inference-device>`__
-  `Instantiate Model using Optimum
   Intel <#instantiate-model-using-optimum-intel>`__
-  `Create an instruction-following inference
   pipeline <#create-an-instruction-following-inference-pipeline>`__

   -  `Setup imports <#setup-imports>`__
   -  `Prepare template for user
      prompt <#prepare-template-for-user-prompt>`__
   -  `Helpers for output parsing <#helpers-for-output-parsing>`__
   -  `Main generation function <#main-generation-function>`__
   -  `Helpers for application <#helpers-for-application>`__

-  `Run instruction-following
   pipeline <#run-instruction-following-pipeline>`__

Installation Instructions
~~~~~~~~~~~~~~~~~~~~~~~~~

This is a self-contained example that relies solely on its own code.

We recommend running the notebook in a virtual environment. You only
need a Jupyter server to start. For details, please refer to
`Installation
Guide <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide>`__.

Prerequisites
-------------



First, we should install the `Hugging Face
Optimum <https://huggingface.co/docs/optimum/installation>`__ library
accelerated by OpenVINO integration. The Hugging Face Optimum Intel API
is a high-level API that enables us to convert and quantize models from
the Hugging Face Transformers library to the OpenVINO™ IR format. For
more details, refer to the `Hugging Face Optimum Intel
documentation <https://huggingface.co/docs/optimum/intel/inference>`__.

.. code:: ipython3

    import os

    os.environ["GIT_CLONE_PROTECTION_ACTIVE"] = "false"

    %pip uninstall -q -y optimum optimum-intel
    %pip install --pre -Uq "openvino>=2024.2.0" openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
    %pip install -q "diffusers>=0.16.1" "transformers>=4.33.0" "torch>=2.1" "nncf>=2.10.0" onnx "gradio>=4.19" --extra-index-url https://download.pytorch.org/whl/cpu
    %pip install -q "git+https://github.com/huggingface/optimum-intel.git"

Convert model using Optimum-CLI tool
------------------------------------



`Optimum Intel <https://huggingface.co/docs/optimum/intel/index>`__ is
the interface between the
`Transformers <https://huggingface.co/docs/transformers/index>`__ and
`Diffusers <https://huggingface.co/docs/diffusers/index>`__ libraries
and OpenVINO to accelerate end-to-end pipelines on Intel architectures.
It provides ease-to-use cli interface for exporting models to `OpenVINO
Intermediate Representation
(IR) <https://docs.openvino.ai/2024/documentation/openvino-ir-format.html>`__
format.

The command bellow demonstrates basic command for model export with
``optimum-cli``

.. code:: bash

   optimum-cli export openvino --model <model_id_or_path> --task <task> <out_dir>

where ``--model`` argument is model id from HuggingFace Hub or local
directory with model (saved using ``.save_pretrained`` method),
``--task`` is one of `supported
task <https://huggingface.co/docs/optimum/exporters/task_manager>`__
that exported model should solve. For LLMs it will be
``text-generation-with-past``. If model initialization requires to use
remote code, ``--trust-remote-code`` flag additionally should be passed.

Compress model weights
----------------------



The `Weights
Compression <https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html>`__
algorithm is aimed at compressing the weights of the models and can be
used to optimize the model footprint and performance of large models
where the size of weights is relatively larger than the size of
activations, for example, Large Language Models (LLM). Compared to INT8
compression, INT4 compression improves performance even more, but
introduces a minor drop in prediction quality.

Weights Compression using Optimum-CLI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



You can also apply fp16, 8-bit or 4-bit weight compression on the
Linear, Convolutional and Embedding layers when exporting your model
with the CLI by setting ``--weight-format`` to respectively fp16, int8
or int4. This type of optimization allows to reduce the memory footprint
and inference latency. By default the quantization scheme for int8/int4
will be
`asymmetric <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Quantization.md#asymmetric-quantization>`__,
to make it
`symmetric <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Quantization.md#symmetric-quantization>`__
you can add ``--sym``.

For INT4 quantization you can also specify the following arguments:

- The ``--group-size`` parameter will define the group size to use for
  quantization, -1 it will results in per-column quantization.
- The ``--ratio`` parameter controls the ratio between 4-bit and 8-bit
  quantization. If set to 0.9, it means that 90% of the layers will be
  quantized to int4 while 10% will be quantized to int8.

Smaller group_size and ratio values usually improve accuracy at the
sacrifice of the model size and inference latency.

   **Note**: There may be no speedup for INT4/INT8 compressed models on
   dGPU.

.. code:: ipython3

    from IPython.display import Markdown, display
    import ipywidgets as widgets

    prepare_int4_model = widgets.Checkbox(
        value=True,
        description="Prepare INT4 model",
        disabled=False,
    )
    prepare_int8_model = widgets.Checkbox(
        value=False,
        description="Prepare INT8 model",
        disabled=False,
    )
    prepare_fp16_model = widgets.Checkbox(
        value=False,
        description="Prepare FP16 model",
        disabled=False,
    )

    display(prepare_int4_model)
    display(prepare_int8_model)
    display(prepare_fp16_model)



.. parsed-literal::

    Checkbox(value=True, description='Prepare INT4 model')



.. parsed-literal::

    Checkbox(value=False, description='Prepare INT8 model')



.. parsed-literal::

    Checkbox(value=False, description='Prepare FP16 model')


.. code:: ipython3

    from pathlib import Path

    model_id = "databricks/dolly-v2-3b"
    model_path = Path("dolly-v2-3b")

    fp16_model_dir = model_path / "FP16"
    int8_model_dir = model_path / "INT8_compressed_weights"
    int4_model_dir = model_path / "INT4_compressed_weights"


    def convert_to_fp16():
        if (fp16_model_dir / "openvino_model.xml").exists():
            return
        fp16_model_dir.mkdir(parents=True, exist_ok=True)
        export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format fp16".format(model_id)
        export_command = export_command_base + " " + str(fp16_model_dir)
        display(Markdown("**Export command:**"))
        display(Markdown(f"`{export_command}`"))
        ! $export_command


    def convert_to_int8():
        if (int8_model_dir / "openvino_model.xml").exists():
            return
        int8_model_dir.mkdir(parents=True, exist_ok=True)
        export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int8".format(model_id)
        export_command = export_command_base + " " + str(int8_model_dir)
        display(Markdown("**Export command:**"))
        display(Markdown(f"`{export_command}`"))
        ! $export_command


    def convert_to_int4():
        if (int4_model_dir / "openvino_model.xml").exists():
            return
        int4_model_dir.mkdir(parents=True, exist_ok=True)
        export_command_base = "optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int4 --ratio 1.0 --group-size 128".format(
            model_id
        )
        export_command = export_command_base + " " + str(int4_model_dir)
        display(Markdown("**Export command:**"))
        display(Markdown(f"`{export_command}`"))
        ! $export_command


    if prepare_fp16_model.value:
        convert_to_fp16()
    if prepare_int8_model.value:
        convert_to_int8()
    if prepare_int4_model.value:
        convert_to_int4()



**Export command:**



``optimum-cli export openvino --model databricks/dolly-v2-3b --task text-generation-with-past --weight-format int4 --ratio 1.0 --group-size 128 dolly-v2-3b/INT4_compressed_weights``


.. parsed-literal::

    2024-07-24 11:40:56.083018: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
    2024-07-24 11:40:56.084962: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
    2024-07-24 11:40:56.121994: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
    2024-07-24 11:40:56.122347: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
    To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2024-07-24 11:40:56.845683: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
      torch.utils._pytree._register_pytree_node(
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
      warn(
    WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
        PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.3.1+cpu)
        Python  3.8.18 (you have 3.8.10)
      Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
      Memory-efficient attention, SwiGLU, sparse and more won't be available.
      Set XFORMERS_MORE_DETAILS=1 for more details
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
      torch.utils._pytree._register_pytree_node(
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
      warn("The installed version of bitsandbytes was compiled without GPU support. "
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
      torch.utils._pytree._register_pytree_node(
    Framework not specified. Using pt to export the model.
    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
    Using framework PyTorch: 2.3.1+cpu
    Overriding 1 configuration item(s)
    	- use_cache -> True
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:934: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      assert batch_size > 0, "batch_size has to be defined and > 0"
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/modeling_attn_mask_utils.py:114: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if (input_shape[-1] > 1 or self.sliding_window is not None) and self.is_causal:
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/optimum/exporters/onnx/model_patcher.py:304: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if past_key_values_length > 0:
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:617: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if seq_len > self.max_seq_len_cached:
    INFO:nncf:Statistics of the bitwidth distribution:
    ┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
    │   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │
    ┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
    │              8 │ 9% (2 / 130)                │ 0% (0 / 128)                           │
    ├────────────────┼─────────────────────────────┼────────────────────────────────────────┤
    │              4 │ 91% (128 / 130)             │ 100% (128 / 128)                       │
    ┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
    [2KApplying Weight Compression ━━━━━━━━━━━━━━━━━━━ 100% 130/130 • 0:01:38 • 0:00:00;0;104;181m0:00:01181m0:00:04


.. code:: ipython3

    fp16_weights = fp16_model_dir / "openvino_model.bin"
    int8_weights = int8_model_dir / "openvino_model.bin"
    int4_weights = int4_model_dir / "openvino_model.bin"

    if fp16_weights.exists():
        print(f"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB")
    for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):
        if compressed_weights.exists():
            print(f"Size of model with INT{precision} compressed weights is {compressed_weights.stat().st_size / 1024 / 1024:.2f} MB")
        if compressed_weights.exists() and fp16_weights.exists():
            print(f"Compression rate for INT{precision} model: {fp16_weights.stat().st_size / compressed_weights.stat().st_size:.3f}")


.. parsed-literal::

    Size of model with INT4 compressed weights is 1497.06 MB


Select model variant and inference device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



select device from dropdown list for running inference using OpenVINO

.. code:: ipython3

    available_models = []
    if int4_model_dir.exists():
        available_models.append("INT4")
    if int8_model_dir.exists():
        available_models.append("INT8")
    if fp16_model_dir.exists():
        available_models.append("FP16")

    model_to_run = widgets.Dropdown(
        options=available_models,
        value=available_models[0],
        description="Model to run:",
        disabled=False,
    )

    model_to_run




.. parsed-literal::

    Dropdown(description='Model to run:', options=('INT4',), value='INT4')



.. code:: ipython3

    import ipywidgets as widgets
    import openvino as ov

    core = ov.Core()

    device = widgets.Dropdown(
        options=core.available_devices + ["AUTO"],
        value="CPU",
        description="Device:",
        disabled=False,
    )

    device




.. parsed-literal::

    Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')



Instantiate Model using Optimum Intel
-------------------------------------



Optimum Intel can be used to load optimized models from the `Hugging
Face Hub <https://huggingface.co/docs/optimum/intel/hf.co/models>`__ and
create pipelines to run an inference with OpenVINO Runtime using Hugging
Face APIs. The Optimum Inference models are API compatible with Hugging
Face Transformers models. This means we just need to replace
``AutoModelForXxx`` class with the corresponding ``OVModelForXxx``
class.

Below is an example of the Dolly model

.. code:: diff

   -from transformers import AutoModelForCausalLM
   +from optimum.intel.openvino import OVModelForCausalLM
   from transformers import AutoTokenizer, pipeline

   model_id = "databricks/dolly-v2-3b"
   -model = AutoModelForCausalLM.from_pretrained(model_id)
   +model = OVModelForCausalLM.from_pretrained(model_id, export=True)

Model class initialization starts with calling ``from_pretrained``
method. When downloading and converting Transformers model, the
parameter ``export=True`` should be added (as we already converted model
before, we do not need to provide this parameter). We can save the
converted model for the next usage with the ``save_pretrained`` method.
Tokenizer class and pipelines API are compatible with Optimum models.

You can find more details about OpenVINO LLM inference using HuggingFace
Optimum API in `LLM inference
guide <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html>`__.

.. code:: ipython3

    from pathlib import Path
    from transformers import AutoTokenizer
    from optimum.intel.openvino import OVModelForCausalLM

    if model_to_run.value == "INT4":
        model_dir = int4_model_dir
    elif model_to_run.value == "INT8":
        model_dir = int8_model_dir
    else:
        model_dir = fp16_model_dir
    print(f"Loading model from {model_dir}")

    tokenizer = AutoTokenizer.from_pretrained(model_dir)

    current_device = device.value

    ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}

    ov_model = OVModelForCausalLM.from_pretrained(model_dir, device=current_device, ov_config=ov_config)


.. parsed-literal::

    2024-07-24 11:43:17.404362: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
    2024-07-24 11:43:17.406313: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
    2024-07-24 11:43:17.443348: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
    2024-07-24 11:43:17.444995: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
    To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2024-07-24 11:43:18.193758: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
      torch.utils._pytree._register_pytree_node(
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
      warn(
    WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
        PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.3.1+cpu)
        Python  3.8.18 (you have 3.8.10)
      Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
      Memory-efficient attention, SwiGLU, sparse and more won't be available.
      Set XFORMERS_MORE_DETAILS=1 for more details
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
      torch.utils._pytree._register_pytree_node(
    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
      warn("The installed version of bitsandbytes was compiled without GPU support. "


.. parsed-literal::

    /home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
    Loading model from dolly-v2-3b/INT4_compressed_weights


.. parsed-literal::

    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
    Compiling the model to CPU ...


Create an instruction-following inference pipeline
--------------------------------------------------



The ``run_generation`` function accepts user-provided text input,
tokenizes it, and runs the generation process. Text generation is an
iterative process, where each next token depends on previously generated
until a maximum number of tokens or stop generation condition is not
reached. To obtain intermediate generation results without waiting until
when generation is finished, we will use
`TextIteratorStreamer <https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.TextIteratorStreamer>`__,
provided as part of HuggingFace `Streaming
API <https://huggingface.co/docs/transformers/main/en/generation_strategies#streaming>`__.

The diagram below illustrates how the instruction-following pipeline
works

.. figure:: https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/e881f4a4-fcc8-427a-afe1-7dd80aebd66e
   :alt: generation pipeline)

   generation pipeline)

As can be seen, on the first iteration, the user provided instructions
converted to token ids using a tokenizer, then prepared input provided
to the model. The model generates probabilities for all tokens in logits
format The way the next token will be selected over predicted
probabilities is driven by the selected decoding methodology. You can
find more information about the most popular decoding methods in this
`blog <https://huggingface.co/blog/how-to-generate>`__.

There are several parameters that can control text generation quality:

-  | ``Temperature`` is a parameter used to control the level of
     creativity in AI-generated text. By adjusting the ``temperature``,
     you can influence the AI model’s probability distribution, making
     the text more focused or diverse.
   | Consider the following example: The AI model has to complete the
     sentence “The cat is \____.” with the following token
     probabilities:

   | playing: 0.5
   | sleeping: 0.25
   | eating: 0.15
   | driving: 0.05
   | flying: 0.05

   -  **Low temperature** (e.g., 0.2): The AI model becomes more focused
      and deterministic, choosing tokens with the highest probability,
      such as “playing.”
   -  **Medium temperature** (e.g., 1.0): The AI model maintains a
      balance between creativity and focus, selecting tokens based on
      their probabilities without significant bias, such as “playing,”
      “sleeping,” or “eating.”
   -  **High temperature** (e.g., 2.0): The AI model becomes more
      adventurous, increasing the chances of selecting less likely
      tokens, such as “driving” and “flying.”

-  ``Top-p``, also known as nucleus sampling, is a parameter used to
   control the range of tokens considered by the AI model based on their
   cumulative probability. By adjusting the ``top-p`` value, you can
   influence the AI model’s token selection, making it more focused or
   diverse. Using the same example with the cat, consider the following
   top_p settings:

   -  **Low top_p** (e.g., 0.5): The AI model considers only tokens with
      the highest cumulative probability, such as “playing.”
   -  **Medium top_p** (e.g., 0.8): The AI model considers tokens with a
      higher cumulative probability, such as “playing,” “sleeping,” and
      “eating.”
   -  **High top_p** (e.g., 1.0): The AI model considers all tokens,
      including those with lower probabilities, such as “driving” and
      “flying.”

-  ``Top-k`` is another popular sampling strategy. In comparison with
   Top-P, which chooses from the smallest possible set of words whose
   cumulative probability exceeds the probability P, in Top-K sampling K
   most likely next words are filtered and the probability mass is
   redistributed among only those K next words. In our example with cat,
   if k=3, then only “playing”, “sleeping” and “eating” will be taken
   into account as possible next word.

To optimize the generation process and use memory more efficiently, the
``use_cache=True`` option is enabled. Since the output side is
auto-regressive, an output token hidden state remains the same once
computed for every further generation step. Therefore, recomputing it
every time you want to generate a new token seems wasteful. With the
cache, the model saves the hidden state once it has been computed. The
model only computes the one for the most recently generated output token
at each time step, re-using the saved ones for hidden tokens. This
reduces the generation complexity from O(n^3) to O(n^2) for a
transformer model. More details about how it works can be found in this
`article <https://scale.com/blog/pytorch-improvements#Text%20Translation>`__.
With this option, the model gets the previous step’s hidden states
(cached attention keys and values) as input and additionally provides
hidden states for the current step as output. It means for all next
iterations, it is enough to provide only a new token obtained from the
previous step and cached key values to get the next token prediction.

The generation cycle repeats until the end of the sequence token is
reached or it also can be interrupted when maximum tokens will be
generated. As already mentioned before, we can enable printing current
generated tokens without waiting until when the whole generation is
finished using Streaming API, it adds a new token to the output queue
and then prints them when they are ready.

Setup imports
~~~~~~~~~~~~~



.. code:: ipython3

    from threading import Thread
    from time import perf_counter
    from typing import List
    import gradio as gr
    from transformers import AutoTokenizer, TextIteratorStreamer
    import numpy as np

Prepare template for user prompt
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



For effective generation, model expects to have input in specific
format. The code below prepare template for passing user instruction
into model with providing additional context.

.. code:: ipython3

    INSTRUCTION_KEY = "### Instruction:"
    RESPONSE_KEY = "### Response:"
    END_KEY = "### End"
    INTRO_BLURB = "Below is an instruction that describes a task. Write a response that appropriately completes the request."

    # This is the prompt that is used for generating responses using an already trained model.  It ends with the response
    # key, where the job of the model is to provide the completion that follows it (i.e. the response itself).
    PROMPT_FOR_GENERATION_FORMAT = """{intro}

    {instruction_key}
    {instruction}

    {response_key}
    """.format(
        intro=INTRO_BLURB,
        instruction_key=INSTRUCTION_KEY,
        instruction="{instruction}",
        response_key=RESPONSE_KEY,
    )

Helpers for output parsing
~~~~~~~~~~~~~~~~~~~~~~~~~~



Model was retrained to finish generation using special token ``### End``
the code below find its id for using it as generation stop-criteria.

.. code:: ipython3

    def get_special_token_id(tokenizer: AutoTokenizer, key: str) -> int:
        """
        Gets the token ID for a given string that has been added to the tokenizer as a special token.

        When training, we configure the tokenizer so that the sequences like "### Instruction:" and "### End" are
        treated specially and converted to a single, new token.  This retrieves the token ID each of these keys map to.

        Args:
            tokenizer (PreTrainedTokenizer): the tokenizer
            key (str): the key to convert to a single token

        Raises:
            RuntimeError: if more than one ID was generated

        Returns:
            int: the token ID for the given key
        """
        token_ids = tokenizer.encode(key)
        if len(token_ids) > 1:
            raise ValueError(f"Expected only a single token for '{key}' but found {token_ids}")
        return token_ids[0]


    tokenizer_response_key = next(
        (token for token in tokenizer.additional_special_tokens if token.startswith(RESPONSE_KEY)),
        None,
    )

    end_key_token_id = None
    if tokenizer_response_key:
        try:
            end_key_token_id = get_special_token_id(tokenizer, END_KEY)
            # Ensure generation stops once it generates "### End"
        except ValueError:
            pass

Main generation function
~~~~~~~~~~~~~~~~~~~~~~~~



As it was discussed above, ``run_generation`` function is the entry
point for starting generation. It gets provided input instruction as
parameter and returns model response.

.. code:: ipython3

    def run_generation(
        user_text: str,
        top_p: float,
        temperature: float,
        top_k: int,
        max_new_tokens: int,
        perf_text: str,
    ):
        """
        Text generation function

        Parameters:
          user_text (str): User-provided instruction for a generation.
          top_p (float):  Nucleus sampling. If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.
          temperature (float): The value used to module the logits distribution.
          top_k (int): The number of highest probability vocabulary tokens to keep for top-k-filtering.
          max_new_tokens (int): Maximum length of generated sequence.
          perf_text (str): Content of text field for printing performance results.
        Returns:
          model_output (str) - model-generated text
          perf_text (str) - updated perf text filed content
        """

        # Prepare input prompt according to model expected template
        prompt_text = PROMPT_FOR_GENERATION_FORMAT.format(instruction=user_text)

        # Tokenize the user text.
        model_inputs = tokenizer(prompt_text, return_tensors="pt")

        # Start generation on a separate thread, so that we don't block the UI. The text is pulled from the streamer
        # in the main thread. Adds timeout to the streamer to handle exceptions in the generation thread.
        streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
        generate_kwargs = dict(
            model_inputs,
            streamer=streamer,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            top_p=top_p,
            temperature=float(temperature),
            top_k=top_k,
            eos_token_id=end_key_token_id,
        )
        t = Thread(target=ov_model.generate, kwargs=generate_kwargs)
        t.start()

        # Pull the generated text from the streamer, and update the model output.
        model_output = ""
        per_token_time = []
        num_tokens = 0
        start = perf_counter()
        for new_text in streamer:
            current_time = perf_counter() - start
            model_output += new_text
            perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)
            yield model_output, perf_text
            start = perf_counter()
        return model_output, perf_text

Helpers for application
~~~~~~~~~~~~~~~~~~~~~~~



For making interactive user interface we will use Gradio library. The
code bellow provides useful functions used for communication with UI
elements.

.. code:: ipython3

    def estimate_latency(
        current_time: float,
        current_perf_text: str,
        new_gen_text: str,
        per_token_time: List[float],
        num_tokens: int,
    ):
        """
        Helper function for performance estimation

        Parameters:
          current_time (float): This step time in seconds.
          current_perf_text (str): Current content of performance UI field.
          new_gen_text (str): New generated text.
          per_token_time (List[float]): history of performance from previous steps.
          num_tokens (int): Total number of generated tokens.

        Returns:
          update for performance text field
          update for a total number of tokens
        """
        num_current_toks = len(tokenizer.encode(new_gen_text))
        num_tokens += num_current_toks
        per_token_time.append(num_current_toks / current_time)
        if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:
            current_bucket = per_token_time[:-10]
            return (
                f"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}",
                num_tokens,
            )
        return current_perf_text, num_tokens


    def select_device(device_str: str, current_text: str = "", progress: gr.Progress = gr.Progress()):
        """
        Helper function for uploading model on the device.

        Parameters:
          device_str (str): Device name.
          current_text (str): Current content of user instruction field (used only for backup purposes, temporally replacing it on the progress bar during model loading).
          progress (gr.Progress): gradio progress tracker
        Returns:
          current_text
        """
        if device_str != ov_model._device:
            ov_model.request = None
            ov_model._device = device_str

            for i in progress.tqdm(range(1), desc=f"Model loading on {device_str}"):
                ov_model.compile()
        return current_text

Run instruction-following pipeline
----------------------------------



Now, we are ready to explore model capabilities. This demo provides a
simple interface that allows communication with a model using text
instruction. Type your instruction into the ``User instruction`` field
or select one from predefined examples and click on the ``Submit``
button to start generation. Additionally, you can modify advanced
generation parameters:

-  ``Device`` - allows switching inference device. Please note, every
   time when new device is selected, model will be recompiled and this
   takes some time.
-  ``Max New Tokens`` - maximum size of generated text.
-  ``Top-p (nucleus sampling)`` - if set to < 1, only the smallest set
   of most probable tokens with probabilities that add up to top_p or
   higher are kept for a generation.
-  ``Top-k`` - the number of highest probability vocabulary tokens to
   keep for top-k-filtering.
-  ``Temperature`` - the value used to module the logits distribution.

.. code:: ipython3

    import requests

    if not Path("gradio_helper.py").exists():
        r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/dolly-2-instruction-following/gradio_helper.py")
        open("gradio_helper.py", "w").write(r.text)

    from gradio_helper import make_demo

    demo = make_demo(run_fn=run_generation, select_device_fn=select_device)

    try:
        demo.queue().launch(debug=False, height=800)
    except Exception:
        demo.queue().launch(debug=False, share=True, height=800)
    # If you are launching remotely, specify server_name and server_port
    # EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`
    # To learn more please refer to the Gradio docs: https://gradio.app/docs/
