Automatic speech recognition using Distil-Whisper and OpenVINO
==============================================================

`Distil-Whisper <https://huggingface.co/distil-whisper/distil-large-v2>`__
is a distilled variant of the
`Whisper <https://huggingface.co/openai/whisper-large-v2>`__ model by
OpenAI. The Distil-Whisper is proposed in the paper `Robust Knowledge
Distillation via Large-Scale Pseudo
Labelling <https://arxiv.org/abs/2311.00430>`__. According to authors,
compared to Whisper, Distil-Whisper runs in several times faster with
50% fewer parameters, while performing to within 1% word error rate
(WER) on out-of-distribution evaluation data.

Whisper is a Transformer based encoder-decoder model, also referred to
as a sequence-to-sequence model. It maps a sequence of audio spectrogram
features to a sequence of text tokens. First, the raw audio inputs are
converted to a log-Mel spectrogram by action of the feature extractor.
Then, the Transformer encoder encodes the spectrogram to form a sequence
of encoder hidden states. Finally, the decoder autoregressively predicts
text tokens, conditional on both the previous tokens and the encoder
hidden states.

You can see the model architecture in the diagram below:

.. figure:: https://user-images.githubusercontent.com/29454499/204536571-8f6d8d77-5fbd-4c6d-8e29-14e734837860.svg
   :alt: whisper_architecture.svg

   whisper_architecture.svg

In this tutorial, we consider how to run Distil-Whisper using OpenVINO.
We will use the pre-trained model from the `Hugging Face
Transformers <https://huggingface.co/docs/transformers/index>`__
library. To simplify the user experience, the `Hugging Face
Optimum <https://huggingface.co/docs/optimum>`__ library is used to
convert the model to OpenVINO™ IR format. To further improve OpenVINO
Distil-Whisper model performance ``INT8`` post-training quantization
from `NNCF <https://github.com/openvinotoolkit/nncf/>`__ is applied.

**Table of contents:**


-  `Prerequisites <#prerequisites>`__
-  `Load PyTorch model <#load-pytorch-model>`__

   -  `Prepare input sample <#prepare-input-sample>`__
   -  `Run model inference <#run-model-inference>`__

-  `Load OpenVINO model using Optimum
   library <#load-openvino-model-using-optimum-library>`__

   -  `Select Inference device <#select-inference-device>`__
   -  `Compile OpenVINO model <#compile-openvino-model>`__
   -  `Run OpenVINO model inference <#run-openvino-model-inference>`__

-  `Compare performance PyTorch vs
   OpenVINO <#compare-performance-pytorch-vs-openvino>`__
-  `Usage OpenVINO model with HuggingFace
   pipelines <#usage-openvino-model-with-huggingface-pipelines>`__
-  `Quantization <#quantization>`__

   -  `Prepare calibration datasets <#prepare-calibration-datasets>`__
   -  `Quantize Distil-Whisper encoder and decoder
      models <#quantize-distil-whisper-encoder-and-decoder-models>`__
   -  `Run quantized model inference <#run-quantized-model-inference>`__
   -  `Compare performance and accuracy of the original and quantized
      models <#compare-performance-and-accuracy-of-the-original-and-quantized-models>`__

-  `Interactive demo <#interactive-demo>`__

Installation Instructions
~~~~~~~~~~~~~~~~~~~~~~~~~

This is a self-contained example that relies solely on its own code.

We recommend running the notebook in a virtual environment. You only
need a Jupyter server to start. For details, please refer to
`Installation
Guide <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide>`__.

Prerequisites
-------------



.. code:: ipython3

    %pip install -q "transformers>=4.35" "torch>=2.1,<2.4.0" "torchvision<0.19.0" onnx "peft==0.6.2" --extra-index-url https://download.pytorch.org/whl/cpu
    %pip install -q "git+https://github.com/huggingface/optimum-intel.git"
    %pip install -q "openvino>=2023.2.0" datasets  "gradio>=4.0" "librosa" "soundfile"
    %pip install -q "nncf>=2.6.0" "jiwer"

Load PyTorch model
------------------



The ``AutoModelForSpeechSeq2Seq.from_pretrained`` method is used for the
initialization of PyTorch Whisper model using the transformers library.
By default, we will use the ``distil-whisper/distil-large-v2`` model as
an example in this tutorial. The model will be downloaded once during
first run and this process may require some time.

You may also choose other models from `Distil-Whisper hugging face
collection <https://huggingface.co/collections/distil-whisper/distil-whisper-models-65411987e6727569748d2eb6>`__
such as ``distil-whisper/distil-medium.en`` or
``distil-whisper/distil-small.en``. Models of the original Whisper
architecture are also available, more on them
`here <https://huggingface.co/openai>`__.

Preprocessing and post-processing are important in this model use.
``AutoProcessor`` class used for initialization ``WhisperProcessor`` is
responsible for preparing audio input data for the model, converting it
to Mel-spectrogram and decoding predicted output token_ids into string
using tokenizer.

.. code:: ipython3

    import ipywidgets as widgets
    
    model_ids = {
        "Distil-Whisper": [
            "distil-whisper/distil-large-v2",
            "distil-whisper/distil-medium.en",
            "distil-whisper/distil-small.en",
        ],
        "Whisper": [
            "openai/whisper-large-v3",
            "openai/whisper-large-v2",
            "openai/whisper-large",
            "openai/whisper-medium",
            "openai/whisper-small",
            "openai/whisper-base",
            "openai/whisper-tiny",
            "openai/whisper-medium.en",
            "openai/whisper-small.en",
            "openai/whisper-base.en",
            "openai/whisper-tiny.en",
        ],
    }
    
    model_type = widgets.Dropdown(
        options=model_ids.keys(),
        value="Distil-Whisper",
        description="Model type:",
        disabled=False,
    )
    
    model_type

.. code:: ipython3

    model_id = widgets.Dropdown(
        options=model_ids[model_type.value],
        value=model_ids[model_type.value][0],
        description="Model:",
        disabled=False,
    )
    
    model_id

.. code:: ipython3

    from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
    
    processor = AutoProcessor.from_pretrained(model_id.value)
    
    pt_model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id.value)
    pt_model.eval();


.. parsed-literal::

    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.


Prepare input sample
~~~~~~~~~~~~~~~~~~~~



The processor expects audio data in numpy array format and information
about the audio sampling rate and returns the ``input_features`` tensor
for making predictions. Conversion of audio to numpy format is handled
by Hugging Face datasets implementation.

.. code:: ipython3

    from datasets import load_dataset
    
    
    def extract_input_features(sample):
        input_features = processor(
            sample["audio"]["array"],
            sampling_rate=sample["audio"]["sampling_rate"],
            return_tensors="pt",
        ).input_features
        return input_features
    
    
    dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation", trust_remote_code=True)
    sample = dataset[0]
    input_features = extract_input_features(sample)

Run model inference
~~~~~~~~~~~~~~~~~~~



To perform speech recognition, one can use ``generate`` interface of the
model. After generation is finished processor.batch_decode can be used
for decoding predicted token_ids into text transcription.

.. code:: ipython3

    import IPython.display as ipd
    
    predicted_ids = pt_model.generate(input_features)
    transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
    
    display(ipd.Audio(sample["audio"]["array"], rate=sample["audio"]["sampling_rate"]))
    print(f"Reference: {sample['text']}")
    print(f"Result: {transcription[0]}")



.. raw:: html

    
    <audio  controls="controls" >
        <source src="data:audio/wav;base64,[encoded data, 249872 characters]" type="audio/wav" />
        Your browser does not support the audio element.
    </audio>



.. parsed-literal::

    Reference: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
    Result:  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.


Load OpenVINO model using Optimum library
-----------------------------------------



The Hugging Face Optimum API is a high-level API that enables us to
convert and quantize models from the Hugging Face Transformers library
to the OpenVINO™ IR format. For more details, refer to the `Hugging Face
Optimum
documentation <https://huggingface.co/docs/optimum/intel/inference>`__.

Optimum Intel can be used to load optimized models from the `Hugging
Face Hub <https://huggingface.co/docs/optimum/intel/hf.co/models>`__ and
create pipelines to run an inference with OpenVINO Runtime using Hugging
Face APIs. The Optimum Inference models are API compatible with Hugging
Face Transformers models. This means we just need to replace the
``AutoModelForXxx`` class with the corresponding ``OVModelForXxx``
class.

Below is an example of the distil-whisper model

.. code:: diff

   -from transformers import AutoModelForSpeechSeq2Seq
   +from optimum.intel.openvino import OVModelForSpeechSeq2Seq
   from transformers import AutoTokenizer, pipeline

   model_id = "distil-whisper/distil-large-v2"
   -model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id)
   +model = OVModelForSpeechSeq2Seq.from_pretrained(model_id, export=True)

Model class initialization starts with calling the ``from_pretrained``
method. When downloading and converting the Transformers model, the
parameter ``export=True`` should be added. We can save the converted
model for the next usage with the ``save_pretrained`` method. Tokenizers
and Processors are distributed with models also compatible with the
OpenVINO model. It means that we can reuse initialized early processor.

.. code:: ipython3

    from pathlib import Path
    from optimum.intel.openvino import OVModelForSpeechSeq2Seq
    
    model_path = Path(model_id.value.replace("/", "_"))
    ov_config = {"CACHE_DIR": ""}
    
    if not model_path.exists():
        ov_model = OVModelForSpeechSeq2Seq.from_pretrained(
            model_id.value,
            ov_config=ov_config,
            export=True,
            compile=False,
            load_in_8bit=False,
        )
        ov_model.half()
        ov_model.save_pretrained(model_path)
    else:
        ov_model = OVModelForSpeechSeq2Seq.from_pretrained(model_path, ov_config=ov_config, compile=False)


.. parsed-literal::

    INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino


Select Inference device
~~~~~~~~~~~~~~~~~~~~~~~



.. code:: ipython3

    import openvino as ov
    import ipywidgets as widgets
    
    core = ov.Core()
    
    device = widgets.Dropdown(
        options=core.available_devices + ["AUTO"],
        value="AUTO",
        description="Device:",
        disabled=False,
    )
    
    device




.. parsed-literal::

    Dropdown(description='Device:', index=4, options=('CPU', 'GPU.0', 'GPU.1', 'GPU.2', 'AUTO'), value='AUTO')



Compile OpenVINO model
~~~~~~~~~~~~~~~~~~~~~~



.. code:: ipython3

    ov_model.to(device.value)
    ov_model.compile()


.. parsed-literal::

    Compiling the encoder to AUTO ...
    Compiling the decoder to AUTO ...
    Compiling the decoder to AUTO ...


Run OpenVINO model inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~



.. code:: ipython3

    predicted_ids = ov_model.generate(input_features)
    transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
    
    display(ipd.Audio(sample["audio"]["array"], rate=sample["audio"]["sampling_rate"]))
    print(f"Reference: {sample['text']}")
    print(f"Result: {transcription[0]}")


.. parsed-literal::

    /home/nsavel/venvs/ov_notebooks_tmp/lib/python3.8/site-packages/optimum/intel/openvino/modeling_seq2seq.py:457: FutureWarning: `shared_memory` is deprecated and will be removed in 2024.0. Value of `shared_memory` is going to override `share_inputs` value. Please use only `share_inputs` explicitly.
      last_hidden_state = torch.from_numpy(self.request(inputs, shared_memory=True)["last_hidden_state"]).to(
    /home/nsavel/venvs/ov_notebooks_tmp/lib/python3.8/site-packages/optimum/intel/openvino/modeling_seq2seq.py:538: FutureWarning: `shared_memory` is deprecated and will be removed in 2024.0. Value of `shared_memory` is going to override `share_inputs` value. Please use only `share_inputs` explicitly.
      self.request.start_async(inputs, shared_memory=True)



.. raw:: html

    
    <audio  controls="controls" >
        <source src="data:audio/wav;base64,[encoded data, 249872 characters]" type="audio/wav" />
        Your browser does not support the audio element.
    </audio>



.. parsed-literal::

    Reference: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
    Result:  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.


Compare performance PyTorch vs OpenVINO
---------------------------------------



.. code:: ipython3

    import time
    import numpy as np
    from tqdm.notebook import tqdm
    
    
    def measure_perf(model, sample, n=10):
        timers = []
        input_features = extract_input_features(sample)
        for _ in tqdm(range(n), desc="Measuring performance"):
            start = time.perf_counter()
            model.generate(input_features)
            end = time.perf_counter()
            timers.append(end - start)
        return np.median(timers)

.. code:: ipython3

    perf_torch = measure_perf(pt_model, sample)
    perf_ov = measure_perf(ov_model, sample)



.. parsed-literal::

    Measuring performance:   0%|          | 0/10 [00:00<?, ?it/s]



.. parsed-literal::

    Measuring performance:   0%|          | 0/10 [00:00<?, ?it/s]


.. code:: ipython3

    print(f"Mean torch {model_id.value} generation time: {perf_torch:.3f}s")
    print(f"Mean openvino {model_id.value} generation time: {perf_ov:.3f}s")
    print(f"Performance {model_id.value} openvino speedup: {perf_torch / perf_ov:.3f}")


.. parsed-literal::

    Mean torch distil-large-v2 generation time: 3.064s
    Mean openvino distil-large-v2 generation time: 1.819s
    Performance distil-large-v2 openvino speedup: 1.684


Usage OpenVINO model with HuggingFace pipelines
-----------------------------------------------



Like the original PyTorch model, the OpenVINO model is also compatible
with HuggingFace
`pipeline <https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline>`__
interface for ``automatic-speech-recognition``. Pipeline can be used for
long audio transcription. Distil-Whisper uses a chunked algorithm to
transcribe long-form audio files. In practice, this chunked long-form
algorithm is 9x faster than the sequential algorithm proposed by OpenAI
in the Whisper paper. To enable chunking, pass the chunk_length_s
parameter to the pipeline. For Distil-Whisper, a chunk length of 15
seconds is optimal. To activate batching, pass the argument batch_size.

.. code:: ipython3

    from transformers import pipeline
    
    ov_model.generation_config = pt_model.generation_config
    
    pipe = pipeline(
        "automatic-speech-recognition",
        model=ov_model,
        tokenizer=processor.tokenizer,
        feature_extractor=processor.feature_extractor,
        max_new_tokens=128,
        chunk_length_s=15,
        batch_size=16,
    )


.. parsed-literal::

    The model 'OVModelForWhisper' is not supported for automatic-speech-recognition. Supported models are ['Pop2PianoForConditionalGeneration', 'SeamlessM4TForSpeechToText', 'SpeechEncoderDecoderModel', 'Speech2TextForConditionalGeneration', 'SpeechT5ForSpeechToText', 'WhisperForConditionalGeneration', 'Data2VecAudioForCTC', 'HubertForCTC', 'MCTCTForCTC', 'SEWForCTC', 'SEWDForCTC', 'UniSpeechForCTC', 'UniSpeechSatForCTC', 'Wav2Vec2ForCTC', 'Wav2Vec2ConformerForCTC', 'WavLMForCTC'].


.. code:: ipython3

    dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation", trust_remote_code=True)
    sample_long = dataset[0]
    
    
    def format_timestamp(seconds: float):
        """
        format time in srt-file expected format
        """
        assert seconds >= 0, "non-negative timestamp expected"
        milliseconds = round(seconds * 1000.0)
    
        hours = milliseconds // 3_600_000
        milliseconds -= hours * 3_600_000
    
        minutes = milliseconds // 60_000
        milliseconds -= minutes * 60_000
    
        seconds = milliseconds // 1_000
        milliseconds -= seconds * 1_000
    
        return (f"{hours}:" if hours > 0 else "00:") + f"{minutes:02d}:{seconds:02d},{milliseconds:03d}"
    
    
    def prepare_srt(transcription):
        """
        Format transcription into srt file format
        """
        segment_lines = []
        for idx, segment in enumerate(transcription["chunks"]):
            segment_lines.append(str(idx + 1) + "\n")
            timestamps = segment["timestamp"]
            time_start = format_timestamp(timestamps[0])
            time_end = format_timestamp(timestamps[1])
            time_str = f"{time_start} --> {time_end}\n"
            segment_lines.append(time_str)
            segment_lines.append(segment["text"] + "\n\n")
        return segment_lines

``return_timestamps`` argument allows getting timestamps of start and
end of speech associated with each processed chunk. It could be useful
in tasks like speech separation or generation of video subtitles. In
this example, we provide output formatting in SRT format, one of the
popular subtitles format.

.. code:: ipython3

    result = pipe(sample_long["audio"].copy(), return_timestamps=True)

.. code:: ipython3

    srt_lines = prepare_srt(result)
    
    display(ipd.Audio(sample_long["audio"]["array"], rate=sample_long["audio"]["sampling_rate"]))
    print("".join(srt_lines))



.. raw:: html

    
    <audio  controls="controls" >
        <source src="data:audio/wav;base64,[encoded data, 2664808 characters]" type="audio/wav" />
        Your browser does not support the audio element.
    </audio>



.. parsed-literal::

    1
    00:00:00,000 --> 00:00:06,560
     Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
    
    2
    00:00:06,560 --> 00:00:11,280
     Nor is Mr. Quilter's manner less interesting than his matter.
    
    3
    00:00:11,280 --> 00:00:16,840
     He tells us that at this festive season of the year, with Christmas and roast beef looming
    
    4
    00:00:16,840 --> 00:00:23,760
     before us, similes drawn from eating and its results occur most readily to the mind.
    
    5
    00:00:23,760 --> 00:00:29,360
     He has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and
    
    6
    00:00:29,360 --> 00:00:33,640
     can discover in it but little of Rocky Ithaca.
    
    7
    00:00:33,640 --> 00:00:39,760
     Lennel's pictures are a sort of upgards and Adam paintings, and Mason's exquisite
    
    8
    00:00:39,760 --> 00:00:44,720
     idles are as national as a jingo poem.
    
    9
    00:00:44,720 --> 00:00:50,320
     Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used
    
    10
    00:00:50,320 --> 00:00:52,920
     to flash his teeth.
    
    11
    00:00:52,920 --> 00:00:58,680
     And Mr. John Collier gives his sitter a cheerful slap on the back, before he says, like
    
    12
    00:00:58,680 --> 00:01:01,120
     a shampooer and a Turkish bath,
    
    13
    00:01:01,120 --> 00:01:02,000
     Next man!
    
    


Quantization
------------



`NNCF <https://github.com/openvinotoolkit/nncf/>`__ enables
post-training quantization by adding the quantization layers into the
model graph and then using a subset of the training dataset to
initialize the parameters of these additional quantization layers. The
framework is designed so that modifications to your original training
code are minor.

The optimization process contains the following steps:

1. Create a calibration dataset for quantization.
2. Run ``nncf.quantize`` to obtain quantized encoder and decoder models.
3. Serialize the ``INT8`` model using ``openvino.save_model`` function.

..

   **Note**: Quantization is time and memory consuming operation.
   Running quantization code below may take some time.

Please select below whether you would like to run Distil-Whisper
quantization.

.. code:: ipython3

    to_quantize = widgets.Checkbox(
        value=True,
        description="Quantization",
        disabled=False,
    )
    
    to_quantize




.. parsed-literal::

    Checkbox(value=True, description='Quantization')



.. code:: ipython3

    # Fetch `skip_kernel_extension` module
    import requests
    
    r = requests.get(
        url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py",
    )
    open("skip_kernel_extension.py", "w").write(r.text)
    
    %load_ext skip_kernel_extension

Prepare calibration datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~



First step is to prepare calibration datasets for quantization. Since we
quantize whisper encoder and decoder separately, we need to prepare a
calibration dataset for each of the models. We import an
``InferRequestWrapper`` class that will intercept model inputs and
collect them to a list. Then we run model inference on some small amount
of audio samples. Generally, increasing the calibration dataset size
improves quantization quality.

.. code:: ipython3

    %%skip not $to_quantize.value
    
    from itertools import islice
    from optimum.intel.openvino.quantization import InferRequestWrapper
    
    
    def collect_calibration_dataset(ov_model: OVModelForSpeechSeq2Seq, calibration_dataset_size: int):
        # Overwrite model request properties, saving the original ones for restoring later
        encoder_calibration_data = []
        decoder_calibration_data = []
        ov_model.encoder.request = InferRequestWrapper(ov_model.encoder.request, encoder_calibration_data, apply_caching=True)
        ov_model.decoder_with_past.request = InferRequestWrapper(ov_model.decoder_with_past.request,
                                                                 decoder_calibration_data,
                                                                 apply_caching=True)
    
        try:
            calibration_dataset = load_dataset("openslr/librispeech_asr", "clean", split="validation", streaming=True, trust_remote_code=True)
            for sample in tqdm(islice(calibration_dataset, calibration_dataset_size), desc="Collecting calibration data",
                               total=calibration_dataset_size):
                input_features = extract_input_features(sample)
                ov_model.generate(input_features)
        finally:
            ov_model.encoder.request = ov_model.encoder.request.request
            ov_model.decoder_with_past.request = ov_model.decoder_with_past.request.request
    
        return encoder_calibration_data, decoder_calibration_data

Quantize Distil-Whisper encoder and decoder models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Below we run the ``quantize`` function which calls ``nncf.quantize`` on
Distil-Whisper encoder and decoder-with-past models. We don’t quantize
first-step-decoder because its share in whole inference time is
negligible.

.. code:: ipython3

    %%skip not $to_quantize.value
    
    import gc
    import shutil
    import nncf
    
    CALIBRATION_DATASET_SIZE = 50
    quantized_model_path = Path(f"{model_path}_quantized")
    
    
    def quantize(ov_model: OVModelForSpeechSeq2Seq, calibration_dataset_size: int):
        if not quantized_model_path.exists():
            encoder_calibration_data, decoder_calibration_data = collect_calibration_dataset(
                ov_model, calibration_dataset_size
            )
            print("Quantizing encoder")
            quantized_encoder = nncf.quantize(
                ov_model.encoder.model,
                nncf.Dataset(encoder_calibration_data),
                subset_size=len(encoder_calibration_data),
                model_type=nncf.ModelType.TRANSFORMER,
                # Smooth Quant algorithm reduces activation quantization error; optimal alpha value was obtained through grid search
                advanced_parameters=nncf.AdvancedQuantizationParameters(smooth_quant_alpha=0.50)
            )
            ov.save_model(quantized_encoder, quantized_model_path / "openvino_encoder_model.xml")
            del quantized_encoder
            del encoder_calibration_data
            gc.collect()
    
            print("Quantizing decoder with past")
            quantized_decoder_with_past = nncf.quantize(
                ov_model.decoder_with_past.model,
                nncf.Dataset(decoder_calibration_data),
                subset_size=len(decoder_calibration_data),
                model_type=nncf.ModelType.TRANSFORMER,
                # Smooth Quant algorithm reduces activation quantization error; optimal alpha value was obtained through grid search
                advanced_parameters=nncf.AdvancedQuantizationParameters(smooth_quant_alpha=0.95)
            )
            ov.save_model(quantized_decoder_with_past, quantized_model_path / "openvino_decoder_with_past_model.xml")
            del quantized_decoder_with_past
            del decoder_calibration_data
            gc.collect()
    
            # Copy the config file and the first-step-decoder manually
            shutil.copy(model_path / "config.json", quantized_model_path / "config.json")
            shutil.copy(model_path / "openvino_decoder_model.xml", quantized_model_path / "openvino_decoder_model.xml")
            shutil.copy(model_path / "openvino_decoder_model.bin", quantized_model_path / "openvino_decoder_model.bin")
    
        quantized_ov_model = OVModelForSpeechSeq2Seq.from_pretrained(quantized_model_path, ov_config=ov_config, compile=False)
        quantized_ov_model.to(device.value)
        quantized_ov_model.compile()
        return quantized_ov_model
    
    
    ov_quantized_model = quantize(ov_model, CALIBRATION_DATASET_SIZE)



.. parsed-literal::

    Collecting calibration data:   0%|          | 0/10 [00:00<?, ?it/s]


.. parsed-literal::

    Quantizing encoder


.. parsed-literal::

    Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:15<00:00,  1.55s/it]
    Applying Smooth Quant: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 128/128 [00:10<00:00, 12.24it/s]


.. parsed-literal::

    INFO:nncf:96 ignored nodes was found by name in the NNCFGraph


.. parsed-literal::

    Statistics collection: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:29<00:00,  2.99s/it]
    Applying Fast Bias correction: 100%|██████████████████████████████████████████████████████████████████████████████████████| 162/162 [00:21<00:00,  7.60it/s]


.. parsed-literal::

    Quantizing decoder with past


.. parsed-literal::

    Statistics collection: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:04<00:00, 85.63it/s]
    Applying Smooth Quant: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 16.09it/s]


.. parsed-literal::

    INFO:nncf:12 ignored nodes was found by name in the NNCFGraph


.. parsed-literal::

    Statistics collection: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:07<00:00, 52.93it/s]
    Applying Fast Bias correction: 100%|████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 18.50it/s]
    Compiling the encoder to AUTO ...
    Compiling the decoder to AUTO ...
    Compiling the decoder to AUTO ...


Run quantized model inference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Let’s compare the transcription results for original and quantized
models.

.. code:: ipython3

    %%skip not $to_quantize.value
    
    dataset = load_dataset(
        "hf-internal-testing/librispeech_asr_dummy", "clean", split="validation", trust_remote_code=True
    )
    sample = dataset[0]
    input_features = extract_input_features(sample)
    
    predicted_ids = ov_model.generate(input_features)
    transcription_original = processor.batch_decode(predicted_ids, skip_special_tokens=True)
    
    predicted_ids = ov_quantized_model.generate(input_features)
    transcription_quantized = processor.batch_decode(predicted_ids, skip_special_tokens=True)
    
    display(ipd.Audio(sample["audio"]["array"], rate=sample["audio"]["sampling_rate"]))
    print(f"Original : {transcription_original[0]}")
    print(f"Quantized: {transcription_quantized[0]}")



.. raw:: html

    
    <audio  controls="controls" >
        <source src="data:audio/wav;base64,[encoded data, 249872 characters]" type="audio/wav" />
        Your browser does not support the audio element.
    </audio>



.. parsed-literal::

    Original :  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
    Quantized:  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.


Results are the same!

Compare performance and accuracy of the original and quantized models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Finally, we compare original and quantized Distil-Whisper models from
accuracy and performance stand-points.

To measure accuracy, we use ``1 - WER`` as a metric, where WER stands
for Word Error Rate.

When measuring inference time, we do it separately for encoder and
decoder-with-past model forwards, and for the whole model inference too.

.. code:: ipython3

    %%skip not $to_quantize.value
    
    import time
    from contextlib import contextmanager
    from jiwer import wer, wer_standardize
    
    
    TEST_DATASET_SIZE = 50
    MEASURE_TIME = False
    
    @contextmanager
    def time_measurement():
        global MEASURE_TIME
        try:
            MEASURE_TIME = True
            yield
        finally:
            MEASURE_TIME = False
    
    def time_fn(obj, fn_name, time_list):
        original_fn = getattr(obj, fn_name)
    
        def wrapper(*args, **kwargs):
            if not MEASURE_TIME:
                return original_fn(\*args, \*\*kwargs)
            start_time = time.perf_counter()
            result = original_fn(\*args, \*\*kwargs)
            end_time = time.perf_counter()
            time_list.append(end_time - start_time)
            return result
    
        setattr(obj, fn_name, wrapper)
    
    def calculate_transcription_time_and_accuracy(ov_model, test_samples):
        encoder_infer_times = []
        decoder_with_past_infer_times = []
        whole_infer_times = []
        time_fn(ov_model, "generate", whole_infer_times)
        time_fn(ov_model.encoder, "forward", encoder_infer_times)
        time_fn(ov_model.decoder_with_past, "forward", decoder_with_past_infer_times)
    
        ground_truths = []
        predictions = []
        for data_item in tqdm(test_samples, desc="Measuring performance and accuracy"):
            input_features = extract_input_features(data_item)
    
            with time_measurement():
                predicted_ids = ov_model.generate(input_features)
            transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
    
            ground_truths.append(data_item["text"])
            predictions.append(transcription[0])
    
        word_accuracy = (1 - wer(ground_truths, predictions, reference_transform=wer_standardize,
                                 hypothesis_transform=wer_standardize)) * 100
        mean_whole_infer_time = sum(whole_infer_times)
        mean_encoder_infer_time = sum(encoder_infer_times)
        mean_decoder_with_time_infer_time = sum(decoder_with_past_infer_times)
        return word_accuracy, (mean_whole_infer_time, mean_encoder_infer_time, mean_decoder_with_time_infer_time)
    
    test_dataset = load_dataset("openslr/librispeech_asr", "clean", split="test", streaming=True, trust_remote_code=True)
    test_dataset = test_dataset.shuffle(seed=42).take(TEST_DATASET_SIZE)
    test_samples = [sample for sample in test_dataset]
    
    accuracy_original, times_original = calculate_transcription_time_and_accuracy(ov_model, test_samples)
    accuracy_quantized, times_quantized = calculate_transcription_time_and_accuracy(ov_quantized_model, test_samples)
    print(f"Encoder performance speedup: {times_original[1] / times_quantized[1]:.3f}")
    print(f"Decoder with past performance speedup: {times_original[2] / times_quantized[2]:.3f}")
    print(f"Whole pipeline performance speedup: {times_original[0] / times_quantized[0]:.3f}")
    print(f"Whisper transcription word accuracy. Original model: {accuracy_original:.2f}%. Quantized model: {accuracy_quantized:.2f}%.")
    print(f"Accuracy drop: {accuracy_original - accuracy_quantized:.2f}%.")


.. parsed-literal::

    Got disconnected from remote data host. Retrying in 5sec [1/20]
    Got disconnected from remote data host. Retrying in 5sec [2/20]



.. parsed-literal::

    Measuring performance and accuracy:   0%|          | 0/50 [00:00<?, ?it/s]



.. parsed-literal::

    Measuring performance and accuracy:   0%|          | 0/50 [00:00<?, ?it/s]


.. parsed-literal::

    Encoder performance speedup: 1.751
    Decoder with past performance speedup: 1.777
    Whole pipeline performance speedup: 1.711
    Whisper transcription word accuracy. Original model: 85.29%. Quantized model: 85.29%.
    Accuracy drop: 0.00%.


As we can see quantization significantly improves model inference time
without major accuracy drop!

Interactive demo
----------------



We are also providing an interactive demo using the Gradio interface,
where you can test model capabilities on your own audio data (using the
upload button) or record using your microphone. Please note, that
Distil-Whisper is currently only available for English speech
recognition. Multilingual support will be provided later.

.. code:: ipython3

    from transformers.pipelines.audio_utils import ffmpeg_read
    import gradio as gr
    
    
    BATCH_SIZE = 16
    MAX_AUDIO_MINS = 30  # maximum audio input in minutes
    
    
    generate_kwargs = {"language": "en", "task": "transcribe"} if not model_id.value.endswith(".en") else {}
    ov_pipe = pipeline(
        "automatic-speech-recognition",
        model=ov_model,
        tokenizer=processor.tokenizer,
        feature_extractor=processor.feature_extractor,
        max_new_tokens=128,
        chunk_length_s=15,
        generate_kwargs=generate_kwargs,
    )
    ov_pipe_forward = ov_pipe._forward
    
    if to_quantize.value:
        ov_quantized_model.generation_config = ov_model.generation_config
        ov_quantized_pipe = pipeline(
            "automatic-speech-recognition",
            model=ov_quantized_model,
            tokenizer=processor.tokenizer,
            feature_extractor=processor.feature_extractor,
            max_new_tokens=128,
            chunk_length_s=15,
            generate_kwargs=generate_kwargs,
        )
        ov_quantized_pipe_forward = ov_quantized_pipe._forward
    
    
    def transcribe(inputs, quantized=False):
        pipe = ov_quantized_pipe if quantized else ov_pipe
        pipe_forward = ov_quantized_pipe_forward if quantized else ov_pipe_forward
    
        if inputs is None:
            raise gr.Error("No audio file submitted! Please record or upload an audio file before submitting your request.")
    
        with open(inputs, "rb") as f:
            inputs = f.read()
    
        inputs = ffmpeg_read(inputs, pipe.feature_extractor.sampling_rate)
        audio_length_mins = len(inputs) / pipe.feature_extractor.sampling_rate / 60
    
        if audio_length_mins > MAX_AUDIO_MINS:
            raise gr.Error(
                f"To ensure fair usage of the Space, the maximum audio length permitted is {MAX_AUDIO_MINS} minutes."
                f"Got an audio of length {round(audio_length_mins, 3)} minutes."
            )
    
        inputs = {"array": inputs, "sampling_rate": pipe.feature_extractor.sampling_rate}
    
        def _forward_ov_time(*args, **kwargs):
            global ov_time
            start_time = time.time()
            result = pipe_forward(*args, **kwargs)
            ov_time = time.time() - start_time
            ov_time = round(ov_time, 2)
            return result
    
        pipe._forward = _forward_ov_time
        ov_text = pipe(inputs.copy(), batch_size=BATCH_SIZE)["text"]
        return ov_text, ov_time

.. code:: ipython3

    if not Path("gradio_helper.py").exists():
        r = requests.get(url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/distil-whisper-asr/gradio_helper.py")
        open("gradio_helper.py", "w").write(r.text)
    
    from gradio_helper import make_demo
    
    demo = make_demo(fn=transcribe, quantized=to_quantize.value)
    
    try:
        demo.launch(debug=False)
    except Exception:
        demo.launch(share=True, debug=False)
    # if you are launching remotely, specify server_name and server_port
    # demo.launch(server_name='your server name', server_port='server port in int')
    # Read more in the docs: https://gradio.app/docs/
