Models Window

This component takes care of the display of the WhisperSeg models. User can select models to annotate the audio or train new custom models.

Get labels from WhisperSeg API

const callWhisperSeg = async (event) => {
    event.preventDefault()

    passWhisperSegIsLoadingToTrack(true)
    const path = import.meta.env.VITE_BACKEND_SERVICE_ADDRESS+'get-labels'

    const annotatedAreas = filterAndConvertAnnotatedAreasForWhisper()
    const convertedLabels = filterAndConvertLabelsForWhisper()

    const requestParameters = {
        audio_id: audioId,
        annotated_areas: annotatedAreas,
        human_labels: convertedLabels,
        model_name: selectedInferenceModel,
        min_frequency: minFreqInference,
        token: tokenInference
    }

    try {
        const response = await axios.post(path, requestParameters)
        const whisperObjects = response.data.labels

        // Create new species, Individuals and Clusternames in the Species panel from the whisper labels
        const updatedSpeciesArray = createSpeciesFromImportedLabels(whisperObjects, speciesArray)
        passSpeciesArrayToApp(updatedSpeciesArray)

        // Assign Species Information to the new labels
        const whisperLabels = assignSpeciesInformationToImportedLabels(updatedSpeciesArray, whisperObjects)

        const annotatedAreaLabels = labels.filter( label => label.species === ANNOTATED_AREA)
        const combinedLabels = whisperLabels.concat(annotatedAreaLabels)
        passLabelsToTrack(combinedLabels)
        passShowModelsWindowToWhisperSeg(false)
    } catch (error){
        if (error.response.status === 403){
            toast.error('Access to WhisperSeg denied due to incorrect access token.')
        } else {
            toast.error('Something went wrong with your request. Check the console to view the error.')
            console.error(error)
        }
    } finally {
        passWhisperSegIsLoadingToTrack(false)
    }
}

callWhisperSeg() makes a request to the WhisperSeg API to generate labels. Before we can make that request we will have to convert our custom label objects into generic objects using filterAndConvertLabelsForWhisper(). In addition, we will filter out the annotated areas from our labels array and pass them to the WhisperSeg API separately, in addition to some other properties (see requestParameters variable). Once the WhisperSeg API has returned a response we first update speciesArray with the newly created species, individuals and clusternames. We then convert the generic objects into our custom label objects using assignSpeciesInformationToImportedLabels(). As a last step remove the annotated areas from the labels array and combine them with the new whisperLabels array, which then becomes the new value of labels.

Train a custom model

const handleClickSubmitTrainingRequestBtn = async (event) => {
    event.preventDefault()

    const annotatedAreas = filterAndConvertAnnotatedAreasForWhisper()
    const convertedLabels = filterAndConvertLabelsForWhisper()

    const allModels = [...modelsAvailableForInference, ...modelsAvailableForFinetuning, ...modelsCurrentlyTrained]

    for (const model of allModels){
        if (model.model_name === newModelName) {
            toast.error(`Model with the name "${newModelName}" already exists.`)
            return
        }
    }

    if (!annotatedAreas.length || !convertedLabels.length){
        toast.error('Provide at least one annotated Area and one label to train the new model on.')
        return
    }

    const path= import.meta.env.VITE_BACKEND_SERVICE_ADDRESS+'finetune-whisperseg'

    const requestParameters = {
        audio_id: audioId,
        annotated_areas: annotatedAreas,
        human_labels: convertedLabels,
        new_model_name: newModelName,
        initial_model_name: selectedFinetuningModel,
        min_frequency: minFreqFinetune,
        token: tokenFinetune
    }

    try {
        await axios.post(path, requestParameters)
        toast.success('Custom model started training and will be available soon.')
        const updatedArray = [...currentlyTrainedModelsNames, newModelName]
        passCurrentlyTrainedModelsNamesToWhisperSeg(updatedArray)
        setNewModelName('')

    } catch (error){
        if (error.response.status === 403){
            toast.error('Access to WhisperSeg denied due to incorrect access token.')
        } else {
            toast.error('Something went wrong with your request. Check the console to view the error.')
            console.error(error)
        }
    }
}

handleClickSubmitTrainingRequestBtn() sends a training request for a custom model to the WhisperSeg API. Before we can make that request we will have to convert our custom label objects into generic objects using filterAndConvertLabelsForWhisper(). In addition, we will filter out the annotated areas from our labels array and pass them to the WhisperSeg API separately, in addition to some other properties (see requestParameters variable). Two checks will be made before that, the user will have to provide at least one annotated area and one label, as well as a new unique model name that follows a naming scheme.