Pushing First version before making full changes

.dockerignore

@@ -0,0 +1,34 @@
+# Include any files or directories that you don't want to be copied to your
+# container here (e.g., local build artifacts, temporary files, etc.).
+#
+# For more help, visit the .dockerignore file reference guide at
+# https://docs.docker.com/engine/reference/builder/#dockerignore-file
+
+**/.DS_Store
+**/__pycache__
+**/.venv
+**/.classpath
+**/.dockerignore
+**/.env
+**/.git
+**/.gitignore
+**/.project
+**/.settings
+**/.toolstarget
+**/.vs
+**/.vscode
+**/*.*proj.user
+**/*.dbmdl
+**/*.jfm
+**/bin
+**/charts
+**/docker-compose*
+**/compose*
+**/Dockerfile*
+**/node_modules
+**/npm-debug.log
+**/obj
+**/secrets.dev.yaml
+**/values.dev.yaml
+LICENSE
+README.md

.gcloudignore

@@ -0,0 +1,19 @@
+# This file specifies files that are *not* uploaded to Google Cloud
+# using gcloud. It follows the same syntax as .gitignore, with the addition of
+# "#!include" directives (which insert the entries of the given .gitignore-style
+# file at that point).
+#
+# For more information, run:
+#   $ gcloud topic gcloudignore
+#
+.gcloudignore
+# If you would like to upload your .git directory, .gitignore file or files
+# from your .gitignore file, remove the corresponding line
+# below:
+.git
+.gitignore
+
+# Python pycache:
+__pycache__/
+# Ignored by the build system
+/setup.cfg

.gitignore

@@ -0,0 +1,91 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# DotEnv configuration
+.env
+
+# Database
+*.db
+*.rdb
+
+# Pycharm
+.idea
+
+# VS Code
+.vscode/
+rediones.code-workspace
+
+# Spyder
+.spyproject/
+
+# Jupyter NB Checkpoints
+.ipynb_checkpoints/
+
+# exclude data from source control by default
+/data/
+
+# Mac OS-specific storage files
+.DS_Store
+
+# vim
+*.swp
+*.swo
+
+# Mypy cache
+.mypy_cache/
+.venv

Dockerfile

@@ -0,0 +1,49 @@
+# syntax=docker/dockerfile:1
+
+# Comments are provided throughout this file to help you get started.
+# If you need more help, visit the Dockerfile reference guide at
+# https://docs.docker.com/engine/reference/builder/
+
+ARG PYTHON_VERSION=3.12
+FROM python:${PYTHON_VERSION}-slim as base
+
+# Prevents Python from writing pyc files.
+ENV PYTHONDONTWRITEBYTECODE=1
+
+# Keeps Python from buffering stdout and stderr to avoid situations where
+# the application crashes without emitting any logs due to buffering.
+ENV PYTHONUNBUFFERED=1
+
+WORKDIR /app
+
+# Create a non-privileged user that the app will run under.
+# See https://docs.docker.com/go/dockerfile-user-best-practices/
+ARG UID=10001
+RUN adduser \
+    --disabled-password \
+    --gecos "" \
+    --home "/nonexistent" \
+    --shell "/sbin/nologin" \
+    --no-create-home \
+    --uid "${UID}" \
+    appuser
+
+# Download dependencies as a separate step to take advantage of Docker's caching.
+# Leverage a cache mount to /root/.cache/pip to speed up subsequent builds.
+# Leverage a bind mount to requirements.txt to avoid having to copy them into
+# into this layer.
+RUN --mount=type=cache,target=/root/.cache/pip \
+    --mount=type=bind,source=requirements.txt,target=requirements.txt \
+    python -m pip install -r requirements.txt
+
+# Switch to the non-privileged user to run the application.
+USER appuser
+
+# Copy the source code into the container.
+COPY . .
+
+# Expose the port that the application listens on.
+EXPOSE 8000
+
+# Run the application.
+CMD uvicorn '.venv.lib.python3.9.site-packages.httpx._transports.asgi:application' --host=0.0.0.0 --port=8000

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Testimony Adekoya
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Procfile

@@ -0,0 +1 @@
+web: uvicorn api:app --host 0.0.0.0 --port 10000

README copy.md

@@ -0,0 +1,100 @@
+# Rediones-AI API Documentation
+
+## Overview
+Welcome to the Rediones-AI API, a marvel of machine learning that powers the mobile application of Rediones. This API is a collection of endpoints designed to enrich your applications with advanced AI capabilities, including topic generation from images and text, and converting text to realistic voices.
+
+## Base URL
+
+All API requests should be made to: [https://api.rediones.com](https://aerial-mission-407204.ue.r.appspot.com/)
+
+## Endpoints
+
+### Home
+
+- **Endpoint**: `/`
+- **Method**: `GET`
+- **Description**: The welcoming endpoint of the API, guiding you to the treasure trove of AI capabilities.
+- **Response**:
+  ```json
+  {
+    "message": "Welcome To Rediones API"
+  }
+  ```
+
+### Health Check
+
+- **Endpoint**: `/health`
+- **Method**: `GET`
+- **Description**: Checks the pulse of the API to ensure it's up and running.
+- **Response**:
+  ```json
+  {
+    "message": "OK"
+  }
+  ```
+
+### Topic Generation
+
+- **Endpoint**: `/topicgen`
+- **Method**: `POST`
+- **Description**: Generates topics based on the provided image or text. For images, it captions the image and then generates topics. For text, it directly generates topics.
+- **Request**:
+  - **Form Data**:
+    - `img`: UploadFile (optional) - The image file for captioning and topic generation.
+    - `text`: string (optional) - The text to generate topics from.
+    - `img_url`: string (optional) - The URL of an image to use for captioning and topic generation.
+      
+  - **JSON For text**:
+    ```json
+      {
+        "text": "The existential crisis of a teapot"
+      }
+      ```
+
+- **Response**:
+  ```json
+  {
+  "topics": ["Philosophy of Inanimate Objects", "The Teapot's Lament"]
+  }
+  ```
+- **Errors**:
+  - 400: "Only one of image_url or image can be accepted."
+  - 400: "Provide at least text or an image."
+
+### Audioverse
+
+- **Endpoint**: `/audioverse`
+- **Method**: `POST`
+- **Description**: Converts provided text into realistic voices, with functionality for voice cloning.
+- **Request**:
+  - **Body**:
+    ```json
+    {
+      "text": "Hello World"
+    }
+    ```
+- **Response**:
+  ```json
+  {
+  "audio_base64": "dGhpcyBpcyBub3QgcmVhbGx5IGJhc2U2NCwgYnV0IHlvdSBnZXQgdGhlIGlkZWE="
+  }
+  ```
+
+## Rate Limits
+You're limited to 1000 requests per day. Exceed this limit, and you'll receive a 429 Too Many Requests error.
+
+## Errors
+
+Understand how our API communicates issues:
+- `400 Bad Request`: Your request is missing something or formatted incorrectly.
+- `401 Unauthorized`: You're not authenticated. Make sure your API key is correct.
+- `429 Too Many Requests`: You've hit the rate limit. Slow down, turbo.
+
+## Versioning
+
+We're always improving. Keep an eye on our changelog for updates. 
+
+## Getting Help
+
+Lost in the wilderness of our API? Have questions or need support? Contact us at [email protected]
+

app.py

@@ -0,0 +1,44 @@
+from utils.image_utils import UrlTest
+from utils.caption_utils import ImageCaptioning
+from utils.topic_generation import TopicGenerator
+import streamlit as st
+
+topic_generator = TopicGenerator()
+img_caption = ImageCaptioning()
+path_out = UrlTest()
+
+
+def return_caption(img):
+    capt = img_caption.get_caption(img)
+    st.image(image=img, caption=capt, width=250, height=250)
+    generated_topics = topic_generator.generate_topics(capt)
+    st.write(f"Topic: {generated_topics}")
+
+
+def main():
+    st.title("Topic/Title Generator")
+
+    # User input
+    user_input = st.selectbox(label="Text Input or Image Input", options=["Text", "Image", "Image URL"])
+    if user_input == "Text":
+        text_input = st.text_input(label="Put in your Idea, Let's generate a matching Topic/Title 🤗🤗")
+        generated_topics = topic_generator.generate_topics(text_input)
+        st.write(f"Topic:{generated_topics}")
+    elif user_input == "Image":
+        img_input = st.file_uploader(label="Drop an Image you have been admiring, Let's see what we can do🤔🤔",
+                                     type=["jpg", "png", "jpeg"],
+                                     accept_multiple_files=True)
+        for in_img in img_input:
+            if in_img is not None:
+                img_loaded = path_out.load_image(in_img)
+                return_caption(img_loaded)
+
+    elif user_input == "Image URL":
+        url_input = st.text_input(label="Do you have a link to the Image you would like to drop, Go Ahead and We got "
+                                        "you covered😉😉")
+        url_img = path_out.check_url(url_input)
+        return_caption(url_img)
+
+
+if __name__ == "__main__":
+    main()

compose.yaml

@@ -0,0 +1,49 @@
+# Comments are provided throughout this file to help you get started.
+# If you need more help, visit the Docker compose reference guide at
+# https://docs.docker.com/compose/compose-file/
+
+# Here the instructions define your application as a service called "server".
+# This service is built from the Dockerfile in the current directory.
+# You can add other services your application may depend on here, such as a
+# database or a cache. For examples, see the Awesome Compose repository:
+# https://github.com/docker/awesome-compose
+services:
+  server:
+    build:
+      context: .
+    ports:
+      - 8000:8000
+
+# The commented out section below is an example of how to define a PostgreSQL
+# database that your application can use. `depends_on` tells Docker Compose to
+# start the database before your application. The `db-data` volume persists the
+# database data between container restarts. The `db-password` secret is used
+# to set the database password. You must create `db/password.txt` and add
+# a password of your choosing to it before running `docker compose up`.
+#     depends_on:
+#       db:
+#         condition: service_healthy
+#   db:
+#     image: postgres
+#     restart: always
+#     user: postgres
+#     secrets:
+#       - db-password
+#     volumes:
+#       - db-data:/var/lib/postgresql/data
+#     environment:
+#       - POSTGRES_DB=example
+#       - POSTGRES_PASSWORD_FILE=/run/secrets/db-password
+#     expose:
+#       - 5432
+#     healthcheck:
+#       test: [ "CMD", "pg_isready" ]
+#       interval: 10s
+#       timeout: 5s
+#       retries: 5
+# volumes:
+#   db-data:
+# secrets:
+#   db-password:
+#     file: db/password.txt
+

config.py

@@ -0,0 +1,15 @@
+from pydantic import AnyHttpUrl
+from pydantic_settings import BaseSettings, SettingsConfigDict
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+class Settings(BaseSettings):
+    PROJECT_NAME: str = "Rediones API"
+    # BACKEND_CORS_ORIGINS: list[AnyHtt)pUrl] = ["*"]
+
+    model_config = SettingsConfigDict(gcase_sensitive=True)
+
+
+settings = Settings()

main.py

@@ -0,0 +1,128 @@
+import base64
+from typing import Annotated, Optional
+from fastapi import (Body, FastAPI, File, Form, HTTPException, UploadFile,
+                     status)
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import JSONResponse
+from pydantic import AnyHttpUrl, UrlConstraints
+
+from config import settings
+import uvicorn
+from utils.audio_utils import AudioUtils
+from utils.caption_utils import ImageCaptioning
+from utils.image_utils import UrlTest
+from utils.topic_generation import TopicGenerator
+
+app = FastAPI(
+    title=settings.PROJECT_NAME,
+)
+
+# CORS
+if settings:
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins='*',
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+topic_generator = TopicGenerator()
+img_caption = ImageCaptioning()
+audio_utils = AudioUtils()
+utils = UrlTest()
+
+
+@app.get("/")
+def root():
+    return {"message": "Welcome To Rediones API"}
+
+
+@app.get("/health")
+def health():
+    return {"message": "OK"}
+
+
+@app.post("/topicgen")
+def generate_topic(
+    img: UploadFile = File(
+        default=None,
+        description="Image file. It mutually excludes ImgUrl",
+        # regex=r"^.+\.(jpg|png|jpeg)$"
+    ),
+    text: Annotated[Optional[str], Form()] = None,
+    img_url: Annotated[
+        Optional[AnyHttpUrl],
+        UrlConstraints(allowed_schemes=["https"]),
+        Form(description=(
+            "Image url only accepts https scheme. It mutually excludes Img"
+        ))
+    ] = None,
+):
+    if img_url and img:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Only one of image_url or img can be accepted"
+        )
+    # if only text is provided
+    elif text is not None and img is None and img_url is None:
+        generated_topics = topic_generator.generate_topics(text)
+        return {"topics": generated_topics}
+
+    # if image/image_url is provided with or without text
+    elif img or img_url or text:
+        img_file_object = None  # initialize img_file_object
+        # decide whether img or img_url is provided
+        if img:
+            # image file must be ended with .jpg, .png, .jpeg
+            if not str(img.filename).endswith(
+                (".jpg", ".png", ".jpeg")
+            ):
+                raise HTTPException(
+                    status_code=status.HTTP_400_BAD_REQUEST,
+                    detail="Image file must be ended with .jpg, .png, .jpeg"
+                )
+            img_file_object = img.file
+        elif img_url:
+            img_file_object = utils.load_image(img_url)
+
+        # decide whether text is provided
+        if text is None:
+            capt = img_caption.get_caption(img_file_object)
+        else:
+            capt = str(text) + "." + img_caption.get_caption(img_file_object)
+
+        generated_topics = topic_generator.generate_topics(capt)
+        return {"topics": generated_topics}
+
+    else:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="enter text or image. "
+            "imageurl and img are mutually exclusive"
+        )
+
+
+@app.post("/audioverse")
+def generate_audio(
+    text: Annotated[str, Body(description="Text to be transcribed.")]
+):
+    if text is not None:
+        audio_bytes = audio_utils.speak(text)
+        audio_base64 = base64.b64encode(audio_bytes).decode("utf-8")
+        return JSONResponse(content={"audio_base64": audio_base64})
+
+@app.post("/transcribe")
+def transcribe_audio(
+    audio: UploadFile = File(
+        default=None,
+        description="Audio file to be transcribed."
+    )
+):
+    if audio is not None:
+        audio_transcribe = audio_utils.improved_transcribe(0.8, audio_file=audio.file)
+        return JSONResponse(content={"audio_transcription": audio_transcribe})
+    
+
+if __name__ == "__main__":
+    uvicorn.run(app, host="127.0.0.1", port=8000)

notebooks/audioverse.ipynb

@@ -0,0 +1,228 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "from dotenv import load_dotenv\n",
+    "import os"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "True"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "load_dotenv()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "openai_key = os.getenv(\"OPENAI\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = OpenAI(api_key=openai_key)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.audio.speech.create(\n",
+    "    model=\"tts-1-hd\",\n",
+    "    voice=\"alloy\",\n",
+    "    input=\"My Name is Testimony[laughs]\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_39704/3198597338.py:1: DeprecationWarning: Due to a bug, this method doesn't actually stream the response content, `.with_streaming_response.method()` should be used instead\n",
+      "  response.stream_to_file(\"../data/output.mp3\")\n"
+     ]
+    }
+   ],
+   "source": [
+    "response.stream_to_file(\"../data/output.mp3\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "audio_file = open(\"../data/output.mp3\", \"rb\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text_response = client.audio.transcriptions.create(\n",
+    "    model=\"whisper-1\",\n",
+    "    file= audio_file,\n",
+    "    response_format= 'text'\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'My name is testimony.\\n'"
+      ]
+     },
+     "execution_count": 19,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "text_response"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "\n",
+    "\n",
+    "def transcribe(audio_file):\n",
+    "  transcription = client.audio.transcriptions.create(\n",
+    "    model=\"whisper-1\", \n",
+    "    file=audio_file, \n",
+    "    response_format=\"text\",\n",
+    "    prompt=\"ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.\"\n",
+    "  )\n",
+    "  return transcription"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "system_prompt = \n",
+    "def generate_corrected_transcript(temperature, system_prompt, audio_file):\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=\"gpt-3.5-turbo-16k\",\n",
+    "        temperature=temperature,\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": system_prompt\n",
+    "            },\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": transcribe(audio_file)\n",
+    "            }\n",
+    "        ]\n",
+    "    )\n",
+    "    return response.choices[0].message.content\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "audio_fil = open(\"../data/output.mp3\", \"rb\")\n",
+    "corrected_text = generate_corrected_transcript(0.7, system_prompt, audio_file=audio_fil)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'My name is Testimony.'"
+      ]
+     },
+     "execution_count": 30,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "corrected_text"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "base",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

requirements.txt

@@ -0,0 +1,36 @@
+requests~=2.27.1
+pillow~=10.2.0
+streamlit
+openai
+annotated-types==0.6.0
+sentencepiece~=0.1.99
+anyio==3.7.1
+certifi==2023.11.17
+click==8.1.7
+dnspython==2.4.2
+email-validator==2.1.0.post1
+exceptiongroup==1.2.0
+fastapi==0.105.0
+h11==0.14.0
+httpcore==1.0.2
+httptools==0.6.1
+httpx==0.25.2
+idna==3.6
+itsdangerous==2.1.2
+Jinja2==3.1.2
+MarkupSafe==2.1.3
+orjson==3.9.10
+pydantic==2.5.2
+pydantic-core==2.14.5
+pydantic-extra-types==2.2.0
+pydantic-settings==2.1.0
+python-dotenv==1.0.0
+python-multipart==0.0.6
+PyYAML==6.0.1
+sniffio==1.3.0
+starlette==0.27.0
+typing-extensions==4.9.0
+ujson==5.9.0
+uvicorn==0.24.0.post1
+watchfiles==0.21.0
+websockets==12.0

utils/__init__.py

@@ -0,0 +1 @@
+

utils/audio_utils.py

@@ -0,0 +1,54 @@
+from openai import OpenAI
+from dotenv import load_dotenv
+import os
+
+load_dotenv()
+
+openai_key = os.getenv("OPENAI")
+
+
+
+class AudioUtils:
+    def __init__(self):
+        self.client = OpenAI(api_key=openai_key)
+
+    def speak(self, text="Hello World!"):
+        print("Model Loaded")
+        response = self.client.audio.speech.create(
+            model="tts-1-hd",
+            voice="alloy",
+            input=text,
+        )
+
+        return response.content
+    
+    def transcribe(self, audio):
+        response = self.client.audio.transcriptions.create(
+            model="whisper-1",
+            file=audio,
+            response_format="text",
+            temperature=0.1,
+        )
+
+        return response
+    
+    def improved_transcribe(self, temperature, audio_file):
+        with open(audio_file, "rb") as file:
+            audio = file.read()
+
+        response = self.client.chat.completions.create(
+            model="gpt-3.5-turbo-1106",
+            temperature=temperature,
+            messages=[
+                {
+                    "role": "system",
+                    "content":"You are a helpful assistant for the company Rediones. Your task is to correct any spelling discrepancies in the transcribed text. Make sure that the names of the following products are spelled correctly: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T. Only add necessary punctuation such as periods, commas, and capitalization, and use only the context provided."
+                },
+                {
+                    "role": "user",
+                    "content": self.transcribe(audio)
+                }
+            ]
+        )
+        return response.choices[0].message.content
+

utils/caption_utils.py

@@ -0,0 +1,73 @@
+from transformers import BlipProcessor, BlipForConditionalGeneration, AutoTokenizer, AutoModelForSeq2SeqLM
+import torch
+import requests
+from dotenv import load_dotenv
+from image_utils import UrlTest
+import os
+
+img = UrlTest()
+
+class ImageCaptioning:
+    def __init__(self):
+        # Initialize Model and Tokenizer
+        self.blip_processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
+        self.blip_model = BlipForConditionalGeneration.from_pretrained('Salesforce/blip-image-captioning-base')
+        self.topic_generator_processor = AutoTokenizer.from_pretrained("google/flan-t5-large")
+        self.topic_generator_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-large")
+        self.blip_model.eval()
+        self.topic_generator_model.eval()
+
+
+    def generate_caption(self, image):
+        # Generate Caption
+        input_text = self.blip_processor(image, return_tensors="pt")
+        outputs = self.blip_model.generate(pixel_values=input_text["pixel_values"], max_new_tokens=128, do_sample=True, temperature=0.9, top_k=50, top_p=0.95)
+        caption_output = [self.blip_processor.decode(output, skip_special_tokens=True) for output in outputs]
+
+        return outputs
+
+    
+    def generate_topics(self, user_input, num_topics=3):
+        query = f"""Generate a topic sentence idea based on the user input. 
+            The generated topics should portray the context or idea behind the given sentences or phrase.
+            For Instance,
+                - "Grocery Shopping" OR "Grocery List" OR "Shopping List": "I'm going grocery shopping tomorrow, 
+                and I would like to get the following things on my grocery list: Milk, Soybeans, Cowpeas, 
+                Saturated Water, Onions, Tomatoes, etc."
+                - "Studying For Exams" OR "Exams Studies": "Exams aare coming up and I have to prepare for the core 
+                courses. I'll be studying for Control Systems, Software Engineering and Circuit Theory."
+                - "Healthy Breakfast": "To prepare a healthy breakfast, I need the appropriate combination of balanced 
+                diet. I'll need oats, yogurt, fresh berries, honey and smoothies."
+                -  "Fitness Routine": "Starting a fitness routine involves workout clothes, running shoes, 
+                a water bottles, and a gym membership. With this, I can start a proper fitness plan."
+                - "Summer Vacation": "Packing swimsuits and enjoy the view of the ocean."
+                - "Coffee Break": "Sipping Coffee at the table."
+                - "Relaxation": "Sitting at the table enjoying."
+                
+            This is what I'm expecting the model to do. Here is the input: {user_input}
+                       """
+            
+        caption_input = self.topic_generator_processor(query, return_tensors="pt", padding=True, truncation=True, max_length=512)
+        caption_output = self.topic_generator_model.generate(**caption_input, temperature=0.1, num_return_sequences=num_topics, do_sample=True, max_length=50, top_k=50, top_p=0.95, num_beams=5) 
+        caption_output = [self.topic_generator_processor.decode(output, skip_special_tokens=True) for output in caption_output]
+
+        return caption_output
+
+    def combo_model(self, image):
+        image = img.load_image(image)
+        caption = self.generate_caption(image)
+        caption = self.blip_processor.decode(caption[0], skip_special_tokens=True)
+        topics = self.generate_topics(caption)
+        topics = [topic for topic in topics if len(topic) > 0]
+        return {"caption": caption,
+                "topics": topics}
+
+    
+if __name__ == "__main__":
+    # Initialize Model
+    model = ImageCaptioning()
+    # Test Image
+    image = "1071642.jpg"
+    # Generate Caption and Topics
+    outputs = model.combo_model(image)
+    print(outputs)

utils/image_utils.py

@@ -0,0 +1,21 @@
+import requests
+from PIL import Image
+import urllib.parse as parse
+import os
+
+
+class UrlTest:
+
+    def check_url(self, url_string):
+        try:
+            result = parse.urlparse(url_string)
+            return all([result.scheme, result.netloc, result.path])
+        except:
+            return url_string
+    # Load an image
+
+    def load_image(self, image_path):
+        if self.check_url(image_path):
+            return Image.open(requests.get(image_path, stream=True).raw)
+        elif os.path.exists(image_path):
+            return Image.open(image_path)

utils/model_serving/blip_serve.py

@@ -0,0 +1,30 @@
+from dotenv import load_dotenv
+from transformers import BlipForConditionalGeneration, BlipProcessor
+import torch
+import litserve as ls
+import os
+
+load_dotenv()
+
+hf_token = os.getenv("HUGGINGFACE")
+
+class RedionesBlipModel():
+    def __init__(self):
+        self.model_name = "Salesforce/blip-image-captioning-base"
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        self.token = hf_token
+
+    def setup(self, device):
+        device = self.device
+        self.model = BlipForConditionalGeneration.from_pretrained(self.model_name,
+                                                                  use_auth_token=self.token,
+                                                                  )
+        self.tokenizer = BlipProcessor.from_pretrained(self.model_name, use_auth_token=self.token)
+        self.model.to(device)
+        self.model.eval()
+        
+    def predict(self, image):
+        input_text = self.tokenizer(image, return_tensors="pt")
+        outputs = self.model.generate(input_ids = input_text["input_ids"].to(self.device), max_new_tokens=50)
+        
+        return outputs

utils/model_serving/model_loader.py

@@ -0,0 +1,28 @@
+from transformers import BlipProcessor, BlipForConditionalGeneration, AutoTokenizer, AutoModelForSeq2SeqLM
+
+
+class ModelLoader:
+    def __init__(self):
+        self.blip_processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
+        self.blip_model = BlipForConditionalGeneration.from_pretrained('Salesforce/blip-image-captioning-base')
+        self.topic_generator_processor = AutoTokenizer.from_pretrained("google/flan-t5-large")
+        self.topic_generator_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-large")
+
+    
+    def load_blip(self):
+        model = self.blip_model
+        processor = self.blip_processor
+        model.eval()
+        return model, processor
+    
+    def load_topic_generator(self):
+        model = self.topic_generator_model
+        processor = self.topic_generator_processor
+        model.eval()
+        return model, processor
+        
+
+# testing the model
+model_load = ModelLoader()
+blip_models, blip_processors = model_load.blip_model()
+topic_generator_models, topic_generator_processors = model_load.load_topic_generator()

utils/topic_generation.py

@@ -0,0 +1,51 @@
+import requests
+from dotenv import load_dotenv
+import os
+
+load_dotenv()
+
+huggingface = os.getenv("HUGGINGFACE")
+
+
+class TopicGenerator:
+
+    def __init__(self):
+        # Initialize API-URL and authorization headers
+        self.url = "https://api-inference.huggingface.co/models/google/flan-t5-large"
+        self.headers = {"Authorization": f"Bearer {huggingface}"}
+
+    def query(self, payload):
+        response = requests.post(self.url, headers=self.headers,
+                                 json=payload)
+        return response
+    
+    def generate_topics(self, user_input, num_topics=3):
+        payload = {
+            "inputs": f"""Generate a topic sentence idea based on the user input. 
+            The generated topics should portray the context or idea behind the given sentences or phrase.
+            For Instance,
+                - "Grocery Shopping" OR "Grocery List" OR "Shopping List": "I'm going grocery shopping tomorrow, 
+                and I would like to get the following things on my grocery list: Milk, Soybeans, Cowpeas, 
+                Saturated Water, Onions, Tomatoes, etc."
+                - "Studying For Exams" OR "Exams Studies": "Exams aare coming up and I have to prepare for the core 
+                courses. I'll be studying for Control Systems, Software Engineering and Circuit Theory."
+                - "Healthy Breakfast": "To prepare a healthy breakfast, I need the appropriate combination of balanced 
+                diet. I'll need oats, yogurt, fresh berries, honey and smoothies."
+                -  "Fitness Routine": "Starting a fitness routine involves workout clothes, running shoes, 
+                a water bottles, and a gym membership. With this, I can start a proper fitness plan."
+                - "Summer Vacation": "Packing swimsuits and enjoy the view of the ocean."
+                - "Coffee Break": "Sipping Coffee at the table."
+                - "Relaxation": "Sitting at the table enjoying."
+                
+            This is what I'm expecting the model to do. Here is the input: {user_input}
+                       """,
+            "do_sample": True,
+            "temperature": 0.7,
+            "num_return_sequences": num_topics
+            }
+        output = self.query(payload)
+        if output.status_code == 200:
+            topic = output.json()
+            return topic
+        else:
+            return f"Error: Received response code {output.status_code}"