LLM integration package

Subpackages

• llmintegration.models package
  • Submodules
  • llmintegration.models.image_describer_results module
    • ImageDescriberResults
      • ImageDescriberResults.get_confidence()
      • ImageDescriberResults.get_description()
      • ImageDescriberResults.get_object_name()
  • llmintegration.models.image_validator_results module
    • ImageValidatorResults
      • ImageValidatorResults.get_description()
      • ImageValidatorResults.get_instructions()
      • ImageValidatorResults.get_is_valid()
      • ImageValidatorResults.get_object_type()
  • Module contents

Submodules

agent_factory module

class dronebuddylib.atoms.llmintegration.agent_factory.AgentFactory(model_name: str, api_key: str, temperature: float | None = None)

Bases: object

get_agent(agent: LLMAgentNames)

agent_types module

gpt_integration module

class dronebuddylib.atoms.llmintegration.gpt_integration.GPTEngine(configs: GPTConfigs)

Bases: object

add_message_with_image(role, content, image_path)

get_response(user_message: str) → str

For the given user_message, get the response from ChatGPT

get_response_for_image_queries(user_message: str, image_path: str) → str

For the given user_message, get the response from ChatGPT

get_response_for_image_queries_as_files(user_message: str, image_path: str) → str

For the given user_message, get the response from ChatGPT

set_system_prompt(system_prompt: str)

i_llm_agent module

class dronebuddylib.atoms.llmintegration.i_llm_agent.ILLMAgent(api_key: str, model_name: str, temperature: float | None = None, logger_location: str | None = None)

Bases: object

get_agent_description() → LLMAgentNames

get_agent_name() → LLMAgentNames

get_llm_session() → ChatSession

get_response_from_llm() → ChatCompletionMessage

get_result()

send_encoded_image_message_to_llm_queue(role, content, image)

send_image_message_to_llm_queue(role, content, image_path)

send_text_message_to_llm_queue(role, content)

set_system_prompt(system_prompt)

image_describer_agent_impl module

class dronebuddylib.atoms.llmintegration.image_describer_agent_impl.ImageDescriberAgentImpl(api_key: str, model_name: str, temperature: float | None = None, logger_location: str | None = None)

Bases: ILLMAgent

A class to implement an image describer agent using an LLM (Large Language Model). This class provides functionalities to describe images to help visually impaired individuals.

SYSTEM_PROMPT_IMAGE_DESCRIBER = '\n    You are a helpful assistant helping a visually impaired person navigate in their day to day life.\n\n    When an image and the instruction DESCRIBE is given, explain the object as much as possible and only explain the object.\n\n    Give the result in the format\n\n    {\n        "object_name": "what kind of object it is",\n        "description": "what the object is, a full description for the person to listen to",\n        "confidence": a numerical value for the confidence\n    }\n    '

get_agent_description()

Gets the description of the LLM agent.

Returns:

The description of the LLM agent.

Return type:

str

get_agent_name()

Gets the name of the LLM agent.

Returns:

The name of the LLM agent.

Return type:

str

get_result() → ImageDescriberResults

Gets the description result from the LLM and formats it into an ImageDescriberResults object.

Returns:

The formatted result of the image description.

Return type:

ImageDescriberResults

image_validator_agent_impl class

class dronebuddylib.atoms.llmintegration.image_validator_agent_impl.ImageValidatorAgentImpl(api_key: str, model_name: str, temperature: float | None = None, logger_location: str | None = None)

Bases: ILLMAgent

A class to implement an image validator agent using an LLM (Large Language Model). This class provides functionalities to validate images for object identification.

SYSTEM_PROMPT_IMAGE_VALIDATOR = '\n    You are a helpful assistant, capable of deciding whether a given image is suitable for remembering to be recognized in future cases. The image should be clear, and have the full object that need to be focused.\n\n    When given the image, with the instruction VALIDATE(type of the object),\n\n    return the result in the format\n\n    { \n    "data":\n        [\n             { \n                "object_type": "type of the object", \n\n                "is_valid": "true if the image is good enough/ false if not", "description": "description of the object", \n                "instructions": \n                            if the area of the image needs to be higher - INCREASE,\n\n                            if the lighting needs to be improved - LIGHTING_IMPROVE,\n\n                            if the object is incomplete : INCOMPLETE, \n\n                            if the object is not in focus : NOT_FOCUSED\n            } \n        ] \n    }\n    '

get_agent_description()

Gets the description of the LLM agent.

Returns:

The description of the LLM agent.

Return type:

str

get_agent_name()

Gets the name of the LLM agent.

Returns:

The name of the LLM agent.

Return type:

str

get_result() → ImageValidatorResults

Gets the validation result from the LLM and formats it into an ImageValidatorResults object.

Returns:

The formatted result of the image validation.

Return type:

ImageValidatorResults

intent_resolver_agent_impl module

class dronebuddylib.atoms.llmintegration.intent_resolver_agent_impl.IntentResolverAgentImpl(api_key: str, model_name: str, temperature: float | None = None, logger_location: str | None = None)

Bases: ILLMAgent

SYSTEM_PROMPT_IMAGE_DESCRIBER = '\n    You are a helpful assistant.\n    '

get_agent_description()

get_agent_name()

object_identifier_agent_impl module

class dronebuddylib.atoms.llmintegration.object_identifier_agent_impl.ObjectIdentifierAgentImpl(api_key: str, model_name: str, temperature: float | None = None, logger_location: str | None = None)

Bases: ILLMAgent

A class to implement an object identifier agent using an LLM (Large Language Model). This class provides functionalities to remember objects and identify objects in images.

SYSTEM_PROMPT_OBJECT_IDENTIFICATION = '\n    You are a helpful assistant.\n\n    When the instruction "REMEMBER_AS(object name)" is given with an image of the object, \n    remember the object and return an acknowledgement in the format of:\n\n    {\n        "status": "SUCCESS" (if successfully added to the memory) / "UNSUCCESSFUL" (if otherwise),\n        "message": "description"\n    }\n\n    Once the instruction "IDENTIFY" is given with the image, \n    return all the identified objects in the form of a JSON object:\n    {\n        "data": [\n            {\n                "class_name": "class the object belongs to",\n                "object_name": "name of the remembered object / unknown if not a previously remembered object",\n                "description": "description of the object",\n                "confidence": confidence as a value\n            }\n        ]\n    }\n    '

get_agent_description()

Gets the description of the LLM agent.

Returns:

The description of the LLM agent.

Return type:

str

get_agent_name()

Gets the name of the LLM agent.

Returns:

The name of the LLM agent.

Return type:

str

Module contents