Making use of OpenAI on a BI Dashboard to generate meaningful insights.
One of our recent projects involved helping a major IOT Services provider, who was facing a significant challenge with their Business Intelligence (BI) dashboard.
The client had invested heavily in a state-of-the-art BI dashboard. This dashboard was supposed to be their go-to tool for analyzing data, customer behavior, and market trends. However, despite the investment, the dashboard was proving to be more of a hurdle than a help. The graphs and charts were numerous and intricate, but clients team struggled to derive meaningful insights from them. They felt overwhelmed by the data and frustrated by their inability to make informed decisions quickly.
The Initial Assessment:
Upon receiving the project, our team at conducted a thorough assessment of customers BI dashboard. We discovered that while the dashboard contained a wealth of data, it lacked effective narrative context. The team was flooded with graphs showing sales trends, customer demographics, and regional performance metrics, but there was no clear explanation of what these metrics meant or how they interconnected.
The sheer volume and complexity of the visual data made it challenging for Client’s team to extract actionable insights. The dashboard failed to translate visual data into concise, understandable summaries, so despite having the data, users couldn’t access or act on it.
Designing the Solution:
Understanding the the situation, we proposed a solution that would bridge the gap between raw data and actionable insights. Our plan involved two key components:
- Textual Insights Generation: We decided to integrate a textual openai insights generation module that would automatically translate the graphical data into clear, actionable summaries. This module would analyze the trends, patterns, and anomalies within the data and provide concise text explanations.
- Enhanced Dashboard Features: We proposed enhancements to the BI dashboard, including interactive features that would allow users to request specific insights and receive tailored textual summaries on demand. This would make the dashboard more intuitive and user-friendly.
How we went about the solution?
We wanted to operate in a non intrusive way, without affecting existing functionalities of the BI dashboard, So we added the extra functionality as a plugin which could be invoked on an need basis.
Step 1: Capture Graphs
- Use the application or tool that generates the graphs and take screenshots of each graph. Ensure that the screenshots are clear and contain all necessary details for analysis.
- Get the image div reference using react hook method useRef()
Specify the ref from the DOM which you need to convert to image
import { useRef } from 'react';
export default function InsightForImage( ) {
const imageReference = useRef(null);
return(
<div ref={imageReference}>
{Your html content for which you need to pass to Open AI}
</div>
)
}The above generated imageReference, we need to convert to a standard image format before passing to the backend API for processing.
Step 2: Save Screenshots
- Save each screenshot in a standard image format such as PNG or JPEG. Organize these images in a directory for easy access.
const handleImageRef = ( ) => {
htmlToImage.toPng(imageReference.current)
.then(function (dataUrl) {
dispatch(yourDispatchAPI({ dataUrl })).then((res) =>{
{Handle response once received}
})
})
.catch(function (error) {
console.error('Error converting to image:', error);
});
}Step 3: Upload Images to a Storage Service (Optional)
- If your API requires the images to be hosted online, upload the screenshots to a cloud storage service like AWS S3, Google Cloud Storage, or any other service that provides a public URL for your images.
//Uploading image to awsS3
export const awsS3PutObject = async (
destinationKey: string,
data: any,
contentType?: string
) => {
const params = {
Bucket: config.get("bucket"), //Specify your bucket
Key: destinationKey,
Body: data,
};
if (contentType) {
params["ContentType"] = contentType;
}
try {
await s3Service.putObject(params).promise();
console.log(`${destinationKey} uploaded to s3 successfully`);
return destinationKey;
} catch (error) {
console.error("Error uploading to S3:", error);
throw error;
}
};//Get the image url from S3 bucket
export const getSignedS3Url = async (
destinationKey: any,
responseContentType: string = "image/png"
) => {
const params = {
Bucket: config.get("bucket"), //Your bucket
Key: destinationKey,
Expires: 120, //Expire time for the generated S3 url
ResponseContentType: responseContentType,
};
let signedURL = await s3Service.getSignedUrlPromise("getObject", params);
return signedURL;
};Step 4: Optimize Images (Optional)
- If needed, optimize the images for size and clarity. Subsequently, this step processes the images efficiently and accurately.
Step 5: Obtain OpenAI API Credentials
- Sign up for the OpenAI API File API (or similar service) and obtain the necessary credentials such as an API key or token.
const conf = config.get("openai.key"); //Get your openai key
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: conf,
});Step 6: Review OpenAI API Documentation
- Go through the OpenAI API documentation to understand the required endpoints, request formats, and parameters needed. Then, use this information for insight generation.
Step 7: Create a Request to the OpenAI API
- To begin, construct an API request to upload the image and request insights. This typically involves, consequently, sending a POST request with the image file along with any necessary parameters.
To illustrate, here is the backend code for integrating with OpenAI:
async getOpenaiResponseForImageUrl(postData: any) {
try {
if (!postData?.dataUrl) {
throw new Error('Invalid request');
}
const dataString: string = postData?.dataUrl;
var regex = /^data:.+\/(.+);base64,(.*)$/;
var matches = dataString.match(regex);
var ext = matches[1];
var data = matches[2];
var buffer = Buffer.from(data, 'base64');
const awsUploadPath = config.get('uploadPath'); //Specify your upload path
const fileName = `data_${new Date().getTime()}.${ext}`;
const destinationKey = `v2_${awsUploadPath}/openai-image/${fileName}`;
await awsS3PutObject(destinationKey, buffer);
const signedS3Url = await getSignedS3Url(destinationKey, `image/${ext}`);
let response: any; // Declare a variable to hold the response
response = await this.openaiCreateCompletionForImageUrl(signedS3Url);
const finishReason = response.choices[0].finish_reason;
const responseData = response.choices[0].message;
const responseContent = responseData.content.replace(/^```json\s+/, '').replace(/\s+```$/, '');;
const insights = [];
const parsedContent = JSON.parse(responseContent);
const insightsArray = Object.keys(parsedContent).map(insightKey => ({
name: insightKey,
description: parsedContent[insightKey]
}));
return {
success: true,
data: insightsArray
}
}catch (err) {
console.error('Error parsing response:', err);
}
}async openaiCreateCompletionForImageUrl(imageUrl: any) {
const chatCompletion = await openai.chat.completions.create({
model: "gpt-4-vision-preview",
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: `Provide a detailed insights of the image in the url\n\n-----
Put the response as a JSON object with three key insights in a paragraph format
\nResponse:`
},
{
type: 'image_url',
image_url: {
url: imageUrl
}
}
]
}
],
max_tokens: 2000,
});
return chatCompletion
}Step 8: Handle the Open AI API Response
Step 9: Interpret the OpenAI Insights
- Analyze the OpenAI insights provided by the API. This analysis could include identifying trends, recognizing patterns, or extracting actionable information from the graphs in your screenshots.
Step 10: Automate the Process (Optional)
- If you need to generate insights frequently, then consider automating the process by using scripts. Alternatively, you might integrate the OpenAI API calls directly into your application.
Step 11: Integrate OpenAI Insights into Your Workflow
- Use the generated OpenAI insights to inform decisions. Additionally, leverage these insights to generate reports, or integrate them with other tools and dashboards as needed.
By following these steps, you can effectively leverage the OpenAI API File API to generate smart insights from screenshots of graphs, making data analysis and decision-making more efficient and informed.
The Result:
The integration of textual openai insights into Client’s IoT platform significantly improved their ability to manage and utilize their data. The once-overwhelming dashboards were now complemented by clear, actionable insights. The team could quickly grasp key information without being bogged down by the complexity of the data.
Success Stories:
- Predictive Maintenance: By leveraging the new insights, Client’s maintenance team was able to identify potential equipment failures before they occurred. This proactive approach led to a 30% reduction in unplanned downtime and maintenance costs.
- Operational Efficiency: The production team used the insights to optimize machine performance and reduce energy consumption. This resulted in a 15% increase in overall production efficiency and significant cost savings.
- Enhanced Monitoring: The ability to receive real-time, comprehensible summaries of sensor data allowed Client to improve their monitoring and response times, leading to quicker resolution of operational issues.
This project showcased our ability to provide tailored solutions that not only address the unique challenges faced by our clients but also deliver tangible improvements in operational efficiency and decision-making. We empowered Client to make informed decisions swiftly and effectively, unlocking the full potential of their IoT data and driving their success forward.
Software engineer working as a full stack developer.