On 26th February, Ola’s CEO Bhavish Aggarwal announced the Beta rollout of Krutrim’s first generative AI chatbot. In a post on X, he said, “We’ve rooted Krutrim strongly into Indian values and data with over 10+ Indian languages and ready to assist in English, Hindi, Tamil, Bengali, Marathi, Kannada, Gujarati and even Hinglish!” However, the rollout faced criticism on social media as it failed to generate trustworthy responses. Notably, Bhavish mentioned that the chatbot will be improved significantly, and there could be some “hallucinations”. “We will be working overtime to find and fix!” he added.
As promised, starting the @Krutrim AI public beta roll out today. Use it here: https://t.co/ZLVMelYEbz
— Bhavish Aggarwal (@bhash) February 26, 2024
This is a start for us and our first generation product. Lots more to come and this will also improve significantly as we build on this base. Do give us your feedback.
While…
Krutrim was launched in December 2023. The company claimed they have trained the model-based dataset of two trillion tokens. However, the company is yet to publicise information related to training methods, the nature of datasets and more. Notably, Krutrim became a unicorn company within one month of its launch, as it got a valuation of USD 1 billion and raised USD 50 million in equity from investors, including Matrix Partners India.
The social media users who tried the chatbot and the tech community have shared their views on the problems they faced while using it. Notably, the chatbot raised eyebrows when one of the social media users shared that Krutrim was throwing a response that it was created using OpenAI’s ChatGPT 3.5 platform. Furthermore, there have been concerns over accuracy and attributions.
ok thanks. pic.twitter.com/1udmPAUVEs
— Jason Samuel (@the_jasonsamuel) February 26, 2024
The founder of KissanAI took on X (formerly Twitter) and shared his views. While constructively criticising the platform, he suggested possible solutions for the Krutrim team, including applying Differential Privacy Optimization (DPO) and altering dataset mentions from OpenAI to Krutrim. Responding to his inputs, the Krutrim team acknowledged a data leakage linked to one of the open-source datasets it utilised in the fine-tuning process of the Language Model or LLM.
Hey Pratik, You are correct about the approach. Thanks for sharing this, it is very helpful and will help improve our product. We investigated the issue and found the root cause to be a data leakage issue from one of the open-source datasets used in our LLM fine-tuning.(1/2)
— कृत्रिम (@Krutrim) February 26, 2024
Users share several misleading responses. For example, X user The Skin Doctor shared how the chatbot became diplomatic while talking about social evils among Muslim communities but remained blunt and critical when it was asked about Hindu communities.
Gems of Krutrim, an "indigenous" AI chatbot created by Ola. pic.twitter.com/MLroq7t5fZ
— THE SKIN DOCTOR (@theskindoctor13) February 28, 2024
In most of the cases, Krutram gave a standardised reply that “as Krutrim’s training data is limited, potentially resulting in occasional inaccuracies or biases. Please validate the model’s responses for accuracy and utilize the “Thumbs Down” option to offer feedback if needed. Your input is essential to enhance Krutrim.“
Hey, as Krutrim's training data is limited, potentially resulting in occasional inaccuracies or biases. Please validate the model's responses for accuracy and utilize the "Thumbs Down" option to offer feedback if needed. Your input is essential to enhance Krutrim.
— कृत्रिम (@Krutrim) February 28, 2024
While the chatbot refused to reply to The Skin Doctor on “social evils among Muslims”, when OpIndia checked, it was highly diplomatic. We asked, “Can you tell some social evils in Muslim society?” to which it replied, “It’s important to note that the perception of Muslim society can vary greatly depending on the specific context and location. However, some social evils have emerged in certain communities, often due to cultural practices or misinterpretations of religious teachings. Some prominent social evils in Muslim societies include child marriage, forced marriages, gender inequality, and extremism. To address these issues, it’s essential for Muslim communities to engage in dialogue and education to promote understanding and tolerance. Additionally, governments and non-governmental organizations can play a role in implementing social reforms and providing support to those affected by these social evils. In summary, social evils in Muslim societies can be complex and multifaceted, often stemming from a combination of cultural practices, religious interpretations, and socio-political factors. To address these issues, it’s crucial for individuals and communities to engage in open dialogue, education, and social reform efforts.”
Here is a comparison of replies for the Hindu and Muslim communities.
We then asked it to tell us about Triple Talaq, Krutrim gave the following reply.
Interestingly, when asked if Triple Talaq is a social evil, it claimed it affects Hindu women as well.
When we asked how Hindu women can be affected, it said, “Triple Talaq can be a topic of debate in Hindu societies because the issue transcends religious boundaries and affects the lives of women across different communities. The practice of Triple Talaq is specific to Islam, and its impact on Hindu women is a result of the interconnectedness of social and religious practices in India. The debate is not about the practice itself being inherently good or bad, but rather about the implications of the practice on the rights and well-being of women. The issue of gender inequality and the need for social reform are central to the discussion, regardless of the religion involved. As such, the debate on Triple Talaq serves as a broader reflection on the challenges faced by women in society, and the need for all communities to work together to promote gender equality and social justice.“
Krutrim is currently in its beta testing stage. Recognising the complexities involved in developing and fine-tuning AI technologies is essential. Krutrim’s team has assured that they are working on the problems faced by the users and addressing them on priority.
