One-shot prompt on DeepSeek V3
Image generated with ImageFX
DeepSeek vs ChatGPT vs Gemini
A single question, which one to use?
The new kid on the block.
The AI landscape is evolving at a rapid pace, making it challenging for users to keep up with the latest developments. New AI models are frequently introduced, often before users have fully mastered the previous ones.
DeepSeek, a new AI model from China, has recently emerged, claiming to be on par with or even superior to ChatGPT. While benchmarks exist to compare different models, they don't always provide clear insights for the average user.
Personal preferences play a significant role in model selection. For example, a user might favor Gemini for its seamless integration within their existing ecosystem, while still utilizing other models for diverse responses.
Evaluation is crucial, and qualitative aspects are subjective, based on individual user preferences and expectations regarding the AI's tone.
Setting aside quantitative comparisons, posing a single question to each model and examining the responses can offer valuable insights into their strengths and weaknesses, facilitating a more informed assessment.
The test
The stochastic nature of LLMs means that results can vary, and most users wouldn't know how to adjust parameters for consistent outcomes, as this isn't a typical use case.
I chose a question for this test that was simple to ask but difficult for the model to reason through, due to the potential for confusion and seemingly plausible but incorrect responses. Additionally, I needed a topic that I was familiar with, as verifying the accuracy of responses to a medical question, for example, would be too time-consuming.
The chosen question was:
Can you help me understand the implication of using hydrogen in a normal combustion engine?
Hydrogen is being promoted as a decarbonisation solution, but its actual use can vary. It can be burned to create mechanical energy or converted into electrical energy using fuel cells. Both of these methods are still in their early stages and far from being commercially available at scale.
Currently, hydrogen production mainly relies on methane reforming, which generates CO2 emissions. This poses a challenge to its image as a clean energy source.
Methodology
One-shot prompt for the initial question.
Follow-up question to inquire about references used to construct the answer.
Verify the response.
Verify the references.
Evaluate (qualitatively) and comment on each answer.
Now let’s see the result of each model:
*For clarity, I have not edited the response in any way except format(bold/italics/spacing) to improve readability, this also helps to understand the quality of the response.
DeepSeek V3
Prompt: Can you help me understand the implication of using hydrogen in a normal combustion engine?
Response:
Certainly! Using hydrogen in a normal combustion engine (often referred to as a hydrogen internal combustion engine, or H2ICE) has several implications, both positive and negative. Here's a breakdown:
Positive Implications:
1. Reduced Greenhouse Gas Emissions:
When hydrogen is burned in a combustion engine, the primary byproduct is water vapor (H2O), rather than carbon dioxide (CO2) and other pollutants associated with fossil fuels. This makes hydrogen a cleaner fuel option in terms of greenhouse gas emissions.
2. Utilization of Existing Infrastructure:
Hydrogen combustion engines can be adapted from existing internal combustion engine (ICE) designs, which means that some of the current manufacturing infrastructure and expertise can be repurposed. This could make the transition to hydrogen easier and more cost-effective compared to developing entirely new technologies like fuel cells.
3. High Energy Content:
Hydrogen has a high energy content by weight (about three times more than gasoline), which can potentially offer good performance in terms of power output.
4. Renewable Potential:
Hydrogen can be produced from renewable sources (e.g., electrolysis of water using renewable electricity), making it a potentially sustainable fuel option if the production process is green.
Negative Implications:
1. Nitrogen Oxides (NOx) Emissions:
While hydrogen combustion produces no CO2, it can still produce nitrogen oxides (NOx) due to the high temperatures in the combustion chamber, especially in air-breathing engines. NOx are harmful pollutants that contribute to smog and acid rain.
2. Storage and Handling Challenges:
Hydrogen has a very low energy density by volume, which means it needs to be stored at high pressures or as a cryogenic liquid to be practical for use in vehicles. This requires specialized storage tanks and handling procedures, which can be costly and complex.
3. Energy Efficiency:
Hydrogen combustion engines are generally less efficient than hydrogen fuel cells. A significant amount of energy is lost as heat during combustion, whereas fuel cells convert hydrogen directly into electricity with higher efficiency.
4. Production and Infrastructure:
Currently, most hydrogen is produced from natural gas through a process called steam methane reforming, which emits CO2. For hydrogen to be truly green, it needs to be produced via electrolysis using renewable energy, which is currently more expensive and less widespread.
Additionally, the infrastructure for hydrogen refueling is still in its infancy, making it less convenient compared to gasoline or electric charging stations.
5. Safety Concerns:
Hydrogen is highly flammable and has a wide range of concentrations in which it can ignite. This requires stringent safety measures to prevent leaks and explosions, which can add to the complexity and cost of using hydrogen as a fuel.
Summary:
Using hydrogen in a normal combustion engine offers a pathway to reduce greenhouse gas emissions and leverage existing engine technology. However, it also presents challenges related to NOx emissions, energy efficiency, storage, and infrastructure. While hydrogen combustion engines can be a transitional technology, many experts believe that hydrogen fuel cells or battery electric vehicles (BEVs) may offer more sustainable and efficient long-term solutions for decarbonizing transportation.
If you're considering hydrogen as a fuel, it's important to weigh these factors and consider the broader context of energy production, distribution, and usage.
Prompt: Can you indicate some references for the above response?
Response:
Certainly! Below are some references and sources that provide further reading on the topic of using hydrogen in internal combustion engines. These references include academic papers, industry reports, and reputable websites that discuss the implications, challenges, and opportunities of hydrogen combustion engines:
1. Academic Papers and Reviews:
Verhelst, S., & Wallner, T. (2009).
Hydrogen-fueled internal combustion engines.
Progress in Energy and Combustion Science, 35(6), 490-527.
DOI: [10.1016/j.pecs.2009.08.001](https://doi.org/10.1016/j.pecs.2009.08.001)
This paper provides a comprehensive review of hydrogen combustion in internal combustion engines, including performance, emissions, and challenges.
White, C. M., Steeper, R. R., & Lutz, A. E. (2006).
The hydrogen-fueled internal combustion engine: A technical review.
International Journal of Hydrogen Energy, 31(10), 1292-1305.
DOI: 10.1016/j.ijhydene.2005.12.001](https://doi.org/10.1016/j.ijhydene.2005.12.001) This article discusses the technical aspects of hydrogen combustion engines, including efficiency, NOx emissions, and storage challenges.
2. Industry Reports:
International Energy Agency (IEA). (2021).
The Future of Hydrogen.
Available at: [https://www.iea.org/reports/the-future-of-hydrogen](https://www.iea.org/reports/the-future-of-hydrogen)
This report provides an overview of hydrogen production, storage, and use, including its application in transportation and combustion engines.
Hydrogen Council. (2021).
Hydrogen Insights Report.
Available at: [https://hydrogencouncil.com/en/hydrogen-insights-2021/](https://hydrogencouncil.com/en/hydrogen-insights-2021/)
This report discusses the role of hydrogen in decarbonizing various sectors, including transportation, and highlights the challenges and opportunities.
3. Government and Organization Resources:
U.S. Department of Energy (DOE).
Hydrogen Internal Combustion Engine Vehicles.
Available at: [https://www.energy.gov/eere/fuelcells/hydrogen-internal-combustion-engine-vehicles](https://www.energy.gov/eere/fuelcells/hydrogen-internal-combustion-engine-vehicles)
This resource provides an overview of hydrogen combustion engines, including their benefits and limitations.
BROKEN LINK
European Commission.
Hydrogen Strategy for a Climate-Neutral Europe.
Available at: [https://ec.europa.eu/energy/topics/energy-strategy/hydrogen-strategy_en](https://ec.europa.eu/energy/topics/energy-strategy/hydrogen-strategy_en)
This document outlines the EU's strategy for hydrogen, including its use in transportation and industrial applications.
BROKEN LINK
4. Reputable Websites and Articles:
National Renewable Energy Laboratory (NREL).
Hydrogen Basics.
Available at: [https://www.nrel.gov/research/eds-hydrogen.html](https://www.nrel.gov/research/eds-hydrogen.html)
This page provides an introduction to hydrogen as a fuel, including its use in combustion engines and fuel cells.
ScienceDirect Topics.
Hydrogen Combustion Engines.
Available at: [https://www.sciencedirect.com/topics/engineering/hydrogen-combustion-engine](https://www.sciencedirect.com/topics/engineering/hydrogen-combustion-engine)
A collection of articles and summaries on hydrogen combustion engines.
BROKEN LINK
5. Case Studies and Real-World Applications:
BMW Hydrogen 7.
BMW's Hydrogen Combustion Engine Vehicle.
Available at: [https://www.bmw.com/en/innovation/bmw-hydrogen-7.html](https://www.bmw.com/en/innovation/bmw-hydrogen-7.html)
BMW's Hydrogen 7 was one of the first production vehicles to use a hydrogen combustion engine. This case study provides insights into the practical challenges and performance of such engines.
BROKEN LINK
Toyota's Hydrogen Engines.
Toyota's Development of Hydrogen Combustion Engines.
Available at: [https://global.toyota/en/newsroom/corporate/36428923.html](https://global.toyota/en/newsroom/corporate/36428923.html)
Toyota has been actively developing hydrogen combustion engines for motorsports and commercial vehicles, offering real-world data and insights.
BROKEN LINK
6. Books:
Heywood, J. B. (2018).
Internal Combustion Engine Fundamentals.
McGraw-Hill Education.
This book provides a foundational understanding of internal combustion engines, including the principles of hydrogen combustion.
Gupta, R. B. (2008).
Hydrogen Fuel: Production, Transport, and Storage.
CRC Press.
This book covers the entire hydrogen value chain, including its use in combustion engines.
These references should provide a solid foundation for understanding the implications of using hydrogen in internal combustion engines. Let me know if you'd like further assistance!
My Comment:
DeepSeek provided a very good response, clearly distinguishing between positive and negative aspects. When asked for references, it supplied authentic published papers with functional DOI links, which is excellent for research purposes. However, it also included some links that were either broken or irrelevant to the question.
Overall, I was surprised by the quality of the response from this new tool. It certainly warrants further attention, and I intend to experiment with it and test more challenging queries.
ChatGPT 4o-mini
Prompt: Can you help me understand the implication of using hydrogen in a normal combustion engine?
Responses: Absolutely! Using hydrogen in a normal combustion engine, which is typically designed for gasoline or diesel, has several implications, both positive and negative.
1. Combustion Characteristics
Higher Flame Speed: Hydrogen has a higher flame speed compared to gasoline or diesel. This means it burns more quickly, which can cause issues in the combustion chamber, potentially leading to knocking (pre-detonation), especially at higher loads or speeds. So, the engine might need to be tuned or modified to handle this faster combustion process.
Higher Peak Temperatures: Because hydrogen burns hotter, there’s a risk of higher peak cylinder temperatures, which can affect engine components over time, potentially leading to overheating or premature wear.
Wide Flammability Range: Hydrogen has a wider flammability range (from about 4% to 75% hydrogen in air), so it’s easier for the engine to misfire or combust at unwanted times, especially in a system not designed for it.
2. Exhaust Emissions
Water Vapor: The main byproduct of burning hydrogen is water vapor, so there are no carbon emissions (CO₂, CO, particulate matter) like you get from gasoline or diesel. This makes hydrogen combustion appealing from an environmental standpoint, as it can help reduce air pollution.
NOx Formation: On the downside, hydrogen combustion at high temperatures can lead to the formation of nitrogen oxides (NOx), which are pollutants that contribute to smog and acid rain. NOx production tends to increase with higher combustion temperatures, and since hydrogen burns hotter than conventional fuels, this could be a problem without additional measures to control it, like using exhaust gas recirculation (EGR) systems or selective catalytic reduction (SCR).
3. Engine Modifications
Fuel Delivery and Injection System: Hydrogen is stored and delivered as either a compressed gas or liquid, requiring specialized tanks and fuel injection systems. Conventional combustion engines aren’t designed to handle hydrogen’s properties, so you’d need to modify or completely replace the fuel system.
Ignition System: Hydrogen requires a different ignition approach. Since it has a lower ignition energy than gasoline, the spark timing and ignition systems would need to be adjusted to prevent pre-ignition or misfires.
Fuel Storage and Safety: Hydrogen must be stored at high pressure (typically 350-700 bar for automotive applications), and the system needs to be robust enough to handle this pressure safely. This could require substantial modifications to the vehicle’s design, especially around fuel tanks and safety mechanisms to prevent leaks or rupture.
4. Energy Density
Lower Energy Density by Volume: Hydrogen has a low energy density by volume compared to gasoline or diesel. This means that to store a significant amount of energy, hydrogen needs to be compressed to very high pressures or liquefied. In practical terms, this could result in larger, heavier fuel tanks, which could reduce vehicle range or payload capacity, though this is mitigated in newer hydrogen fuel cell vehicles.
Higher Energy Density by Mass: On a mass basis, hydrogen contains more energy per kilogram than gasoline, but the challenges lie in its storage and delivery rather than the energy per weight.
5. Fuel Efficiency and Range
Potentially Lower Efficiency: A hydrogen-powered internal combustion engine (ICE) could be less efficient than a hydrogen fuel cell or battery-electric vehicle. Combustion engines have inherent thermodynamic losses, meaning a hydrogen ICE is unlikely to achieve the high efficiency of a fuel cell, which directly converts chemical energy into electricity with fewer steps and losses.
Range: While hydrogen fuel cells have an advantage here in terms of efficiency, a well-designed hydrogen ICE could still have a comparable range to traditional gasoline or diesel vehicles, depending on the fuel tank size and energy density. However, range anxiety could still be a challenge due to refueling infrastructure and storage limitations.
6. Infrastructure and Adoption
Refueling Infrastructure: Just like fuel cells, using hydrogen in a combustion engine would require hydrogen refueling stations, which are currently limited and expensive to build. Widespread adoption would need significant investment in hydrogen distribution infrastructure, especially if we want to scale up beyond niche applications.
Public Perception: Hydrogen combustion might face challenges in terms of public perception, as many people associate hydrogen with fuel cells rather than traditional combustion engines, which could influence consumer acceptance and market viability.
7. Future Potential
Transitional Technology: For some, hydrogen combustion could be seen as a bridge technology—an easier way to adapt existing engines to run on a cleaner fuel while we transition to fully electric or hydrogen fuel cell-powered vehicles. However, fuel cells are generally considered a more efficient and scalable solution for hydrogen-based transport, while hydrogen combustion could face more obstacles due to inefficiency and emissions (NOx).
In summary, using hydrogen in a normal combustion engine offers several environmental benefits, particularly in terms of eliminating carbon emissions, but it comes with a host of technical challenges related to combustion characteristics, engine efficiency, emissions control, and infrastructure. Hydrogen combustion could be useful in certain applications, but hydrogen fuel cells are generally considered a better fit for the long-term vision of hydrogen-powered transportation.
Prompt: Can you indicate some references for the above response?
Certainly! Here are some references and sources that can provide more detailed information on hydrogen combustion engines and the points mentioned in the response:
1. Hydrogen Combustion Characteristics
“Combustion of Hydrogen and Hydrogen Blends in Internal Combustion Engines” – by B. Yüceer, et al. (Elsevier, 2017)
This review discusses the combustion characteristics of hydrogen in internal combustion engines, including flame speed, temperature effects, and knocking.
“Hydrogen Combustion in Internal Combustion Engines: A Review” – by E. O. Reh, et al. (Journal of Energy, 2020)
This paper explores the effects of hydrogen combustion on engine performance, including combustion stability and emissions.
2. Exhaust Emissions
"Hydrogen Combustion: Emissions and Strategies to Control NOx" – by S. R. Arpa, et al. (Journal of Engineering for Gas Turbines and Power, 2013)
Discusses the formation of nitrogen oxides (NOx) in hydrogen combustion and strategies for reducing them in internal combustion engines.
"Hydrogen as a Combustion Fuel in the Context of Emissions Control" – by A. Asadullah, et al. (Renewable and Sustainable Energy Reviews, 2021)
Explores the environmental impact of hydrogen combustion, particularly NOx and other pollutants.
3. Engine Modifications
"The Potential for Hydrogen as a Transport Fuel: A Review of Internal Combustion Engine Modifications" – by J. F. P. Hibbard, et al. (Energy Policy, 2007)
This paper looks at the modifications needed for traditional combustion engines to run on hydrogen, including changes to the fuel delivery, injection system, and ignition.
"Modifications of IC Engine to Run on Hydrogen" – by R. S. Tiwari and D. K. Agrawal (Journal of Scientific & Industrial Research, 2015)
Details on how internal combustion engines can be modified to use hydrogen, including changes to the fuel injection system and combustion chamber.
4. Energy Density and Efficiency
"Comparative Study of Hydrogen and Gasoline Internal Combustion Engines" – by M. R. Riazi, et al. (International Journal of Hydrogen Energy, 2012)
Discusses the energy densities of hydrogen and gasoline and compares the performance and efficiency of hydrogen combustion engines.
"Energy Efficiency of Hydrogen and Other Fuels in Internal Combustion Engines" – by J. G. Hwang, et al. (International Journal of Energy Research, 2014)
Provides a detailed analysis of the efficiency of hydrogen internal combustion engines and how they compare to other fuels.
5. Infrastructure and Adoption
"Hydrogen Infrastructure for Vehicle Applications" – by A. J. H. Laird and M. J. Forster (International Journal of Hydrogen Energy, 2016)
Discusses the current state of hydrogen infrastructure for transport, focusing on refueling stations and the challenges of scaling hydrogen use in vehicles.
"Hydrogen Economy and the Transition to Fuel Cell Vehicles" – by A. D. Millington and M. E. Williams (Journal of Power Sources, 2019)
Examines the current and future role of hydrogen in the transportation sector, including both fuel cell and combustion engine applications.
6. Environmental Impact and NOx Control
"Hydrogen Combustion and NOx Emissions in a Spark-Ignition Engine" – by M. A. Chien, et al. (Fuel, 2015)
A detailed look at how hydrogen combustion affects NOx emissions in spark-ignition engines and strategies to mitigate these emissions.
"Strategies to Reduce NOx Emissions in Hydrogen Combustion Engines" – by J. L. Fernández, et al. (Environmental Science & Technology, 2018)
Focuses on various methods and technologies, like exhaust gas recirculation and lean combustion strategies, to reduce NOx when using hydrogen.
7. General Resources on Hydrogen Use in Combustion Engines
"Hydrogen as a Fuel for Internal Combustion Engines" – by M. R. Singal and M. S. Bhatti (Springer, 2019)
A comprehensive book that looks at the fundamentals of using hydrogen in internal combustion engines, covering everything from fuel systems to emissions and performance.
"Hydrogen Combustion in Spark-Ignition Engines" – by E. S. Afshari and M. H. Kermani (Energy Conversion and Management, 2015)
Provides an in-depth review of hydrogen combustion in spark-ignition engines, covering the thermodynamic and emission-related aspects.
These references should provide a solid foundation for deeper understanding and exploration of hydrogen combustion engines. They cover the technical aspects, environmental impact, efficiency comparisons, and infrastructure considerations related to hydrogen use in internal combustion engines.
My Comment:
The first publicly available LLM has the most verbose response, nearly 1500 words in total (approximately 800 for Gemini and approximately 1000 for DeepSeek), but is the response quality good or bad? I see two problematic issues:
While it mentions pros and cons like the other LLMs, it doesn't clearly indicate which points are positive and which are negative. The reader has to infer this from the tone of the sentence, which can be challenging for those with a weaker grasp of English.
Many of the references appear to be fabricated; they seem like they should be published papers, but they can't be found. This is unacceptable behavior from an LLM.
Gemini 2.0 Flash Experimental
Prompt: Can you help me understand the implication of using hydrogen in a normal combustion engine?
Response: Using hydrogen in a normal combustion engine has several implications, both positive and negative:1
Pros:
Reduced carbon emissions: Hydrogen combustion produces mainly water vapor as a byproduct, significantly reducing greenhouse gas emissions compared to traditional fossil fuels.
Relatively simple modification: Existing combustion engine technology can be adapted to run on hydrogen, potentially requiring less extensive infrastructure changes compared to other alternatives like fuel cells.2
Higher efficiency potential: Hydrogen can offer higher thermal efficiency compared to gasoline in some engine designs.
Cons:
NOx emissions: While carbon emissions are reduced, hydrogen combustion can still produce nitrogen oxides (NOx), which are harmful air pollutants.3 This can be mitigated with exhaust treatment systems but adds complexity and cost.4
Lower energy density: Hydrogen has a lower energy density by volume compared to gasoline, meaning more storage space is needed to achieve the same driving range.5
Hydrogen production and storage: The environmental benefits of hydrogen are dependent on how it's produced.6"Green" hydrogen, produced from renewable energy sources, is ideal but currently more expensive. Storage and transportation of hydrogen also present challenges.7
Engine wear and material compatibility: Hydrogen combustion can lead to higher temperatures and potential issues with material embrittlement, requiring engine modifications and potentially reducing engine life.
Overall, using hydrogen in a normal combustion engine offers a potential pathway to reduce carbon emissions from the transportation sector, but it also presents challenges related to emissions, efficiency, and infrastructure.8
Are Hydrogens Engines A Really Bad Idea For Commercial Vehicles? interactanalysis.com
Hydrogen fuel cells and combustion engines - Volvo Group
Does the use of hydrogen produce air pollutants such as nitrogen oxides?
Hydrogen Factsheet - Center for Sustainable Systems - University of Michigan
Hydrogen Benefits and Considerations - Alternative Fuels Data Center
Adfc link doesn’t refer to ICE using H2 but to FCEV
Hydrogen storage: a challenge for the development of the industry - Teréga
Link to storage but not to ICE using H2
Prompt: Can you indicate some references for the above response?
{Gemini had reported some reference as part of the first response, however to respect the procedure, also the follow up prompt was asked}
Response: You're right to ask for references! My previous response was a summary of commonly known information, but it's important to back it up with sources. Finding specific academic papers that perfectly encapsulate every point can be tricky, as research is often focused on specific aspects. However, I can provide some general areas and types of resources that support the points I made:
General Overviews and Reports:
International Energy Agency (IEA) reports on hydrogen: The IEA publishes comprehensive reports on hydrogen, covering production, applications, and policy. These are excellent sources for overall information on hydrogen's role in the energy transition. Look for reports like "The Future of Hydrogen" and related publications.
National laboratory reports (e.g., US Department of Energy): National labs like the National Renewable Energy Laboratory (NREL) in the US conduct extensive research on hydrogen and publish reports and fact sheets. These often contain detailed technical information.
Industry publications and websites: Organizations like the Fuel Cell and Hydrogen Energy Association (FCHEA) provide information on hydrogen technologies and applications.
Specific Areas and Examples:
NOx emissions from hydrogen combustion: Research papers on hydrogen combustion in internal combustion engines often discuss NOx formation. Search terms like "hydrogen ICE NOx emissions," "hydrogen engine NOx reduction," and "lean burn hydrogen combustion" can lead to relevant studies.
Hydrogen storage: Research on hydrogen storage covers various methods, including compressed gas, liquid hydrogen, and materials-based storage. Search terms like "hydrogen storage technologies," "metal hydride hydrogen storage," and "compressed hydrogen storage" can provide relevant information.
Engine modifications for hydrogen: Studies on adapting existing engines for hydrogen operation often discuss necessary modifications to fuel injection systems, ignition timing, and materials. Search terms like "hydrogen engine conversion," "hydrogen engine design," and "hydrogen fuel injection" can be helpful.
Life Cycle Assessment of Hydrogen: Studies comparing the environmental impact of different hydrogen production methods and end uses. Search terms like "hydrogen LCA" and "well-to-wheel hydrogen emissions" will be useful.
Example of a more specific type of reference:
A paper discussing NOx emissions might be titled something like: "Experimental Investigation of NOx Emissions from a Hydrogen-Fueled Internal Combustion Engine under Lean Burn Conditions."
It's important to note that the field of hydrogen technology is constantly evolving, so it's best to look for recent publications and reports to get the most up-to-date information. Using academic search engines like Google Scholar, IEEE Xplore, and ScienceDirect can be helpful in finding relevant research papers.
By searching within these areas, you should be able to find credible sources to support the information I provided.
My Comment:
Gemini provided a clear list of pros and cons, using simple language appropriate for the straightforward question. It also included references without needing to be asked, although some were not directly related to hydrogen combustion but rather to hydrogen storage and fuel cells. Unfortunately, when asked for additional references, the response was lacking and placed the research burden on the user.
Conclusion
DeepSeek, the latest addition to the LLMs catalog, is causing a stir in the AI world. It's open-source, trained at a reportedly lower cost and record speed of 60 days, and offers cheaper inference costs via API than established models.
In a singular test, it provided good references and functioning DOI links - a big plus for researchers and scientists.
The biggest potential downside is that any disputes must be resolved under the laws of the People's Republic of China. While I wouldn't use it for business right now, it's a great option for research and development.
ChatGPT had the first-mover advantage, but the competition is catching up fast. Its test response was disappointing in terms of clarity and fabricated references. In past supervised learning work, I checked the veracity of answers, and non-existent or broken links would have lowered the score significantly. This makes ChatGPT the least favoured in this specific test case.
Gemini is a solid model, providing a clear and concise initial response in this test, even for those with a weaker grasp of grammar. However, it fell short when asked for references, leaving the research burden to the user. Given the powerhouse behind the model, this is unacceptable - why couldn't it extract information from Google Scholar?
Gemini remains my personal favourite due to its excellent integration within my ecosystem. However, I'll conduct a more challenging programmatic test between Gemini and DeepSeek using API calls.
Which other model should I include in a new three-way test? Llama is a possibility, but I'm open to suggestions.
Author: Daniele Ventriglia
First post: 26 January 2025
Last edition: NA
Copyright: Engineering and Data Limited, Hampshire, UK.
Comments
Post a Comment