How Claude’s new AI data analysis tool compares to ChatGPT’s version (hint: it doesn’t)

Final week, Anthropic (which has a $4B funding from Amazon) introduced that Claude 3.5 Sonnet now helps knowledge evaluation.

TechCruch’s headline for this functionality was “Anthropic’s AI can now run and write code.” That’s technically appropriate, however do not get too excited. Claude is not going to sit down there and simulate all of your code for you. The truth is much more restricted.

The function that Anthropic introduced is just like ChatGPT’s Superior Knowledge Evaluation. One distinction is that Claude’s evaluation software is accessible to everybody, together with free customers. ChatGPT’s Superior Knowledge Evaluation is just accessible to Plus and Enterprise account customers paying $20 or extra a month.

Producing code

Each ChatGPT Plus and Claude carry out their knowledge evaluation by writing and working snippets of code that parse and course of the information. One key distinction is that Claude writes its code in JavaScript whereas ChatGPT writes its code in Python.

These are attention-grabbing decisions. Python has a wealthy ecosystem of numerical evaluation libraries like Pandas, NumPy, and SciPy. JavaScript additionally has a wealthy ecosystem, however its knowledge and AI choices should not fairly as intensive as these for Python. Python may be very robust in machine studying and AI, with frameworks like TensorFlow, PyTorch, and Keras. Python additionally supplies glorious help for large knowledge, though, as you will see, nothing about Claude’s present evaluation software will be thought-about even medium knowledge.

JavaScript, against this, is ideally suited to knowledge visualization in internet pages. The Anthropic answer makes use of React, however there are additionally nice visualization libraries like D3.js and chart.js accessible for info presentation. I did discover it odd that with such nice visualization instruments, the pie charts I generated utilizing Claude tended to cut off the information labels for a few of the classes.

Once you ask Claude to course of knowledge, it offers you its output but additionally lets you take a look at the underlying code it generates to do this knowledge evaluation. Here is an instance.

Utilization limits

I made a decision to make use of Claude to check out its evaluation capabilities. I restricted my use to the free model. In keeping with Claude’s FAQ, the $20/mo Professional model will increase utilization limits by 5 occasions.

That is in all probability needed for critical use as a result of after about 20 minutes of testing, I received shut down.

I did attempt opening a brand new chat, however it did not let me again in. After ready an hour, I used to be capable of ask some extra questions.

Writing code to scrub up knowledge

To check Claude’s knowledge evaluation capabilities, I went to the information.gov web site and downloaded a Social Safety Administration dataset on child identify utilization derived from social safety card functions.

This knowledge got here within the type of a ZIP file. I extracted 145 comma separated worth (CSV) textual content recordsdata containing child identify knowledge from 1880 to 2023, one file per yr.

I first tried to pick all of the recordsdata and import them as a bunch into Claude. I used to be knowledgeable that Claude would solely import 5 recordsdata directly.

So, I made a decision I would write a script that will create a single file containing all the information. The gotcha was that every particular person file did not comprise the yr as one of many fields. So my script must add the yr from the file’s identify to every document within the file, after which do that for all of the recordsdata.

Reasonably than coding this myself, I requested Claude to do it for me.

I have to rapidly mix 145 textual content recordsdata on a Macintosh. Every file identify consists of the letters yob adopted by 4 numbers, indicating the yr adopted by .txt. The recordsdata themselves are comma separated values. For every file, I have to prepend the yr contained within the file identify, adopted by a comma, to each line in its corresponding file. I then want to mix all 145 recordsdata into one grasp file. How can I do that rapidly?

It created a shell script that appeared like it might do the job.

I saved the script and ran it. 

It labored and did precisely as I requested.  The results of working that shell script was a 37MB file. Sadly, I quickly discovered that 37MB exceeded Claude’s add restrict of 30MB. I wanted a dataset that was significantly smaller.

Reasonably than utilizing identify knowledge from every year, I figured that if I used identify knowledge from just one file per decade, I would cut back my dataset measurement to 10% of the unique measurement. So I modified my immediate and fed it again to Claude.

I have to rapidly mix 145 textual content recordsdata on a Macintosh. Every file identify consists of the letters yob adopted by 4 numbers, indicating the yr adopted by .txt. The recordsdata themselves are comma separated values. For every file has a file identify ending in 0.txt, prepend the yr contained within the file identify adopted by a comma to each line in its corresponding file. Then want to mix all recordsdata ending in 0.txt into one grasp file. Write a shell script to do this.

That labored simply in addition to the primary immediate, and I used to be given a 3.9MB file.

General, I used to be fairly happy with in the present day’s Claude 3.5 Sonnet’s coding work. I’ve beforehand run that LLM via my battery of coding checks with out a lot success. So it was good to see it run easily this time. Sadly, that was the final a part of in the present day’s testing course of that ran easily.

Extra limits in Claude

So let us take a look at knowledge evaluation in Claude. Sadly, Claude seems to be very restricted by way of the quantity of information in can ingest. Claude says that its Professional model permits “at the very least 5x the utilization in comparison with our free service” and that “in case your conversations are comparatively brief, you’ll be able to count on to ship at the very least 45 messages each 5 hours.”

That is not quite a bit. And whereas Claude does say which you could add 5 recordsdata and 30MB, I discovered that my mixed 3.9MB file was thought-about a whopping 9124% over its size restrict. That file comprises 219,181 information.

Okay, nice. So then I attempted a file for only one yr. The file yob2020.txt is just 561KB and comprises simply 31,550 information. That file is outwardly 1239% over Claude’s size limits.

Performing some math, and assuming you have not hit their message utilization limits, it seems like Claude limits its knowledge evaluation to round 2,000 traces of about 25 characters every.

Let’s examine that to ChatGPT Plus, lets?

Now, sure, I am utilizing the free Claude model, but when Claude Professional supplies 5X capability, we will generalize (as a result of the corporate does not present arduous limits) that Claude Professional would max out at about 10,000 25-character traces.

In contrast, I fed 69,215 information with a mean of fifty characters per line into ChatGPT Plus and it labored simply nice. I fed a 22,797 document dataset consisting of sentiment knowledge from customers who uninstalled my apps (with most information containing sentiment phrases in addition to fastened knowledge) into ChatGPT Plus and it labored simply nice. I fed two recordsdata consisting of 170,000+ traces of 3D printer G-code into ChatGPT Plus and it labored simply nice.

I’ve discovered ChatGPT Plus’s knowledge evaluation genuinely useful and productivity-enhancing. But when a professional account was restricted to only 10,000 information or much less, as Claude Professional appears to do, I in all probability would have discovered it an attention-grabbing expertise demonstration, however not one thing I might reliably add to my workflow equipment bag.

Really testing Claude’s knowledge evaluation

I downloaded about 30 datasets from knowledge.gov earlier than I discovered one sufficiently small for Claude to look at. That is a November 2020 dataset of adoptable pets from the Montgomery County Animal Companies and Adoption Heart in Derwood, Maryland.

This dataset has 85 information of about 190 characters every. Let’s examine what it may inform us.

With a immediate of “What are you able to inform me about this knowledge?” Claude recognized the most typical pet sort (canines), the most typical consumption sorts (proprietor give up then strays — that simply appears so unhappy), notable patterns and distinctive options (Molly is a standard identify).

I requested for a pie chart representing animal distribution. It gave me this, which confirmed the primary animal sorts however left “Different” to almost 50% of the bar graph.

I wished to know what that “Different” class represented. There’s one thing a bit poignant about the concept that 30-something p.c of the “Different” class consists of tropical fish. I’ve this miserable imaginative and prescient in my head of row upon row of goldfish bowls, every containing one lone goldfish.

Check out that chart and the one simply above it. Discover that whereas there’s loads of house for the chart to point out the labels, they’re lower off in each charts. I do know there are 30-something p.c of tropical fish, however I do not know the precise proportion as a result of all that is proven is a “3”.

JavaScript has glorious charting libraries. I might suppose Anthropic would have been capable of tweak the output to totally characterize the chart knowledge, particularly in panorama view.

Effectively, that is a bummer

I used to be actually hoping that Claude’s knowledge evaluation options can be on par with that of ChatGPT Plus. Even when Claude’s free model might solely do one-fifth of what ChatGPT Plus might, I may need signed up for a subscription.

I actually like the concept of sending my knowledge via a number of evaluation instruments and evaluating the outcomes. That alone would have justified my signing up for one more $240/yr of AI invoice.

However since its clear from my extrapolations above that the Claude Professional model could not deal with even the smallest of the datasets I’ve beforehand efficiently fed into ChatGPT Plus, it actually does not appear definitely worth the funding.

I’ve reached out to Anthropic for remark however have not but heard again. If the corporate responds, I am going to replace this text with its suggestions.

In the meantime, what do you suppose? Have you ever used Superior Knowledge Evaluation in ChatGPT Plus? Are you a Claude or ChatGPT consumer? When would you or would you ever think about using Claude as a substitute of ChatGPT? Tell us within the feedback under.

You’ll be able to observe my day-to-day undertaking updates on social media. Remember to subscribe to my weekly replace publication, and observe me on Twitter/X at @DavidGewirtz, on Fb at Fb.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.