SEO Article Writing Experiment: AI vs Human

The spread of AI (Artificial Intelligence) articles has led to a growing belief that human intervention is no longer necessary in SEO content writing. This news has spread worldwide and raised concerns among content writers. However, it's important to note that this statement remains unproven, yet can not be entirely denied.

ChatGPT, a prominent AI model, has explicitly stated that its purpose isn't to replace human writers but to assist them throughout the content creation process.

A number of SEO practitioners have undertaken case studies on this subject, with varying results. Some indicate that AI-generated SEO content has positive SEO outcomes, while others report the opposite.

Given this background, cmlabs decided to conduct a similar experiment comparing the performance of AI-written and human-written SEO articles on Google search results pages using keyword rank metrics.

The main objective of this experiment is to determine whether articles written by AI can achieve higher rankings on Google compared to articles written by humans.

This experiment took place over two months. The first month was used by the SEO and Content Writer team to research keywords and produce articles, and the following month was allocated for the SEO and data teams to track the ranking of each keyword and analyze its progress.

After conducting this experiment, additional data was obtained to support the experiment's findings. The following are the stages of the experiment we conducted.

Stages of the AI vs. Human Experiment

All stages in this experiment were conducted fairly, with no differentiation or special treatment given to articles written by humans or AI. We applied the same guidelines to both types of content to avoid factors that could influence the performance of specific content.

1. Websites Selection

In this experiment, we chose six websites, each of which already had different levels of content assets and organic traffic. These websites were tiketresmi.com, museumnusantara.com, inklusiuang.com, animalogi.com, teknohacks.com, and bundatimes.com.

Each website would later publish five SEO articles written by humans and five SEO articles written by AI, all with topics related to the website's niche.

2. Keyword Research

The SEO team used third-party SEO tools such as Google Keyword Planner, Semrush, and Sequence Keyword Research Tool to conduct keyword research. This step is critical because each keyword has its own competitive level that can impact its ranking on Google.

To ensure that all keywords, both for AI-generated articles and human content had an equal level of competition, we followed the following guidelines:

• Keyword selection is aligned with each website's niche.
• All keywords had an "informational" search intent.
• The search volume for the chosen keywords had similar gaps.
• All keywords had a "low" level of competition.
• We ensured that none of the selected keywords had previously been used in articles on the respective websites.

Here is the list of keywords used in this experiment:

3. AI Selection

The launch of ChatGPT by OpenAI has captured the attention of a wide audience. Although ChatGPT is not the first AI of its kind, since its release on November 30, 2022, it has prompted various platforms to compete in providing features supported by ChatGPT or AI-based solutions.

In our case, we have opted to utilize ChatGPT through a Chrome extension known as AIPRM for ChatGPT. Additionally, we have used several available prompt templates within this extension. Our primary objective is to ensure that all AI-generated articles remain consistent in quality, thereby preventing biases and disparities among AI-generated articles, which could impact their performance in Google rankings.

Furthermore, we conducted preliminary testing of these prompt templates to produce an article using the "informational" keyword and compared it with articles featuring the same keyword on the Search Engine Results Page (SERP). The results indicated that the AI-generated article using the prompt template contained information similar to that found in the articles we encountered on the SERP. Therefore, it can be concluded that the prompt template is suitable for this AI vs. Human writing experiment.

4. Content Creation and Publication

The purpose of this experiment is to assess the performance of articles written by AI in comparison to those written by humans on the SERP. To achieve this, we have minimized or avoided any human intervention in the production of AI-generated articles. However, it is essential to ensure that all articles adhere to the same guidelines for consistent optimization.

Here are the guidelines for the creation of all articles:

• Use only one primary keyword.
• Maintain a minimum word count of 500 for each article.
• Include the primary keyword in the meta-information and the introduction of the article.
• Incorporate a minimum of one internal link.

In addition, there are supplementary guidelines to be followed for AI articles:

• Write articles using the prompt template "Human Written | 100% Unique | SEO Optimized Article," including the introduction, headings, and closing.
• Create meta-information using the prompt template “Title + Meta description + 10 Hn tags.”

Furthermore, for the article cover images in both human and AI articles, we use licensed images downloaded from stock image providers.

Once 30 AI articles and 30 human articles have been produced, we proceed with their publication on the respective websites.

5. Creation of Looker Studio Dashboard

The Data Team used Looker Studio to analyze the performance of each article (based on their URLs) in theSERP. Several key metrics that have been analyzed include clicks (organic traffic) and impressions. This data will serve as supporting information for the final results of the experiment.

6. Analyzing and Tracking Keyword Rankings

The process of analyzing and tracking the rankings of each keyword is undertaken by the SEO and Data teams over a month. As previously mentioned, the Data Team utilizes Looker Studio to obtain the necessary data.

Meanwhile, the SEO team used Sequence Stats to track the keyword positions of each article. The activity of tracking keyword rankings is conducted daily, since the first day following the publication of the article (D+1 after publication).

To facilitate the keyword rank tracking process, the SEO team creates two types of views in Sequence Stats for each domain, namely (a) a view dedicated to tracking keyword rankings for human writing and (b) a view specialized for tracking keyword rankings for AI articles. These views automatically display the average keyword rank for each.

Here is a view for tracking the keyword ranking of human writing on museumnusantara.com. This data was collected on October 12, 2023. On that date, the top five keywords from human-written articles secured a market share of 6.34%. This is a positive development, as museumnusantara.com is now positioned as the third-highest domain among all competitors vying for market share with the same keywords.

Furthermore, here is a view for tracking keyword rankings for AI-generated articles on tiketresmi.com. This data was also collected on October 12, 2023. It is proven that the keywords from AI-authored articles only obtained a market share of 0.21% (position 25th). This is relatively low, as approximately 99.79% of the market share is still held by competitors for the same keywords.

Results of AI vs. Human Writing

Over  30 days, we observed changes in keyword rankings. However, it is known that among the six websites used in the experiment, five of them were predominantly influenced by articles authored by humans, as indicated by the keyword ranking metrics. In these five websites, keywords from human articles had higher average ranks compared to keywords from AI articles.

Here are the average keyword ranks for each website, along with supporting data such as clicks and impressions, on the last day of the experiment:
Then, from the data above, we found another intriguing aspect that caught our attention - the accumulation of User Engagement, as indicated by the Clicks metric, which has positive results. Human articles received significantly more clicks (1302) compared to AI articles (233). In other words, the clicks gained by AI articles only accounted for approximately 17% of the total clicks on human-authored articles.

It's worth noting that the clicks on human-authored articles came from various queries, in addition to the primary keywords, which were picked up by Google. These queries include:

Human articles also managed to achieve higher total impressions than AI articles on four out of six websites. It's important to note that high impressions indicate that content is exposed to a larger audience, contributing to increased visibility.

Throughout these 30 days, human articles consistently improved their performance in terms of clicks and impressions over time, while AI-authored articles tended to remain stagnant or experience a decline.

This experiment also demonstrates that writing articles with AI that provide useful information has the potential to rank on Google. This aligns with Google's guidelines about AI-generated content.

Challenges During the AI vs. Human Writing Experiment

Several challenges were encountered during the AI vs. Human writing experiment, which may have influenced the final results. However, it's hard to figure out how much these challenges affected the experiment's results. Nevertheless, we took various measures, including following guidelines at each stage to ensure the accuracy of the data. With an awareness of these challenges, we hope to address or avoid them in future experiments to obtain even more accurate data and results.

Here, we outline the challenges faced by each team involved:

Challenges Faced by the SEO Team

As mentioned earlier in the Keyword Research phase, keyword selection is a critical aspect. Each keyword has its own potential to rank highly on Google's pages, which is beyond the control of the SEO team.

The SEO team's efforts included choosing keywords with the same level of competition, i.e., low, and having similar search volumes or minimal differences between human and AI content using third-party SEO tools.

Unfortunately, despite low competition levels being indicated, the SEO team found that some specific keywords still had a competitive landscape in SERP. This was seen from the numerous websites already featuring articles using those keywords on the first page of Google.

Another challenge was the addition of internal links in each article. While all articles had internal links, each internal link naturally had different effects on an article's performance that couldn't be measured precisely.

Challenges Faced by the Content Writer Team

The Content Writer team faced more challenges while producing AI articles (AIPRM for ChatGPT). Despite using prompt templates, ChatGPT occasionally presented information that was less relevant, particularly in the case of local topics related to Indonesian culture and transportation information in Indonesia, such as bus ticket prices and bus routes.

For example, in the article "Busana Pranakan," Busana pranakan refers to the daily attire worn by male Abdi Dalem during their service at the Yogyakarta Palace. However, here's an example of information provided by ChatGPT:

Upon further investigation, it appears that ChatGPT may have misunderstood "busana pranakan" as related to Chinese Peranakan culture. Consequently, we needed to add more specific information to the prompt template and correct several inaccuracies in the articles to align the content with the targeted keywords.

This resulted in some AI articles receiving a human touch, which ideally should have been avoided during the experiment.

Additionally, we encountered challenges while creating meta titles and meta descriptions for each AI article. Since the article creation prompt template only provided a title, we decided to use another prompt template for generating meta titles and descriptions.

However, it was not uncommon for the meta titles or descriptions to become too lengthy (meta titles should ideally not exceed 65 characters, and meta descriptions should not exceed 155 characters). Consequently, adjustments or a human touch were needed for these elements.

Challenges Faced by the Data Team

The challenges faced by the Data team were primarily technical in nature, such as the presence of null data after blending data from published articles, Google Search Console, and Google Analytics 4. This occurred because the total number of users accessing published articles, despite having clicks and impressions, could not be precisely counted. This phenomenon is normal due to variations in metric calculations between Google Search Console and Google Analytics 4.

Furthermore, due to the small scale of the AI vs. Human writing experiment (utilizing only 60 articles) and its short duration, the data obtained was relatively limited and could be further optimized.

Conclusion

From the AI vs. Human writing experiment we conducted, it can be concluded that human writing, on the whole, outperformed AI articles based on average rank, clicks, and impressions metrics.

Human-written articles also have a higher potential to attract organic traffic. This is because, during the writing process, the human writer team had the opportunity to use a variety of related keywords within the articles, allowing Google to capture more queries.

The use of AI for article writing still has the potential for good performance on Google's pages, however, it requires human intervention in the production process.

Therefore, it can be said that human content creation still plays a crucial role in SEO, and the presence of AI cannot entirely replace human writers as.

Hence, here at cmlabs, we continue to implement human content creation, carried out by our Content Writer team. We even have guidelines in place regarding the Prohibition of Artificial Intelligence (AI) Tools for Client Content Writing. We will maintain this approach until further data becomes available regarding the use of AI in content writing. We are committed to producing valuable content for readers through collaboration with our SEO and other involved teams.

Furthermore, it's not unlikely that there will be further experiments related to AI vs. Human, whether using the same method or different methods, to obtain more extensive and accurate data.